Sep 16 18:43:14.186226 ip-10-0-129-194 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 18:43:14.598616 ip-10-0-129-194 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:43:14.598616 ip-10-0-129-194 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 18:43:14.598616 ip-10-0-129-194 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:43:14.598616 ip-10-0-129-194 kubenswrapper[2582]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 18:43:14.598616 ip-10-0-129-194 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:43:14.601220 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.601100    2582 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 18:43:14.605385 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605369    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:43:14.605385 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605385    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:43:14.605449 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605390    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:43:14.605449 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605394    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:43:14.605449 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605397    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:43:14.605449 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605400    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:43:14.605449 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605403    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:43:14.605449 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605406    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:43:14.605449 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605409    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:43:14.605449 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605412    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:43:14.605449 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605415    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:43:14.605449 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605419    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:43:14.605449 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605424    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:43:14.605449 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605428    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:43:14.605449 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605431    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:43:14.605449 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605434    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:43:14.605449 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605437    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:43:14.605449 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605440    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:43:14.605449 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605443    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:43:14.605449 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605446    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:43:14.605449 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605449    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:43:14.605449 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605452    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:43:14.605933 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605455    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:43:14.605933 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605458    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:43:14.605933 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605461    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:43:14.605933 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605464    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:43:14.605933 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605467    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:43:14.605933 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605476    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:43:14.605933 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605478    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:43:14.605933 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605482    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:43:14.605933 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605485    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:43:14.605933 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605488    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:43:14.605933 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605491    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:43:14.605933 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605494    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:43:14.605933 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605496    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:43:14.605933 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605499    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:43:14.605933 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605502    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:43:14.605933 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605504    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:43:14.605933 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605507    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:43:14.605933 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605510    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:43:14.605933 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605512    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:43:14.605933 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605515    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:43:14.606425 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605517    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:43:14.606425 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605520    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:43:14.606425 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605523    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:43:14.606425 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605525    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:43:14.606425 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605528    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:43:14.606425 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605531    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:43:14.606425 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605533    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:43:14.606425 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605536    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:43:14.606425 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605538    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:43:14.606425 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605541    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:43:14.606425 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605543    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:43:14.606425 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605546    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:43:14.606425 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605549    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:43:14.606425 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605553    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:43:14.606425 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605556    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:43:14.606425 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605558    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:43:14.606425 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605561    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:43:14.606425 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605564    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:43:14.606425 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605568    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:43:14.606893 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605573    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:43:14.606893 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605577    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:43:14.606893 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605581    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:43:14.606893 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605584    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:43:14.606893 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605586    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:43:14.606893 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605591    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:43:14.606893 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605594    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:43:14.606893 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605597    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:43:14.606893 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605599    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:43:14.606893 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605602    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:43:14.606893 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605605    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:43:14.606893 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605608    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:43:14.606893 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605611    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:43:14.606893 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605614    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:43:14.606893 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605617    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:43:14.606893 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605619    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:43:14.606893 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605623    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:43:14.606893 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605625    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:43:14.606893 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605628    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:43:14.606893 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605631    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:43:14.607402 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605633    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:43:14.607402 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605636    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:43:14.607402 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605638    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:43:14.607402 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605641    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:43:14.607402 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605643    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:43:14.607402 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605646    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:43:14.607402 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605649    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:43:14.607402 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605652    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:43:14.607402 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605655    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:43:14.607402 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605658    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:43:14.607402 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605660    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:43:14.607402 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605663    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:43:14.607402 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605666    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:43:14.607402 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605668    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:43:14.607402 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605671    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:43:14.607402 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605673    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:43:14.607402 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605676    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:43:14.607402 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.605679    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:43:14.607402 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606392    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:43:14.607402 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606399    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:43:14.607887 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606402    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:43:14.607887 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606406    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:43:14.607887 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606409    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:43:14.607887 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606412    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:43:14.607887 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606414    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:43:14.607887 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606417    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:43:14.607887 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606420    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:43:14.607887 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606422    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:43:14.607887 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606426    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:43:14.607887 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606430    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:43:14.607887 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606433    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:43:14.607887 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606436    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:43:14.607887 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606439    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:43:14.607887 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606442    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:43:14.607887 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606445    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:43:14.607887 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606447    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:43:14.607887 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606451    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:43:14.607887 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606455    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:43:14.607887 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606458    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:43:14.607887 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606461    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:43:14.608405 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606464    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:43:14.608405 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606466    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:43:14.608405 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606469    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:43:14.608405 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606472    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:43:14.608405 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606475    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:43:14.608405 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606477    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:43:14.608405 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606480    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:43:14.608405 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606483    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:43:14.608405 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606485    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:43:14.608405 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606488    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:43:14.608405 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606491    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:43:14.608405 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606493    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:43:14.608405 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606496    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:43:14.608405 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606499    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:43:14.608405 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606502    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:43:14.608405 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606504    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:43:14.608405 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606507    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:43:14.608405 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606510    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:43:14.608405 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606512    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:43:14.608405 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606515    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:43:14.608405 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606518    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:43:14.608919 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606520    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:43:14.608919 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606525    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:43:14.608919 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606527    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:43:14.608919 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606530    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:43:14.608919 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606533    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:43:14.608919 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606536    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:43:14.608919 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606539    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:43:14.608919 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606541    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:43:14.608919 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606545    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:43:14.608919 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606548    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:43:14.608919 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606551    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:43:14.608919 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606554    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:43:14.608919 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606556    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:43:14.608919 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606559    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:43:14.608919 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606561    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:43:14.608919 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606564    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:43:14.608919 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606567    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:43:14.608919 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606570    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:43:14.608919 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606573    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:43:14.608919 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606575    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:43:14.609444 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606578    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:43:14.609444 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606581    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:43:14.609444 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606583    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:43:14.609444 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606588    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:43:14.609444 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606592    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:43:14.609444 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606595    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:43:14.609444 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606598    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:43:14.609444 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606601    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:43:14.609444 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606604    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:43:14.609444 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606607    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:43:14.609444 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606610    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:43:14.609444 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606612    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:43:14.609444 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606615    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:43:14.609444 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606618    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:43:14.609444 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606621    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:43:14.609444 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606624    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:43:14.609444 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606629    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:43:14.609444 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606632    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:43:14.609444 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606635    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:43:14.609915 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606638    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:43:14.609915 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606641    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:43:14.609915 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606644    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:43:14.609915 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606648    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:43:14.609915 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606652    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:43:14.609915 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606655    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:43:14.609915 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606658    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:43:14.609915 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606661    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:43:14.609915 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606664    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:43:14.609915 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606667    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:43:14.609915 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606669    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:43:14.609915 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606672    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:43:14.609915 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606675    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:43:14.609915 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606677    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:43:14.609915 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606680    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:43:14.609915 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606682    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:43:14.609915 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.606685    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:43:14.609915 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607142    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:43:14.609915 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607149    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:43:14.610398 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607152    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:43:14.610398 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607155    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:43:14.610398 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607158    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:43:14.610398 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607161    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:43:14.610398 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607164    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:43:14.610398 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607167    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:43:14.610398 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607170    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:43:14.610398 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607173    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:43:14.610398 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607177    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:43:14.610398 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607180    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:43:14.610398 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607183    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:43:14.610398 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607186    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:43:14.610398 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607189    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:43:14.610398 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607192    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:43:14.610398 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607197    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:43:14.610398 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607200    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:43:14.610398 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607203    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:43:14.610398 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607206    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:43:14.610398 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607209    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:43:14.610398 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607211    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:43:14.610910 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607214    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:43:14.610910 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607217    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:43:14.610910 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607220    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:43:14.610910 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607222    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:43:14.610910 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607225    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:43:14.610910 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607228    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:43:14.610910 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607231    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:43:14.610910 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607234    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:43:14.610910 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607236    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:43:14.610910 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607239    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:43:14.610910 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607241    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:43:14.610910 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607244    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:43:14.610910 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607247    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:43:14.610910 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607250    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:43:14.610910 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607252    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:43:14.610910 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607255    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:43:14.610910 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607257    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:43:14.610910 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607260    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:43:14.610910 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607263    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:43:14.610910 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607265    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:43:14.611428 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607268    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:43:14.611428 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607272    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:43:14.611428 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607275    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:43:14.611428 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607278    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:43:14.611428 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607283    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:43:14.611428 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607286    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:43:14.611428 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607289    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:43:14.611428 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607292    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:43:14.611428 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607296    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:43:14.611428 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607298    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:43:14.611428 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607301    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:43:14.611428 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607304    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:43:14.611428 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607307    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:43:14.611428 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607310    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:43:14.611428 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607312    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:43:14.611428 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607315    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:43:14.611428 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607318    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:43:14.611428 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607320    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:43:14.611428 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607323    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:43:14.611902 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607326    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:43:14.611902 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607328    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:43:14.611902 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607331    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:43:14.611902 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607334    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:43:14.611902 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607346    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:43:14.611902 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607350    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:43:14.611902 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607352    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:43:14.611902 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607355    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:43:14.611902 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607358    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:43:14.611902 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607360    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:43:14.611902 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607363    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:43:14.611902 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607366    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:43:14.611902 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607369    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:43:14.611902 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607372    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:43:14.611902 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607374    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:43:14.611902 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607378    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:43:14.611902 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607380    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:43:14.611902 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607384    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:43:14.611902 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607387    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:43:14.611902 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607390    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:43:14.612444 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607393    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:43:14.612444 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607395    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:43:14.612444 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607398    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:43:14.612444 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607401    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:43:14.612444 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607403    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:43:14.612444 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607406    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:43:14.612444 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607409    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:43:14.612444 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607414    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:43:14.612444 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607418    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:43:14.612444 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607421    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:43:14.612444 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607424    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:43:14.612444 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607427    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:43:14.612444 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607429    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:43:14.612444 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607432    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:43:14.612444 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607435    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:43:14.612444 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607438    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:43:14.612444 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607441    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:43:14.612444 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.607445    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:43:14.612444 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607525    2582 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 18:43:14.612969 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607571    2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 18:43:14.612969 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607581    2582 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 18:43:14.612969 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607586    2582 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 18:43:14.612969 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607591    2582 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 18:43:14.612969 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607594    2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 18:43:14.612969 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607599    2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 18:43:14.612969 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607604    2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 18:43:14.612969 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607607    2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 18:43:14.612969 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607610    2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 18:43:14.612969 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607615    2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 18:43:14.612969 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607620    2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 18:43:14.612969 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607623    2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 18:43:14.612969 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607626    2582 flags.go:64] FLAG: --cgroup-root=""
Sep 16 18:43:14.612969 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607630    2582 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 18:43:14.612969 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607633    2582 flags.go:64] FLAG: --client-ca-file=""
Sep 16 18:43:14.612969 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607636    2582 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 18:43:14.612969 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607639    2582 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 18:43:14.612969 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607644    2582 flags.go:64] FLAG: --cluster-domain=""
Sep 16 18:43:14.612969 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607647    2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 18:43:14.612969 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607650    2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 18:43:14.612969 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607654    2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 18:43:14.612969 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607657    2582 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 18:43:14.612969 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607661    2582 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 18:43:14.612969 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607665    2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 18:43:14.613546 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607668    2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 18:43:14.613546 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607671    2582 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 18:43:14.613546 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607675    2582 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 18:43:14.613546 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607678    2582 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 18:43:14.613546 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607681    2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 18:43:14.613546 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607684    2582 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 18:43:14.613546 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607687    2582 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 18:43:14.613546 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607691    2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 18:43:14.613546 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607695    2582 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 18:43:14.613546 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607697    2582 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 18:43:14.613546 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607700    2582 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 18:43:14.613546 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607703    2582 flags.go:64] FLAG: --enable-server="true"
Sep 16 18:43:14.613546 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607706    2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 18:43:14.613546 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607711    2582 flags.go:64] FLAG: --event-burst="100"
Sep 16 18:43:14.613546 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607714    2582 flags.go:64] FLAG: --event-qps="50"
Sep 16 18:43:14.613546 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607717    2582 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 18:43:14.613546 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607720    2582 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 18:43:14.613546 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607723    2582 flags.go:64] FLAG: --eviction-hard=""
Sep 16 18:43:14.613546 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607728    2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 18:43:14.613546 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607731    2582 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 18:43:14.613546 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607734    2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 18:43:14.613546 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607738    2582 flags.go:64] FLAG: --eviction-soft=""
Sep 16 18:43:14.613546 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607740    2582 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 18:43:14.613546 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607744    2582 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 18:43:14.613546 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607747    2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 18:43:14.614185 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607750    2582 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 18:43:14.614185 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607753    2582 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 18:43:14.614185 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607758    2582 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 18:43:14.614185 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607761    2582 flags.go:64] FLAG: --feature-gates=""
Sep 16 18:43:14.614185 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607765    2582 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 18:43:14.614185 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607768    2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 18:43:14.614185 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607771    2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 18:43:14.614185 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607774    2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 18:43:14.614185 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607778    2582 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 18:43:14.614185 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607781    2582 flags.go:64] FLAG: --help="false"
Sep 16 18:43:14.614185 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607784    2582 flags.go:64] FLAG: --hostname-override="ip-10-0-129-194.ec2.internal"
Sep 16 18:43:14.614185 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607788    2582 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 18:43:14.614185 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607791    2582 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 18:43:14.614185 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607794    2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 18:43:14.614185 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607797    2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 18:43:14.614185 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607802    2582 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 18:43:14.614185 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607805    2582 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 18:43:14.614185 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607808    2582 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 18:43:14.614185 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607811    2582 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 18:43:14.614185 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607814    2582 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 18:43:14.614185 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607817    2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 18:43:14.614185 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607820    2582 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 18:43:14.614185 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607823    2582 flags.go:64] FLAG: --kube-reserved=""
Sep 16 18:43:14.614185 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607826    2582 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 18:43:14.614776 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607829    2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 18:43:14.614776 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607832    2582 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 18:43:14.614776 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607836    2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 18:43:14.614776 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607839    2582 flags.go:64] FLAG: --lock-file=""
Sep 16 18:43:14.614776 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607842    2582 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 18:43:14.614776 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607845    2582 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 18:43:14.614776 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607849    2582 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 18:43:14.614776 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607854    2582 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 18:43:14.614776 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607857    2582 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 18:43:14.614776 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607860    2582 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 18:43:14.614776 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607865    2582 flags.go:64] FLAG: --logging-format="text"
Sep 16 18:43:14.614776 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607868    2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 18:43:14.614776 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607872    2582 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 18:43:14.614776 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607875    2582 flags.go:64] FLAG: --manifest-url=""
Sep 16 18:43:14.614776 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607878    2582 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 18:43:14.614776 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607882    2582 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 18:43:14.614776 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607886    2582 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 18:43:14.614776 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607890    2582 flags.go:64] FLAG: --max-pods="110"
Sep 16 18:43:14.614776 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607894    2582 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 18:43:14.614776 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607897    2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 18:43:14.614776 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607901    2582 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 18:43:14.614776 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607904    2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 18:43:14.614776 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607907    2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 18:43:14.614776 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607910    2582 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 18:43:14.614776 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607913    2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 18:43:14.615445 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607921    2582 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 18:43:14.615445 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607924    2582 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 18:43:14.615445 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607927    2582 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 18:43:14.615445 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607930    2582 flags.go:64] FLAG: --pod-cidr=""
Sep 16 18:43:14.615445 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607933    2582 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 18:43:14.615445 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607939    2582 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 18:43:14.615445 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607954    2582 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 18:43:14.615445 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607958    2582 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 18:43:14.615445 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607961    2582 flags.go:64] FLAG: --port="10250"
Sep 16 18:43:14.615445 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607965    2582 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 18:43:14.615445 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607969    2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04afe19f5793171c3"
Sep 16 18:43:14.615445 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607972    2582 flags.go:64] FLAG: --qos-reserved=""
Sep 16 18:43:14.615445 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607975    2582 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 18:43:14.615445 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607978    2582 flags.go:64] FLAG: --register-node="true"
Sep 16 18:43:14.615445 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607981    2582 flags.go:64] FLAG: --register-with-taints=""
Sep 16 18:43:14.615445 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607985    2582 flags.go:64] FLAG: --registry-burst="10"
Sep 16 18:43:14.615445 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607988    2582 flags.go:64] FLAG: --registry-qps="5"
Sep 16 18:43:14.615445 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607993    2582 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 18:43:14.615445 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.607996    2582 flags.go:64] FLAG: --reserved-memory=""
Sep 16 18:43:14.615445 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.608000    2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 18:43:14.615445 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.608003    2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 18:43:14.615445 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.608006    2582 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 18:43:14.615445 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.608009    2582 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 18:43:14.615445 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.608012    2582 flags.go:64] FLAG: --runonce="false"
Sep 16 18:43:14.615445 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.608015    2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 18:43:14.616182 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.608018    2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 18:43:14.616182 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.608021    2582 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 18:43:14.616182 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.608025    2582 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 18:43:14.616182 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.608027    2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 18:43:14.616182 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.608049    2582 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 18:43:14.616182 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.608053    2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 18:43:14.616182 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.608057    2582 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 18:43:14.616182 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.608060    2582 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 18:43:14.616182 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.608063    2582 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 18:43:14.616182 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.608066    2582 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 18:43:14.616182 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.608069    2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 18:43:14.616182 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.608072    2582 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 18:43:14.616182 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.608076    2582 flags.go:64] FLAG: --system-cgroups=""
Sep 16 18:43:14.616182 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.608079    2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 18:43:14.616182 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.608084    2582 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 18:43:14.616182 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.608087    2582 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 18:43:14.616182 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.608093    2582 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 18:43:14.616182 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.608097    2582 flags.go:64] FLAG: --tls-min-version=""
Sep 16 18:43:14.616182 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.608100    2582 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 18:43:14.616182 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.608103    2582 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 18:43:14.616182 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.608106    2582 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 18:43:14.616182 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.608109    2582 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 18:43:14.616182 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.608113    2582 flags.go:64] FLAG: --v="2"
Sep 16 18:43:14.616182 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.608117    2582 flags.go:64] FLAG: --version="false"
Sep 16 18:43:14.616182 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.608123    2582 flags.go:64] FLAG: --vmodule=""
Sep 16 18:43:14.616761 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.608127    2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 18:43:14.616761 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.608131    2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 18:43:14.616761 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608251    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:43:14.616761 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608255    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:43:14.616761 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608258    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:43:14.616761 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608261    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:43:14.616761 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608263    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:43:14.616761 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608266    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:43:14.616761 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608269    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:43:14.616761 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608272    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:43:14.616761 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608275    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:43:14.616761 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608278    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:43:14.616761 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608281    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:43:14.616761 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608283    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:43:14.616761 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608286    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:43:14.616761 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608289    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:43:14.616761 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608292    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:43:14.616761 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608294    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:43:14.616761 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608298    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:43:14.617236 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608301    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:43:14.617236 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608303    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:43:14.617236 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608306    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:43:14.617236 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608309    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:43:14.617236 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608313    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:43:14.617236 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608315    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:43:14.617236 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608318    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:43:14.617236 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608321    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:43:14.617236 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608324    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:43:14.617236 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608327    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:43:14.617236 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608330    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:43:14.617236 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608333    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:43:14.617236 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608337    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:43:14.617236 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608340    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:43:14.617236 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608343    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:43:14.617236 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608346    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:43:14.617236 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608348    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:43:14.617236 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608351    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:43:14.617236 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608354    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:43:14.617697 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608358    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:43:14.617697 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608362    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:43:14.617697 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608365    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:43:14.617697 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608368    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:43:14.617697 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608371    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:43:14.617697 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608374    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:43:14.617697 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608377    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:43:14.617697 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608380    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:43:14.617697 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608382    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:43:14.617697 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608385    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:43:14.617697 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608388    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:43:14.617697 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608391    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:43:14.617697 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608394    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:43:14.617697 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608396    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:43:14.617697 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608399    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:43:14.617697 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608402    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:43:14.617697 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608405    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:43:14.617697 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608409    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:43:14.617697 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608412    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:43:14.617697 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608415    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:43:14.618209 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608418    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:43:14.618209 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608421    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:43:14.618209 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608424    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:43:14.618209 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608427    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:43:14.618209 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608431    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:43:14.618209 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608437    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:43:14.618209 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608440    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:43:14.618209 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608443    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:43:14.618209 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608446    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:43:14.618209 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608450    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:43:14.618209 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608455    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:43:14.618209 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608459    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:43:14.618209 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608463    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:43:14.618209 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608466    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:43:14.618209 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608469    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:43:14.618209 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608473    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:43:14.618209 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608476    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:43:14.618209 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608479    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:43:14.618209 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608481    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:43:14.618209 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608484    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:43:14.618697 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608487    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:43:14.618697 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608490    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:43:14.618697 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608492    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:43:14.618697 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608495    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:43:14.618697 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608498    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:43:14.618697 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608501    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:43:14.618697 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608503    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:43:14.618697 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608506    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:43:14.618697 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608509    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:43:14.618697 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608513    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:43:14.618697 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608516    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:43:14.618697 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608519    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:43:14.618697 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608521    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:43:14.618697 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608525    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:43:14.618697 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608528    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:43:14.618697 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608531    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:43:14.618697 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608533    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:43:14.618697 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608537    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:43:14.618697 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608540    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:43:14.618697 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608543    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:43:14.619218 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608546    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:43:14.619218 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608549    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:43:14.619218 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.608552    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:43:14.619218 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.608557    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:43:14.619218 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.616478    2582 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 18:43:14.619218 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.616495    2582 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 18:43:14.619218 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.616513    2582 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 18:43:14.619218 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.616523    2582 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 18:43:14.619218 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616583    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:43:14.619218 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616588    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:43:14.619218 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616592    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:43:14.619218 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616595    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:43:14.619218 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616599    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:43:14.619218 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616602    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:43:14.619218 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616605    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:43:14.619627 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616608    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:43:14.619627 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616611    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:43:14.619627 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616614    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:43:14.619627 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616617    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:43:14.619627 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616620    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:43:14.619627 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616624    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:43:14.619627 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616628    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:43:14.619627 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616632    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:43:14.619627 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616635    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:43:14.619627 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616638    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:43:14.619627 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616641    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:43:14.619627 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616643    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:43:14.619627 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616646    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:43:14.619627 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616649    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:43:14.619627 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616652    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:43:14.619627 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616655    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:43:14.619627 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616658    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:43:14.619627 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616661    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:43:14.619627 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616663    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:43:14.620180 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616666    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:43:14.620180 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616669    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:43:14.620180 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616672    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:43:14.620180 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616680    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:43:14.620180 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616684    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:43:14.620180 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616686    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:43:14.620180 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616689    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:43:14.620180 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616691    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:43:14.620180 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616694    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:43:14.620180 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616697    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:43:14.620180 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616700    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:43:14.620180 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616702    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:43:14.620180 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616705    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:43:14.620180 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616708    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:43:14.620180 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616711    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:43:14.620180 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616713    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:43:14.620180 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616716    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:43:14.620180 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616719    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:43:14.620180 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616721    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:43:14.620180 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616724    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:43:14.620683 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616727    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:43:14.620683 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616730    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:43:14.620683 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616733    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:43:14.620683 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616736    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:43:14.620683 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616738    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:43:14.620683 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616741    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:43:14.620683 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616744    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:43:14.620683 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616747    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:43:14.620683 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616750    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:43:14.620683 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616752    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:43:14.620683 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616755    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:43:14.620683 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616758    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:43:14.620683 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616760    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:43:14.620683 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616763    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:43:14.620683 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616766    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:43:14.620683 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616769    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:43:14.620683 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616778    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:43:14.620683 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616781    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:43:14.620683 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616783    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:43:14.620683 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616786    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:43:14.621180 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616789    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:43:14.621180 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616791    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:43:14.621180 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616795    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:43:14.621180 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616798    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:43:14.621180 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616802    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:43:14.621180 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616805    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:43:14.621180 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616807    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:43:14.621180 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616810    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:43:14.621180 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616813    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:43:14.621180 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616816    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:43:14.621180 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616819    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:43:14.621180 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616822    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:43:14.621180 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616825    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:43:14.621180 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616827    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:43:14.621180 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616830    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:43:14.621180 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616833    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:43:14.621180 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616835    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:43:14.621180 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616838    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:43:14.621180 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616841    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:43:14.621180 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616844    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:43:14.621683 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616847    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:43:14.621683 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616849    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:43:14.621683 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616852    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:43:14.621683 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616855    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:43:14.621683 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616857    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:43:14.621683 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616860    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:43:14.621683 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616862    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:43:14.621683 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616865    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:43:14.621683 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616868    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:43:14.621683 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616878    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:43:14.621683 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616882    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:43:14.621683 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616885    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:43:14.621683 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.616888    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:43:14.621683 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.616893    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:43:14.621683 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617059    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:43:14.622065 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617065    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:43:14.622065 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617068    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:43:14.622065 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617072    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:43:14.622065 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617075    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:43:14.622065 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617078    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:43:14.622065 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617080    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:43:14.622065 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617083    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:43:14.622065 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617087    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:43:14.622065 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617090    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:43:14.622065 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617093    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:43:14.622065 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617096    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:43:14.622065 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617099    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:43:14.622065 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617102    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:43:14.622065 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617105    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:43:14.622065 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617108    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:43:14.622065 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617111    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:43:14.622065 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617114    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:43:14.622065 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617118    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:43:14.622065 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617122    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:43:14.622517 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617125    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:43:14.622517 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617128    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:43:14.622517 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617131    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:43:14.622517 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617133    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:43:14.622517 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617136    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:43:14.622517 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617139    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:43:14.622517 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617141    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:43:14.622517 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617144    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:43:14.622517 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617152    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:43:14.622517 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617155    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:43:14.622517 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617158    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:43:14.622517 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617161    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:43:14.622517 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617163    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:43:14.622517 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617166    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:43:14.622517 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617169    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:43:14.622517 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617171    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:43:14.622517 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617174    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:43:14.622517 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617177    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:43:14.622517 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617179    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:43:14.623005 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617182    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:43:14.623005 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617185    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:43:14.623005 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617187    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:43:14.623005 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617190    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:43:14.623005 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617193    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:43:14.623005 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617196    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:43:14.623005 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617199    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:43:14.623005 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617201    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:43:14.623005 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617204    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:43:14.623005 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617206    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:43:14.623005 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617209    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:43:14.623005 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617212    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:43:14.623005 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617216    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:43:14.623005 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617219    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:43:14.623005 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617223    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:43:14.623005 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617227    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:43:14.623005 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617230    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:43:14.623005 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617233    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:43:14.623005 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617236    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:43:14.623474 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617239    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:43:14.623474 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617242    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:43:14.623474 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617245    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:43:14.623474 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617253    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:43:14.623474 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617257    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:43:14.623474 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617259    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:43:14.623474 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617262    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:43:14.623474 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617265    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:43:14.623474 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617267    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:43:14.623474 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617270    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:43:14.623474 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617272    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:43:14.623474 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617275    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:43:14.623474 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617278    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:43:14.623474 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617280    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:43:14.623474 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617283    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:43:14.623474 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617285    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:43:14.623474 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617288    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:43:14.623474 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617291    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:43:14.623474 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617294    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:43:14.623474 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617296    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:43:14.624045 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617299    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:43:14.624045 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617301    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:43:14.624045 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617304    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:43:14.624045 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617307    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:43:14.624045 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617310    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:43:14.624045 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617312    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:43:14.624045 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617315    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:43:14.624045 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617318    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:43:14.624045 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617321    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:43:14.624045 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617323    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:43:14.624045 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617326    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:43:14.624045 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617328    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:43:14.624045 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617331    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:43:14.624045 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617334    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:43:14.624045 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617336    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:43:14.624045 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617339    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:43:14.624045 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617347    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:43:14.624045 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617350    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:43:14.624045 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617353    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:43:14.624045 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617355    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:43:14.624557 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:14.617358    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:43:14.624557 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.617363    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:43:14.624557 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.617486    2582 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 18:43:14.624557 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.620051    2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 18:43:14.624557 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.621197    2582 server.go:1019] "Starting client certificate rotation"
Sep 16 18:43:14.624557 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.621295    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:43:14.624557 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.621330    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:43:14.624557 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.624072    2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:43:14.625566 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.625532    2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:43:14.636200 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.636181    2582 log.go:25] "Validated CRI v1 runtime API"
Sep 16 18:43:14.640274 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.640160    2582 log.go:25] "Validated CRI v1 image API"
Sep 16 18:43:14.641143 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.641123    2582 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 18:43:14.644579 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.644557    2582 fs.go:135] Filesystem UUIDs: map[38b8415f-d1a2-4322-8cd6-084eb45a6fda:/dev/nvme0n1p3 47696f96-a0b1-4b4b-a95b-84aec40ecc38:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Sep 16 18:43:14.644647 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.644578    2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 18:43:14.648756 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.648652    2582 manager.go:216] Machine: {Timestamp:2026-09-16 18:43:14.648192491 +0000 UTC m=+0.364680444 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100311 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21c642759211a33c61d8cd2d764871 SystemUUID:ec21c642-7592-11a3-3c61-d8cd2d764871 BootID:97c6421e-cdd1-43f0-910c-3e16e6d9db3e Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:fb:50:3b:d3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:fb:50:3b:d3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:de:31:7d:96:2f:1a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 18:43:14.648756 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.648752    2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 18:43:14.648915 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.648826    2582 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 18:43:14.652565 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.652539    2582 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 18:43:14.652707 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.652566    2582 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-194.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 18:43:14.653413 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.653398    2582 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 18:43:14.653463 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.653417    2582 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 18:43:14.653463 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.653426    2582 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:43:14.653463 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.653441    2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:43:14.654714 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.654704    2582 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 18:43:14.656308 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.656293    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:43:14.660091 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.660078    2582 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:43:14.660220 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.660210    2582 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 18:43:14.660248 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.660241    2582 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 18:43:14.660285 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.660256    2582 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 18:43:14.660285 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.660270    2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 18:43:14.660285 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.660279    2582 kubelet.go:398] "Adding apiserver pod source"
Sep 16 18:43:14.660375 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.660294    2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 18:43:14.660660 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.660647    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:43:14.660713 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.660672    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:43:14.662988 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.662969    2582 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 18:43:14.666010 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.665996    2582 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 18:43:14.666057 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.666026    2582 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 18:43:14.668090 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.668078    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 18:43:14.668144 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.668094    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 18:43:14.668144 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.668101    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 18:43:14.668144 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.668106    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 18:43:14.668144 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.668116    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 18:43:14.668144 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.668122    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 18:43:14.668144 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.668136    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 18:43:14.668144 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.668142    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 18:43:14.668144 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.668148    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 18:43:14.668490 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.668155    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 18:43:14.668490 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.668164    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 18:43:14.668490 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.668169    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 18:43:14.668490 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.668190    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 18:43:14.668490 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.668195    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 18:43:14.672022 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.672009    2582 server.go:1268] "Started kubelet"
Sep 16 18:43:14.672117 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.672090    2582 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 18:43:14.672175 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.672090    2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 18:43:14.672175 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.672163    2582 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 18:43:14.672862 ip-10-0-129-194 systemd[1]: Started Kubernetes Kubelet.
Sep 16 18:43:14.677375 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.677340    2582 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 18:43:14.679287 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.679264    2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-194.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 18:43:14.679407 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:14.679377    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-194.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 18:43:14.679506 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:14.679399    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 18:43:14.680431 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:14.680398    2582 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 18:43:14.681220 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.681205    2582 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 18:43:14.684984 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:14.683821    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-194.ec2.internal.18d5e157eb13c085  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-194.ec2.internal,UID:ip-10-0-129-194.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-194.ec2.internal,},FirstTimestamp:2026-09-16 18:43:14.671984773 +0000 UTC m=+0.388472722,LastTimestamp:2026-09-16 18:43:14.671984773 +0000 UTC m=+0.388472722,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-194.ec2.internal,}"
Sep 16 18:43:14.685775 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.685762    2582 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 18:43:14.685828 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.685779    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 18:43:14.686000 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:14.685959    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-194.ec2.internal\" not found"
Sep 16 18:43:14.686543 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.686526    2582 factory.go:55] Registering systemd factory
Sep 16 18:43:14.686642 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.686547    2582 factory.go:223] Registration of the systemd container factory successfully
Sep 16 18:43:14.686642 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.686546    2582 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 18:43:14.686642 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.686562    2582 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 18:43:14.686642 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.686545    2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 18:43:14.686821 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.686694    2582 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 18:43:14.686821 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.686703    2582 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 18:43:14.686821 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.686773    2582 factory.go:153] Registering CRI-O factory
Sep 16 18:43:14.686821 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.686783    2582 factory.go:223] Registration of the crio container factory successfully
Sep 16 18:43:14.686976 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.686825    2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 18:43:14.686976 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.686842    2582 factory.go:103] Registering Raw factory
Sep 16 18:43:14.686976 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.686856    2582 manager.go:1196] Started watching for new ooms in manager
Sep 16 18:43:14.687251 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.687240    2582 manager.go:349] Starting recovery of all containers
Sep 16 18:43:14.690370 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:14.690342    2582 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-194.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 18:43:14.690516 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:14.690492    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 18:43:14.697818 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.697806    2582 manager.go:354] Recovery completed
Sep 16 18:43:14.700678 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:14.700647    2582 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Sep 16 18:43:14.704305 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.704293    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:43:14.705812 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.705794    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-194.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:43:14.705875 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.705825    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-194.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:43:14.705875 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.705843    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-194.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:43:14.706401 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.706388    2582 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 18:43:14.706453 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.706399    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:43:14.706453 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.706400    2582 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 18:43:14.706514 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.706460    2582 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:43:14.707172 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.707158    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-194.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:43:14.707230 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.707184    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-194.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:43:14.707230 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.707196    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-194.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:43:14.707924 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:14.707852    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-194.ec2.internal.18d5e157ed17e249  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-194.ec2.internal,UID:ip-10-0-129-194.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-194.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-194.ec2.internal,},FirstTimestamp:2026-09-16 18:43:14.705809993 +0000 UTC m=+0.422297943,LastTimestamp:2026-09-16 18:43:14.705809993 +0000 UTC m=+0.422297943,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-194.ec2.internal,}"
Sep 16 18:43:14.708663 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.708652    2582 policy_none.go:49] "None policy: Start"
Sep 16 18:43:14.708698 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.708667    2582 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 18:43:14.708698 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.708677    2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 18:43:14.710439 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.710420    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lgqg5"
Sep 16 18:43:14.710714 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.710703    2582 policy_none.go:47] "Start"
Sep 16 18:43:14.718146 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:14.717940    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-194.ec2.internal.18d5e157ed1839a6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-194.ec2.internal,UID:ip-10-0-129-194.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-194.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-194.ec2.internal,},FirstTimestamp:2026-09-16 18:43:14.705832358 +0000 UTC m=+0.422320312,LastTimestamp:2026-09-16 18:43:14.705832358 +0000 UTC m=+0.422320312,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-194.ec2.internal,}"
Sep 16 18:43:14.719845 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.719827    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lgqg5"
Sep 16 18:43:14.744690 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.744673    2582 manager.go:337] "Starting Device Plugin manager"
Sep 16 18:43:14.768344 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:14.744724    2582 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 18:43:14.768344 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.744738    2582 server.go:87] "Starting device plugin registration server"
Sep 16 18:43:14.768344 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.745098    2582 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 18:43:14.768344 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.745111    2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 18:43:14.768344 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.745206    2582 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 18:43:14.768344 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.745287    2582 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 18:43:14.768344 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.745298    2582 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 18:43:14.768344 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:14.746089    2582 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 18:43:14.768344 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:14.746138    2582 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-194.ec2.internal\" not found"
Sep 16 18:43:14.790366 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.790331    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 18:43:14.791626 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.791600    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 18:43:14.791742 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.791666    2582 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 18:43:14.791742 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.791697    2582 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 18:43:14.791818 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:14.791742    2582 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 18:43:14.794100 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.794079    2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:43:14.845751 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.845710    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:43:14.846754 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.846738    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-194.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:43:14.846833 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.846769    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-194.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:43:14.846833 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.846780    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-194.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:43:14.846833 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.846802    2582 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-194.ec2.internal"
Sep 16 18:43:14.855920 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.855852    2582 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-194.ec2.internal"
Sep 16 18:43:14.855920 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:14.855878    2582 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-194.ec2.internal\": node \"ip-10-0-129-194.ec2.internal\" not found"
Sep 16 18:43:14.879117 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:14.879088    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-194.ec2.internal\" not found"
Sep 16 18:43:14.892448 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.892419    2582 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-194.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-194.ec2.internal"]
Sep 16 18:43:14.892527 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.892499    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:43:14.893418 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.893405    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-194.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:43:14.893487 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.893431    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-194.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:43:14.893487 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.893443    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-194.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:43:14.895856 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.895843    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:43:14.896033 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.896008    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-194.ec2.internal"
Sep 16 18:43:14.896078 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.896055    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:43:14.896578 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.896562    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-194.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:43:14.896643 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.896594    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-194.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:43:14.896643 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.896604    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-194.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:43:14.896716 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.896562    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-194.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:43:14.896716 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.896666    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-194.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:43:14.896716 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.896678    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-194.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:43:14.898957 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.898933    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-194.ec2.internal"
Sep 16 18:43:14.899030 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.898972    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:43:14.899660 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.899642    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-194.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:43:14.899733 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.899677    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-194.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:43:14.899733 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:14.899691    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-194.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:43:14.910522 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:14.910504    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-194.ec2.internal\" not found" node="ip-10-0-129-194.ec2.internal"
Sep 16 18:43:14.913874 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:14.913858    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-194.ec2.internal\" not found" node="ip-10-0-129-194.ec2.internal"
Sep 16 18:43:14.979356 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:14.979322    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-194.ec2.internal\" not found"
Sep 16 18:43:15.079781 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:15.079750    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-194.ec2.internal\" not found"
Sep 16 18:43:15.088064 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:15.088035    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a8fee4f0c396919249581e1ee599be3f-config\") pod \"kube-apiserver-proxy-ip-10-0-129-194.ec2.internal\" (UID: \"a8fee4f0c396919249581e1ee599be3f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-194.ec2.internal"
Sep 16 18:43:15.088163 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:15.088069    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/65b8cacc050a0fe82b20ca75521a82f8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-194.ec2.internal\" (UID: \"65b8cacc050a0fe82b20ca75521a82f8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-194.ec2.internal"
Sep 16 18:43:15.088163 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:15.088097    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65b8cacc050a0fe82b20ca75521a82f8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-194.ec2.internal\" (UID: \"65b8cacc050a0fe82b20ca75521a82f8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-194.ec2.internal"
Sep 16 18:43:15.180457 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:15.180427    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-194.ec2.internal\" not found"
Sep 16 18:43:15.188815 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:15.188788    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/65b8cacc050a0fe82b20ca75521a82f8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-194.ec2.internal\" (UID: \"65b8cacc050a0fe82b20ca75521a82f8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-194.ec2.internal"
Sep 16 18:43:15.188877 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:15.188821    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65b8cacc050a0fe82b20ca75521a82f8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-194.ec2.internal\" (UID: \"65b8cacc050a0fe82b20ca75521a82f8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-194.ec2.internal"
Sep 16 18:43:15.188877 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:15.188839    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a8fee4f0c396919249581e1ee599be3f-config\") pod \"kube-apiserver-proxy-ip-10-0-129-194.ec2.internal\" (UID: \"a8fee4f0c396919249581e1ee599be3f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-194.ec2.internal"
Sep 16 18:43:15.188942 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:15.188896    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/65b8cacc050a0fe82b20ca75521a82f8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-194.ec2.internal\" (UID: \"65b8cacc050a0fe82b20ca75521a82f8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-194.ec2.internal"
Sep 16 18:43:15.188942 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:15.188906    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65b8cacc050a0fe82b20ca75521a82f8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-194.ec2.internal\" (UID: \"65b8cacc050a0fe82b20ca75521a82f8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-194.ec2.internal"
Sep 16 18:43:15.189025 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:15.188980    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a8fee4f0c396919249581e1ee599be3f-config\") pod \"kube-apiserver-proxy-ip-10-0-129-194.ec2.internal\" (UID: \"a8fee4f0c396919249581e1ee599be3f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-194.ec2.internal"
Sep 16 18:43:15.212957 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:15.212924    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-194.ec2.internal"
Sep 16 18:43:15.216522 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:15.216498    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-194.ec2.internal"
Sep 16 18:43:17.356178 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:17.356143    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-194.ec2.internal\" not found"
Sep 16 18:43:17.456790 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:17.456752    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-194.ec2.internal\" not found"
Sep 16 18:43:17.557384 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:17.557303    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-194.ec2.internal\" not found"
Sep 16 18:43:17.657814 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:17.657781    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-194.ec2.internal\" not found"
Sep 16 18:43:17.695315 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:17.695290    2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 18:43:17.695795 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:17.695469    2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:43:17.758699 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:17.758465    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-194.ec2.internal\" not found"
Sep 16 18:43:17.800179 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:17.800128    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 18:38:16 +0000 UTC" deadline="2028-07-02 07:33:32.581063686 +0000 UTC"
Sep 16 18:43:17.800179 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:17.800174    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15708h50m14.78089343s"
Sep 16 18:43:17.806966 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:17.806943    2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:43:17.846593 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:17.846560    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65b8cacc050a0fe82b20ca75521a82f8.slice/crio-95643a65399664eca12516fdbc73b38ad4c1ae619170bc0f74fcac7baaef782d WatchSource:0}: Error finding container 95643a65399664eca12516fdbc73b38ad4c1ae619170bc0f74fcac7baaef782d: Status 404 returned error can't find the container with id 95643a65399664eca12516fdbc73b38ad4c1ae619170bc0f74fcac7baaef782d
Sep 16 18:43:17.847264 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:17.847246    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8fee4f0c396919249581e1ee599be3f.slice/crio-b4c7f86c89b5091e7cb3a191e1eae5675f6c0f04bebb52abd5c897a81141227d WatchSource:0}: Error finding container b4c7f86c89b5091e7cb3a191e1eae5675f6c0f04bebb52abd5c897a81141227d: Status 404 returned error can't find the container with id b4c7f86c89b5091e7cb3a191e1eae5675f6c0f04bebb52abd5c897a81141227d
Sep 16 18:43:17.851025 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:17.851012    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:43:17.859489 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:17.859472    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-194.ec2.internal\" not found"
Sep 16 18:43:17.870229 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:17.870191    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-194.ec2.internal" event={"ID":"a8fee4f0c396919249581e1ee599be3f","Type":"ContainerStarted","Data":"b4c7f86c89b5091e7cb3a191e1eae5675f6c0f04bebb52abd5c897a81141227d"}
Sep 16 18:43:17.871238 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:17.871220    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-194.ec2.internal" event={"ID":"65b8cacc050a0fe82b20ca75521a82f8","Type":"ContainerStarted","Data":"95643a65399664eca12516fdbc73b38ad4c1ae619170bc0f74fcac7baaef782d"}
Sep 16 18:43:17.960475 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:17.960438    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-194.ec2.internal\" not found"
Sep 16 18:43:17.987438 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:17.987413    2582 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:43:18.061389 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:18.061340    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-194.ec2.internal\" not found"
Sep 16 18:43:18.161855 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:18.161775    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-194.ec2.internal\" not found"
Sep 16 18:43:18.262804 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:18.262770    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-194.ec2.internal\" not found"
Sep 16 18:43:18.278935 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.278749    2582 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:43:18.361193 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.361157    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-194.ec2.internal"
Sep 16 18:43:18.375619 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.375592    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:43:18.376609 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.376583    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-194.ec2.internal"
Sep 16 18:43:18.386850 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.386820    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:43:18.736281 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.736246    2582 apiserver.go:52] "Watching apiserver"
Sep 16 18:43:18.744640 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.744610    2582 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 18:43:18.747054 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.747023    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-tplxv","kube-system/konnectivity-agent-pr4kj","kube-system/kube-apiserver-proxy-ip-10-0-129-194.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5hzq","openshift-cluster-node-tuning-operator/tuned-7kkxp","openshift-image-registry/node-ca-w8t8m","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-194.ec2.internal","openshift-multus/network-metrics-daemon-g7q9m","openshift-dns/node-resolver-wdq46","openshift-multus/multus-6nmxb","openshift-multus/multus-additional-cni-plugins-rkgjp","openshift-network-diagnostics/network-check-target-dgl4n","openshift-network-operator/iptables-alerter-vmbb4","openshift-ovn-kubernetes/ovnkube-node-pv8wf"]
Sep 16 18:43:18.749606 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.749581    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vmbb4"
Sep 16 18:43:18.751789 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.751764    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pr4kj"
Sep 16 18:43:18.752237 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.752212    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wxf5q\""
Sep 16 18:43:18.752406 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.752383    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 18:43:18.752517 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.752499    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:43:18.752616 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.752600    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 18:43:18.753966 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.753943    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6nmxb"
Sep 16 18:43:18.754145 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.754125    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7lj9r\""
Sep 16 18:43:18.754381 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.754368    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 18:43:18.754675 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.754623    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 18:43:18.756143 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.756127    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5hzq"
Sep 16 18:43:18.756375 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.756356    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 18:43:18.756680 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.756497    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-7ksft\""
Sep 16 18:43:18.756825 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.756809    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 18:43:18.756895 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.756819    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 18:43:18.756895 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.756809    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 18:43:18.758506 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.758489    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7kkxp"
Sep 16 18:43:18.758595 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.758530    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 18:43:18.758668 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.758650    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 18:43:18.758739 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.758676    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-twbgl\""
Sep 16 18:43:18.758895 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.758877    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 18:43:18.760803 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.760781    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-w8t8m"
Sep 16 18:43:18.761163 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.761147    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ttpm5\""
Sep 16 18:43:18.761251 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.761182    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:43:18.762051 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.761854    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 18:43:18.762051 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.761858    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 18:43:18.763347 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.763080    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g7q9m"
Sep 16 18:43:18.763347 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.763089    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 18:43:18.763347 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.763106    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 18:43:18.763347 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.763081    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 18:43:18.763347 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:18.763193    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g7q9m" podUID="de8ddfa3-fd54-49d7-8f90-035aa4439e98"
Sep 16 18:43:18.763347 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.763246    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lljrs\""
Sep 16 18:43:18.765386 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.765368    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wdq46"
Sep 16 18:43:18.767618 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.767599    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tplxv"
Sep 16 18:43:18.767711 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:18.767671    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tplxv" podUID="e6428035-ca16-4098-ac4e-1ca4931bb7a7"
Sep 16 18:43:18.768052 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.768030    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 18:43:18.768137 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.768107    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rx2ts\""
Sep 16 18:43:18.768188 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.768034    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 18:43:18.770136 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.770117    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rkgjp"
Sep 16 18:43:18.771031 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.771008    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/870ff296-8673-44d5-9fc3-619486d607a1-serviceca\") pod \"node-ca-w8t8m\" (UID: \"870ff296-8673-44d5-9fc3-619486d607a1\") " pod="openshift-image-registry/node-ca-w8t8m"
Sep 16 18:43:18.771119 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.771038    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/245475f1-ca6f-40bb-8ab5-a2ab8f46c538-system-cni-dir\") pod \"multus-6nmxb\" (UID: \"245475f1-ca6f-40bb-8ab5-a2ab8f46c538\") " pod="openshift-multus/multus-6nmxb"
Sep 16 18:43:18.771119 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.771062    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a1140d90-5f02-4921-b4ce-40e3a7129314-etc-systemd\") pod \"tuned-7kkxp\" (UID: \"a1140d90-5f02-4921-b4ce-40e3a7129314\") " pod="openshift-cluster-node-tuning-operator/tuned-7kkxp"
Sep 16 18:43:18.771119 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.771090    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9f7c6c50-3969-4103-a47f-d4eea4196e74-host-slash\") pod \"iptables-alerter-vmbb4\" (UID: \"9f7c6c50-3969-4103-a47f-d4eea4196e74\") " pod="openshift-network-operator/iptables-alerter-vmbb4"
Sep 16 18:43:18.771246 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.771122    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bfffe95c-37f4-4aab-a7aa-19c64f66e810-agent-certs\") pod \"konnectivity-agent-pr4kj\" (UID: \"bfffe95c-37f4-4aab-a7aa-19c64f66e810\") " pod="kube-system/konnectivity-agent-pr4kj"
Sep 16 18:43:18.771246 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.771146    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bfffe95c-37f4-4aab-a7aa-19c64f66e810-konnectivity-ca\") pod \"konnectivity-agent-pr4kj\" (UID: \"bfffe95c-37f4-4aab-a7aa-19c64f66e810\") " pod="kube-system/konnectivity-agent-pr4kj"
Sep 16 18:43:18.771246 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.771186    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de8ddfa3-fd54-49d7-8f90-035aa4439e98-metrics-certs\") pod \"network-metrics-daemon-g7q9m\" (UID: \"de8ddfa3-fd54-49d7-8f90-035aa4439e98\") " pod="openshift-multus/network-metrics-daemon-g7q9m"
Sep 16 18:43:18.771373 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.771251    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/245475f1-ca6f-40bb-8ab5-a2ab8f46c538-cni-binary-copy\") pod \"multus-6nmxb\" (UID: \"245475f1-ca6f-40bb-8ab5-a2ab8f46c538\") " pod="openshift-multus/multus-6nmxb"
Sep 16 18:43:18.771373 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.771276    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a1140d90-5f02-4921-b4ce-40e3a7129314-host\") pod \"tuned-7kkxp\" (UID: \"a1140d90-5f02-4921-b4ce-40e3a7129314\") " pod="openshift-cluster-node-tuning-operator/tuned-7kkxp"
Sep 16 18:43:18.771460 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.771376    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnnw4\" (UniqueName: \"kubernetes.io/projected/245475f1-ca6f-40bb-8ab5-a2ab8f46c538-kube-api-access-rnnw4\") pod \"multus-6nmxb\" (UID: \"245475f1-ca6f-40bb-8ab5-a2ab8f46c538\") " pod="openshift-multus/multus-6nmxb"
Sep 16 18:43:18.771460 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.771413    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6kt5\" (UniqueName: \"kubernetes.io/projected/de8ddfa3-fd54-49d7-8f90-035aa4439e98-kube-api-access-b6kt5\") pod \"network-metrics-daemon-g7q9m\" (UID: \"de8ddfa3-fd54-49d7-8f90-035aa4439e98\") " pod="openshift-multus/network-metrics-daemon-g7q9m"
Sep 16 18:43:18.771460 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.771436    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/245475f1-ca6f-40bb-8ab5-a2ab8f46c538-multus-socket-dir-parent\") pod \"multus-6nmxb\" (UID: \"245475f1-ca6f-40bb-8ab5-a2ab8f46c538\") " pod="openshift-multus/multus-6nmxb"
Sep 16 18:43:18.771606 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.771458    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/245475f1-ca6f-40bb-8ab5-a2ab8f46c538-host-var-lib-kubelet\") pod \"multus-6nmxb\" (UID: \"245475f1-ca6f-40bb-8ab5-a2ab8f46c538\") " pod="openshift-multus/multus-6nmxb"
Sep 16 18:43:18.771606 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.771513    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9f7c6c50-3969-4103-a47f-d4eea4196e74-iptables-alerter-script\") pod \"iptables-alerter-vmbb4\" (UID: \"9f7c6c50-3969-4103-a47f-d4eea4196e74\") " pod="openshift-network-operator/iptables-alerter-vmbb4"
Sep 16 18:43:18.771606 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.771548    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/47ed2ebe-2b00-4044-b292-59a9ab8fafca-device-dir\") pod \"aws-ebs-csi-driver-node-f5hzq\" (UID: \"47ed2ebe-2b00-4044-b292-59a9ab8fafca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5hzq"
Sep 16 18:43:18.771606 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.771574    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mhp7\" (UniqueName: \"kubernetes.io/projected/47ed2ebe-2b00-4044-b292-59a9ab8fafca-kube-api-access-8mhp7\") pod \"aws-ebs-csi-driver-node-f5hzq\" (UID: \"47ed2ebe-2b00-4044-b292-59a9ab8fafca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5hzq"
Sep 16 18:43:18.771606 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.771599    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/245475f1-ca6f-40bb-8ab5-a2ab8f46c538-cnibin\") pod \"multus-6nmxb\" (UID: \"245475f1-ca6f-40bb-8ab5-a2ab8f46c538\") " pod="openshift-multus/multus-6nmxb"
Sep 16 18:43:18.771883 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.771625    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/245475f1-ca6f-40bb-8ab5-a2ab8f46c538-multus-conf-dir\") pod \"multus-6nmxb\" (UID: \"245475f1-ca6f-40bb-8ab5-a2ab8f46c538\") " pod="openshift-multus/multus-6nmxb"
Sep 16 18:43:18.771883 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.771693    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a1140d90-5f02-4921-b4ce-40e3a7129314-run\") pod \"tuned-7kkxp\" (UID: \"a1140d90-5f02-4921-b4ce-40e3a7129314\") " pod="openshift-cluster-node-tuning-operator/tuned-7kkxp"
Sep 16 18:43:18.771883 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.771801    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c7ns\" (UniqueName: \"kubernetes.io/projected/a1140d90-5f02-4921-b4ce-40e3a7129314-kube-api-access-9c7ns\") pod \"tuned-7kkxp\" (UID: \"a1140d90-5f02-4921-b4ce-40e3a7129314\") " pod="openshift-cluster-node-tuning-operator/tuned-7kkxp"
Sep 16 18:43:18.771883 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.771826    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/870ff296-8673-44d5-9fc3-619486d607a1-host\") pod \"node-ca-w8t8m\" (UID: \"870ff296-8673-44d5-9fc3-619486d607a1\") " pod="openshift-image-registry/node-ca-w8t8m"
Sep 16 18:43:18.771883 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.771848    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/245475f1-ca6f-40bb-8ab5-a2ab8f46c538-host-run-k8s-cni-cncf-io\") pod \"multus-6nmxb\" (UID: \"245475f1-ca6f-40bb-8ab5-a2ab8f46c538\") " pod="openshift-multus/multus-6nmxb"
Sep 16 18:43:18.771883 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.771871    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/245475f1-ca6f-40bb-8ab5-a2ab8f46c538-host-var-lib-cni-multus\") pod \"multus-6nmxb\" (UID: \"245475f1-ca6f-40bb-8ab5-a2ab8f46c538\") " pod="openshift-multus/multus-6nmxb"
Sep 16 18:43:18.772179 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.771906    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/245475f1-ca6f-40bb-8ab5-a2ab8f46c538-multus-daemon-config\") pod \"multus-6nmxb\" (UID: \"245475f1-ca6f-40bb-8ab5-a2ab8f46c538\") " pod="openshift-multus/multus-6nmxb"
Sep 16 18:43:18.772179 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.771933    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/245475f1-ca6f-40bb-8ab5-a2ab8f46c538-host-run-multus-certs\") pod \"multus-6nmxb\" (UID: \"245475f1-ca6f-40bb-8ab5-a2ab8f46c538\") " pod="openshift-multus/multus-6nmxb"
Sep 16 18:43:18.772179 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.771955    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1140d90-5f02-4921-b4ce-40e3a7129314-etc-kubernetes\") pod \"tuned-7kkxp\" (UID: \"a1140d90-5f02-4921-b4ce-40e3a7129314\") " pod="openshift-cluster-node-tuning-operator/tuned-7kkxp"
Sep 16 18:43:18.772179 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.771994    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1140d90-5f02-4921-b4ce-40e3a7129314-sys\") pod \"tuned-7kkxp\" (UID: \"a1140d90-5f02-4921-b4ce-40e3a7129314\") " pod="openshift-cluster-node-tuning-operator/tuned-7kkxp"
Sep 16 18:43:18.772179 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.772039    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/47ed2ebe-2b00-4044-b292-59a9ab8fafca-socket-dir\") pod \"aws-ebs-csi-driver-node-f5hzq\" (UID: \"47ed2ebe-2b00-4044-b292-59a9ab8fafca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5hzq"
Sep 16 18:43:18.772179 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.772084    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/47ed2ebe-2b00-4044-b292-59a9ab8fafca-sys-fs\") pod \"aws-ebs-csi-driver-node-f5hzq\" (UID: \"47ed2ebe-2b00-4044-b292-59a9ab8fafca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5hzq"
Sep 16 18:43:18.772179 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.772118    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/245475f1-ca6f-40bb-8ab5-a2ab8f46c538-etc-kubernetes\") pod \"multus-6nmxb\" (UID: \"245475f1-ca6f-40bb-8ab5-a2ab8f46c538\") " pod="openshift-multus/multus-6nmxb"
Sep 16 18:43:18.772179 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.772142    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a1140d90-5f02-4921-b4ce-40e3a7129314-etc-sysconfig\") pod \"tuned-7kkxp\" (UID: \"a1140d90-5f02-4921-b4ce-40e3a7129314\") " pod="openshift-cluster-node-tuning-operator/tuned-7kkxp"
Sep 16 18:43:18.772179 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.772164    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a1140d90-5f02-4921-b4ce-40e3a7129314-lib-modules\") pod \"tuned-7kkxp\" (UID: \"a1140d90-5f02-4921-b4ce-40e3a7129314\") " pod="openshift-cluster-node-tuning-operator/tuned-7kkxp"
Sep 16 18:43:18.772658 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.772204    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z79jh\" (UniqueName: \"kubernetes.io/projected/870ff296-8673-44d5-9fc3-619486d607a1-kube-api-access-z79jh\") pod \"node-ca-w8t8m\" (UID: \"870ff296-8673-44d5-9fc3-619486d607a1\") " pod="openshift-image-registry/node-ca-w8t8m"
Sep 16 18:43:18.772658 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.772232    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hrb6\" (UniqueName: \"kubernetes.io/projected/9f7c6c50-3969-4103-a47f-d4eea4196e74-kube-api-access-7hrb6\") pod \"iptables-alerter-vmbb4\" (UID: \"9f7c6c50-3969-4103-a47f-d4eea4196e74\") " pod="openshift-network-operator/iptables-alerter-vmbb4"
Sep 16 18:43:18.772658 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.772259    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/47ed2ebe-2b00-4044-b292-59a9ab8fafca-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f5hzq\" (UID: \"47ed2ebe-2b00-4044-b292-59a9ab8fafca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5hzq"
Sep 16 18:43:18.772658 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.772282    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/47ed2ebe-2b00-4044-b292-59a9ab8fafca-registration-dir\") pod \"aws-ebs-csi-driver-node-f5hzq\" (UID: \"47ed2ebe-2b00-4044-b292-59a9ab8fafca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5hzq"
Sep 16 18:43:18.772658 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.772350    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/245475f1-ca6f-40bb-8ab5-a2ab8f46c538-host-var-lib-cni-bin\") pod \"multus-6nmxb\" (UID: \"245475f1-ca6f-40bb-8ab5-a2ab8f46c538\") " pod="openshift-multus/multus-6nmxb"
Sep 16 18:43:18.772658 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.772406    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/245475f1-ca6f-40bb-8ab5-a2ab8f46c538-hostroot\") pod \"multus-6nmxb\" (UID: \"245475f1-ca6f-40bb-8ab5-a2ab8f46c538\") " pod="openshift-multus/multus-6nmxb"
Sep 16 18:43:18.772658 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.772446    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a1140d90-5f02-4921-b4ce-40e3a7129314-etc-modprobe-d\") pod \"tuned-7kkxp\" (UID: \"a1140d90-5f02-4921-b4ce-40e3a7129314\") " pod="openshift-cluster-node-tuning-operator/tuned-7kkxp"
Sep 16 18:43:18.772658 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.772469    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a1140d90-5f02-4921-b4ce-40e3a7129314-etc-sysctl-conf\") pod \"tuned-7kkxp\" (UID: \"a1140d90-5f02-4921-b4ce-40e3a7129314\") " pod="openshift-cluster-node-tuning-operator/tuned-7kkxp"
Sep 16 18:43:18.772658 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.772498    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1140d90-5f02-4921-b4ce-40e3a7129314-var-lib-kubelet\") pod \"tuned-7kkxp\" (UID: \"a1140d90-5f02-4921-b4ce-40e3a7129314\") " pod="openshift-cluster-node-tuning-operator/tuned-7kkxp"
Sep 16 18:43:18.772658 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.772534    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/47ed2ebe-2b00-4044-b292-59a9ab8fafca-etc-selinux\") pod \"aws-ebs-csi-driver-node-f5hzq\" (UID: \"47ed2ebe-2b00-4044-b292-59a9ab8fafca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5hzq"
Sep 16 18:43:18.772658 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.772555    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fh4l5\""
Sep 16 18:43:18.772658 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.772563    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/245475f1-ca6f-40bb-8ab5-a2ab8f46c538-multus-cni-dir\") pod \"multus-6nmxb\" (UID: \"245475f1-ca6f-40bb-8ab5-a2ab8f46c538\") " pod="openshift-multus/multus-6nmxb"
Sep 16 18:43:18.772658 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.772581    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/245475f1-ca6f-40bb-8ab5-a2ab8f46c538-host-run-netns\") pod \"multus-6nmxb\" (UID: \"245475f1-ca6f-40bb-8ab5-a2ab8f46c538\") " pod="openshift-multus/multus-6nmxb"
Sep 16 18:43:18.772658 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.772628    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a1140d90-5f02-4921-b4ce-40e3a7129314-etc-sysctl-d\") pod \"tuned-7kkxp\" (UID: \"a1140d90-5f02-4921-b4ce-40e3a7129314\") " pod="openshift-cluster-node-tuning-operator/tuned-7kkxp"
Sep 16 18:43:18.772658 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.772651    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a1140d90-5f02-4921-b4ce-40e3a7129314-etc-tuned\") pod \"tuned-7kkxp\" (UID: \"a1140d90-5f02-4921-b4ce-40e3a7129314\") " pod="openshift-cluster-node-tuning-operator/tuned-7kkxp"
Sep 16 18:43:18.773399 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.772685    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a1140d90-5f02-4921-b4ce-40e3a7129314-tmp\") pod \"tuned-7kkxp\" (UID: \"a1140d90-5f02-4921-b4ce-40e3a7129314\") " pod="openshift-cluster-node-tuning-operator/tuned-7kkxp"
Sep 16 18:43:18.773399 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.772713    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 18:43:18.773399 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.772797    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 18:43:18.773399 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.772985    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgl4n"
Sep 16 18:43:18.773399 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:18.773040    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgl4n" podUID="f79ebb7c-f4b5-4e48-9506-81110189ba43"
Sep 16 18:43:18.774844 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.774823    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:43:18.775253 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.775236    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.778032 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.778016    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 18:43:18.779683 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.779663    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 18:43:18.779792 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.779750    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 18:43:18.779792 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.779783    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ql55z\""
Sep 16 18:43:18.779919 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.779802    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 18:43:18.779919 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.779882    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 18:43:18.780508 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.780490    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 18:43:18.804434 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.804411    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-s6gxb"
Sep 16 18:43:18.812423 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.812397    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-s6gxb"
Sep 16 18:43:18.863094 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.863066    2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 18:43:18.873695 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.873667    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz9ps\" (UniqueName: \"kubernetes.io/projected/b8dac3a1-52c7-4402-a075-59d997b756ef-kube-api-access-fz9ps\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.873879 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.873704    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/47ed2ebe-2b00-4044-b292-59a9ab8fafca-device-dir\") pod \"aws-ebs-csi-driver-node-f5hzq\" (UID: \"47ed2ebe-2b00-4044-b292-59a9ab8fafca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5hzq"
Sep 16 18:43:18.873879 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.873745    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mhp7\" (UniqueName: \"kubernetes.io/projected/47ed2ebe-2b00-4044-b292-59a9ab8fafca-kube-api-access-8mhp7\") pod \"aws-ebs-csi-driver-node-f5hzq\" (UID: \"47ed2ebe-2b00-4044-b292-59a9ab8fafca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5hzq"
Sep 16 18:43:18.873879 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.873766    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/245475f1-ca6f-40bb-8ab5-a2ab8f46c538-multus-conf-dir\") pod \"multus-6nmxb\" (UID: \"245475f1-ca6f-40bb-8ab5-a2ab8f46c538\") " pod="openshift-multus/multus-6nmxb"
Sep 16 18:43:18.873879 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.873787    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c7ns\" (UniqueName: \"kubernetes.io/projected/a1140d90-5f02-4921-b4ce-40e3a7129314-kube-api-access-9c7ns\") pod \"tuned-7kkxp\" (UID: \"a1140d90-5f02-4921-b4ce-40e3a7129314\") " pod="openshift-cluster-node-tuning-operator/tuned-7kkxp"
Sep 16 18:43:18.873879 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.873807    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/870ff296-8673-44d5-9fc3-619486d607a1-host\") pod \"node-ca-w8t8m\" (UID: \"870ff296-8673-44d5-9fc3-619486d607a1\") " pod="openshift-image-registry/node-ca-w8t8m"
Sep 16 18:43:18.873879 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.873820    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/47ed2ebe-2b00-4044-b292-59a9ab8fafca-device-dir\") pod \"aws-ebs-csi-driver-node-f5hzq\" (UID: \"47ed2ebe-2b00-4044-b292-59a9ab8fafca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5hzq"
Sep 16 18:43:18.873879 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.873830    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b8dac3a1-52c7-4402-a075-59d997b756ef-host-cni-bin\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.874229 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.873958    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/245475f1-ca6f-40bb-8ab5-a2ab8f46c538-multus-conf-dir\") pod \"multus-6nmxb\" (UID: \"245475f1-ca6f-40bb-8ab5-a2ab8f46c538\") " pod="openshift-multus/multus-6nmxb"
Sep 16 18:43:18.874229 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.873973    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/245475f1-ca6f-40bb-8ab5-a2ab8f46c538-host-run-k8s-cni-cncf-io\") pod \"multus-6nmxb\" (UID: \"245475f1-ca6f-40bb-8ab5-a2ab8f46c538\") " pod="openshift-multus/multus-6nmxb"
Sep 16 18:43:18.874229 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.873981    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/870ff296-8673-44d5-9fc3-619486d607a1-host\") pod \"node-ca-w8t8m\" (UID: \"870ff296-8673-44d5-9fc3-619486d607a1\") " pod="openshift-image-registry/node-ca-w8t8m"
Sep 16 18:43:18.874229 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.874042    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1140d90-5f02-4921-b4ce-40e3a7129314-etc-kubernetes\") pod \"tuned-7kkxp\" (UID: \"a1140d90-5f02-4921-b4ce-40e3a7129314\") " pod="openshift-cluster-node-tuning-operator/tuned-7kkxp"
Sep 16 18:43:18.874229 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.874106    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/245475f1-ca6f-40bb-8ab5-a2ab8f46c538-host-run-k8s-cni-cncf-io\") pod \"multus-6nmxb\" (UID: \"245475f1-ca6f-40bb-8ab5-a2ab8f46c538\") " pod="openshift-multus/multus-6nmxb"
Sep 16 18:43:18.874229 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.874123    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b8dac3a1-52c7-4402-a075-59d997b756ef-host-run-ovn-kubernetes\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.874229 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.874160    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c7890694-9f4a-4bf0-9647-1d84e8f43321-cni-binary-copy\") pod \"multus-additional-cni-plugins-rkgjp\" (UID: \"c7890694-9f4a-4bf0-9647-1d84e8f43321\") " pod="openshift-multus/multus-additional-cni-plugins-rkgjp"
Sep 16 18:43:18.874229 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.874169    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1140d90-5f02-4921-b4ce-40e3a7129314-etc-kubernetes\") pod \"tuned-7kkxp\" (UID: \"a1140d90-5f02-4921-b4ce-40e3a7129314\") " pod="openshift-cluster-node-tuning-operator/tuned-7kkxp"
Sep 16 18:43:18.874229 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.874181    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c7890694-9f4a-4bf0-9647-1d84e8f43321-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rkgjp\" (UID: \"c7890694-9f4a-4bf0-9647-1d84e8f43321\") " pod="openshift-multus/multus-additional-cni-plugins-rkgjp"
Sep 16 18:43:18.874229 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.874212    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z79jh\" (UniqueName: \"kubernetes.io/projected/870ff296-8673-44d5-9fc3-619486d607a1-kube-api-access-z79jh\") pod \"node-ca-w8t8m\" (UID: \"870ff296-8673-44d5-9fc3-619486d607a1\") " pod="openshift-image-registry/node-ca-w8t8m"
Sep 16 18:43:18.874675 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.874238    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c60bca6d-3f8c-4725-94a2-eedd9a075586-hosts-file\") pod \"node-resolver-wdq46\" (UID: \"c60bca6d-3f8c-4725-94a2-eedd9a075586\") " pod="openshift-dns/node-resolver-wdq46"
Sep 16 18:43:18.874675 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.874269    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b8dac3a1-52c7-4402-a075-59d997b756ef-host-slash\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.874675 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.874294    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/245475f1-ca6f-40bb-8ab5-a2ab8f46c538-hostroot\") pod \"multus-6nmxb\" (UID: \"245475f1-ca6f-40bb-8ab5-a2ab8f46c538\") " pod="openshift-multus/multus-6nmxb"
Sep 16 18:43:18.874675 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.874315    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a1140d90-5f02-4921-b4ce-40e3a7129314-etc-modprobe-d\") pod \"tuned-7kkxp\" (UID: \"a1140d90-5f02-4921-b4ce-40e3a7129314\") " pod="openshift-cluster-node-tuning-operator/tuned-7kkxp"
Sep 16 18:43:18.874675 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.874332    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/870ff296-8673-44d5-9fc3-619486d607a1-serviceca\") pod \"node-ca-w8t8m\" (UID: \"870ff296-8673-44d5-9fc3-619486d607a1\") " pod="openshift-image-registry/node-ca-w8t8m"
Sep 16 18:43:18.874675 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.874377    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/245475f1-ca6f-40bb-8ab5-a2ab8f46c538-hostroot\") pod \"multus-6nmxb\" (UID: \"245475f1-ca6f-40bb-8ab5-a2ab8f46c538\") " pod="openshift-multus/multus-6nmxb"
Sep 16 18:43:18.874675 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.874404    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b8dac3a1-52c7-4402-a075-59d997b756ef-host-kubelet\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.874675 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.874435    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a1140d90-5f02-4921-b4ce-40e3a7129314-etc-modprobe-d\") pod \"tuned-7kkxp\" (UID: \"a1140d90-5f02-4921-b4ce-40e3a7129314\") " pod="openshift-cluster-node-tuning-operator/tuned-7kkxp"
Sep 16 18:43:18.874675 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.874500    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b8dac3a1-52c7-4402-a075-59d997b756ef-run-ovn\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.874675 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.874516    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b8dac3a1-52c7-4402-a075-59d997b756ef-log-socket\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.874675 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.874533    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/245475f1-ca6f-40bb-8ab5-a2ab8f46c538-multus-cni-dir\") pod \"multus-6nmxb\" (UID: \"245475f1-ca6f-40bb-8ab5-a2ab8f46c538\") " pod="openshift-multus/multus-6nmxb"
Sep 16 18:43:18.874675 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.874555    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a1140d90-5f02-4921-b4ce-40e3a7129314-tmp\") pod \"tuned-7kkxp\" (UID: \"a1140d90-5f02-4921-b4ce-40e3a7129314\") " pod="openshift-cluster-node-tuning-operator/tuned-7kkxp"
Sep 16 18:43:18.874675 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.874577    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c7890694-9f4a-4bf0-9647-1d84e8f43321-system-cni-dir\") pod \"multus-additional-cni-plugins-rkgjp\" (UID: \"c7890694-9f4a-4bf0-9647-1d84e8f43321\") " pod="openshift-multus/multus-additional-cni-plugins-rkgjp"
Sep 16 18:43:18.874675 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.874599    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lbsd\" (UniqueName: \"kubernetes.io/projected/c60bca6d-3f8c-4725-94a2-eedd9a075586-kube-api-access-4lbsd\") pod \"node-resolver-wdq46\" (UID: \"c60bca6d-3f8c-4725-94a2-eedd9a075586\") " pod="openshift-dns/node-resolver-wdq46"
Sep 16 18:43:18.874675 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.874620    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b8dac3a1-52c7-4402-a075-59d997b756ef-host-run-netns\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.874675 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.874640    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b8dac3a1-52c7-4402-a075-59d997b756ef-ovnkube-config\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.874675 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.874664    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/245475f1-ca6f-40bb-8ab5-a2ab8f46c538-system-cni-dir\") pod \"multus-6nmxb\" (UID: \"245475f1-ca6f-40bb-8ab5-a2ab8f46c538\") " pod="openshift-multus/multus-6nmxb"
Sep 16 18:43:18.875574 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.874672    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/245475f1-ca6f-40bb-8ab5-a2ab8f46c538-multus-cni-dir\") pod \"multus-6nmxb\" (UID: \"245475f1-ca6f-40bb-8ab5-a2ab8f46c538\") " pod="openshift-multus/multus-6nmxb"
Sep 16 18:43:18.875574 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.874689    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9f7c6c50-3969-4103-a47f-d4eea4196e74-host-slash\") pod \"iptables-alerter-vmbb4\" (UID: \"9f7c6c50-3969-4103-a47f-d4eea4196e74\") " pod="openshift-network-operator/iptables-alerter-vmbb4"
Sep 16 18:43:18.875574 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.874761    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bfffe95c-37f4-4aab-a7aa-19c64f66e810-konnectivity-ca\") pod \"konnectivity-agent-pr4kj\" (UID: \"bfffe95c-37f4-4aab-a7aa-19c64f66e810\") " pod="kube-system/konnectivity-agent-pr4kj"
Sep 16 18:43:18.875574 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.874778    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/245475f1-ca6f-40bb-8ab5-a2ab8f46c538-system-cni-dir\") pod \"multus-6nmxb\" (UID: \"245475f1-ca6f-40bb-8ab5-a2ab8f46c538\") " pod="openshift-multus/multus-6nmxb"
Sep 16 18:43:18.875574 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.874785    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b8dac3a1-52c7-4402-a075-59d997b756ef-env-overrides\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.875574 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.874809    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/245475f1-ca6f-40bb-8ab5-a2ab8f46c538-cni-binary-copy\") pod \"multus-6nmxb\" (UID: \"245475f1-ca6f-40bb-8ab5-a2ab8f46c538\") " pod="openshift-multus/multus-6nmxb"
Sep 16 18:43:18.875574 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.874822    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9f7c6c50-3969-4103-a47f-d4eea4196e74-host-slash\") pod \"iptables-alerter-vmbb4\" (UID: \"9f7c6c50-3969-4103-a47f-d4eea4196e74\") " pod="openshift-network-operator/iptables-alerter-vmbb4"
Sep 16 18:43:18.875574 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.874833    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a1140d90-5f02-4921-b4ce-40e3a7129314-host\") pod \"tuned-7kkxp\" (UID: \"a1140d90-5f02-4921-b4ce-40e3a7129314\") " pod="openshift-cluster-node-tuning-operator/tuned-7kkxp"
Sep 16 18:43:18.875574 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.874864    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c7890694-9f4a-4bf0-9647-1d84e8f43321-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rkgjp\" (UID: \"c7890694-9f4a-4bf0-9647-1d84e8f43321\") " pod="openshift-multus/multus-additional-cni-plugins-rkgjp"
Sep 16 18:43:18.875574 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.874888    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/870ff296-8673-44d5-9fc3-619486d607a1-serviceca\") pod \"node-ca-w8t8m\" (UID: \"870ff296-8673-44d5-9fc3-619486d607a1\") " pod="openshift-image-registry/node-ca-w8t8m"
Sep 16 18:43:18.875574 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.874891    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmpfz\" (UniqueName: \"kubernetes.io/projected/f79ebb7c-f4b5-4e48-9506-81110189ba43-kube-api-access-pmpfz\") pod \"network-check-target-dgl4n\" (UID: \"f79ebb7c-f4b5-4e48-9506-81110189ba43\") " pod="openshift-network-diagnostics/network-check-target-dgl4n"
Sep 16 18:43:18.875574 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.874930    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a1140d90-5f02-4921-b4ce-40e3a7129314-host\") pod \"tuned-7kkxp\" (UID: \"a1140d90-5f02-4921-b4ce-40e3a7129314\") " pod="openshift-cluster-node-tuning-operator/tuned-7kkxp"
Sep 16 18:43:18.875574 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.874942    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e6428035-ca16-4098-ac4e-1ca4931bb7a7-kubelet-config\") pod \"global-pull-secret-syncer-tplxv\" (UID: \"e6428035-ca16-4098-ac4e-1ca4931bb7a7\") " pod="kube-system/global-pull-secret-syncer-tplxv"
Sep 16 18:43:18.875574 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.874973    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e6428035-ca16-4098-ac4e-1ca4931bb7a7-dbus\") pod \"global-pull-secret-syncer-tplxv\" (UID: \"e6428035-ca16-4098-ac4e-1ca4931bb7a7\") " pod="kube-system/global-pull-secret-syncer-tplxv"
Sep 16 18:43:18.875574 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.875085    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnnw4\" (UniqueName: \"kubernetes.io/projected/245475f1-ca6f-40bb-8ab5-a2ab8f46c538-kube-api-access-rnnw4\") pod \"multus-6nmxb\" (UID: \"245475f1-ca6f-40bb-8ab5-a2ab8f46c538\") " pod="openshift-multus/multus-6nmxb"
Sep 16 18:43:18.875574 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.875192    2582 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 18:43:18.875574 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.875363    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/245475f1-ca6f-40bb-8ab5-a2ab8f46c538-host-var-lib-kubelet\") pod \"multus-6nmxb\" (UID: \"245475f1-ca6f-40bb-8ab5-a2ab8f46c538\") " pod="openshift-multus/multus-6nmxb"
Sep 16 18:43:18.875574 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.875408    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b8dac3a1-52c7-4402-a075-59d997b756ef-var-lib-openvswitch\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.876531 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.875412    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bfffe95c-37f4-4aab-a7aa-19c64f66e810-konnectivity-ca\") pod \"konnectivity-agent-pr4kj\" (UID: \"bfffe95c-37f4-4aab-a7aa-19c64f66e810\") " pod="kube-system/konnectivity-agent-pr4kj"
Sep 16 18:43:18.876531 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.875435    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b8dac3a1-52c7-4402-a075-59d997b756ef-host-cni-netd\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.876531 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.875432    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/245475f1-ca6f-40bb-8ab5-a2ab8f46c538-cni-binary-copy\") pod \"multus-6nmxb\" (UID: \"245475f1-ca6f-40bb-8ab5-a2ab8f46c538\") " pod="openshift-multus/multus-6nmxb"
Sep 16 18:43:18.876531 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.875471    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/245475f1-ca6f-40bb-8ab5-a2ab8f46c538-host-var-lib-kubelet\") pod \"multus-6nmxb\" (UID: \"245475f1-ca6f-40bb-8ab5-a2ab8f46c538\") " pod="openshift-multus/multus-6nmxb"
Sep 16 18:43:18.876531 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.875477    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b8dac3a1-52c7-4402-a075-59d997b756ef-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.876531 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.875511    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/245475f1-ca6f-40bb-8ab5-a2ab8f46c538-cnibin\") pod \"multus-6nmxb\" (UID: \"245475f1-ca6f-40bb-8ab5-a2ab8f46c538\") " pod="openshift-multus/multus-6nmxb"
Sep 16 18:43:18.876531 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.875536    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a1140d90-5f02-4921-b4ce-40e3a7129314-run\") pod \"tuned-7kkxp\" (UID: \"a1140d90-5f02-4921-b4ce-40e3a7129314\") " pod="openshift-cluster-node-tuning-operator/tuned-7kkxp"
Sep 16 18:43:18.876531 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.875560    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b8dac3a1-52c7-4402-a075-59d997b756ef-node-log\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.876531 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.875580    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/245475f1-ca6f-40bb-8ab5-a2ab8f46c538-host-var-lib-cni-multus\") pod \"multus-6nmxb\" (UID: \"245475f1-ca6f-40bb-8ab5-a2ab8f46c538\") " pod="openshift-multus/multus-6nmxb"
Sep 16 18:43:18.876531 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.875595    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/245475f1-ca6f-40bb-8ab5-a2ab8f46c538-cnibin\") pod \"multus-6nmxb\" (UID: \"245475f1-ca6f-40bb-8ab5-a2ab8f46c538\") " pod="openshift-multus/multus-6nmxb"
Sep 16 18:43:18.876531 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.875614    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a1140d90-5f02-4921-b4ce-40e3a7129314-run\") pod \"tuned-7kkxp\" (UID: \"a1140d90-5f02-4921-b4ce-40e3a7129314\") " pod="openshift-cluster-node-tuning-operator/tuned-7kkxp"
Sep 16 18:43:18.876531 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.875622    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/245475f1-ca6f-40bb-8ab5-a2ab8f46c538-multus-daemon-config\") pod \"multus-6nmxb\" (UID: \"245475f1-ca6f-40bb-8ab5-a2ab8f46c538\") " pod="openshift-multus/multus-6nmxb"
Sep 16 18:43:18.876531 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.875645    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/245475f1-ca6f-40bb-8ab5-a2ab8f46c538-host-run-multus-certs\") pod \"multus-6nmxb\" (UID: \"245475f1-ca6f-40bb-8ab5-a2ab8f46c538\") " pod="openshift-multus/multus-6nmxb"
Sep 16 18:43:18.876531 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.875680    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1140d90-5f02-4921-b4ce-40e3a7129314-sys\") pod \"tuned-7kkxp\" (UID: \"a1140d90-5f02-4921-b4ce-40e3a7129314\") " pod="openshift-cluster-node-tuning-operator/tuned-7kkxp"
Sep 16 18:43:18.876531 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.875703    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c60bca6d-3f8c-4725-94a2-eedd9a075586-tmp-dir\") pod \"node-resolver-wdq46\" (UID: \"c60bca6d-3f8c-4725-94a2-eedd9a075586\") " pod="openshift-dns/node-resolver-wdq46"
Sep 16 18:43:18.876531 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.875734    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/245475f1-ca6f-40bb-8ab5-a2ab8f46c538-host-run-multus-certs\") pod \"multus-6nmxb\" (UID: \"245475f1-ca6f-40bb-8ab5-a2ab8f46c538\") " pod="openshift-multus/multus-6nmxb"
Sep 16 18:43:18.876531 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.875743    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e6428035-ca16-4098-ac4e-1ca4931bb7a7-original-pull-secret\") pod \"global-pull-secret-syncer-tplxv\" (UID: \"e6428035-ca16-4098-ac4e-1ca4931bb7a7\") " pod="kube-system/global-pull-secret-syncer-tplxv"
Sep 16 18:43:18.876531 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.875736    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/245475f1-ca6f-40bb-8ab5-a2ab8f46c538-host-var-lib-cni-multus\") pod \"multus-6nmxb\" (UID: \"245475f1-ca6f-40bb-8ab5-a2ab8f46c538\") " pod="openshift-multus/multus-6nmxb"
Sep 16 18:43:18.877409 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.875751    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1140d90-5f02-4921-b4ce-40e3a7129314-sys\") pod \"tuned-7kkxp\" (UID: \"a1140d90-5f02-4921-b4ce-40e3a7129314\") " pod="openshift-cluster-node-tuning-operator/tuned-7kkxp"
Sep 16 18:43:18.877409 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.875777    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/47ed2ebe-2b00-4044-b292-59a9ab8fafca-socket-dir\") pod \"aws-ebs-csi-driver-node-f5hzq\" (UID: \"47ed2ebe-2b00-4044-b292-59a9ab8fafca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5hzq"
Sep 16 18:43:18.877409 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.875799    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/47ed2ebe-2b00-4044-b292-59a9ab8fafca-sys-fs\") pod \"aws-ebs-csi-driver-node-f5hzq\" (UID: \"47ed2ebe-2b00-4044-b292-59a9ab8fafca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5hzq"
Sep 16 18:43:18.877409 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.875813    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/245475f1-ca6f-40bb-8ab5-a2ab8f46c538-etc-kubernetes\") pod \"multus-6nmxb\" (UID: \"245475f1-ca6f-40bb-8ab5-a2ab8f46c538\") " pod="openshift-multus/multus-6nmxb"
Sep 16 18:43:18.877409 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.875847    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a1140d90-5f02-4921-b4ce-40e3a7129314-etc-sysconfig\") pod \"tuned-7kkxp\" (UID: \"a1140d90-5f02-4921-b4ce-40e3a7129314\") " pod="openshift-cluster-node-tuning-operator/tuned-7kkxp"
Sep 16 18:43:18.877409 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.875865    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a1140d90-5f02-4921-b4ce-40e3a7129314-lib-modules\") pod \"tuned-7kkxp\" (UID: \"a1140d90-5f02-4921-b4ce-40e3a7129314\") " pod="openshift-cluster-node-tuning-operator/tuned-7kkxp"
Sep 16 18:43:18.877409 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.875880    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hrb6\" (UniqueName: \"kubernetes.io/projected/9f7c6c50-3969-4103-a47f-d4eea4196e74-kube-api-access-7hrb6\") pod \"iptables-alerter-vmbb4\" (UID: \"9f7c6c50-3969-4103-a47f-d4eea4196e74\") " pod="openshift-network-operator/iptables-alerter-vmbb4"
Sep 16 18:43:18.877409 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.875900    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/47ed2ebe-2b00-4044-b292-59a9ab8fafca-sys-fs\") pod \"aws-ebs-csi-driver-node-f5hzq\" (UID: \"47ed2ebe-2b00-4044-b292-59a9ab8fafca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5hzq"
Sep 16 18:43:18.877409 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.875905    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gxdj\" (UniqueName: \"kubernetes.io/projected/c7890694-9f4a-4bf0-9647-1d84e8f43321-kube-api-access-6gxdj\") pod \"multus-additional-cni-plugins-rkgjp\" (UID: \"c7890694-9f4a-4bf0-9647-1d84e8f43321\") " pod="openshift-multus/multus-additional-cni-plugins-rkgjp"
Sep 16 18:43:18.877409 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.875923    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/47ed2ebe-2b00-4044-b292-59a9ab8fafca-socket-dir\") pod \"aws-ebs-csi-driver-node-f5hzq\" (UID: \"47ed2ebe-2b00-4044-b292-59a9ab8fafca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5hzq"
Sep 16 18:43:18.877409 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.875937    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/47ed2ebe-2b00-4044-b292-59a9ab8fafca-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f5hzq\" (UID: \"47ed2ebe-2b00-4044-b292-59a9ab8fafca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5hzq"
Sep 16 18:43:18.877409 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.875960    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/245475f1-ca6f-40bb-8ab5-a2ab8f46c538-etc-kubernetes\") pod \"multus-6nmxb\" (UID: \"245475f1-ca6f-40bb-8ab5-a2ab8f46c538\") " pod="openshift-multus/multus-6nmxb"
Sep 16 18:43:18.877409 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.875963    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/47ed2ebe-2b00-4044-b292-59a9ab8fafca-registration-dir\") pod \"aws-ebs-csi-driver-node-f5hzq\" (UID: \"47ed2ebe-2b00-4044-b292-59a9ab8fafca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5hzq"
Sep 16 18:43:18.877409 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.875997    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/245475f1-ca6f-40bb-8ab5-a2ab8f46c538-host-var-lib-cni-bin\") pod \"multus-6nmxb\" (UID: \"245475f1-ca6f-40bb-8ab5-a2ab8f46c538\") " pod="openshift-multus/multus-6nmxb"
Sep 16 18:43:18.877409 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.876024    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a1140d90-5f02-4921-b4ce-40e3a7129314-etc-sysctl-conf\") pod \"tuned-7kkxp\" (UID: \"a1140d90-5f02-4921-b4ce-40e3a7129314\") " pod="openshift-cluster-node-tuning-operator/tuned-7kkxp"
Sep 16 18:43:18.877409 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.876036    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a1140d90-5f02-4921-b4ce-40e3a7129314-etc-sysconfig\") pod \"tuned-7kkxp\" (UID: \"a1140d90-5f02-4921-b4ce-40e3a7129314\") " pod="openshift-cluster-node-tuning-operator/tuned-7kkxp"
Sep 16 18:43:18.877409 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.876021    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/47ed2ebe-2b00-4044-b292-59a9ab8fafca-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f5hzq\" (UID: \"47ed2ebe-2b00-4044-b292-59a9ab8fafca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5hzq"
Sep 16 18:43:18.878320 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.876051    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1140d90-5f02-4921-b4ce-40e3a7129314-var-lib-kubelet\") pod \"tuned-7kkxp\" (UID: \"a1140d90-5f02-4921-b4ce-40e3a7129314\") " pod="openshift-cluster-node-tuning-operator/tuned-7kkxp"
Sep 16 18:43:18.878320 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.876083    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a1140d90-5f02-4921-b4ce-40e3a7129314-etc-tuned\") pod \"tuned-7kkxp\" (UID: \"a1140d90-5f02-4921-b4ce-40e3a7129314\") " pod="openshift-cluster-node-tuning-operator/tuned-7kkxp"
Sep 16 18:43:18.878320 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.876092    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1140d90-5f02-4921-b4ce-40e3a7129314-var-lib-kubelet\") pod \"tuned-7kkxp\" (UID: \"a1140d90-5f02-4921-b4ce-40e3a7129314\") " pod="openshift-cluster-node-tuning-operator/tuned-7kkxp"
Sep 16 18:43:18.878320 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.876114    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b8dac3a1-52c7-4402-a075-59d997b756ef-run-systemd\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.878320 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.876134    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a1140d90-5f02-4921-b4ce-40e3a7129314-etc-sysctl-conf\") pod \"tuned-7kkxp\" (UID: \"a1140d90-5f02-4921-b4ce-40e3a7129314\") " pod="openshift-cluster-node-tuning-operator/tuned-7kkxp"
Sep 16 18:43:18.878320 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.876137    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/47ed2ebe-2b00-4044-b292-59a9ab8fafca-registration-dir\") pod \"aws-ebs-csi-driver-node-f5hzq\" (UID: \"47ed2ebe-2b00-4044-b292-59a9ab8fafca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5hzq"
Sep 16 18:43:18.878320 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.876139    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b8dac3a1-52c7-4402-a075-59d997b756ef-run-openvswitch\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.878320 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.876161    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/245475f1-ca6f-40bb-8ab5-a2ab8f46c538-multus-daemon-config\") pod \"multus-6nmxb\" (UID: \"245475f1-ca6f-40bb-8ab5-a2ab8f46c538\") " pod="openshift-multus/multus-6nmxb"
Sep 16 18:43:18.878320 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.876167    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/245475f1-ca6f-40bb-8ab5-a2ab8f46c538-host-var-lib-cni-bin\") pod \"multus-6nmxb\" (UID: \"245475f1-ca6f-40bb-8ab5-a2ab8f46c538\") " pod="openshift-multus/multus-6nmxb"
Sep 16 18:43:18.878320 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.876214    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c7890694-9f4a-4bf0-9647-1d84e8f43321-cnibin\") pod \"multus-additional-cni-plugins-rkgjp\" (UID: \"c7890694-9f4a-4bf0-9647-1d84e8f43321\") " pod="openshift-multus/multus-additional-cni-plugins-rkgjp"
Sep 16 18:43:18.878320 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.876276    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/47ed2ebe-2b00-4044-b292-59a9ab8fafca-etc-selinux\") pod \"aws-ebs-csi-driver-node-f5hzq\" (UID: \"47ed2ebe-2b00-4044-b292-59a9ab8fafca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5hzq"
Sep 16 18:43:18.878320 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.876303    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/245475f1-ca6f-40bb-8ab5-a2ab8f46c538-host-run-netns\") pod \"multus-6nmxb\" (UID: \"245475f1-ca6f-40bb-8ab5-a2ab8f46c538\") " pod="openshift-multus/multus-6nmxb"
Sep 16 18:43:18.878320 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.876324    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a1140d90-5f02-4921-b4ce-40e3a7129314-etc-sysctl-d\") pod \"tuned-7kkxp\" (UID: \"a1140d90-5f02-4921-b4ce-40e3a7129314\") " pod="openshift-cluster-node-tuning-operator/tuned-7kkxp"
Sep 16 18:43:18.878320 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.876349    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b8dac3a1-52c7-4402-a075-59d997b756ef-systemd-units\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.878320 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.876358    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/47ed2ebe-2b00-4044-b292-59a9ab8fafca-etc-selinux\") pod \"aws-ebs-csi-driver-node-f5hzq\" (UID: \"47ed2ebe-2b00-4044-b292-59a9ab8fafca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5hzq"
Sep 16 18:43:18.878320 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.876367    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/245475f1-ca6f-40bb-8ab5-a2ab8f46c538-host-run-netns\") pod \"multus-6nmxb\" (UID: \"245475f1-ca6f-40bb-8ab5-a2ab8f46c538\") " pod="openshift-multus/multus-6nmxb"
Sep 16 18:43:18.878320 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.876374    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a1140d90-5f02-4921-b4ce-40e3a7129314-etc-systemd\") pod \"tuned-7kkxp\" (UID: \"a1140d90-5f02-4921-b4ce-40e3a7129314\") " pod="openshift-cluster-node-tuning-operator/tuned-7kkxp"
Sep 16 18:43:18.879211 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.876422    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a1140d90-5f02-4921-b4ce-40e3a7129314-etc-systemd\") pod \"tuned-7kkxp\" (UID: \"a1140d90-5f02-4921-b4ce-40e3a7129314\") " pod="openshift-cluster-node-tuning-operator/tuned-7kkxp"
Sep 16 18:43:18.879211 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.876443    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a1140d90-5f02-4921-b4ce-40e3a7129314-etc-sysctl-d\") pod \"tuned-7kkxp\" (UID: \"a1140d90-5f02-4921-b4ce-40e3a7129314\") " pod="openshift-cluster-node-tuning-operator/tuned-7kkxp"
Sep 16 18:43:18.879211 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.876448    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bfffe95c-37f4-4aab-a7aa-19c64f66e810-agent-certs\") pod \"konnectivity-agent-pr4kj\" (UID: \"bfffe95c-37f4-4aab-a7aa-19c64f66e810\") " pod="kube-system/konnectivity-agent-pr4kj"
Sep 16 18:43:18.879211 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.876475    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b8dac3a1-52c7-4402-a075-59d997b756ef-etc-openvswitch\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.879211 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.876498    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a1140d90-5f02-4921-b4ce-40e3a7129314-lib-modules\") pod \"tuned-7kkxp\" (UID: \"a1140d90-5f02-4921-b4ce-40e3a7129314\") " pod="openshift-cluster-node-tuning-operator/tuned-7kkxp"
Sep 16 18:43:18.879211 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.876501    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b8dac3a1-52c7-4402-a075-59d997b756ef-ovn-node-metrics-cert\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.879211 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.876537    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de8ddfa3-fd54-49d7-8f90-035aa4439e98-metrics-certs\") pod \"network-metrics-daemon-g7q9m\" (UID: \"de8ddfa3-fd54-49d7-8f90-035aa4439e98\") " pod="openshift-multus/network-metrics-daemon-g7q9m"
Sep 16 18:43:18.879211 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.876582    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c7890694-9f4a-4bf0-9647-1d84e8f43321-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rkgjp\" (UID: \"c7890694-9f4a-4bf0-9647-1d84e8f43321\") " pod="openshift-multus/multus-additional-cni-plugins-rkgjp"
Sep 16 18:43:18.879211 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:18.876656    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:18.879211 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.876674    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b8dac3a1-52c7-4402-a075-59d997b756ef-ovnkube-script-lib\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.879211 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:18.876781    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de8ddfa3-fd54-49d7-8f90-035aa4439e98-metrics-certs podName:de8ddfa3-fd54-49d7-8f90-035aa4439e98 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:19.37674327 +0000 UTC m=+3.018458930 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de8ddfa3-fd54-49d7-8f90-035aa4439e98-metrics-certs") pod "network-metrics-daemon-g7q9m" (UID: "de8ddfa3-fd54-49d7-8f90-035aa4439e98") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:18.879211 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.876802    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6kt5\" (UniqueName: \"kubernetes.io/projected/de8ddfa3-fd54-49d7-8f90-035aa4439e98-kube-api-access-b6kt5\") pod \"network-metrics-daemon-g7q9m\" (UID: \"de8ddfa3-fd54-49d7-8f90-035aa4439e98\") " pod="openshift-multus/network-metrics-daemon-g7q9m"
Sep 16 18:43:18.879211 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.876832    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/245475f1-ca6f-40bb-8ab5-a2ab8f46c538-multus-socket-dir-parent\") pod \"multus-6nmxb\" (UID: \"245475f1-ca6f-40bb-8ab5-a2ab8f46c538\") " pod="openshift-multus/multus-6nmxb"
Sep 16 18:43:18.879211 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.876883    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9f7c6c50-3969-4103-a47f-d4eea4196e74-iptables-alerter-script\") pod \"iptables-alerter-vmbb4\" (UID: \"9f7c6c50-3969-4103-a47f-d4eea4196e74\") " pod="openshift-network-operator/iptables-alerter-vmbb4"
Sep 16 18:43:18.879211 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.876947    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/245475f1-ca6f-40bb-8ab5-a2ab8f46c538-multus-socket-dir-parent\") pod \"multus-6nmxb\" (UID: \"245475f1-ca6f-40bb-8ab5-a2ab8f46c538\") " pod="openshift-multus/multus-6nmxb"
Sep 16 18:43:18.879211 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.877429    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9f7c6c50-3969-4103-a47f-d4eea4196e74-iptables-alerter-script\") pod \"iptables-alerter-vmbb4\" (UID: \"9f7c6c50-3969-4103-a47f-d4eea4196e74\") " pod="openshift-network-operator/iptables-alerter-vmbb4"
Sep 16 18:43:18.880070 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.879484    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a1140d90-5f02-4921-b4ce-40e3a7129314-etc-tuned\") pod \"tuned-7kkxp\" (UID: \"a1140d90-5f02-4921-b4ce-40e3a7129314\") " pod="openshift-cluster-node-tuning-operator/tuned-7kkxp"
Sep 16 18:43:18.880070 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.879570    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a1140d90-5f02-4921-b4ce-40e3a7129314-tmp\") pod \"tuned-7kkxp\" (UID: \"a1140d90-5f02-4921-b4ce-40e3a7129314\") " pod="openshift-cluster-node-tuning-operator/tuned-7kkxp"
Sep 16 18:43:18.880070 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.879847    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bfffe95c-37f4-4aab-a7aa-19c64f66e810-agent-certs\") pod \"konnectivity-agent-pr4kj\" (UID: \"bfffe95c-37f4-4aab-a7aa-19c64f66e810\") " pod="kube-system/konnectivity-agent-pr4kj"
Sep 16 18:43:18.882678 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.882580    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mhp7\" (UniqueName: \"kubernetes.io/projected/47ed2ebe-2b00-4044-b292-59a9ab8fafca-kube-api-access-8mhp7\") pod \"aws-ebs-csi-driver-node-f5hzq\" (UID: \"47ed2ebe-2b00-4044-b292-59a9ab8fafca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5hzq"
Sep 16 18:43:18.882678 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.882582    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z79jh\" (UniqueName: \"kubernetes.io/projected/870ff296-8673-44d5-9fc3-619486d607a1-kube-api-access-z79jh\") pod \"node-ca-w8t8m\" (UID: \"870ff296-8673-44d5-9fc3-619486d607a1\") " pod="openshift-image-registry/node-ca-w8t8m"
Sep 16 18:43:18.883008 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.882987    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c7ns\" (UniqueName: \"kubernetes.io/projected/a1140d90-5f02-4921-b4ce-40e3a7129314-kube-api-access-9c7ns\") pod \"tuned-7kkxp\" (UID: \"a1140d90-5f02-4921-b4ce-40e3a7129314\") " pod="openshift-cluster-node-tuning-operator/tuned-7kkxp"
Sep 16 18:43:18.883272 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.883251    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnnw4\" (UniqueName: \"kubernetes.io/projected/245475f1-ca6f-40bb-8ab5-a2ab8f46c538-kube-api-access-rnnw4\") pod \"multus-6nmxb\" (UID: \"245475f1-ca6f-40bb-8ab5-a2ab8f46c538\") " pod="openshift-multus/multus-6nmxb"
Sep 16 18:43:18.883601 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.883579    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hrb6\" (UniqueName: \"kubernetes.io/projected/9f7c6c50-3969-4103-a47f-d4eea4196e74-kube-api-access-7hrb6\") pod \"iptables-alerter-vmbb4\" (UID: \"9f7c6c50-3969-4103-a47f-d4eea4196e74\") " pod="openshift-network-operator/iptables-alerter-vmbb4"
Sep 16 18:43:18.891133 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.891106    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6kt5\" (UniqueName: \"kubernetes.io/projected/de8ddfa3-fd54-49d7-8f90-035aa4439e98-kube-api-access-b6kt5\") pod \"network-metrics-daemon-g7q9m\" (UID: \"de8ddfa3-fd54-49d7-8f90-035aa4439e98\") " pod="openshift-multus/network-metrics-daemon-g7q9m"
Sep 16 18:43:18.977192 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.977159    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b8dac3a1-52c7-4402-a075-59d997b756ef-run-systemd\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.977192 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.977200    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b8dac3a1-52c7-4402-a075-59d997b756ef-run-openvswitch\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.977429 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.977228    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c7890694-9f4a-4bf0-9647-1d84e8f43321-cnibin\") pod \"multus-additional-cni-plugins-rkgjp\" (UID: \"c7890694-9f4a-4bf0-9647-1d84e8f43321\") " pod="openshift-multus/multus-additional-cni-plugins-rkgjp"
Sep 16 18:43:18.977429 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.977251    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b8dac3a1-52c7-4402-a075-59d997b756ef-systemd-units\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.977429 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.977272    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b8dac3a1-52c7-4402-a075-59d997b756ef-etc-openvswitch\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.977429 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.977292    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b8dac3a1-52c7-4402-a075-59d997b756ef-ovn-node-metrics-cert\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.977429 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.977292    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b8dac3a1-52c7-4402-a075-59d997b756ef-run-systemd\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.977429 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.977327    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c7890694-9f4a-4bf0-9647-1d84e8f43321-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rkgjp\" (UID: \"c7890694-9f4a-4bf0-9647-1d84e8f43321\") " pod="openshift-multus/multus-additional-cni-plugins-rkgjp"
Sep 16 18:43:18.977429 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.977339    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c7890694-9f4a-4bf0-9647-1d84e8f43321-cnibin\") pod \"multus-additional-cni-plugins-rkgjp\" (UID: \"c7890694-9f4a-4bf0-9647-1d84e8f43321\") " pod="openshift-multus/multus-additional-cni-plugins-rkgjp"
Sep 16 18:43:18.977429 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.977352    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b8dac3a1-52c7-4402-a075-59d997b756ef-ovnkube-script-lib\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.977429 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.977378    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz9ps\" (UniqueName: \"kubernetes.io/projected/b8dac3a1-52c7-4402-a075-59d997b756ef-kube-api-access-fz9ps\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.977429 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.977384    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b8dac3a1-52c7-4402-a075-59d997b756ef-run-openvswitch\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.977429 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.977405    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b8dac3a1-52c7-4402-a075-59d997b756ef-host-cni-bin\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.977429 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.977417    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b8dac3a1-52c7-4402-a075-59d997b756ef-etc-openvswitch\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.977429 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.977431    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b8dac3a1-52c7-4402-a075-59d997b756ef-host-run-ovn-kubernetes\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.978067 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.977451    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b8dac3a1-52c7-4402-a075-59d997b756ef-systemd-units\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.978067 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.977454    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c7890694-9f4a-4bf0-9647-1d84e8f43321-cni-binary-copy\") pod \"multus-additional-cni-plugins-rkgjp\" (UID: \"c7890694-9f4a-4bf0-9647-1d84e8f43321\") " pod="openshift-multus/multus-additional-cni-plugins-rkgjp"
Sep 16 18:43:18.978067 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.977488    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b8dac3a1-52c7-4402-a075-59d997b756ef-host-cni-bin\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.978067 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.977515    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c7890694-9f4a-4bf0-9647-1d84e8f43321-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rkgjp\" (UID: \"c7890694-9f4a-4bf0-9647-1d84e8f43321\") " pod="openshift-multus/multus-additional-cni-plugins-rkgjp"
Sep 16 18:43:18.978067 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.977544    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c60bca6d-3f8c-4725-94a2-eedd9a075586-hosts-file\") pod \"node-resolver-wdq46\" (UID: \"c60bca6d-3f8c-4725-94a2-eedd9a075586\") " pod="openshift-dns/node-resolver-wdq46"
Sep 16 18:43:18.978067 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.977570    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b8dac3a1-52c7-4402-a075-59d997b756ef-host-slash\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.978067 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.977595    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b8dac3a1-52c7-4402-a075-59d997b756ef-host-kubelet\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.978067 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.977616    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b8dac3a1-52c7-4402-a075-59d997b756ef-run-ovn\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.978067 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.977639    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b8dac3a1-52c7-4402-a075-59d997b756ef-log-socket\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.978067 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.977666    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c7890694-9f4a-4bf0-9647-1d84e8f43321-system-cni-dir\") pod \"multus-additional-cni-plugins-rkgjp\" (UID: \"c7890694-9f4a-4bf0-9647-1d84e8f43321\") " pod="openshift-multus/multus-additional-cni-plugins-rkgjp"
Sep 16 18:43:18.978067 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.977690    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lbsd\" (UniqueName: \"kubernetes.io/projected/c60bca6d-3f8c-4725-94a2-eedd9a075586-kube-api-access-4lbsd\") pod \"node-resolver-wdq46\" (UID: \"c60bca6d-3f8c-4725-94a2-eedd9a075586\") " pod="openshift-dns/node-resolver-wdq46"
Sep 16 18:43:18.978067 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.977733    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b8dac3a1-52c7-4402-a075-59d997b756ef-host-run-netns\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.978067 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.977738    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b8dac3a1-52c7-4402-a075-59d997b756ef-host-run-ovn-kubernetes\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.978067 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.977758    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b8dac3a1-52c7-4402-a075-59d997b756ef-ovnkube-config\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.978067 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.977811    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c60bca6d-3f8c-4725-94a2-eedd9a075586-hosts-file\") pod \"node-resolver-wdq46\" (UID: \"c60bca6d-3f8c-4725-94a2-eedd9a075586\") " pod="openshift-dns/node-resolver-wdq46"
Sep 16 18:43:18.978067 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.977822    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b8dac3a1-52c7-4402-a075-59d997b756ef-env-overrides\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.978067 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.977849    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b8dac3a1-52c7-4402-a075-59d997b756ef-host-slash\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.978843 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.977853    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c7890694-9f4a-4bf0-9647-1d84e8f43321-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rkgjp\" (UID: \"c7890694-9f4a-4bf0-9647-1d84e8f43321\") " pod="openshift-multus/multus-additional-cni-plugins-rkgjp"
Sep 16 18:43:18.978843 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.977904    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmpfz\" (UniqueName: \"kubernetes.io/projected/f79ebb7c-f4b5-4e48-9506-81110189ba43-kube-api-access-pmpfz\") pod \"network-check-target-dgl4n\" (UID: \"f79ebb7c-f4b5-4e48-9506-81110189ba43\") " pod="openshift-network-diagnostics/network-check-target-dgl4n"
Sep 16 18:43:18.978843 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.977933    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e6428035-ca16-4098-ac4e-1ca4931bb7a7-kubelet-config\") pod \"global-pull-secret-syncer-tplxv\" (UID: \"e6428035-ca16-4098-ac4e-1ca4931bb7a7\") " pod="kube-system/global-pull-secret-syncer-tplxv"
Sep 16 18:43:18.978843 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.977957    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e6428035-ca16-4098-ac4e-1ca4931bb7a7-dbus\") pod \"global-pull-secret-syncer-tplxv\" (UID: \"e6428035-ca16-4098-ac4e-1ca4931bb7a7\") " pod="kube-system/global-pull-secret-syncer-tplxv"
Sep 16 18:43:18.978843 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.977962    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c7890694-9f4a-4bf0-9647-1d84e8f43321-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rkgjp\" (UID: \"c7890694-9f4a-4bf0-9647-1d84e8f43321\") " pod="openshift-multus/multus-additional-cni-plugins-rkgjp"
Sep 16 18:43:18.978843 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.977969    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c7890694-9f4a-4bf0-9647-1d84e8f43321-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rkgjp\" (UID: \"c7890694-9f4a-4bf0-9647-1d84e8f43321\") " pod="openshift-multus/multus-additional-cni-plugins-rkgjp"
Sep 16 18:43:18.978843 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.977987    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b8dac3a1-52c7-4402-a075-59d997b756ef-var-lib-openvswitch\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.978843 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.977993    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c7890694-9f4a-4bf0-9647-1d84e8f43321-cni-binary-copy\") pod \"multus-additional-cni-plugins-rkgjp\" (UID: \"c7890694-9f4a-4bf0-9647-1d84e8f43321\") " pod="openshift-multus/multus-additional-cni-plugins-rkgjp"
Sep 16 18:43:18.978843 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.978011    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b8dac3a1-52c7-4402-a075-59d997b756ef-host-cni-netd\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.978843 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.978017    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e6428035-ca16-4098-ac4e-1ca4931bb7a7-kubelet-config\") pod \"global-pull-secret-syncer-tplxv\" (UID: \"e6428035-ca16-4098-ac4e-1ca4931bb7a7\") " pod="kube-system/global-pull-secret-syncer-tplxv"
Sep 16 18:43:18.978843 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.978039    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b8dac3a1-52c7-4402-a075-59d997b756ef-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.978843 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.978045    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b8dac3a1-52c7-4402-a075-59d997b756ef-run-ovn\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.978843 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.978061    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b8dac3a1-52c7-4402-a075-59d997b756ef-host-run-netns\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.978843 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.978077    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c7890694-9f4a-4bf0-9647-1d84e8f43321-system-cni-dir\") pod \"multus-additional-cni-plugins-rkgjp\" (UID: \"c7890694-9f4a-4bf0-9647-1d84e8f43321\") " pod="openshift-multus/multus-additional-cni-plugins-rkgjp"
Sep 16 18:43:18.978843 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.978065    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b8dac3a1-52c7-4402-a075-59d997b756ef-node-log\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.978843 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.978100    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b8dac3a1-52c7-4402-a075-59d997b756ef-node-log\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.978843 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.978115    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c60bca6d-3f8c-4725-94a2-eedd9a075586-tmp-dir\") pod \"node-resolver-wdq46\" (UID: \"c60bca6d-3f8c-4725-94a2-eedd9a075586\") " pod="openshift-dns/node-resolver-wdq46"
Sep 16 18:43:18.979617 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.978141    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e6428035-ca16-4098-ac4e-1ca4931bb7a7-original-pull-secret\") pod \"global-pull-secret-syncer-tplxv\" (UID: \"e6428035-ca16-4098-ac4e-1ca4931bb7a7\") " pod="kube-system/global-pull-secret-syncer-tplxv"
Sep 16 18:43:18.979617 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.978163    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gxdj\" (UniqueName: \"kubernetes.io/projected/c7890694-9f4a-4bf0-9647-1d84e8f43321-kube-api-access-6gxdj\") pod \"multus-additional-cni-plugins-rkgjp\" (UID: \"c7890694-9f4a-4bf0-9647-1d84e8f43321\") " pod="openshift-multus/multus-additional-cni-plugins-rkgjp"
Sep 16 18:43:18.979617 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.978259    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b8dac3a1-52c7-4402-a075-59d997b756ef-var-lib-openvswitch\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.979617 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.978266    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c7890694-9f4a-4bf0-9647-1d84e8f43321-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rkgjp\" (UID: \"c7890694-9f4a-4bf0-9647-1d84e8f43321\") " pod="openshift-multus/multus-additional-cni-plugins-rkgjp"
Sep 16 18:43:18.979617 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.978279    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e6428035-ca16-4098-ac4e-1ca4931bb7a7-dbus\") pod \"global-pull-secret-syncer-tplxv\" (UID: \"e6428035-ca16-4098-ac4e-1ca4931bb7a7\") " pod="kube-system/global-pull-secret-syncer-tplxv"
Sep 16 18:43:18.979617 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.978322    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b8dac3a1-52c7-4402-a075-59d997b756ef-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.979617 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.978355    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b8dac3a1-52c7-4402-a075-59d997b756ef-log-socket\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.979617 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.978381    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b8dac3a1-52c7-4402-a075-59d997b756ef-host-cni-netd\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.979617 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.978412    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b8dac3a1-52c7-4402-a075-59d997b756ef-host-kubelet\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.979617 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:18.978438    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:18.979617 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:18.978493    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6428035-ca16-4098-ac4e-1ca4931bb7a7-original-pull-secret podName:e6428035-ca16-4098-ac4e-1ca4931bb7a7 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:19.478475483 +0000 UTC m=+3.120191139 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e6428035-ca16-4098-ac4e-1ca4931bb7a7-original-pull-secret") pod "global-pull-secret-syncer-tplxv" (UID: "e6428035-ca16-4098-ac4e-1ca4931bb7a7") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:18.979617 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.978531    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c60bca6d-3f8c-4725-94a2-eedd9a075586-tmp-dir\") pod \"node-resolver-wdq46\" (UID: \"c60bca6d-3f8c-4725-94a2-eedd9a075586\") " pod="openshift-dns/node-resolver-wdq46"
Sep 16 18:43:18.979617 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.978771    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b8dac3a1-52c7-4402-a075-59d997b756ef-ovnkube-script-lib\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.979617 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.978787    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b8dac3a1-52c7-4402-a075-59d997b756ef-env-overrides\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.979617 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.978801    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b8dac3a1-52c7-4402-a075-59d997b756ef-ovnkube-config\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.980225 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.979868    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b8dac3a1-52c7-4402-a075-59d997b756ef-ovn-node-metrics-cert\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:18.985598 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:18.985573    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:43:18.985598 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:18.985599    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:43:18.985789 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:18.985609    2582 projected.go:196] Error preparing data for projected volume kube-api-access-pmpfz for pod openshift-network-diagnostics/network-check-target-dgl4n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:18.985789 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:18.985676    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f79ebb7c-f4b5-4e48-9506-81110189ba43-kube-api-access-pmpfz podName:f79ebb7c-f4b5-4e48-9506-81110189ba43 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:19.485655137 +0000 UTC m=+3.127370795 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-pmpfz" (UniqueName: "kubernetes.io/projected/f79ebb7c-f4b5-4e48-9506-81110189ba43-kube-api-access-pmpfz") pod "network-check-target-dgl4n" (UID: "f79ebb7c-f4b5-4e48-9506-81110189ba43") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:18.987446 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.987390    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gxdj\" (UniqueName: \"kubernetes.io/projected/c7890694-9f4a-4bf0-9647-1d84e8f43321-kube-api-access-6gxdj\") pod \"multus-additional-cni-plugins-rkgjp\" (UID: \"c7890694-9f4a-4bf0-9647-1d84e8f43321\") " pod="openshift-multus/multus-additional-cni-plugins-rkgjp"
Sep 16 18:43:18.987657 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.987640    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lbsd\" (UniqueName: \"kubernetes.io/projected/c60bca6d-3f8c-4725-94a2-eedd9a075586-kube-api-access-4lbsd\") pod \"node-resolver-wdq46\" (UID: \"c60bca6d-3f8c-4725-94a2-eedd9a075586\") " pod="openshift-dns/node-resolver-wdq46"
Sep 16 18:43:18.987850 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:18.987832    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz9ps\" (UniqueName: \"kubernetes.io/projected/b8dac3a1-52c7-4402-a075-59d997b756ef-kube-api-access-fz9ps\") pod \"ovnkube-node-pv8wf\" (UID: \"b8dac3a1-52c7-4402-a075-59d997b756ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:19.063234 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:19.063196    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vmbb4"
Sep 16 18:43:19.070255 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:19.070229    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pr4kj"
Sep 16 18:43:19.078100 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:19.078071    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6nmxb"
Sep 16 18:43:19.085959 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:19.085931    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rkgjp"
Sep 16 18:43:19.092853 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:19.092829    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5hzq"
Sep 16 18:43:19.102623 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:19.102600    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7kkxp"
Sep 16 18:43:19.108440 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:19.108414    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-w8t8m"
Sep 16 18:43:19.115221 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:19.115196    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wdq46"
Sep 16 18:43:19.116385 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:19.116362    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:19.296502 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:19.296472    2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:43:19.381509 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:19.381481    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de8ddfa3-fd54-49d7-8f90-035aa4439e98-metrics-certs\") pod \"network-metrics-daemon-g7q9m\" (UID: \"de8ddfa3-fd54-49d7-8f90-035aa4439e98\") " pod="openshift-multus/network-metrics-daemon-g7q9m"
Sep 16 18:43:19.381679 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:19.381624    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:19.381795 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:19.381695    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de8ddfa3-fd54-49d7-8f90-035aa4439e98-metrics-certs podName:de8ddfa3-fd54-49d7-8f90-035aa4439e98 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:20.381675761 +0000 UTC m=+4.023391420 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de8ddfa3-fd54-49d7-8f90-035aa4439e98-metrics-certs") pod "network-metrics-daemon-g7q9m" (UID: "de8ddfa3-fd54-49d7-8f90-035aa4439e98") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:19.482734 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:19.482679    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e6428035-ca16-4098-ac4e-1ca4931bb7a7-original-pull-secret\") pod \"global-pull-secret-syncer-tplxv\" (UID: \"e6428035-ca16-4098-ac4e-1ca4931bb7a7\") " pod="kube-system/global-pull-secret-syncer-tplxv"
Sep 16 18:43:19.482915 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:19.482846    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:19.482976 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:19.482924    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6428035-ca16-4098-ac4e-1ca4931bb7a7-original-pull-secret podName:e6428035-ca16-4098-ac4e-1ca4931bb7a7 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:20.482900089 +0000 UTC m=+4.124615731 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e6428035-ca16-4098-ac4e-1ca4931bb7a7-original-pull-secret") pod "global-pull-secret-syncer-tplxv" (UID: "e6428035-ca16-4098-ac4e-1ca4931bb7a7") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:19.584054 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:19.583968    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmpfz\" (UniqueName: \"kubernetes.io/projected/f79ebb7c-f4b5-4e48-9506-81110189ba43-kube-api-access-pmpfz\") pod \"network-check-target-dgl4n\" (UID: \"f79ebb7c-f4b5-4e48-9506-81110189ba43\") " pod="openshift-network-diagnostics/network-check-target-dgl4n"
Sep 16 18:43:19.584192 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:19.584138    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:43:19.584192 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:19.584160    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:43:19.584192 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:19.584171    2582 projected.go:196] Error preparing data for projected volume kube-api-access-pmpfz for pod openshift-network-diagnostics/network-check-target-dgl4n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:19.584287 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:19.584217    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f79ebb7c-f4b5-4e48-9506-81110189ba43-kube-api-access-pmpfz podName:f79ebb7c-f4b5-4e48-9506-81110189ba43 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:20.584203473 +0000 UTC m=+4.225919116 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-pmpfz" (UniqueName: "kubernetes.io/projected/f79ebb7c-f4b5-4e48-9506-81110189ba43-kube-api-access-pmpfz") pod "network-check-target-dgl4n" (UID: "f79ebb7c-f4b5-4e48-9506-81110189ba43") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:19.663372 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:19.663342    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfffe95c_37f4_4aab_a7aa_19c64f66e810.slice/crio-714fad343b44fb516cd6a3a88b6ccb6d8ec93a892203362d4444b99d7cb34f08 WatchSource:0}: Error finding container 714fad343b44fb516cd6a3a88b6ccb6d8ec93a892203362d4444b99d7cb34f08: Status 404 returned error can't find the container with id 714fad343b44fb516cd6a3a88b6ccb6d8ec93a892203362d4444b99d7cb34f08
Sep 16 18:43:19.665378 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:19.665246    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod870ff296_8673_44d5_9fc3_619486d607a1.slice/crio-a7ce6adcecd32a681112e90a0dee5d0731164ab8a29c9975469e5b19d4814d15 WatchSource:0}: Error finding container a7ce6adcecd32a681112e90a0dee5d0731164ab8a29c9975469e5b19d4814d15: Status 404 returned error can't find the container with id a7ce6adcecd32a681112e90a0dee5d0731164ab8a29c9975469e5b19d4814d15
Sep 16 18:43:19.668459 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:19.668432    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1140d90_5f02_4921_b4ce_40e3a7129314.slice/crio-1e22c60a2584ac9b95ca4ba4fe6a4a37cec94a7fed3efa356b86c6274daf8f5a WatchSource:0}: Error finding container 1e22c60a2584ac9b95ca4ba4fe6a4a37cec94a7fed3efa356b86c6274daf8f5a: Status 404 returned error can't find the container with id 1e22c60a2584ac9b95ca4ba4fe6a4a37cec94a7fed3efa356b86c6274daf8f5a
Sep 16 18:43:19.690382 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:19.690356    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7890694_9f4a_4bf0_9647_1d84e8f43321.slice/crio-04f8b21f4748857177030c1cdc9902d780e1d7f2e6387f6c18187159070d036f WatchSource:0}: Error finding container 04f8b21f4748857177030c1cdc9902d780e1d7f2e6387f6c18187159070d036f: Status 404 returned error can't find the container with id 04f8b21f4748857177030c1cdc9902d780e1d7f2e6387f6c18187159070d036f
Sep 16 18:43:19.690831 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:19.690806    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod245475f1_ca6f_40bb_8ab5_a2ab8f46c538.slice/crio-fbfdaf0147835d8b57baddcbea9d3e6b13416fe87e5ad52e319a742ec31e3739 WatchSource:0}: Error finding container fbfdaf0147835d8b57baddcbea9d3e6b13416fe87e5ad52e319a742ec31e3739: Status 404 returned error can't find the container with id fbfdaf0147835d8b57baddcbea9d3e6b13416fe87e5ad52e319a742ec31e3739
Sep 16 18:43:19.692250 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:19.692232    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8dac3a1_52c7_4402_a075_59d997b756ef.slice/crio-1623065fd52d3f7a529235c67f7acd6b8731bd2097d1491e5a7ac5c320d93445 WatchSource:0}: Error finding container 1623065fd52d3f7a529235c67f7acd6b8731bd2097d1491e5a7ac5c320d93445: Status 404 returned error can't find the container with id 1623065fd52d3f7a529235c67f7acd6b8731bd2097d1491e5a7ac5c320d93445
Sep 16 18:43:19.693534 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:19.693351    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc60bca6d_3f8c_4725_94a2_eedd9a075586.slice/crio-3a86c785f0f40e6dafe81131152e02c8cc4996992a0ef4a750df6d7d1a20ff32 WatchSource:0}: Error finding container 3a86c785f0f40e6dafe81131152e02c8cc4996992a0ef4a750df6d7d1a20ff32: Status 404 returned error can't find the container with id 3a86c785f0f40e6dafe81131152e02c8cc4996992a0ef4a750df6d7d1a20ff32
Sep 16 18:43:19.694008 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:19.693855    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f7c6c50_3969_4103_a47f_d4eea4196e74.slice/crio-fd3a519637db881a238587cd9aa63f72d95fe848ceb16130bb978cb26f360a2d WatchSource:0}: Error finding container fd3a519637db881a238587cd9aa63f72d95fe848ceb16130bb978cb26f360a2d: Status 404 returned error can't find the container with id fd3a519637db881a238587cd9aa63f72d95fe848ceb16130bb978cb26f360a2d
Sep 16 18:43:19.694683 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:19.694666    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47ed2ebe_2b00_4044_b292_59a9ab8fafca.slice/crio-5f3b08476a9355f88ad5d312d6c67d3294e7c4d03216c5cb95261db282119504 WatchSource:0}: Error finding container 5f3b08476a9355f88ad5d312d6c67d3294e7c4d03216c5cb95261db282119504: Status 404 returned error can't find the container with id 5f3b08476a9355f88ad5d312d6c67d3294e7c4d03216c5cb95261db282119504
Sep 16 18:43:19.815899 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:19.815860    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:38:18 +0000 UTC" deadline="2028-05-02 05:42:40.368835759 +0000 UTC"
Sep 16 18:43:19.815899 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:19.815894    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14242h59m20.552946247s"
Sep 16 18:43:19.867384 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:19.867313    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgl4n"
Sep 16 18:43:19.867515 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:19.867421    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgl4n" podUID="f79ebb7c-f4b5-4e48-9506-81110189ba43"
Sep 16 18:43:19.867515 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:19.867465    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g7q9m"
Sep 16 18:43:19.867603 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:19.867564    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g7q9m" podUID="de8ddfa3-fd54-49d7-8f90-035aa4439e98"
Sep 16 18:43:19.875128 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:19.875096    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wdq46" event={"ID":"c60bca6d-3f8c-4725-94a2-eedd9a075586","Type":"ContainerStarted","Data":"3a86c785f0f40e6dafe81131152e02c8cc4996992a0ef4a750df6d7d1a20ff32"}
Sep 16 18:43:19.876200 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:19.876164    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vmbb4" event={"ID":"9f7c6c50-3969-4103-a47f-d4eea4196e74","Type":"ContainerStarted","Data":"fd3a519637db881a238587cd9aa63f72d95fe848ceb16130bb978cb26f360a2d"}
Sep 16 18:43:19.877331 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:19.877305    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf" event={"ID":"b8dac3a1-52c7-4402-a075-59d997b756ef","Type":"ContainerStarted","Data":"1623065fd52d3f7a529235c67f7acd6b8731bd2097d1491e5a7ac5c320d93445"}
Sep 16 18:43:19.878492 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:19.878468    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7kkxp" event={"ID":"a1140d90-5f02-4921-b4ce-40e3a7129314","Type":"ContainerStarted","Data":"1e22c60a2584ac9b95ca4ba4fe6a4a37cec94a7fed3efa356b86c6274daf8f5a"}
Sep 16 18:43:19.879780 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:19.879697    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-w8t8m" event={"ID":"870ff296-8673-44d5-9fc3-619486d607a1","Type":"ContainerStarted","Data":"a7ce6adcecd32a681112e90a0dee5d0731164ab8a29c9975469e5b19d4814d15"}
Sep 16 18:43:19.880933 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:19.880908    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pr4kj" event={"ID":"bfffe95c-37f4-4aab-a7aa-19c64f66e810","Type":"ContainerStarted","Data":"714fad343b44fb516cd6a3a88b6ccb6d8ec93a892203362d4444b99d7cb34f08"}
Sep 16 18:43:19.882228 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:19.882208    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5hzq" event={"ID":"47ed2ebe-2b00-4044-b292-59a9ab8fafca","Type":"ContainerStarted","Data":"5f3b08476a9355f88ad5d312d6c67d3294e7c4d03216c5cb95261db282119504"}
Sep 16 18:43:19.883217 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:19.883194    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6nmxb" event={"ID":"245475f1-ca6f-40bb-8ab5-a2ab8f46c538","Type":"ContainerStarted","Data":"fbfdaf0147835d8b57baddcbea9d3e6b13416fe87e5ad52e319a742ec31e3739"}
Sep 16 18:43:19.885100 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:19.885073    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rkgjp" event={"ID":"c7890694-9f4a-4bf0-9647-1d84e8f43321","Type":"ContainerStarted","Data":"04f8b21f4748857177030c1cdc9902d780e1d7f2e6387f6c18187159070d036f"}
Sep 16 18:43:19.889713 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:19.889685    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-194.ec2.internal" event={"ID":"a8fee4f0c396919249581e1ee599be3f","Type":"ContainerStarted","Data":"c924dfea107b4364300cf03a0d3433a881c30fe269c92303e1b5577b489f784b"}
Sep 16 18:43:19.905825 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:19.905767    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-194.ec2.internal" podStartSLOduration=1.905751372 podStartE2EDuration="1.905751372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:19.905388583 +0000 UTC m=+3.547104250" watchObservedRunningTime="2026-09-16 18:43:19.905751372 +0000 UTC m=+3.547467035"
Sep 16 18:43:20.389647 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:20.389560    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de8ddfa3-fd54-49d7-8f90-035aa4439e98-metrics-certs\") pod \"network-metrics-daemon-g7q9m\" (UID: \"de8ddfa3-fd54-49d7-8f90-035aa4439e98\") " pod="openshift-multus/network-metrics-daemon-g7q9m"
Sep 16 18:43:20.389856 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:20.389779    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:20.389856 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:20.389845    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de8ddfa3-fd54-49d7-8f90-035aa4439e98-metrics-certs podName:de8ddfa3-fd54-49d7-8f90-035aa4439e98 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:22.389826205 +0000 UTC m=+6.031541849 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de8ddfa3-fd54-49d7-8f90-035aa4439e98-metrics-certs") pod "network-metrics-daemon-g7q9m" (UID: "de8ddfa3-fd54-49d7-8f90-035aa4439e98") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:20.490692 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:20.490219    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e6428035-ca16-4098-ac4e-1ca4931bb7a7-original-pull-secret\") pod \"global-pull-secret-syncer-tplxv\" (UID: \"e6428035-ca16-4098-ac4e-1ca4931bb7a7\") " pod="kube-system/global-pull-secret-syncer-tplxv"
Sep 16 18:43:20.490692 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:20.490362    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:20.490692 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:20.490426    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6428035-ca16-4098-ac4e-1ca4931bb7a7-original-pull-secret podName:e6428035-ca16-4098-ac4e-1ca4931bb7a7 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:22.490409004 +0000 UTC m=+6.132124651 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e6428035-ca16-4098-ac4e-1ca4931bb7a7-original-pull-secret") pod "global-pull-secret-syncer-tplxv" (UID: "e6428035-ca16-4098-ac4e-1ca4931bb7a7") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:20.591283 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:20.591239    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmpfz\" (UniqueName: \"kubernetes.io/projected/f79ebb7c-f4b5-4e48-9506-81110189ba43-kube-api-access-pmpfz\") pod \"network-check-target-dgl4n\" (UID: \"f79ebb7c-f4b5-4e48-9506-81110189ba43\") " pod="openshift-network-diagnostics/network-check-target-dgl4n"
Sep 16 18:43:20.591450 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:20.591434    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:43:20.591523 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:20.591457    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:43:20.591523 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:20.591473    2582 projected.go:196] Error preparing data for projected volume kube-api-access-pmpfz for pod openshift-network-diagnostics/network-check-target-dgl4n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:20.591617 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:20.591545    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f79ebb7c-f4b5-4e48-9506-81110189ba43-kube-api-access-pmpfz podName:f79ebb7c-f4b5-4e48-9506-81110189ba43 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:22.591525799 +0000 UTC m=+6.233241444 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-pmpfz" (UniqueName: "kubernetes.io/projected/f79ebb7c-f4b5-4e48-9506-81110189ba43-kube-api-access-pmpfz") pod "network-check-target-dgl4n" (UID: "f79ebb7c-f4b5-4e48-9506-81110189ba43") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:20.816840 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:20.816746    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:38:18 +0000 UTC" deadline="2028-07-04 13:19:23.787573522 +0000 UTC"
Sep 16 18:43:20.816840 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:20.816806    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15762h36m2.970771213s"
Sep 16 18:43:20.867856 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:20.867827    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tplxv"
Sep 16 18:43:20.868037 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:20.867950    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tplxv" podUID="e6428035-ca16-4098-ac4e-1ca4931bb7a7"
Sep 16 18:43:20.912005 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:20.911892    2582 generic.go:356] "Generic (PLEG): container finished" podID="65b8cacc050a0fe82b20ca75521a82f8" containerID="4d950595b5f3daadec4b544963efa5990190e236f06569163890325d3e32eadc" exitCode=0
Sep 16 18:43:20.912005 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:20.911978    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-194.ec2.internal" event={"ID":"65b8cacc050a0fe82b20ca75521a82f8","Type":"ContainerDied","Data":"4d950595b5f3daadec4b544963efa5990190e236f06569163890325d3e32eadc"}
Sep 16 18:43:21.867320 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:21.867200    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgl4n"
Sep 16 18:43:21.867803 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:21.867327    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgl4n" podUID="f79ebb7c-f4b5-4e48-9506-81110189ba43"
Sep 16 18:43:21.867803 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:21.867744    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g7q9m"
Sep 16 18:43:21.867943 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:21.867852    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g7q9m" podUID="de8ddfa3-fd54-49d7-8f90-035aa4439e98"
Sep 16 18:43:21.938088 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:21.937385    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-194.ec2.internal" event={"ID":"65b8cacc050a0fe82b20ca75521a82f8","Type":"ContainerStarted","Data":"ff0c37ce92085efa49edadbc3fa81dd63c3b059e74652f44115d068f12a1a61a"}
Sep 16 18:43:22.405500 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:22.405413    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de8ddfa3-fd54-49d7-8f90-035aa4439e98-metrics-certs\") pod \"network-metrics-daemon-g7q9m\" (UID: \"de8ddfa3-fd54-49d7-8f90-035aa4439e98\") " pod="openshift-multus/network-metrics-daemon-g7q9m"
Sep 16 18:43:22.405700 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:22.405561    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:22.405700 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:22.405623    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de8ddfa3-fd54-49d7-8f90-035aa4439e98-metrics-certs podName:de8ddfa3-fd54-49d7-8f90-035aa4439e98 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:26.405605747 +0000 UTC m=+10.047321394 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de8ddfa3-fd54-49d7-8f90-035aa4439e98-metrics-certs") pod "network-metrics-daemon-g7q9m" (UID: "de8ddfa3-fd54-49d7-8f90-035aa4439e98") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:22.506094 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:22.506046    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e6428035-ca16-4098-ac4e-1ca4931bb7a7-original-pull-secret\") pod \"global-pull-secret-syncer-tplxv\" (UID: \"e6428035-ca16-4098-ac4e-1ca4931bb7a7\") " pod="kube-system/global-pull-secret-syncer-tplxv"
Sep 16 18:43:22.506284 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:22.506209    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:22.506348 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:22.506286    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6428035-ca16-4098-ac4e-1ca4931bb7a7-original-pull-secret podName:e6428035-ca16-4098-ac4e-1ca4931bb7a7 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:26.506266137 +0000 UTC m=+10.147981782 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e6428035-ca16-4098-ac4e-1ca4931bb7a7-original-pull-secret") pod "global-pull-secret-syncer-tplxv" (UID: "e6428035-ca16-4098-ac4e-1ca4931bb7a7") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:22.606640 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:22.606601    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmpfz\" (UniqueName: \"kubernetes.io/projected/f79ebb7c-f4b5-4e48-9506-81110189ba43-kube-api-access-pmpfz\") pod \"network-check-target-dgl4n\" (UID: \"f79ebb7c-f4b5-4e48-9506-81110189ba43\") " pod="openshift-network-diagnostics/network-check-target-dgl4n"
Sep 16 18:43:22.606844 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:22.606802    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:43:22.606844 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:22.606823    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:43:22.606844 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:22.606836    2582 projected.go:196] Error preparing data for projected volume kube-api-access-pmpfz for pod openshift-network-diagnostics/network-check-target-dgl4n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:22.606998 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:22.606893    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f79ebb7c-f4b5-4e48-9506-81110189ba43-kube-api-access-pmpfz podName:f79ebb7c-f4b5-4e48-9506-81110189ba43 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:26.606875884 +0000 UTC m=+10.248591528 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-pmpfz" (UniqueName: "kubernetes.io/projected/f79ebb7c-f4b5-4e48-9506-81110189ba43-kube-api-access-pmpfz") pod "network-check-target-dgl4n" (UID: "f79ebb7c-f4b5-4e48-9506-81110189ba43") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:22.867630 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:22.867596    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tplxv"
Sep 16 18:43:22.868112 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:22.867750    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tplxv" podUID="e6428035-ca16-4098-ac4e-1ca4931bb7a7"
Sep 16 18:43:23.866676 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:23.866642    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgl4n"
Sep 16 18:43:23.866894 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:23.866683    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g7q9m"
Sep 16 18:43:23.866894 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:23.866815    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgl4n" podUID="f79ebb7c-f4b5-4e48-9506-81110189ba43"
Sep 16 18:43:23.867010 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:23.866918    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g7q9m" podUID="de8ddfa3-fd54-49d7-8f90-035aa4439e98"
Sep 16 18:43:24.866858 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:24.866823    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tplxv"
Sep 16 18:43:24.867345 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:24.866985    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tplxv" podUID="e6428035-ca16-4098-ac4e-1ca4931bb7a7"
Sep 16 18:43:25.867162 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:25.867121    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgl4n"
Sep 16 18:43:25.867706 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:25.867255    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgl4n" podUID="f79ebb7c-f4b5-4e48-9506-81110189ba43"
Sep 16 18:43:25.867706 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:25.867343    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g7q9m"
Sep 16 18:43:25.867706 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:25.867441    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g7q9m" podUID="de8ddfa3-fd54-49d7-8f90-035aa4439e98"
Sep 16 18:43:26.435784 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:26.435742    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de8ddfa3-fd54-49d7-8f90-035aa4439e98-metrics-certs\") pod \"network-metrics-daemon-g7q9m\" (UID: \"de8ddfa3-fd54-49d7-8f90-035aa4439e98\") " pod="openshift-multus/network-metrics-daemon-g7q9m"
Sep 16 18:43:26.435984 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:26.435926    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:26.436050 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:26.436008    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de8ddfa3-fd54-49d7-8f90-035aa4439e98-metrics-certs podName:de8ddfa3-fd54-49d7-8f90-035aa4439e98 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:34.43598726 +0000 UTC m=+18.077702905 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de8ddfa3-fd54-49d7-8f90-035aa4439e98-metrics-certs") pod "network-metrics-daemon-g7q9m" (UID: "de8ddfa3-fd54-49d7-8f90-035aa4439e98") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:26.536890 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:26.536851    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e6428035-ca16-4098-ac4e-1ca4931bb7a7-original-pull-secret\") pod \"global-pull-secret-syncer-tplxv\" (UID: \"e6428035-ca16-4098-ac4e-1ca4931bb7a7\") " pod="kube-system/global-pull-secret-syncer-tplxv"
Sep 16 18:43:26.537057 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:26.537016    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:26.537136 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:26.537080    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6428035-ca16-4098-ac4e-1ca4931bb7a7-original-pull-secret podName:e6428035-ca16-4098-ac4e-1ca4931bb7a7 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:34.537065008 +0000 UTC m=+18.178780649 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e6428035-ca16-4098-ac4e-1ca4931bb7a7-original-pull-secret") pod "global-pull-secret-syncer-tplxv" (UID: "e6428035-ca16-4098-ac4e-1ca4931bb7a7") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:26.637700 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:26.637658    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmpfz\" (UniqueName: \"kubernetes.io/projected/f79ebb7c-f4b5-4e48-9506-81110189ba43-kube-api-access-pmpfz\") pod \"network-check-target-dgl4n\" (UID: \"f79ebb7c-f4b5-4e48-9506-81110189ba43\") " pod="openshift-network-diagnostics/network-check-target-dgl4n"
Sep 16 18:43:26.637898 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:26.637881    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:43:26.637949 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:26.637907    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:43:26.637949 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:26.637922    2582 projected.go:196] Error preparing data for projected volume kube-api-access-pmpfz for pod openshift-network-diagnostics/network-check-target-dgl4n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:26.638039 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:26.637985    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f79ebb7c-f4b5-4e48-9506-81110189ba43-kube-api-access-pmpfz podName:f79ebb7c-f4b5-4e48-9506-81110189ba43 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:34.637967117 +0000 UTC m=+18.279682762 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-pmpfz" (UniqueName: "kubernetes.io/projected/f79ebb7c-f4b5-4e48-9506-81110189ba43-kube-api-access-pmpfz") pod "network-check-target-dgl4n" (UID: "f79ebb7c-f4b5-4e48-9506-81110189ba43") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:26.868052 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:26.868018    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tplxv"
Sep 16 18:43:26.868526 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:26.868137    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tplxv" podUID="e6428035-ca16-4098-ac4e-1ca4931bb7a7"
Sep 16 18:43:27.867224 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:27.867146    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgl4n"
Sep 16 18:43:27.867403 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:27.867277    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgl4n" podUID="f79ebb7c-f4b5-4e48-9506-81110189ba43"
Sep 16 18:43:27.867489 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:27.867460    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g7q9m"
Sep 16 18:43:27.867610 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:27.867572    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g7q9m" podUID="de8ddfa3-fd54-49d7-8f90-035aa4439e98"
Sep 16 18:43:28.867127 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:28.867091    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tplxv"
Sep 16 18:43:28.867613 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:28.867208    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tplxv" podUID="e6428035-ca16-4098-ac4e-1ca4931bb7a7"
Sep 16 18:43:29.866916 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:29.866874    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgl4n"
Sep 16 18:43:29.867092 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:29.866890    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g7q9m"
Sep 16 18:43:29.867092 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:29.867001    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgl4n" podUID="f79ebb7c-f4b5-4e48-9506-81110189ba43"
Sep 16 18:43:29.867092 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:29.867059    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g7q9m" podUID="de8ddfa3-fd54-49d7-8f90-035aa4439e98"
Sep 16 18:43:30.867214 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:30.867178    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tplxv"
Sep 16 18:43:30.867691 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:30.867303    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tplxv" podUID="e6428035-ca16-4098-ac4e-1ca4931bb7a7"
Sep 16 18:43:31.867606 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:31.867564    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g7q9m"
Sep 16 18:43:31.868058 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:31.867616    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgl4n"
Sep 16 18:43:31.868058 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:31.867699    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g7q9m" podUID="de8ddfa3-fd54-49d7-8f90-035aa4439e98"
Sep 16 18:43:31.868058 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:31.867842    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgl4n" podUID="f79ebb7c-f4b5-4e48-9506-81110189ba43"
Sep 16 18:43:32.867241 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:32.867203    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tplxv"
Sep 16 18:43:32.867428 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:32.867347    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tplxv" podUID="e6428035-ca16-4098-ac4e-1ca4931bb7a7"
Sep 16 18:43:33.867362 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:33.867329    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgl4n"
Sep 16 18:43:33.867795 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:33.867457    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgl4n" podUID="f79ebb7c-f4b5-4e48-9506-81110189ba43"
Sep 16 18:43:33.867795 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:33.867517    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g7q9m"
Sep 16 18:43:33.867795 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:33.867635    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g7q9m" podUID="de8ddfa3-fd54-49d7-8f90-035aa4439e98"
Sep 16 18:43:34.495273 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:34.495238    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de8ddfa3-fd54-49d7-8f90-035aa4439e98-metrics-certs\") pod \"network-metrics-daemon-g7q9m\" (UID: \"de8ddfa3-fd54-49d7-8f90-035aa4439e98\") " pod="openshift-multus/network-metrics-daemon-g7q9m"
Sep 16 18:43:34.495437 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:34.495390    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:34.495498 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:34.495466    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de8ddfa3-fd54-49d7-8f90-035aa4439e98-metrics-certs podName:de8ddfa3-fd54-49d7-8f90-035aa4439e98 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:50.495446476 +0000 UTC m=+34.137162117 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de8ddfa3-fd54-49d7-8f90-035aa4439e98-metrics-certs") pod "network-metrics-daemon-g7q9m" (UID: "de8ddfa3-fd54-49d7-8f90-035aa4439e98") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:34.596516 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:34.596474    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e6428035-ca16-4098-ac4e-1ca4931bb7a7-original-pull-secret\") pod \"global-pull-secret-syncer-tplxv\" (UID: \"e6428035-ca16-4098-ac4e-1ca4931bb7a7\") " pod="kube-system/global-pull-secret-syncer-tplxv"
Sep 16 18:43:34.596663 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:34.596618    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:34.596711 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:34.596677    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6428035-ca16-4098-ac4e-1ca4931bb7a7-original-pull-secret podName:e6428035-ca16-4098-ac4e-1ca4931bb7a7 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:50.596662269 +0000 UTC m=+34.238377916 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e6428035-ca16-4098-ac4e-1ca4931bb7a7-original-pull-secret") pod "global-pull-secret-syncer-tplxv" (UID: "e6428035-ca16-4098-ac4e-1ca4931bb7a7") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:34.697291 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:34.697247    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmpfz\" (UniqueName: \"kubernetes.io/projected/f79ebb7c-f4b5-4e48-9506-81110189ba43-kube-api-access-pmpfz\") pod \"network-check-target-dgl4n\" (UID: \"f79ebb7c-f4b5-4e48-9506-81110189ba43\") " pod="openshift-network-diagnostics/network-check-target-dgl4n"
Sep 16 18:43:34.697483 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:34.697455    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:43:34.697483 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:34.697481    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:43:34.697595 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:34.697497    2582 projected.go:196] Error preparing data for projected volume kube-api-access-pmpfz for pod openshift-network-diagnostics/network-check-target-dgl4n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:34.697595 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:34.697561    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f79ebb7c-f4b5-4e48-9506-81110189ba43-kube-api-access-pmpfz podName:f79ebb7c-f4b5-4e48-9506-81110189ba43 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:50.697540954 +0000 UTC m=+34.339256603 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-pmpfz" (UniqueName: "kubernetes.io/projected/f79ebb7c-f4b5-4e48-9506-81110189ba43-kube-api-access-pmpfz") pod "network-check-target-dgl4n" (UID: "f79ebb7c-f4b5-4e48-9506-81110189ba43") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:34.867670 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:34.867633    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tplxv"
Sep 16 18:43:34.868140 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:34.867773    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tplxv" podUID="e6428035-ca16-4098-ac4e-1ca4931bb7a7"
Sep 16 18:43:35.867734 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:35.867692    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgl4n"
Sep 16 18:43:35.867734 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:35.867713    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g7q9m"
Sep 16 18:43:35.868186 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:35.867831    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgl4n" podUID="f79ebb7c-f4b5-4e48-9506-81110189ba43"
Sep 16 18:43:35.868186 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:35.867973    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g7q9m" podUID="de8ddfa3-fd54-49d7-8f90-035aa4439e98"
Sep 16 18:43:36.867919 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:36.867885    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tplxv"
Sep 16 18:43:36.868348 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:36.868008    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tplxv" podUID="e6428035-ca16-4098-ac4e-1ca4931bb7a7"
Sep 16 18:43:37.867159 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:37.866994    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgl4n"
Sep 16 18:43:37.867272 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:37.867046    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g7q9m"
Sep 16 18:43:37.867336 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:37.867264    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgl4n" podUID="f79ebb7c-f4b5-4e48-9506-81110189ba43"
Sep 16 18:43:37.867336 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:37.867313    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g7q9m" podUID="de8ddfa3-fd54-49d7-8f90-035aa4439e98"
Sep 16 18:43:37.972895 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:37.972690    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wdq46" event={"ID":"c60bca6d-3f8c-4725-94a2-eedd9a075586","Type":"ContainerStarted","Data":"527dc3a85f54c92f89b88b8a22ac4148aff4f42b4460fde35ff89f543dc85289"}
Sep 16 18:43:37.974414 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:37.974378    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7kkxp" event={"ID":"a1140d90-5f02-4921-b4ce-40e3a7129314","Type":"ContainerStarted","Data":"f1d05f6425a90583616ca5305befa07af130d11cab6f1b3ef27cb4c779c196f5"}
Sep 16 18:43:37.976153 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:37.976120    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-w8t8m" event={"ID":"870ff296-8673-44d5-9fc3-619486d607a1","Type":"ContainerStarted","Data":"742b17dbfe4f330be69d44e0941ef5bd3fd938b2d476b08b396b3d4fd7b49ed3"}
Sep 16 18:43:37.977746 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:37.977674    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pr4kj" event={"ID":"bfffe95c-37f4-4aab-a7aa-19c64f66e810","Type":"ContainerStarted","Data":"5d04a193a9f8dfb476804ef6a9849ca26ea0f59bb62881794f4819dd12d8ca33"}
Sep 16 18:43:37.979183 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:37.979156    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5hzq" event={"ID":"47ed2ebe-2b00-4044-b292-59a9ab8fafca","Type":"ContainerStarted","Data":"f7596dfac0daf147c52bab1acbdba4730b9f5b186ec128785fa0e35894499be8"}
Sep 16 18:43:37.980709 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:37.980681    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6nmxb" event={"ID":"245475f1-ca6f-40bb-8ab5-a2ab8f46c538","Type":"ContainerStarted","Data":"20dcedd4ab0bfb91924bb55e86b465dec05361d8b6eafedc8a710810a70fed1c"}
Sep 16 18:43:37.982349 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:37.982320    2582 generic.go:356] "Generic (PLEG): container finished" podID="c7890694-9f4a-4bf0-9647-1d84e8f43321" containerID="fa977528108f315643b483f7023a42bde40168dc053bb8cc61113022fece9028" exitCode=0
Sep 16 18:43:37.982452 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:37.982353    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rkgjp" event={"ID":"c7890694-9f4a-4bf0-9647-1d84e8f43321","Type":"ContainerDied","Data":"fa977528108f315643b483f7023a42bde40168dc053bb8cc61113022fece9028"}
Sep 16 18:43:37.991531 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:37.991487    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-194.ec2.internal" podStartSLOduration=19.991473097 podStartE2EDuration="19.991473097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:21.955941539 +0000 UTC m=+5.597657206" watchObservedRunningTime="2026-09-16 18:43:37.991473097 +0000 UTC m=+21.633188776"
Sep 16 18:43:37.991678 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:37.991626    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-wdq46" podStartSLOduration=3.303769374 podStartE2EDuration="20.991615772s" totalImagesPullingTime="17.687846398s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:37.990865678 +0000 UTC m=+21.632581343" watchObservedRunningTime="2026-09-16 18:43:37.991615772 +0000 UTC m=+21.633331451"
Sep 16 18:43:38.010772 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:38.010710    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-w8t8m" podStartSLOduration=3.233746774 podStartE2EDuration="21.010694957s" totalImagesPullingTime="17.776948183s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:38.010487681 +0000 UTC m=+21.652203345" watchObservedRunningTime="2026-09-16 18:43:38.010694957 +0000 UTC m=+21.652410621"
Sep 16 18:43:38.025739 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:38.025685    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-pr4kj" podStartSLOduration=4.305375692 podStartE2EDuration="22.025670094s" totalImagesPullingTime="17.720294402s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:43:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:38.02558932 +0000 UTC m=+21.667304985" watchObservedRunningTime="2026-09-16 18:43:38.025670094 +0000 UTC m=+21.667385760"
Sep 16 18:43:38.043961 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:38.043919    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-6nmxb" podStartSLOduration=3.029439709 podStartE2EDuration="21.043903635s" totalImagesPullingTime="18.014463926s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:38.043712426 +0000 UTC m=+21.685428092" watchObservedRunningTime="2026-09-16 18:43:38.043903635 +0000 UTC m=+21.685619298"
Sep 16 18:43:38.064178 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:38.063965    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7kkxp" podStartSLOduration=3.306527884 podStartE2EDuration="21.063944925s" totalImagesPullingTime="17.757417041s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:38.063018696 +0000 UTC m=+21.704734361" watchObservedRunningTime="2026-09-16 18:43:38.063944925 +0000 UTC m=+21.705660591"
Sep 16 18:43:38.530730 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:38.530680    2582 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 18:43:38.832853 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:38.832738    2582 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T18:43:38.530703774Z","UUID":"39c7ada0-ae69-4b41-9139-634f1dc60fa5","Handler":null,"Name":"","Endpoint":""}
Sep 16 18:43:38.835230 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:38.835189    2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 18:43:38.835230 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:38.835226    2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 18:43:38.856378 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:38.856344    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-pr4kj"
Sep 16 18:43:38.866903 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:38.866811    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tplxv"
Sep 16 18:43:38.867098 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:38.866936    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tplxv" podUID="e6428035-ca16-4098-ac4e-1ca4931bb7a7"
Sep 16 18:43:38.989005 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:38.988967    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf" event={"ID":"b8dac3a1-52c7-4402-a075-59d997b756ef","Type":"ContainerStarted","Data":"bed07722791f6d77ad36e4c70cb4be2714835e0566b9c6edc1ad2a1e332f935a"}
Sep 16 18:43:38.990082 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:38.989056    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf" event={"ID":"b8dac3a1-52c7-4402-a075-59d997b756ef","Type":"ContainerStarted","Data":"a976794462f3745bee4bedb319dd0d19d61032cb4fd84af389302e1e5884ec28"}
Sep 16 18:43:38.990082 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:38.989089    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf" event={"ID":"b8dac3a1-52c7-4402-a075-59d997b756ef","Type":"ContainerStarted","Data":"1caaa562e26f8bd8a1f846f6d4e9d9ca1e1255cb9911eb3b5feb85af835a16c4"}
Sep 16 18:43:38.990082 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:38.989126    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf" event={"ID":"b8dac3a1-52c7-4402-a075-59d997b756ef","Type":"ContainerStarted","Data":"f499abb6bca30220770621540355a5d9a2be1dc5e753da937630c1b0f7086023"}
Sep 16 18:43:38.990082 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:38.989171    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf" event={"ID":"b8dac3a1-52c7-4402-a075-59d997b756ef","Type":"ContainerStarted","Data":"e7d1f5a6891a3b8584c54cec57512a304f8748a0ef39ddca774c8514b01381f5"}
Sep 16 18:43:38.990082 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:38.989217    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf" event={"ID":"b8dac3a1-52c7-4402-a075-59d997b756ef","Type":"ContainerStarted","Data":"42fb1668f5dad84e06cae60c1c3598cf3244bfc51e65352b513ae69864ddd0dc"}
Sep 16 18:43:38.991675 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:38.991486    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5hzq" event={"ID":"47ed2ebe-2b00-4044-b292-59a9ab8fafca","Type":"ContainerStarted","Data":"b47f773bcf2e124338cda1191c7b18cec3c0e780a8a7ae3d460a32599e7e9f5b"}
Sep 16 18:43:39.866833 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:39.866798    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgl4n"
Sep 16 18:43:39.866986 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:39.866846    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g7q9m"
Sep 16 18:43:39.866986 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:39.866923    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgl4n" podUID="f79ebb7c-f4b5-4e48-9506-81110189ba43"
Sep 16 18:43:39.867071 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:39.867052    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g7q9m" podUID="de8ddfa3-fd54-49d7-8f90-035aa4439e98"
Sep 16 18:43:39.995705 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:39.995663    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vmbb4" event={"ID":"9f7c6c50-3969-4103-a47f-d4eea4196e74","Type":"ContainerStarted","Data":"5ff6a8a6fef5e73c019cc4347385ecd9e4de849e6334115269b95e62ed439aa8"}
Sep 16 18:43:39.998481 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:39.998455    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5hzq" event={"ID":"47ed2ebe-2b00-4044-b292-59a9ab8fafca","Type":"ContainerStarted","Data":"06255185c9329e9658b1cc348474e60440a6f2f894d791d6f1ec630027abf03b"}
Sep 16 18:43:40.011393 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:40.011342    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vmbb4" podStartSLOduration=6.259898905 podStartE2EDuration="24.011326839s" totalImagesPullingTime="17.751427934s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:40.010824096 +0000 UTC m=+23.652539761" watchObservedRunningTime="2026-09-16 18:43:40.011326839 +0000 UTC m=+23.653042502"
Sep 16 18:43:40.034337 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:40.032759    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5hzq" podStartSLOduration=3.667262534 podStartE2EDuration="23.032738537s" totalImagesPullingTime="19.365476003s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:40.031807112 +0000 UTC m=+23.673522775" watchObservedRunningTime="2026-09-16 18:43:40.032738537 +0000 UTC m=+23.674454198"
Sep 16 18:43:40.223730 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:40.223631    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-pr4kj"
Sep 16 18:43:40.224325 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:40.224302    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-pr4kj"
Sep 16 18:43:40.867667 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:40.867629    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tplxv"
Sep 16 18:43:40.867845 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:40.867765    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tplxv" podUID="e6428035-ca16-4098-ac4e-1ca4931bb7a7"
Sep 16 18:43:41.002206 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:41.002178    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pr4kj"
Sep 16 18:43:41.867413 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:41.867377    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g7q9m"
Sep 16 18:43:41.867600 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:41.867394    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgl4n"
Sep 16 18:43:41.867600 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:41.867515    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g7q9m" podUID="de8ddfa3-fd54-49d7-8f90-035aa4439e98"
Sep 16 18:43:41.867731 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:41.867595    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgl4n" podUID="f79ebb7c-f4b5-4e48-9506-81110189ba43"
Sep 16 18:43:42.867614 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:42.867443    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tplxv"
Sep 16 18:43:42.868091 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:42.867690    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tplxv" podUID="e6428035-ca16-4098-ac4e-1ca4931bb7a7"
Sep 16 18:43:43.009026 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:43.008992    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf" event={"ID":"b8dac3a1-52c7-4402-a075-59d997b756ef","Type":"ContainerStarted","Data":"bc3fb6ac54794dc315de16ab448546197b66053ef2b2480e25e6ad7b02bac655"}
Sep 16 18:43:43.010686 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:43.010661    2582 generic.go:356] "Generic (PLEG): container finished" podID="c7890694-9f4a-4bf0-9647-1d84e8f43321" containerID="fc0cce79df5367d13c99ab42f011d7c6e75a3854547e526b4de1cf5bd7b6d4d1" exitCode=0
Sep 16 18:43:43.010819 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:43.010755    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rkgjp" event={"ID":"c7890694-9f4a-4bf0-9647-1d84e8f43321","Type":"ContainerDied","Data":"fc0cce79df5367d13c99ab42f011d7c6e75a3854547e526b4de1cf5bd7b6d4d1"}
Sep 16 18:43:43.867293 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:43.867207    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgl4n"
Sep 16 18:43:43.867418 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:43.867313    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgl4n" podUID="f79ebb7c-f4b5-4e48-9506-81110189ba43"
Sep 16 18:43:43.867418 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:43.867365    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g7q9m"
Sep 16 18:43:43.867493 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:43.867462    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g7q9m" podUID="de8ddfa3-fd54-49d7-8f90-035aa4439e98"
Sep 16 18:43:44.014564 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:44.014529    2582 generic.go:356] "Generic (PLEG): container finished" podID="c7890694-9f4a-4bf0-9647-1d84e8f43321" containerID="7b0bb6c849854efad3c0e6a386a8189b1f1ddbb15959b83e24c95814caad70d5" exitCode=0
Sep 16 18:43:44.015046 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:44.014573    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rkgjp" event={"ID":"c7890694-9f4a-4bf0-9647-1d84e8f43321","Type":"ContainerDied","Data":"7b0bb6c849854efad3c0e6a386a8189b1f1ddbb15959b83e24c95814caad70d5"}
Sep 16 18:43:44.867481 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:44.867189    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tplxv"
Sep 16 18:43:44.867611 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:44.867533    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tplxv" podUID="e6428035-ca16-4098-ac4e-1ca4931bb7a7"
Sep 16 18:43:45.019676 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:45.019641    2582 generic.go:356] "Generic (PLEG): container finished" podID="c7890694-9f4a-4bf0-9647-1d84e8f43321" containerID="112f8b77f114e0906c67fb4720ff36c12fe70e01d295b7700da9d25f6955198e" exitCode=0
Sep 16 18:43:45.020271 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:45.019738    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rkgjp" event={"ID":"c7890694-9f4a-4bf0-9647-1d84e8f43321","Type":"ContainerDied","Data":"112f8b77f114e0906c67fb4720ff36c12fe70e01d295b7700da9d25f6955198e"}
Sep 16 18:43:45.023696 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:45.023667    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf" event={"ID":"b8dac3a1-52c7-4402-a075-59d997b756ef","Type":"ContainerStarted","Data":"b2e941b3bdce7f9d8dffe2d9a9f3971e387d21b291adc955b26532e18d9d45b1"}
Sep 16 18:43:45.023929 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:45.023915    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:45.038777 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:45.038755    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:45.071164 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:45.071115    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf" podStartSLOduration=9.850270876 podStartE2EDuration="28.071101189s" totalImagesPullingTime="18.220830313s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:45.069452769 +0000 UTC m=+28.711168435" watchObservedRunningTime="2026-09-16 18:43:45.071101189 +0000 UTC m=+28.712816853"
Sep 16 18:43:45.866952 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:45.866918    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgl4n"
Sep 16 18:43:45.867144 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:45.867047    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgl4n" podUID="f79ebb7c-f4b5-4e48-9506-81110189ba43"
Sep 16 18:43:45.867144 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:45.867114    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g7q9m"
Sep 16 18:43:45.867271 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:45.867246    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g7q9m" podUID="de8ddfa3-fd54-49d7-8f90-035aa4439e98"
Sep 16 18:43:46.026842 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:46.026818    2582 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 16 18:43:46.027401 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:46.027377    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:46.047152 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:46.047124    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:46.182381 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:46.182288    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tplxv"]
Sep 16 18:43:46.182528 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:46.182464    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tplxv"
Sep 16 18:43:46.182592 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:46.182573    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tplxv" podUID="e6428035-ca16-4098-ac4e-1ca4931bb7a7"
Sep 16 18:43:46.186141 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:46.186085    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dgl4n"]
Sep 16 18:43:46.187020 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:46.186840    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-g7q9m"]
Sep 16 18:43:46.187020 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:46.186857    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgl4n"
Sep 16 18:43:46.187020 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:46.186967    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgl4n" podUID="f79ebb7c-f4b5-4e48-9506-81110189ba43"
Sep 16 18:43:46.187020 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:46.187022    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g7q9m"
Sep 16 18:43:46.187340 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:46.187118    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g7q9m" podUID="de8ddfa3-fd54-49d7-8f90-035aa4439e98"
Sep 16 18:43:46.218845 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:46.218808    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:43:47.867098 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:47.866879    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tplxv"
Sep 16 18:43:47.867521 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:47.866896    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g7q9m"
Sep 16 18:43:47.867521 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:47.867196    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tplxv" podUID="e6428035-ca16-4098-ac4e-1ca4931bb7a7"
Sep 16 18:43:47.867521 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:47.866896    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgl4n"
Sep 16 18:43:47.867521 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:47.867260    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g7q9m" podUID="de8ddfa3-fd54-49d7-8f90-035aa4439e98"
Sep 16 18:43:47.867521 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:47.867352    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgl4n" podUID="f79ebb7c-f4b5-4e48-9506-81110189ba43"
Sep 16 18:43:49.867249 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:49.867194    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g7q9m"
Sep 16 18:43:49.867249 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:49.867247    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgl4n"
Sep 16 18:43:49.867760 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:49.867217    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tplxv"
Sep 16 18:43:49.867760 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:49.867327    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g7q9m" podUID="de8ddfa3-fd54-49d7-8f90-035aa4439e98"
Sep 16 18:43:49.867760 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:49.867415    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tplxv" podUID="e6428035-ca16-4098-ac4e-1ca4931bb7a7"
Sep 16 18:43:49.867760 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:49.867462    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgl4n" podUID="f79ebb7c-f4b5-4e48-9506-81110189ba43"
Sep 16 18:43:50.038109 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.038076    2582 generic.go:356] "Generic (PLEG): container finished" podID="c7890694-9f4a-4bf0-9647-1d84e8f43321" containerID="5b5084ce22662cc7eb97d88b55c9855b439e05b55a11274ca45e10bf2290a18c" exitCode=0
Sep 16 18:43:50.038273 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.038141    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rkgjp" event={"ID":"c7890694-9f4a-4bf0-9647-1d84e8f43321","Type":"ContainerDied","Data":"5b5084ce22662cc7eb97d88b55c9855b439e05b55a11274ca45e10bf2290a18c"}
Sep 16 18:43:50.519371 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.519336    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de8ddfa3-fd54-49d7-8f90-035aa4439e98-metrics-certs\") pod \"network-metrics-daemon-g7q9m\" (UID: \"de8ddfa3-fd54-49d7-8f90-035aa4439e98\") " pod="openshift-multus/network-metrics-daemon-g7q9m"
Sep 16 18:43:50.519549 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:50.519459    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:50.519549 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:50.519513    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de8ddfa3-fd54-49d7-8f90-035aa4439e98-metrics-certs podName:de8ddfa3-fd54-49d7-8f90-035aa4439e98 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:22.519498866 +0000 UTC m=+66.161214512 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de8ddfa3-fd54-49d7-8f90-035aa4439e98-metrics-certs") pod "network-metrics-daemon-g7q9m" (UID: "de8ddfa3-fd54-49d7-8f90-035aa4439e98") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:50.619673 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.619638    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e6428035-ca16-4098-ac4e-1ca4931bb7a7-original-pull-secret\") pod \"global-pull-secret-syncer-tplxv\" (UID: \"e6428035-ca16-4098-ac4e-1ca4931bb7a7\") " pod="kube-system/global-pull-secret-syncer-tplxv"
Sep 16 18:43:50.619847 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:50.619766    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:50.619847 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:50.619817    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6428035-ca16-4098-ac4e-1ca4931bb7a7-original-pull-secret podName:e6428035-ca16-4098-ac4e-1ca4931bb7a7 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:22.619803567 +0000 UTC m=+66.261519209 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e6428035-ca16-4098-ac4e-1ca4931bb7a7-original-pull-secret") pod "global-pull-secret-syncer-tplxv" (UID: "e6428035-ca16-4098-ac4e-1ca4931bb7a7") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:50.638865 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.638844    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-194.ec2.internal" event="NodeReady"
Sep 16 18:43:50.638976 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.638967    2582 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 18:43:50.678939 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.678896    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-5znf9"]
Sep 16 18:43:50.700571 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.700550    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-79f6f8f56-gk2mv"]
Sep 16 18:43:50.700713 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.700683    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5znf9"
Sep 16 18:43:50.710468 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.710448    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 16 18:43:50.711373 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.711356    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:43:50.711921 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.711905    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 16 18:43:50.711963 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.711941    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-nsz85\""
Sep 16 18:43:50.718402 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.718377    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 16 18:43:50.720870 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.720849    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmpfz\" (UniqueName: \"kubernetes.io/projected/f79ebb7c-f4b5-4e48-9506-81110189ba43-kube-api-access-pmpfz\") pod \"network-check-target-dgl4n\" (UID: \"f79ebb7c-f4b5-4e48-9506-81110189ba43\") " pod="openshift-network-diagnostics/network-check-target-dgl4n"
Sep 16 18:43:50.721054 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:50.721032    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:43:50.721114 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:50.721062    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:43:50.721114 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:50.721077    2582 projected.go:196] Error preparing data for projected volume kube-api-access-pmpfz for pod openshift-network-diagnostics/network-check-target-dgl4n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:50.721210 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:50.721144    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f79ebb7c-f4b5-4e48-9506-81110189ba43-kube-api-access-pmpfz podName:f79ebb7c-f4b5-4e48-9506-81110189ba43 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:22.721123237 +0000 UTC m=+66.362838886 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-pmpfz" (UniqueName: "kubernetes.io/projected/f79ebb7c-f4b5-4e48-9506-81110189ba43-kube-api-access-pmpfz") pod "network-check-target-dgl4n" (UID: "f79ebb7c-f4b5-4e48-9506-81110189ba43") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:50.724734 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.724695    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dvjf7"]
Sep 16 18:43:50.724835 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.724819    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79f6f8f56-gk2mv"
Sep 16 18:43:50.729672 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.729654    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 18:43:50.730062 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.730047    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 18:43:50.730201 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.730105    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4tghj\""
Sep 16 18:43:50.731054 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.730994    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 18:43:50.734603 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.734585    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 18:43:50.745025 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.745006    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-h28sc"]
Sep 16 18:43:50.745134 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.745119    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dvjf7"
Sep 16 18:43:50.748311 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.748291    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 16 18:43:50.748398 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.748332    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:43:50.748441 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.748430    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-fhxx4\""
Sep 16 18:43:50.748791 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.748775    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 16 18:43:50.768516 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.768469    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-l56nd"]
Sep 16 18:43:50.768636 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.768599    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-h28sc"
Sep 16 18:43:50.776942 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.776920    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 16 18:43:50.776942 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.776938    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-2qmgb\""
Sep 16 18:43:50.777104 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.776971    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 16 18:43:50.777104 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.777009    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 18:43:50.777270 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.777256    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 18:43:50.781708 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.781690    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 16 18:43:50.790998 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.790978    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-7qwl2"]
Sep 16 18:43:50.791093 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.791083    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-l56nd"
Sep 16 18:43:50.797278 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.797243    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 16 18:43:50.797278 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.797258    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 16 18:43:50.800979 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.800960    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-vzfhx\""
Sep 16 18:43:50.815265 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.815247    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dvjf7"]
Sep 16 18:43:50.815372 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.815280    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-l56nd"]
Sep 16 18:43:50.815372 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.815294    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-h28sc"]
Sep 16 18:43:50.815372 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.815313    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-5znf9"]
Sep 16 18:43:50.815372 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.815329    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79f6f8f56-gk2mv"]
Sep 16 18:43:50.815372 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.815343    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-nw2m8"]
Sep 16 18:43:50.815554 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.815372    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-7qwl2"
Sep 16 18:43:50.820419 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.820399    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 16 18:43:50.820522 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.820507    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:43:50.820589 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.820510    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-vl5cq\""
Sep 16 18:43:50.820657 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.820549    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 16 18:43:50.821286 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.821262    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/40a3ba8a-40a8-48eb-99df-da46c59ee997-bound-sa-token\") pod \"image-registry-79f6f8f56-gk2mv\" (UID: \"40a3ba8a-40a8-48eb-99df-da46c59ee997\") " pod="openshift-image-registry/image-registry-79f6f8f56-gk2mv"
Sep 16 18:43:50.821398 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.821300    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q92ps\" (UniqueName: \"kubernetes.io/projected/fe6d9245-1609-46f1-9e0e-4fbdc31145d1-kube-api-access-q92ps\") pod \"cluster-samples-operator-86499c89c4-dvjf7\" (UID: \"fe6d9245-1609-46f1-9e0e-4fbdc31145d1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dvjf7"
Sep 16 18:43:50.821398 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.821305    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 16 18:43:50.821398 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.821330    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fa83f1c9-b7b9-4c1e-b0c4-09328df33051-serving-cert\") pod \"service-ca-operator-cf7c776b7-5znf9\" (UID: \"fa83f1c9-b7b9-4c1e-b0c4-09328df33051\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5znf9"
Sep 16 18:43:50.821563 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.821426    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/40a3ba8a-40a8-48eb-99df-da46c59ee997-image-registry-private-configuration\") pod \"image-registry-79f6f8f56-gk2mv\" (UID: \"40a3ba8a-40a8-48eb-99df-da46c59ee997\") " pod="openshift-image-registry/image-registry-79f6f8f56-gk2mv"
Sep 16 18:43:50.821563 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.821451    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40a3ba8a-40a8-48eb-99df-da46c59ee997-registry-tls\") pod \"image-registry-79f6f8f56-gk2mv\" (UID: \"40a3ba8a-40a8-48eb-99df-da46c59ee997\") " pod="openshift-image-registry/image-registry-79f6f8f56-gk2mv"
Sep 16 18:43:50.821563 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.821476    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcdvf\" (UniqueName: \"kubernetes.io/projected/40a3ba8a-40a8-48eb-99df-da46c59ee997-kube-api-access-vcdvf\") pod \"image-registry-79f6f8f56-gk2mv\" (UID: \"40a3ba8a-40a8-48eb-99df-da46c59ee997\") " pod="openshift-image-registry/image-registry-79f6f8f56-gk2mv"
Sep 16 18:43:50.821563 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.821508    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fvqg\" (UniqueName: \"kubernetes.io/projected/fa83f1c9-b7b9-4c1e-b0c4-09328df33051-kube-api-access-7fvqg\") pod \"service-ca-operator-cf7c776b7-5znf9\" (UID: \"fa83f1c9-b7b9-4c1e-b0c4-09328df33051\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5znf9"
Sep 16 18:43:50.821563 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.821531    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/40a3ba8a-40a8-48eb-99df-da46c59ee997-trusted-ca\") pod \"image-registry-79f6f8f56-gk2mv\" (UID: \"40a3ba8a-40a8-48eb-99df-da46c59ee997\") " pod="openshift-image-registry/image-registry-79f6f8f56-gk2mv"
Sep 16 18:43:50.821855 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.821570    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/40a3ba8a-40a8-48eb-99df-da46c59ee997-registry-certificates\") pod \"image-registry-79f6f8f56-gk2mv\" (UID: \"40a3ba8a-40a8-48eb-99df-da46c59ee997\") " pod="openshift-image-registry/image-registry-79f6f8f56-gk2mv"
Sep 16 18:43:50.821855 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.821604    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/40a3ba8a-40a8-48eb-99df-da46c59ee997-ca-trust-extracted\") pod \"image-registry-79f6f8f56-gk2mv\" (UID: \"40a3ba8a-40a8-48eb-99df-da46c59ee997\") " pod="openshift-image-registry/image-registry-79f6f8f56-gk2mv"
Sep 16 18:43:50.821855 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.821630    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fe6d9245-1609-46f1-9e0e-4fbdc31145d1-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-dvjf7\" (UID: \"fe6d9245-1609-46f1-9e0e-4fbdc31145d1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dvjf7"
Sep 16 18:43:50.821855 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.821656    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fa83f1c9-b7b9-4c1e-b0c4-09328df33051-config\") pod \"service-ca-operator-cf7c776b7-5znf9\" (UID: \"fa83f1c9-b7b9-4c1e-b0c4-09328df33051\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5znf9"
Sep 16 18:43:50.821855 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.821691    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/40a3ba8a-40a8-48eb-99df-da46c59ee997-installation-pull-secrets\") pod \"image-registry-79f6f8f56-gk2mv\" (UID: \"40a3ba8a-40a8-48eb-99df-da46c59ee997\") " pod="openshift-image-registry/image-registry-79f6f8f56-gk2mv"
Sep 16 18:43:50.827276 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.827260    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 16 18:43:50.839428 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.839408    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-nccjc"]
Sep 16 18:43:50.839561 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.839546    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nw2m8"
Sep 16 18:43:50.845902 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.845880    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 18:43:50.846002 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.845923    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 18:43:50.846077 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.846031    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xrh57\""
Sep 16 18:43:50.856579 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.856560    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-7qwl2"]
Sep 16 18:43:50.856666 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.856584    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7465b55b9d-njqmh"]
Sep 16 18:43:50.856709 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.856665    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-nccjc"
Sep 16 18:43:50.866169 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.866152    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 16 18:43:50.866378 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.866362    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 16 18:43:50.866458 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.866404    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 16 18:43:50.866626 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.866610    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-58t9p\""
Sep 16 18:43:50.871062 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.871041    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:43:50.877747 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.877714    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7465b55b9d-njqmh"
Sep 16 18:43:50.881165 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.881147    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wxss9"]
Sep 16 18:43:50.881517 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.881499    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 16 18:43:50.881775 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.881757    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 18:43:50.881861 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.881767    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 16 18:43:50.881861 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.881793    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-mq4xp\""
Sep 16 18:43:50.881861 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.881765    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 18:43:50.881861 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.881823    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 16 18:43:50.882030 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.881755    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 16 18:43:50.897860 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.897841    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f4d45b76-rpnnc"]
Sep 16 18:43:50.897938 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.897920    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wxss9"
Sep 16 18:43:50.903440 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.903422    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 18:43:50.909197 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.909183    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 18:43:50.909260 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.909189    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 18:43:50.909496 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.909483    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-ktvzd\""
Sep 16 18:43:50.920172 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.920153    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-674c5bb56f-6xn2v"]
Sep 16 18:43:50.920271 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.920257    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f4d45b76-rpnnc"
Sep 16 18:43:50.922324 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.922307    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed1b0dcc-4b23-4906-bc1d-4efa04e932cf-tmp\") pod \"insights-operator-5844b856db-h28sc\" (UID: \"ed1b0dcc-4b23-4906-bc1d-4efa04e932cf\") " pod="openshift-insights/insights-operator-5844b856db-h28sc"
Sep 16 18:43:50.922415 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.922336    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/40a3ba8a-40a8-48eb-99df-da46c59ee997-image-registry-private-configuration\") pod \"image-registry-79f6f8f56-gk2mv\" (UID: \"40a3ba8a-40a8-48eb-99df-da46c59ee997\") " pod="openshift-image-registry/image-registry-79f6f8f56-gk2mv"
Sep 16 18:43:50.922415 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.922355    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40a3ba8a-40a8-48eb-99df-da46c59ee997-registry-tls\") pod \"image-registry-79f6f8f56-gk2mv\" (UID: \"40a3ba8a-40a8-48eb-99df-da46c59ee997\") " pod="openshift-image-registry/image-registry-79f6f8f56-gk2mv"
Sep 16 18:43:50.922415 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.922372    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d76fc261-072a-4625-818d-e8faa5710cbd-tmp-dir\") pod \"dns-default-nw2m8\" (UID: \"d76fc261-072a-4625-818d-e8faa5710cbd\") " pod="openshift-dns/dns-default-nw2m8"
Sep 16 18:43:50.922415 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.922409    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed1b0dcc-4b23-4906-bc1d-4efa04e932cf-trusted-ca-bundle\") pod \"insights-operator-5844b856db-h28sc\" (UID: \"ed1b0dcc-4b23-4906-bc1d-4efa04e932cf\") " pod="openshift-insights/insights-operator-5844b856db-h28sc"
Sep 16 18:43:50.922612 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.922433    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed1b0dcc-4b23-4906-bc1d-4efa04e932cf-service-ca-bundle\") pod \"insights-operator-5844b856db-h28sc\" (UID: \"ed1b0dcc-4b23-4906-bc1d-4efa04e932cf\") " pod="openshift-insights/insights-operator-5844b856db-h28sc"
Sep 16 18:43:50.922612 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.922458    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkqdz\" (UniqueName: \"kubernetes.io/projected/ed1b0dcc-4b23-4906-bc1d-4efa04e932cf-kube-api-access-hkqdz\") pod \"insights-operator-5844b856db-h28sc\" (UID: \"ed1b0dcc-4b23-4906-bc1d-4efa04e932cf\") " pod="openshift-insights/insights-operator-5844b856db-h28sc"
Sep 16 18:43:50.922612 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:50.922487    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:43:50.922612 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:50.922507    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79f6f8f56-gk2mv: secret "image-registry-tls" not found
Sep 16 18:43:50.922612 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.922519    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/40a3ba8a-40a8-48eb-99df-da46c59ee997-ca-trust-extracted\") pod \"image-registry-79f6f8f56-gk2mv\" (UID: \"40a3ba8a-40a8-48eb-99df-da46c59ee997\") " pod="openshift-image-registry/image-registry-79f6f8f56-gk2mv"
Sep 16 18:43:50.922612 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:50.922566    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40a3ba8a-40a8-48eb-99df-da46c59ee997-registry-tls podName:40a3ba8a-40a8-48eb-99df-da46c59ee997 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:51.422548013 +0000 UTC m=+35.064263657 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/40a3ba8a-40a8-48eb-99df-da46c59ee997-registry-tls") pod "image-registry-79f6f8f56-gk2mv" (UID: "40a3ba8a-40a8-48eb-99df-da46c59ee997") : secret "image-registry-tls" not found
Sep 16 18:43:50.922612 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.922591    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b6695f86-08c0-4836-87b1-f2a9d76084bd-trusted-ca\") pod \"console-operator-696884fb9c-7qwl2\" (UID: \"b6695f86-08c0-4836-87b1-f2a9d76084bd\") " pod="openshift-console-operator/console-operator-696884fb9c-7qwl2"
Sep 16 18:43:50.922978 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.922626    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d76fc261-072a-4625-818d-e8faa5710cbd-metrics-tls\") pod \"dns-default-nw2m8\" (UID: \"d76fc261-072a-4625-818d-e8faa5710cbd\") " pod="openshift-dns/dns-default-nw2m8"
Sep 16 18:43:50.922978 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.922667    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ed1b0dcc-4b23-4906-bc1d-4efa04e932cf-snapshots\") pod \"insights-operator-5844b856db-h28sc\" (UID: \"ed1b0dcc-4b23-4906-bc1d-4efa04e932cf\") " pod="openshift-insights/insights-operator-5844b856db-h28sc"
Sep 16 18:43:50.922978 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.922702    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgv5f\" (UniqueName: \"kubernetes.io/projected/c1f25728-6d57-4c30-9d73-d90c127bd06c-kube-api-access-rgv5f\") pod \"kube-storage-version-migrator-operator-6c8f84f897-nccjc\" (UID: \"c1f25728-6d57-4c30-9d73-d90c127bd06c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-nccjc"
Sep 16 18:43:50.922978 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.922750    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b6695f86-08c0-4836-87b1-f2a9d76084bd-serving-cert\") pod \"console-operator-696884fb9c-7qwl2\" (UID: \"b6695f86-08c0-4836-87b1-f2a9d76084bd\") " pod="openshift-console-operator/console-operator-696884fb9c-7qwl2"
Sep 16 18:43:50.922978 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.922783    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/40a3ba8a-40a8-48eb-99df-da46c59ee997-bound-sa-token\") pod \"image-registry-79f6f8f56-gk2mv\" (UID: \"40a3ba8a-40a8-48eb-99df-da46c59ee997\") " pod="openshift-image-registry/image-registry-79f6f8f56-gk2mv"
Sep 16 18:43:50.922978 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.922814    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d76fc261-072a-4625-818d-e8faa5710cbd-config-volume\") pod \"dns-default-nw2m8\" (UID: \"d76fc261-072a-4625-818d-e8faa5710cbd\") " pod="openshift-dns/dns-default-nw2m8"
Sep 16 18:43:50.922978 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.922875    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c1f25728-6d57-4c30-9d73-d90c127bd06c-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-nccjc\" (UID: \"c1f25728-6d57-4c30-9d73-d90c127bd06c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-nccjc"
Sep 16 18:43:50.922978 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.922902    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fvqg\" (UniqueName: \"kubernetes.io/projected/fa83f1c9-b7b9-4c1e-b0c4-09328df33051-kube-api-access-7fvqg\") pod \"service-ca-operator-cf7c776b7-5znf9\" (UID: \"fa83f1c9-b7b9-4c1e-b0c4-09328df33051\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5znf9"
Sep 16 18:43:50.922978 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.922930    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ed1b0dcc-4b23-4906-bc1d-4efa04e932cf-serving-cert\") pod \"insights-operator-5844b856db-h28sc\" (UID: \"ed1b0dcc-4b23-4906-bc1d-4efa04e932cf\") " pod="openshift-insights/insights-operator-5844b856db-h28sc"
Sep 16 18:43:50.922978 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.922955    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/40a3ba8a-40a8-48eb-99df-da46c59ee997-ca-trust-extracted\") pod \"image-registry-79f6f8f56-gk2mv\" (UID: \"40a3ba8a-40a8-48eb-99df-da46c59ee997\") " pod="openshift-image-registry/image-registry-79f6f8f56-gk2mv"
Sep 16 18:43:50.923484 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.922961    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcdvf\" (UniqueName: \"kubernetes.io/projected/40a3ba8a-40a8-48eb-99df-da46c59ee997-kube-api-access-vcdvf\") pod \"image-registry-79f6f8f56-gk2mv\" (UID: \"40a3ba8a-40a8-48eb-99df-da46c59ee997\") " pod="openshift-image-registry/image-registry-79f6f8f56-gk2mv"
Sep 16 18:43:50.923484 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.923021    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ba5d0219-c94f-41a7-a6fc-86b14d2bccd2-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-l56nd\" (UID: \"ba5d0219-c94f-41a7-a6fc-86b14d2bccd2\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-l56nd"
Sep 16 18:43:50.923484 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.923057    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/40a3ba8a-40a8-48eb-99df-da46c59ee997-trusted-ca\") pod \"image-registry-79f6f8f56-gk2mv\" (UID: \"40a3ba8a-40a8-48eb-99df-da46c59ee997\") " pod="openshift-image-registry/image-registry-79f6f8f56-gk2mv"
Sep 16 18:43:50.923484 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.923089    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/40a3ba8a-40a8-48eb-99df-da46c59ee997-registry-certificates\") pod \"image-registry-79f6f8f56-gk2mv\" (UID: \"40a3ba8a-40a8-48eb-99df-da46c59ee997\") " pod="openshift-image-registry/image-registry-79f6f8f56-gk2mv"
Sep 16 18:43:50.923484 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.923117    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fe6d9245-1609-46f1-9e0e-4fbdc31145d1-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-dvjf7\" (UID: \"fe6d9245-1609-46f1-9e0e-4fbdc31145d1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dvjf7"
Sep 16 18:43:50.923484 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.923146    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c1f25728-6d57-4c30-9d73-d90c127bd06c-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-nccjc\" (UID: \"c1f25728-6d57-4c30-9d73-d90c127bd06c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-nccjc"
Sep 16 18:43:50.923484 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.923175    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fa83f1c9-b7b9-4c1e-b0c4-09328df33051-config\") pod \"service-ca-operator-cf7c776b7-5znf9\" (UID: \"fa83f1c9-b7b9-4c1e-b0c4-09328df33051\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5znf9"
Sep 16 18:43:50.923484 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.923229    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/40a3ba8a-40a8-48eb-99df-da46c59ee997-installation-pull-secrets\") pod \"image-registry-79f6f8f56-gk2mv\" (UID: \"40a3ba8a-40a8-48eb-99df-da46c59ee997\") " pod="openshift-image-registry/image-registry-79f6f8f56-gk2mv"
Sep 16 18:43:50.923484 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:50.923248    2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:43:50.923484 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.923255    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpzvf\" (UniqueName: \"kubernetes.io/projected/b6695f86-08c0-4836-87b1-f2a9d76084bd-kube-api-access-kpzvf\") pod \"console-operator-696884fb9c-7qwl2\" (UID: \"b6695f86-08c0-4836-87b1-f2a9d76084bd\") " pod="openshift-console-operator/console-operator-696884fb9c-7qwl2"
Sep 16 18:43:50.923484 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.923280    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b6695f86-08c0-4836-87b1-f2a9d76084bd-config\") pod \"console-operator-696884fb9c-7qwl2\" (UID: \"b6695f86-08c0-4836-87b1-f2a9d76084bd\") " pod="openshift-console-operator/console-operator-696884fb9c-7qwl2"
Sep 16 18:43:50.923484 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:50.923313    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe6d9245-1609-46f1-9e0e-4fbdc31145d1-samples-operator-tls podName:fe6d9245-1609-46f1-9e0e-4fbdc31145d1 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:51.423297595 +0000 UTC m=+35.065013245 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fe6d9245-1609-46f1-9e0e-4fbdc31145d1-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-dvjf7" (UID: "fe6d9245-1609-46f1-9e0e-4fbdc31145d1") : secret "samples-operator-tls" not found
Sep 16 18:43:50.923484 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.923335    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ba5d0219-c94f-41a7-a6fc-86b14d2bccd2-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-l56nd\" (UID: \"ba5d0219-c94f-41a7-a6fc-86b14d2bccd2\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-l56nd"
Sep 16 18:43:50.923484 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.923384    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw5xq\" (UniqueName: \"kubernetes.io/projected/d76fc261-072a-4625-818d-e8faa5710cbd-kube-api-access-vw5xq\") pod \"dns-default-nw2m8\" (UID: \"d76fc261-072a-4625-818d-e8faa5710cbd\") " pod="openshift-dns/dns-default-nw2m8"
Sep 16 18:43:50.923484 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.923413    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q92ps\" (UniqueName: \"kubernetes.io/projected/fe6d9245-1609-46f1-9e0e-4fbdc31145d1-kube-api-access-q92ps\") pod \"cluster-samples-operator-86499c89c4-dvjf7\" (UID: \"fe6d9245-1609-46f1-9e0e-4fbdc31145d1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dvjf7"
Sep 16 18:43:50.924170 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.923437    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fa83f1c9-b7b9-4c1e-b0c4-09328df33051-serving-cert\") pod \"service-ca-operator-cf7c776b7-5znf9\" (UID: \"fa83f1c9-b7b9-4c1e-b0c4-09328df33051\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5znf9"
Sep 16 18:43:50.924170 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.923699    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/40a3ba8a-40a8-48eb-99df-da46c59ee997-registry-certificates\") pod \"image-registry-79f6f8f56-gk2mv\" (UID: \"40a3ba8a-40a8-48eb-99df-da46c59ee997\") " pod="openshift-image-registry/image-registry-79f6f8f56-gk2mv"
Sep 16 18:43:50.924170 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.923810    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fa83f1c9-b7b9-4c1e-b0c4-09328df33051-config\") pod \"service-ca-operator-cf7c776b7-5znf9\" (UID: \"fa83f1c9-b7b9-4c1e-b0c4-09328df33051\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5znf9"
Sep 16 18:43:50.924170 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.924128    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/40a3ba8a-40a8-48eb-99df-da46c59ee997-trusted-ca\") pod \"image-registry-79f6f8f56-gk2mv\" (UID: \"40a3ba8a-40a8-48eb-99df-da46c59ee997\") " pod="openshift-image-registry/image-registry-79f6f8f56-gk2mv"
Sep 16 18:43:50.926753 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.926709    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fa83f1c9-b7b9-4c1e-b0c4-09328df33051-serving-cert\") pod \"service-ca-operator-cf7c776b7-5znf9\" (UID: \"fa83f1c9-b7b9-4c1e-b0c4-09328df33051\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5znf9"
Sep 16 18:43:50.926847 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.926782    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/40a3ba8a-40a8-48eb-99df-da46c59ee997-image-registry-private-configuration\") pod \"image-registry-79f6f8f56-gk2mv\" (UID: \"40a3ba8a-40a8-48eb-99df-da46c59ee997\") " pod="openshift-image-registry/image-registry-79f6f8f56-gk2mv"
Sep 16 18:43:50.926847 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.926809    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/40a3ba8a-40a8-48eb-99df-da46c59ee997-installation-pull-secrets\") pod \"image-registry-79f6f8f56-gk2mv\" (UID: \"40a3ba8a-40a8-48eb-99df-da46c59ee997\") " pod="openshift-image-registry/image-registry-79f6f8f56-gk2mv"
Sep 16 18:43:50.927381 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.927364    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 16 18:43:50.927653 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.927639    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 16 18:43:50.927710 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.927671    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 16 18:43:50.927791 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.927671    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 16 18:43:50.927919 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.927907    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-4q8jh\""
Sep 16 18:43:50.943713 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.943693    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fvqg\" (UniqueName: \"kubernetes.io/projected/fa83f1c9-b7b9-4c1e-b0c4-09328df33051-kube-api-access-7fvqg\") pod \"service-ca-operator-cf7c776b7-5znf9\" (UID: \"fa83f1c9-b7b9-4c1e-b0c4-09328df33051\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5znf9"
Sep 16 18:43:50.946793 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.946770    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q92ps\" (UniqueName: \"kubernetes.io/projected/fe6d9245-1609-46f1-9e0e-4fbdc31145d1-kube-api-access-q92ps\") pod \"cluster-samples-operator-86499c89c4-dvjf7\" (UID: \"fe6d9245-1609-46f1-9e0e-4fbdc31145d1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dvjf7"
Sep 16 18:43:50.951753 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.951713    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/40a3ba8a-40a8-48eb-99df-da46c59ee997-bound-sa-token\") pod \"image-registry-79f6f8f56-gk2mv\" (UID: \"40a3ba8a-40a8-48eb-99df-da46c59ee997\") " pod="openshift-image-registry/image-registry-79f6f8f56-gk2mv"
Sep 16 18:43:50.952619 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.952602    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-578df9684-v6zn8"]
Sep 16 18:43:50.952786 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.952770    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-674c5bb56f-6xn2v"
Sep 16 18:43:50.953851 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.953829    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcdvf\" (UniqueName: \"kubernetes.io/projected/40a3ba8a-40a8-48eb-99df-da46c59ee997-kube-api-access-vcdvf\") pod \"image-registry-79f6f8f56-gk2mv\" (UID: \"40a3ba8a-40a8-48eb-99df-da46c59ee997\") " pod="openshift-image-registry/image-registry-79f6f8f56-gk2mv"
Sep 16 18:43:50.959706 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.959680    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 16 18:43:50.959859 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.959843    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 16 18:43:50.960203 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.960189    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 16 18:43:50.961102 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.961087    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 16 18:43:50.975096 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.975076    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-czbqf"]
Sep 16 18:43:50.975225 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.975208    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578df9684-v6zn8"
Sep 16 18:43:50.993319 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.993297    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-x6kvz"]
Sep 16 18:43:50.993424 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.993395    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-czbqf"
Sep 16 18:43:50.993849 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.993828    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 16 18:43:50.996127 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.996100    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:43:50.996603 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.996589    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-c2ds5\""
Sep 16 18:43:50.996777 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:50.996765    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:43:51.009561 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.009535    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5znf9"
Sep 16 18:43:51.015357 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.015336    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-nccjc"]
Sep 16 18:43:51.015450 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.015371    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-578df9684-v6zn8"]
Sep 16 18:43:51.015450 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.015385    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-czbqf"]
Sep 16 18:43:51.015450 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.015401    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-x6kvz"]
Sep 16 18:43:51.015450 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.015412    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nw2m8"]
Sep 16 18:43:51.015450 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.015427    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-674c5bb56f-6xn2v"]
Sep 16 18:43:51.015450 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.015439    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f4d45b76-rpnnc"]
Sep 16 18:43:51.015450 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.015448    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wxss9"]
Sep 16 18:43:51.015692 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.015460    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7465b55b9d-njqmh"]
Sep 16 18:43:51.015692 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.015460    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-x6kvz"
Sep 16 18:43:51.015692 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.015475    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-cfz6r"]
Sep 16 18:43:51.023956 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.023909    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ff403ffb-51c2-45d7-b90f-cf1ac24054d6-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-674c5bb56f-6xn2v\" (UID: \"ff403ffb-51c2-45d7-b90f-cf1ac24054d6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-674c5bb56f-6xn2v"
Sep 16 18:43:51.023956 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.023942    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpzvf\" (UniqueName: \"kubernetes.io/projected/b6695f86-08c0-4836-87b1-f2a9d76084bd-kube-api-access-kpzvf\") pod \"console-operator-696884fb9c-7qwl2\" (UID: \"b6695f86-08c0-4836-87b1-f2a9d76084bd\") " pod="openshift-console-operator/console-operator-696884fb9c-7qwl2"
Sep 16 18:43:51.024081 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.023961    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9ac2b8ef-ff60-4791-9e56-5ac64263bf1e-default-certificate\") pod \"router-default-7465b55b9d-njqmh\" (UID: \"9ac2b8ef-ff60-4791-9e56-5ac64263bf1e\") " pod="openshift-ingress/router-default-7465b55b9d-njqmh"
Sep 16 18:43:51.024081 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.023986    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b6695f86-08c0-4836-87b1-f2a9d76084bd-config\") pod \"console-operator-696884fb9c-7qwl2\" (UID: \"b6695f86-08c0-4836-87b1-f2a9d76084bd\") " pod="openshift-console-operator/console-operator-696884fb9c-7qwl2"
Sep 16 18:43:51.024081 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.024011    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ba5d0219-c94f-41a7-a6fc-86b14d2bccd2-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-l56nd\" (UID: \"ba5d0219-c94f-41a7-a6fc-86b14d2bccd2\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-l56nd"
Sep 16 18:43:51.024210 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:51.024082    2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 18:43:51.024210 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:51.024127    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba5d0219-c94f-41a7-a6fc-86b14d2bccd2-networking-console-plugin-cert podName:ba5d0219-c94f-41a7-a6fc-86b14d2bccd2 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:51.524109524 +0000 UTC m=+35.165825166 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ba5d0219-c94f-41a7-a6fc-86b14d2bccd2-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-l56nd" (UID: "ba5d0219-c94f-41a7-a6fc-86b14d2bccd2") : secret "networking-console-plugin-cert" not found
Sep 16 18:43:51.024210 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.024149    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw5xq\" (UniqueName: \"kubernetes.io/projected/d76fc261-072a-4625-818d-e8faa5710cbd-kube-api-access-vw5xq\") pod \"dns-default-nw2m8\" (UID: \"d76fc261-072a-4625-818d-e8faa5710cbd\") " pod="openshift-dns/dns-default-nw2m8"
Sep 16 18:43:51.024210 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.024166    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ac2b8ef-ff60-4791-9e56-5ac64263bf1e-metrics-certs\") pod \"router-default-7465b55b9d-njqmh\" (UID: \"9ac2b8ef-ff60-4791-9e56-5ac64263bf1e\") " pod="openshift-ingress/router-default-7465b55b9d-njqmh"
Sep 16 18:43:51.024210 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.024187    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ff403ffb-51c2-45d7-b90f-cf1ac24054d6-hub\") pod \"cluster-proxy-proxy-agent-674c5bb56f-6xn2v\" (UID: \"ff403ffb-51c2-45d7-b90f-cf1ac24054d6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-674c5bb56f-6xn2v"
Sep 16 18:43:51.024431 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.024225    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ac2b8ef-ff60-4791-9e56-5ac64263bf1e-service-ca-bundle\") pod \"router-default-7465b55b9d-njqmh\" (UID: \"9ac2b8ef-ff60-4791-9e56-5ac64263bf1e\") " pod="openshift-ingress/router-default-7465b55b9d-njqmh"
Sep 16 18:43:51.024431 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.024247    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/031b5959-191e-47b0-93c6-85b69794a6ba-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-f4d45b76-rpnnc\" (UID: \"031b5959-191e-47b0-93c6-85b69794a6ba\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f4d45b76-rpnnc"
Sep 16 18:43:51.024431 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.024271    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed1b0dcc-4b23-4906-bc1d-4efa04e932cf-tmp\") pod \"insights-operator-5844b856db-h28sc\" (UID: \"ed1b0dcc-4b23-4906-bc1d-4efa04e932cf\") " pod="openshift-insights/insights-operator-5844b856db-h28sc"
Sep 16 18:43:51.024431 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.024317    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff403ffb-51c2-45d7-b90f-cf1ac24054d6-additional-service-ca\") pod \"cluster-proxy-proxy-agent-674c5bb56f-6xn2v\" (UID: \"ff403ffb-51c2-45d7-b90f-cf1ac24054d6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-674c5bb56f-6xn2v"
Sep 16 18:43:51.024431 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.024349    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d76fc261-072a-4625-818d-e8faa5710cbd-tmp-dir\") pod \"dns-default-nw2m8\" (UID: \"d76fc261-072a-4625-818d-e8faa5710cbd\") " pod="openshift-dns/dns-default-nw2m8"
Sep 16 18:43:51.024431 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.024377    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed1b0dcc-4b23-4906-bc1d-4efa04e932cf-trusted-ca-bundle\") pod \"insights-operator-5844b856db-h28sc\" (UID: \"ed1b0dcc-4b23-4906-bc1d-4efa04e932cf\") " pod="openshift-insights/insights-operator-5844b856db-h28sc"
Sep 16 18:43:51.024431 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.024392    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed1b0dcc-4b23-4906-bc1d-4efa04e932cf-service-ca-bundle\") pod \"insights-operator-5844b856db-h28sc\" (UID: \"ed1b0dcc-4b23-4906-bc1d-4efa04e932cf\") " pod="openshift-insights/insights-operator-5844b856db-h28sc"
Sep 16 18:43:51.024431 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.024414    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ff403ffb-51c2-45d7-b90f-cf1ac24054d6-ca\") pod \"cluster-proxy-proxy-agent-674c5bb56f-6xn2v\" (UID: \"ff403ffb-51c2-45d7-b90f-cf1ac24054d6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-674c5bb56f-6xn2v"
Sep 16 18:43:51.024843 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.024455    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6l4j\" (UniqueName: \"kubernetes.io/projected/ff403ffb-51c2-45d7-b90f-cf1ac24054d6-kube-api-access-z6l4j\") pod \"cluster-proxy-proxy-agent-674c5bb56f-6xn2v\" (UID: \"ff403ffb-51c2-45d7-b90f-cf1ac24054d6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-674c5bb56f-6xn2v"
Sep 16 18:43:51.024843 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.024521    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b6695f86-08c0-4836-87b1-f2a9d76084bd-config\") pod \"console-operator-696884fb9c-7qwl2\" (UID: \"b6695f86-08c0-4836-87b1-f2a9d76084bd\") " pod="openshift-console-operator/console-operator-696884fb9c-7qwl2"
Sep 16 18:43:51.024843 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.024577    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkqdz\" (UniqueName: \"kubernetes.io/projected/ed1b0dcc-4b23-4906-bc1d-4efa04e932cf-kube-api-access-hkqdz\") pod \"insights-operator-5844b856db-h28sc\" (UID: \"ed1b0dcc-4b23-4906-bc1d-4efa04e932cf\") " pod="openshift-insights/insights-operator-5844b856db-h28sc"
Sep 16 18:43:51.024843 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.024607    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed1b0dcc-4b23-4906-bc1d-4efa04e932cf-tmp\") pod \"insights-operator-5844b856db-h28sc\" (UID: \"ed1b0dcc-4b23-4906-bc1d-4efa04e932cf\") " pod="openshift-insights/insights-operator-5844b856db-h28sc"
Sep 16 18:43:51.024843 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.024644    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d76fc261-072a-4625-818d-e8faa5710cbd-tmp-dir\") pod \"dns-default-nw2m8\" (UID: \"d76fc261-072a-4625-818d-e8faa5710cbd\") " pod="openshift-dns/dns-default-nw2m8"
Sep 16 18:43:51.024843 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.024674    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b6695f86-08c0-4836-87b1-f2a9d76084bd-trusted-ca\") pod \"console-operator-696884fb9c-7qwl2\" (UID: \"b6695f86-08c0-4836-87b1-f2a9d76084bd\") " pod="openshift-console-operator/console-operator-696884fb9c-7qwl2"
Sep 16 18:43:51.024843 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.024706    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d76fc261-072a-4625-818d-e8faa5710cbd-metrics-tls\") pod \"dns-default-nw2m8\" (UID: \"d76fc261-072a-4625-818d-e8faa5710cbd\") " pod="openshift-dns/dns-default-nw2m8"
Sep 16 18:43:51.024843 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.024750    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ed1b0dcc-4b23-4906-bc1d-4efa04e932cf-snapshots\") pod \"insights-operator-5844b856db-h28sc\" (UID: \"ed1b0dcc-4b23-4906-bc1d-4efa04e932cf\") " pod="openshift-insights/insights-operator-5844b856db-h28sc"
Sep 16 18:43:51.024843 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.024778    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnf2f\" (UniqueName: \"kubernetes.io/projected/9ac2b8ef-ff60-4791-9e56-5ac64263bf1e-kube-api-access-qnf2f\") pod \"router-default-7465b55b9d-njqmh\" (UID: \"9ac2b8ef-ff60-4791-9e56-5ac64263bf1e\") " pod="openshift-ingress/router-default-7465b55b9d-njqmh"
Sep 16 18:43:51.024843 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.024810    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v4dt\" (UniqueName: \"kubernetes.io/projected/d5d54719-5c41-4b38-a2a1-367dc4bd6801-kube-api-access-6v4dt\") pod \"ingress-canary-wxss9\" (UID: \"d5d54719-5c41-4b38-a2a1-367dc4bd6801\") " pod="openshift-ingress-canary/ingress-canary-wxss9"
Sep 16 18:43:51.024843 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:51.024817    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:43:51.024843 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.024845    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqdkw\" (UniqueName: \"kubernetes.io/projected/031b5959-191e-47b0-93c6-85b69794a6ba-kube-api-access-sqdkw\") pod \"managed-serviceaccount-addon-agent-f4d45b76-rpnnc\" (UID: \"031b5959-191e-47b0-93c6-85b69794a6ba\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f4d45b76-rpnnc"
Sep 16 18:43:51.025422 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:51.024860    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d76fc261-072a-4625-818d-e8faa5710cbd-metrics-tls podName:d76fc261-072a-4625-818d-e8faa5710cbd nodeName:}" failed. No retries permitted until 2026-09-16 18:43:51.524848327 +0000 UTC m=+35.166563968 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d76fc261-072a-4625-818d-e8faa5710cbd-metrics-tls") pod "dns-default-nw2m8" (UID: "d76fc261-072a-4625-818d-e8faa5710cbd") : secret "dns-default-metrics-tls" not found
Sep 16 18:43:51.025422 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.024897    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgv5f\" (UniqueName: \"kubernetes.io/projected/c1f25728-6d57-4c30-9d73-d90c127bd06c-kube-api-access-rgv5f\") pod \"kube-storage-version-migrator-operator-6c8f84f897-nccjc\" (UID: \"c1f25728-6d57-4c30-9d73-d90c127bd06c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-nccjc"
Sep 16 18:43:51.025422 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.024925    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b6695f86-08c0-4836-87b1-f2a9d76084bd-serving-cert\") pod \"console-operator-696884fb9c-7qwl2\" (UID: \"b6695f86-08c0-4836-87b1-f2a9d76084bd\") " pod="openshift-console-operator/console-operator-696884fb9c-7qwl2"
Sep 16 18:43:51.025422 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.024936    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed1b0dcc-4b23-4906-bc1d-4efa04e932cf-service-ca-bundle\") pod \"insights-operator-5844b856db-h28sc\" (UID: \"ed1b0dcc-4b23-4906-bc1d-4efa04e932cf\") " pod="openshift-insights/insights-operator-5844b856db-h28sc"
Sep 16 18:43:51.025422 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.024955    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d76fc261-072a-4625-818d-e8faa5710cbd-config-volume\") pod \"dns-default-nw2m8\" (UID: \"d76fc261-072a-4625-818d-e8faa5710cbd\") " pod="openshift-dns/dns-default-nw2m8"
Sep 16 18:43:51.025422 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.024995    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c1f25728-6d57-4c30-9d73-d90c127bd06c-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-nccjc\" (UID: \"c1f25728-6d57-4c30-9d73-d90c127bd06c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-nccjc"
Sep 16 18:43:51.025422 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.025020    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9ac2b8ef-ff60-4791-9e56-5ac64263bf1e-stats-auth\") pod \"router-default-7465b55b9d-njqmh\" (UID: \"9ac2b8ef-ff60-4791-9e56-5ac64263bf1e\") " pod="openshift-ingress/router-default-7465b55b9d-njqmh"
Sep 16 18:43:51.025422 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.025062    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ed1b0dcc-4b23-4906-bc1d-4efa04e932cf-serving-cert\") pod \"insights-operator-5844b856db-h28sc\" (UID: \"ed1b0dcc-4b23-4906-bc1d-4efa04e932cf\") " pod="openshift-insights/insights-operator-5844b856db-h28sc"
Sep 16 18:43:51.025422 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.025094    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ba5d0219-c94f-41a7-a6fc-86b14d2bccd2-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-l56nd\" (UID: \"ba5d0219-c94f-41a7-a6fc-86b14d2bccd2\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-l56nd"
Sep 16 18:43:51.025422 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.025121    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ff403ffb-51c2-45d7-b90f-cf1ac24054d6-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-674c5bb56f-6xn2v\" (UID: \"ff403ffb-51c2-45d7-b90f-cf1ac24054d6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-674c5bb56f-6xn2v"
Sep 16 18:43:51.025422 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.025145    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5d54719-5c41-4b38-a2a1-367dc4bd6801-cert\") pod \"ingress-canary-wxss9\" (UID: \"d5d54719-5c41-4b38-a2a1-367dc4bd6801\") " pod="openshift-ingress-canary/ingress-canary-wxss9"
Sep 16 18:43:51.025422 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.025186    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c1f25728-6d57-4c30-9d73-d90c127bd06c-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-nccjc\" (UID: \"c1f25728-6d57-4c30-9d73-d90c127bd06c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-nccjc"
Sep 16 18:43:51.025422 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.025234    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed1b0dcc-4b23-4906-bc1d-4efa04e932cf-trusted-ca-bundle\") pod \"insights-operator-5844b856db-h28sc\" (UID: \"ed1b0dcc-4b23-4906-bc1d-4efa04e932cf\") " pod="openshift-insights/insights-operator-5844b856db-h28sc"
Sep 16 18:43:51.025422 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.025296    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ed1b0dcc-4b23-4906-bc1d-4efa04e932cf-snapshots\") pod \"insights-operator-5844b856db-h28sc\" (UID: \"ed1b0dcc-4b23-4906-bc1d-4efa04e932cf\") " pod="openshift-insights/insights-operator-5844b856db-h28sc"
Sep 16 18:43:51.026002 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.025497    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b6695f86-08c0-4836-87b1-f2a9d76084bd-trusted-ca\") pod \"console-operator-696884fb9c-7qwl2\" (UID: \"b6695f86-08c0-4836-87b1-f2a9d76084bd\") " pod="openshift-console-operator/console-operator-696884fb9c-7qwl2"
Sep 16 18:43:51.026002 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.025641    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d76fc261-072a-4625-818d-e8faa5710cbd-config-volume\") pod \"dns-default-nw2m8\" (UID: \"d76fc261-072a-4625-818d-e8faa5710cbd\") " pod="openshift-dns/dns-default-nw2m8"
Sep 16 18:43:51.026002 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.025701    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c1f25728-6d57-4c30-9d73-d90c127bd06c-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-nccjc\" (UID: \"c1f25728-6d57-4c30-9d73-d90c127bd06c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-nccjc"
Sep 16 18:43:51.026102 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.026061    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ba5d0219-c94f-41a7-a6fc-86b14d2bccd2-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-l56nd\" (UID: \"ba5d0219-c94f-41a7-a6fc-86b14d2bccd2\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-l56nd"
Sep 16 18:43:51.027519 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.027499    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ed1b0dcc-4b23-4906-bc1d-4efa04e932cf-serving-cert\") pod \"insights-operator-5844b856db-h28sc\" (UID: \"ed1b0dcc-4b23-4906-bc1d-4efa04e932cf\") " pod="openshift-insights/insights-operator-5844b856db-h28sc"
Sep 16 18:43:51.027632 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.027538    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b6695f86-08c0-4836-87b1-f2a9d76084bd-serving-cert\") pod \"console-operator-696884fb9c-7qwl2\" (UID: \"b6695f86-08c0-4836-87b1-f2a9d76084bd\") " pod="openshift-console-operator/console-operator-696884fb9c-7qwl2"
Sep 16 18:43:51.027793 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.027775    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c1f25728-6d57-4c30-9d73-d90c127bd06c-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-nccjc\" (UID: \"c1f25728-6d57-4c30-9d73-d90c127bd06c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-nccjc"
Sep 16 18:43:51.030398 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.030382    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 16 18:43:51.030533 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.030428    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 18:43:51.030644 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.030625    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 18:43:51.031947 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.031927    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-bw4mq\""
Sep 16 18:43:51.032106 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.031953    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 16 18:43:51.042489 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.042471    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-cfz6r"]
Sep 16 18:43:51.042597 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.042567    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-cfz6r"
Sep 16 18:43:51.044031 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.044007    2582 generic.go:356] "Generic (PLEG): container finished" podID="c7890694-9f4a-4bf0-9647-1d84e8f43321" containerID="bb4ac720c37f80a2d6605ebf2b07405e8eb624ea53801dfbc5ba9924ce776dc9" exitCode=0
Sep 16 18:43:51.044140 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.044046    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rkgjp" event={"ID":"c7890694-9f4a-4bf0-9647-1d84e8f43321","Type":"ContainerDied","Data":"bb4ac720c37f80a2d6605ebf2b07405e8eb624ea53801dfbc5ba9924ce776dc9"}
Sep 16 18:43:51.048143 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.048104    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 16 18:43:51.048621 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.048597    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-b6ft2\""
Sep 16 18:43:51.049542 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.049207    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:43:51.053753 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.052521    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw5xq\" (UniqueName: \"kubernetes.io/projected/d76fc261-072a-4625-818d-e8faa5710cbd-kube-api-access-vw5xq\") pod \"dns-default-nw2m8\" (UID: \"d76fc261-072a-4625-818d-e8faa5710cbd\") " pod="openshift-dns/dns-default-nw2m8"
Sep 16 18:43:51.054090 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.053902    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkqdz\" (UniqueName: \"kubernetes.io/projected/ed1b0dcc-4b23-4906-bc1d-4efa04e932cf-kube-api-access-hkqdz\") pod \"insights-operator-5844b856db-h28sc\" (UID: \"ed1b0dcc-4b23-4906-bc1d-4efa04e932cf\") " pod="openshift-insights/insights-operator-5844b856db-h28sc"
Sep 16 18:43:51.056799 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.056751    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpzvf\" (UniqueName: \"kubernetes.io/projected/b6695f86-08c0-4836-87b1-f2a9d76084bd-kube-api-access-kpzvf\") pod \"console-operator-696884fb9c-7qwl2\" (UID: \"b6695f86-08c0-4836-87b1-f2a9d76084bd\") " pod="openshift-console-operator/console-operator-696884fb9c-7qwl2"
Sep 16 18:43:51.062760 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.062740    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgv5f\" (UniqueName: \"kubernetes.io/projected/c1f25728-6d57-4c30-9d73-d90c127bd06c-kube-api-access-rgv5f\") pod \"kube-storage-version-migrator-operator-6c8f84f897-nccjc\" (UID: \"c1f25728-6d57-4c30-9d73-d90c127bd06c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-nccjc"
Sep 16 18:43:51.076008 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.075981    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-h28sc"
Sep 16 18:43:51.124314 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.124122    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-7qwl2"
Sep 16 18:43:51.126870 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.126266    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ff403ffb-51c2-45d7-b90f-cf1ac24054d6-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-674c5bb56f-6xn2v\" (UID: \"ff403ffb-51c2-45d7-b90f-cf1ac24054d6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-674c5bb56f-6xn2v"
Sep 16 18:43:51.126870 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.126295    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9ac2b8ef-ff60-4791-9e56-5ac64263bf1e-default-certificate\") pod \"router-default-7465b55b9d-njqmh\" (UID: \"9ac2b8ef-ff60-4791-9e56-5ac64263bf1e\") " pod="openshift-ingress/router-default-7465b55b9d-njqmh"
Sep 16 18:43:51.126870 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.126344    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ac2b8ef-ff60-4791-9e56-5ac64263bf1e-metrics-certs\") pod \"router-default-7465b55b9d-njqmh\" (UID: \"9ac2b8ef-ff60-4791-9e56-5ac64263bf1e\") " pod="openshift-ingress/router-default-7465b55b9d-njqmh"
Sep 16 18:43:51.126870 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.126363    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ff403ffb-51c2-45d7-b90f-cf1ac24054d6-hub\") pod \"cluster-proxy-proxy-agent-674c5bb56f-6xn2v\" (UID: \"ff403ffb-51c2-45d7-b90f-cf1ac24054d6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-674c5bb56f-6xn2v"
Sep 16 18:43:51.126870 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:51.126442    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:43:51.126870 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:51.126496    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ac2b8ef-ff60-4791-9e56-5ac64263bf1e-metrics-certs podName:9ac2b8ef-ff60-4791-9e56-5ac64263bf1e nodeName:}" failed. No retries permitted until 2026-09-16 18:43:51.626477555 +0000 UTC m=+35.268193201 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9ac2b8ef-ff60-4791-9e56-5ac64263bf1e-metrics-certs") pod "router-default-7465b55b9d-njqmh" (UID: "9ac2b8ef-ff60-4791-9e56-5ac64263bf1e") : secret "router-metrics-certs-default" not found
Sep 16 18:43:51.126870 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.126784    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e040ac61-5af8-4092-9a00-ec184466deaf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-x6kvz\" (UID: \"e040ac61-5af8-4092-9a00-ec184466deaf\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-x6kvz"
Sep 16 18:43:51.126870 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.126849    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ac2b8ef-ff60-4791-9e56-5ac64263bf1e-service-ca-bundle\") pod \"router-default-7465b55b9d-njqmh\" (UID: \"9ac2b8ef-ff60-4791-9e56-5ac64263bf1e\") " pod="openshift-ingress/router-default-7465b55b9d-njqmh"
Sep 16 18:43:51.127227 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.126898    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/031b5959-191e-47b0-93c6-85b69794a6ba-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-f4d45b76-rpnnc\" (UID: \"031b5959-191e-47b0-93c6-85b69794a6ba\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f4d45b76-rpnnc"
Sep 16 18:43:51.127227 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:51.127010    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9ac2b8ef-ff60-4791-9e56-5ac64263bf1e-service-ca-bundle podName:9ac2b8ef-ff60-4791-9e56-5ac64263bf1e nodeName:}" failed. No retries permitted until 2026-09-16 18:43:51.626993277 +0000 UTC m=+35.268708924 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9ac2b8ef-ff60-4791-9e56-5ac64263bf1e-service-ca-bundle") pod "router-default-7465b55b9d-njqmh" (UID: "9ac2b8ef-ff60-4791-9e56-5ac64263bf1e") : configmap references non-existent config key: service-ca.crt
Sep 16 18:43:51.127227 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.127034    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff403ffb-51c2-45d7-b90f-cf1ac24054d6-additional-service-ca\") pod \"cluster-proxy-proxy-agent-674c5bb56f-6xn2v\" (UID: \"ff403ffb-51c2-45d7-b90f-cf1ac24054d6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-674c5bb56f-6xn2v"
Sep 16 18:43:51.127227 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.127066    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckcnp\" (UniqueName: \"kubernetes.io/projected/e4d69278-94eb-4b53-8967-c787e4575efd-kube-api-access-ckcnp\") pod \"volume-data-source-validator-86695b7cc7-cfz6r\" (UID: \"e4d69278-94eb-4b53-8967-c787e4575efd\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-cfz6r"
Sep 16 18:43:51.127227 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.127097    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jr5l\" (UniqueName: \"kubernetes.io/projected/45d7cc49-fa21-4780-b9d5-a5d6a0c6b65d-kube-api-access-4jr5l\") pod \"klusterlet-addon-workmgr-578df9684-v6zn8\" (UID: \"45d7cc49-fa21-4780-b9d5-a5d6a0c6b65d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578df9684-v6zn8"
Sep 16 18:43:51.127227 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.127138    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ff403ffb-51c2-45d7-b90f-cf1ac24054d6-ca\") pod \"cluster-proxy-proxy-agent-674c5bb56f-6xn2v\" (UID: \"ff403ffb-51c2-45d7-b90f-cf1ac24054d6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-674c5bb56f-6xn2v"
Sep 16 18:43:51.127227 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.127162    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6l4j\" (UniqueName: \"kubernetes.io/projected/ff403ffb-51c2-45d7-b90f-cf1ac24054d6-kube-api-access-z6l4j\") pod \"cluster-proxy-proxy-agent-674c5bb56f-6xn2v\" (UID: \"ff403ffb-51c2-45d7-b90f-cf1ac24054d6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-674c5bb56f-6xn2v"
Sep 16 18:43:51.127227 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.127206    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/45d7cc49-fa21-4780-b9d5-a5d6a0c6b65d-klusterlet-config\") pod \"klusterlet-addon-workmgr-578df9684-v6zn8\" (UID: \"45d7cc49-fa21-4780-b9d5-a5d6a0c6b65d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578df9684-v6zn8"
Sep 16 18:43:51.127572 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.127235    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/45d7cc49-fa21-4780-b9d5-a5d6a0c6b65d-tmp\") pod \"klusterlet-addon-workmgr-578df9684-v6zn8\" (UID: \"45d7cc49-fa21-4780-b9d5-a5d6a0c6b65d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578df9684-v6zn8"
Sep 16 18:43:51.127572 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.127280    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnf2f\" (UniqueName: \"kubernetes.io/projected/9ac2b8ef-ff60-4791-9e56-5ac64263bf1e-kube-api-access-qnf2f\") pod \"router-default-7465b55b9d-njqmh\" (UID: \"9ac2b8ef-ff60-4791-9e56-5ac64263bf1e\") " pod="openshift-ingress/router-default-7465b55b9d-njqmh"
Sep 16 18:43:51.127572 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.127305    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e040ac61-5af8-4092-9a00-ec184466deaf-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-x6kvz\" (UID: \"e040ac61-5af8-4092-9a00-ec184466deaf\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-x6kvz"
Sep 16 18:43:51.127572 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.127343    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v4dt\" (UniqueName: \"kubernetes.io/projected/d5d54719-5c41-4b38-a2a1-367dc4bd6801-kube-api-access-6v4dt\") pod \"ingress-canary-wxss9\" (UID: \"d5d54719-5c41-4b38-a2a1-367dc4bd6801\") " pod="openshift-ingress-canary/ingress-canary-wxss9"
Sep 16 18:43:51.127572 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.127378    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqdkw\" (UniqueName: \"kubernetes.io/projected/031b5959-191e-47b0-93c6-85b69794a6ba-kube-api-access-sqdkw\") pod \"managed-serviceaccount-addon-agent-f4d45b76-rpnnc\" (UID: \"031b5959-191e-47b0-93c6-85b69794a6ba\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f4d45b76-rpnnc"
Sep 16 18:43:51.127572 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.127452    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9ac2b8ef-ff60-4791-9e56-5ac64263bf1e-stats-auth\") pod \"router-default-7465b55b9d-njqmh\" (UID: \"9ac2b8ef-ff60-4791-9e56-5ac64263bf1e\") " pod="openshift-ingress/router-default-7465b55b9d-njqmh"
Sep 16 18:43:51.127572 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.127483    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ff403ffb-51c2-45d7-b90f-cf1ac24054d6-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-674c5bb56f-6xn2v\" (UID: \"ff403ffb-51c2-45d7-b90f-cf1ac24054d6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-674c5bb56f-6xn2v"
Sep 16 18:43:51.127572 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.127524    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5d54719-5c41-4b38-a2a1-367dc4bd6801-cert\") pod \"ingress-canary-wxss9\" (UID: \"d5d54719-5c41-4b38-a2a1-367dc4bd6801\") " pod="openshift-ingress-canary/ingress-canary-wxss9"
Sep 16 18:43:51.127572 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.127550    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4627c\" (UniqueName: \"kubernetes.io/projected/e040ac61-5af8-4092-9a00-ec184466deaf-kube-api-access-4627c\") pod \"cluster-monitoring-operator-6cd954b897-x6kvz\" (UID: \"e040ac61-5af8-4092-9a00-ec184466deaf\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-x6kvz"
Sep 16 18:43:51.127999 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.127578    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87gsk\" (UniqueName: \"kubernetes.io/projected/18ccea05-a541-499e-abd4-68798bb34a45-kube-api-access-87gsk\") pod \"network-check-source-6f8b768455-czbqf\" (UID: \"18ccea05-a541-499e-abd4-68798bb34a45\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-czbqf"
Sep 16 18:43:51.128401 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.128158    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff403ffb-51c2-45d7-b90f-cf1ac24054d6-additional-service-ca\") pod \"cluster-proxy-proxy-agent-674c5bb56f-6xn2v\" (UID: \"ff403ffb-51c2-45d7-b90f-cf1ac24054d6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-674c5bb56f-6xn2v"
Sep 16 18:43:51.132531 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.132486    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/031b5959-191e-47b0-93c6-85b69794a6ba-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-f4d45b76-rpnnc\" (UID: \"031b5959-191e-47b0-93c6-85b69794a6ba\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f4d45b76-rpnnc"
Sep 16 18:43:51.132531 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.132465    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9ac2b8ef-ff60-4791-9e56-5ac64263bf1e-default-certificate\") pod \"router-default-7465b55b9d-njqmh\" (UID: \"9ac2b8ef-ff60-4791-9e56-5ac64263bf1e\") " pod="openshift-ingress/router-default-7465b55b9d-njqmh"
Sep 16 18:43:51.132686 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.132573    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ff403ffb-51c2-45d7-b90f-cf1ac24054d6-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-674c5bb56f-6xn2v\" (UID: \"ff403ffb-51c2-45d7-b90f-cf1ac24054d6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-674c5bb56f-6xn2v"
Sep 16 18:43:51.132686 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.132598    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5d54719-5c41-4b38-a2a1-367dc4bd6801-cert\") pod \"ingress-canary-wxss9\" (UID: \"d5d54719-5c41-4b38-a2a1-367dc4bd6801\") " pod="openshift-ingress-canary/ingress-canary-wxss9"
Sep 16 18:43:51.133021 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.132996    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ff403ffb-51c2-45d7-b90f-cf1ac24054d6-hub\") pod \"cluster-proxy-proxy-agent-674c5bb56f-6xn2v\" (UID: \"ff403ffb-51c2-45d7-b90f-cf1ac24054d6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-674c5bb56f-6xn2v"
Sep 16 18:43:51.133901 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.133878    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9ac2b8ef-ff60-4791-9e56-5ac64263bf1e-stats-auth\") pod \"router-default-7465b55b9d-njqmh\" (UID: \"9ac2b8ef-ff60-4791-9e56-5ac64263bf1e\") " pod="openshift-ingress/router-default-7465b55b9d-njqmh"
Sep 16 18:43:51.134241 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.134222    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ff403ffb-51c2-45d7-b90f-cf1ac24054d6-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-674c5bb56f-6xn2v\" (UID: \"ff403ffb-51c2-45d7-b90f-cf1ac24054d6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-674c5bb56f-6xn2v"
Sep 16 18:43:51.134496 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.134480    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ff403ffb-51c2-45d7-b90f-cf1ac24054d6-ca\") pod \"cluster-proxy-proxy-agent-674c5bb56f-6xn2v\" (UID: \"ff403ffb-51c2-45d7-b90f-cf1ac24054d6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-674c5bb56f-6xn2v"
Sep 16 18:43:51.144357 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.144240    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnf2f\" (UniqueName: \"kubernetes.io/projected/9ac2b8ef-ff60-4791-9e56-5ac64263bf1e-kube-api-access-qnf2f\") pod \"router-default-7465b55b9d-njqmh\" (UID: \"9ac2b8ef-ff60-4791-9e56-5ac64263bf1e\") " pod="openshift-ingress/router-default-7465b55b9d-njqmh"
Sep 16 18:43:51.144357 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.144540    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqdkw\" (UniqueName: \"kubernetes.io/projected/031b5959-191e-47b0-93c6-85b69794a6ba-kube-api-access-sqdkw\") pod \"managed-serviceaccount-addon-agent-f4d45b76-rpnnc\" (UID: \"031b5959-191e-47b0-93c6-85b69794a6ba\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f4d45b76-rpnnc"
Sep 16 18:43:51.145153 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.145135    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v4dt\" (UniqueName: \"kubernetes.io/projected/d5d54719-5c41-4b38-a2a1-367dc4bd6801-kube-api-access-6v4dt\") pod \"ingress-canary-wxss9\" (UID: \"d5d54719-5c41-4b38-a2a1-367dc4bd6801\") " pod="openshift-ingress-canary/ingress-canary-wxss9"
Sep 16 18:43:51.145304 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.145281    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6l4j\" (UniqueName: \"kubernetes.io/projected/ff403ffb-51c2-45d7-b90f-cf1ac24054d6-kube-api-access-z6l4j\") pod \"cluster-proxy-proxy-agent-674c5bb56f-6xn2v\" (UID: \"ff403ffb-51c2-45d7-b90f-cf1ac24054d6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-674c5bb56f-6xn2v"
Sep 16 18:43:51.164614 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.164172    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-nccjc"
Sep 16 18:43:51.179783 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.179753    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pr4kj_bfffe95c-37f4-4aab-a7aa-19c64f66e810/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:51.206641 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.206036    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wxss9"
Sep 16 18:43:51.229324 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.229297    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4627c\" (UniqueName: \"kubernetes.io/projected/e040ac61-5af8-4092-9a00-ec184466deaf-kube-api-access-4627c\") pod \"cluster-monitoring-operator-6cd954b897-x6kvz\" (UID: \"e040ac61-5af8-4092-9a00-ec184466deaf\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-x6kvz"
Sep 16 18:43:51.229411 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.229334    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87gsk\" (UniqueName: \"kubernetes.io/projected/18ccea05-a541-499e-abd4-68798bb34a45-kube-api-access-87gsk\") pod \"network-check-source-6f8b768455-czbqf\" (UID: \"18ccea05-a541-499e-abd4-68798bb34a45\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-czbqf"
Sep 16 18:43:51.229458 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.229414    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e040ac61-5af8-4092-9a00-ec184466deaf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-x6kvz\" (UID: \"e040ac61-5af8-4092-9a00-ec184466deaf\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-x6kvz"
Sep 16 18:43:51.229495 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.229472    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckcnp\" (UniqueName: \"kubernetes.io/projected/e4d69278-94eb-4b53-8967-c787e4575efd-kube-api-access-ckcnp\") pod \"volume-data-source-validator-86695b7cc7-cfz6r\" (UID: \"e4d69278-94eb-4b53-8967-c787e4575efd\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-cfz6r"
Sep 16 18:43:51.229549 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.229502    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jr5l\" (UniqueName: \"kubernetes.io/projected/45d7cc49-fa21-4780-b9d5-a5d6a0c6b65d-kube-api-access-4jr5l\") pod \"klusterlet-addon-workmgr-578df9684-v6zn8\" (UID: \"45d7cc49-fa21-4780-b9d5-a5d6a0c6b65d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578df9684-v6zn8"
Sep 16 18:43:51.229603 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.229550    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/45d7cc49-fa21-4780-b9d5-a5d6a0c6b65d-klusterlet-config\") pod \"klusterlet-addon-workmgr-578df9684-v6zn8\" (UID: \"45d7cc49-fa21-4780-b9d5-a5d6a0c6b65d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578df9684-v6zn8"
Sep 16 18:43:51.229603 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.229577    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/45d7cc49-fa21-4780-b9d5-a5d6a0c6b65d-tmp\") pod \"klusterlet-addon-workmgr-578df9684-v6zn8\" (UID: \"45d7cc49-fa21-4780-b9d5-a5d6a0c6b65d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578df9684-v6zn8"
Sep 16 18:43:51.229696 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.229618    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e040ac61-5af8-4092-9a00-ec184466deaf-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-x6kvz\" (UID: \"e040ac61-5af8-4092-9a00-ec184466deaf\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-x6kvz"
Sep 16 18:43:51.229771 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:51.229692    2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 18:43:51.229820 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:51.229781    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e040ac61-5af8-4092-9a00-ec184466deaf-cluster-monitoring-operator-tls podName:e040ac61-5af8-4092-9a00-ec184466deaf nodeName:}" failed. No retries permitted until 2026-09-16 18:43:51.729760107 +0000 UTC m=+35.371475762 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e040ac61-5af8-4092-9a00-ec184466deaf-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-x6kvz" (UID: "e040ac61-5af8-4092-9a00-ec184466deaf") : secret "cluster-monitoring-operator-tls" not found
Sep 16 18:43:51.230384 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.230361    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e040ac61-5af8-4092-9a00-ec184466deaf-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-x6kvz\" (UID: \"e040ac61-5af8-4092-9a00-ec184466deaf\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-x6kvz"
Sep 16 18:43:51.230593 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.230572    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/45d7cc49-fa21-4780-b9d5-a5d6a0c6b65d-tmp\") pod \"klusterlet-addon-workmgr-578df9684-v6zn8\" (UID: \"45d7cc49-fa21-4780-b9d5-a5d6a0c6b65d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578df9684-v6zn8"
Sep 16 18:43:51.233324 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.233288    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/45d7cc49-fa21-4780-b9d5-a5d6a0c6b65d-klusterlet-config\") pod \"klusterlet-addon-workmgr-578df9684-v6zn8\" (UID: \"45d7cc49-fa21-4780-b9d5-a5d6a0c6b65d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578df9684-v6zn8"
Sep 16 18:43:51.244010 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.243912    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckcnp\" (UniqueName: \"kubernetes.io/projected/e4d69278-94eb-4b53-8967-c787e4575efd-kube-api-access-ckcnp\") pod \"volume-data-source-validator-86695b7cc7-cfz6r\" (UID: \"e4d69278-94eb-4b53-8967-c787e4575efd\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-cfz6r"
Sep 16 18:43:51.244820 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.244788    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87gsk\" (UniqueName: \"kubernetes.io/projected/18ccea05-a541-499e-abd4-68798bb34a45-kube-api-access-87gsk\") pod \"network-check-source-6f8b768455-czbqf\" (UID: \"18ccea05-a541-499e-abd4-68798bb34a45\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-czbqf"
Sep 16 18:43:51.244970 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.244951    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jr5l\" (UniqueName: \"kubernetes.io/projected/45d7cc49-fa21-4780-b9d5-a5d6a0c6b65d-kube-api-access-4jr5l\") pod \"klusterlet-addon-workmgr-578df9684-v6zn8\" (UID: \"45d7cc49-fa21-4780-b9d5-a5d6a0c6b65d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578df9684-v6zn8"
Sep 16 18:43:51.246287 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.246258    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4627c\" (UniqueName: \"kubernetes.io/projected/e040ac61-5af8-4092-9a00-ec184466deaf-kube-api-access-4627c\") pod \"cluster-monitoring-operator-6cd954b897-x6kvz\" (UID: \"e040ac61-5af8-4092-9a00-ec184466deaf\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-x6kvz"
Sep 16 18:43:51.251307 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.251240    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f4d45b76-rpnnc"
Sep 16 18:43:51.266685 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.266637    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-674c5bb56f-6xn2v"
Sep 16 18:43:51.281000 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.280927    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pr4kj_bfffe95c-37f4-4aab-a7aa-19c64f66e810/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:51.297498 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.297450    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578df9684-v6zn8"
Sep 16 18:43:51.303605 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.303580    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-czbqf"
Sep 16 18:43:51.319954 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.319908    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-5znf9"]
Sep 16 18:43:51.327911 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:51.327876    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa83f1c9_b7b9_4c1e_b0c4_09328df33051.slice/crio-87567ce5710c94be0d975a9d699790c943297df9a237111b40a0ed58b2252498 WatchSource:0}: Error finding container 87567ce5710c94be0d975a9d699790c943297df9a237111b40a0ed58b2252498: Status 404 returned error can't find the container with id 87567ce5710c94be0d975a9d699790c943297df9a237111b40a0ed58b2252498
Sep 16 18:43:51.358311 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.358271    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-cfz6r"
Sep 16 18:43:51.382115 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.380609    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-h28sc"]
Sep 16 18:43:51.387999 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:51.387967    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded1b0dcc_4b23_4906_bc1d_4efa04e932cf.slice/crio-6bef060d919d70bb265d884e06346aec7649974f94f0930695caf6d4a7d10dc5 WatchSource:0}: Error finding container 6bef060d919d70bb265d884e06346aec7649974f94f0930695caf6d4a7d10dc5: Status 404 returned error can't find the container with id 6bef060d919d70bb265d884e06346aec7649974f94f0930695caf6d4a7d10dc5
Sep 16 18:43:51.424464 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.424436    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-7qwl2"]
Sep 16 18:43:51.427742 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:51.427706    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6695f86_08c0_4836_87b1_f2a9d76084bd.slice/crio-637104c262941b75221755f4dbcd41a293c21f781d46e0d7946676f63acc0ed8 WatchSource:0}: Error finding container 637104c262941b75221755f4dbcd41a293c21f781d46e0d7946676f63acc0ed8: Status 404 returned error can't find the container with id 637104c262941b75221755f4dbcd41a293c21f781d46e0d7946676f63acc0ed8
Sep 16 18:43:51.432614 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.432588    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40a3ba8a-40a8-48eb-99df-da46c59ee997-registry-tls\") pod \"image-registry-79f6f8f56-gk2mv\" (UID: \"40a3ba8a-40a8-48eb-99df-da46c59ee997\") " pod="openshift-image-registry/image-registry-79f6f8f56-gk2mv"
Sep 16 18:43:51.432780 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.432760    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fe6d9245-1609-46f1-9e0e-4fbdc31145d1-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-dvjf7\" (UID: \"fe6d9245-1609-46f1-9e0e-4fbdc31145d1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dvjf7"
Sep 16 18:43:51.432914 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:51.432816    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:43:51.432914 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:51.432836    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79f6f8f56-gk2mv: secret "image-registry-tls" not found
Sep 16 18:43:51.432914 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:51.432880    2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:43:51.432914 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:51.432899    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40a3ba8a-40a8-48eb-99df-da46c59ee997-registry-tls podName:40a3ba8a-40a8-48eb-99df-da46c59ee997 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:52.432880427 +0000 UTC m=+36.074596072 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/40a3ba8a-40a8-48eb-99df-da46c59ee997-registry-tls") pod "image-registry-79f6f8f56-gk2mv" (UID: "40a3ba8a-40a8-48eb-99df-da46c59ee997") : secret "image-registry-tls" not found
Sep 16 18:43:51.433052 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:51.432928    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe6d9245-1609-46f1-9e0e-4fbdc31145d1-samples-operator-tls podName:fe6d9245-1609-46f1-9e0e-4fbdc31145d1 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:52.432912874 +0000 UTC m=+36.074628522 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fe6d9245-1609-46f1-9e0e-4fbdc31145d1-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-dvjf7" (UID: "fe6d9245-1609-46f1-9e0e-4fbdc31145d1") : secret "samples-operator-tls" not found
Sep 16 18:43:51.466309 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.466285    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-nccjc"]
Sep 16 18:43:51.469495 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:51.469054    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1f25728_6d57_4c30_9d73_d90c127bd06c.slice/crio-2c43755608b8cbb2a1682f3ef153d42a0df4350d79b9915f79ea9ab0f8b59713 WatchSource:0}: Error finding container 2c43755608b8cbb2a1682f3ef153d42a0df4350d79b9915f79ea9ab0f8b59713: Status 404 returned error can't find the container with id 2c43755608b8cbb2a1682f3ef153d42a0df4350d79b9915f79ea9ab0f8b59713
Sep 16 18:43:51.492630 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.492603    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wxss9"]
Sep 16 18:43:51.495292 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:51.495268    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5d54719_5c41_4b38_a2a1_367dc4bd6801.slice/crio-ba31397305e8f03df798c2dbb9cfad52f9b5eb8a3fcdf040637c811b4d077399 WatchSource:0}: Error finding container ba31397305e8f03df798c2dbb9cfad52f9b5eb8a3fcdf040637c811b4d077399: Status 404 returned error can't find the container with id ba31397305e8f03df798c2dbb9cfad52f9b5eb8a3fcdf040637c811b4d077399
Sep 16 18:43:51.533366 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.533296    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ba5d0219-c94f-41a7-a6fc-86b14d2bccd2-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-l56nd\" (UID: \"ba5d0219-c94f-41a7-a6fc-86b14d2bccd2\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-l56nd"
Sep 16 18:43:51.533500 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.533429    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d76fc261-072a-4625-818d-e8faa5710cbd-metrics-tls\") pod \"dns-default-nw2m8\" (UID: \"d76fc261-072a-4625-818d-e8faa5710cbd\") " pod="openshift-dns/dns-default-nw2m8"
Sep 16 18:43:51.533500 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:51.533456    2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 18:43:51.533608 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:51.533520    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba5d0219-c94f-41a7-a6fc-86b14d2bccd2-networking-console-plugin-cert podName:ba5d0219-c94f-41a7-a6fc-86b14d2bccd2 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:52.533500878 +0000 UTC m=+36.175216534 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ba5d0219-c94f-41a7-a6fc-86b14d2bccd2-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-l56nd" (UID: "ba5d0219-c94f-41a7-a6fc-86b14d2bccd2") : secret "networking-console-plugin-cert" not found
Sep 16 18:43:51.533608 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:51.533564    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:43:51.533608 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:51.533603    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d76fc261-072a-4625-818d-e8faa5710cbd-metrics-tls podName:d76fc261-072a-4625-818d-e8faa5710cbd nodeName:}" failed. No retries permitted until 2026-09-16 18:43:52.533591484 +0000 UTC m=+36.175307139 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d76fc261-072a-4625-818d-e8faa5710cbd-metrics-tls") pod "dns-default-nw2m8" (UID: "d76fc261-072a-4625-818d-e8faa5710cbd") : secret "dns-default-metrics-tls" not found
Sep 16 18:43:51.534274 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.534252    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f4d45b76-rpnnc"]
Sep 16 18:43:51.537594 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:51.537574    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod031b5959_191e_47b0_93c6_85b69794a6ba.slice/crio-383882bfe68427a4491735ca35039312f5e65246bacfac76b3c3e712ecbb0576 WatchSource:0}: Error finding container 383882bfe68427a4491735ca35039312f5e65246bacfac76b3c3e712ecbb0576: Status 404 returned error can't find the container with id 383882bfe68427a4491735ca35039312f5e65246bacfac76b3c3e712ecbb0576
Sep 16 18:43:51.568680 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.568651    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-674c5bb56f-6xn2v"]
Sep 16 18:43:51.571565 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:51.571539    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff403ffb_51c2_45d7_b90f_cf1ac24054d6.slice/crio-b6c69aa5b5c705d081e8ba00c01b79143a758ca70121cede27bfd3f114e550be WatchSource:0}: Error finding container b6c69aa5b5c705d081e8ba00c01b79143a758ca70121cede27bfd3f114e550be: Status 404 returned error can't find the container with id b6c69aa5b5c705d081e8ba00c01b79143a758ca70121cede27bfd3f114e550be
Sep 16 18:43:51.607277 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.607118    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-578df9684-v6zn8"]
Sep 16 18:43:51.609629 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:51.609601    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45d7cc49_fa21_4780_b9d5_a5d6a0c6b65d.slice/crio-94ee9a79d81491c5135d9d9fbab4b2925da101c6432489b299b5de1fc18222d6 WatchSource:0}: Error finding container 94ee9a79d81491c5135d9d9fbab4b2925da101c6432489b299b5de1fc18222d6: Status 404 returned error can't find the container with id 94ee9a79d81491c5135d9d9fbab4b2925da101c6432489b299b5de1fc18222d6
Sep 16 18:43:51.612314 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.612232    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-czbqf"]
Sep 16 18:43:51.614460 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:51.614439    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18ccea05_a541_499e_abd4_68798bb34a45.slice/crio-25f0b7202085c39edbffb2d2ff32bdf69223bff06d054f4ae2d251058049f541 WatchSource:0}: Error finding container 25f0b7202085c39edbffb2d2ff32bdf69223bff06d054f4ae2d251058049f541: Status 404 returned error can't find the container with id 25f0b7202085c39edbffb2d2ff32bdf69223bff06d054f4ae2d251058049f541
Sep 16 18:43:51.634328 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.634304    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ac2b8ef-ff60-4791-9e56-5ac64263bf1e-metrics-certs\") pod \"router-default-7465b55b9d-njqmh\" (UID: \"9ac2b8ef-ff60-4791-9e56-5ac64263bf1e\") " pod="openshift-ingress/router-default-7465b55b9d-njqmh"
Sep 16 18:43:51.634454 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.634341    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ac2b8ef-ff60-4791-9e56-5ac64263bf1e-service-ca-bundle\") pod \"router-default-7465b55b9d-njqmh\" (UID: \"9ac2b8ef-ff60-4791-9e56-5ac64263bf1e\") " pod="openshift-ingress/router-default-7465b55b9d-njqmh"
Sep 16 18:43:51.634511 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:51.634460    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:43:51.634511 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:51.634488    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9ac2b8ef-ff60-4791-9e56-5ac64263bf1e-service-ca-bundle podName:9ac2b8ef-ff60-4791-9e56-5ac64263bf1e nodeName:}" failed. No retries permitted until 2026-09-16 18:43:52.634475944 +0000 UTC m=+36.276191586 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9ac2b8ef-ff60-4791-9e56-5ac64263bf1e-service-ca-bundle") pod "router-default-7465b55b9d-njqmh" (UID: "9ac2b8ef-ff60-4791-9e56-5ac64263bf1e") : configmap references non-existent config key: service-ca.crt
Sep 16 18:43:51.634606 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:51.634516    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ac2b8ef-ff60-4791-9e56-5ac64263bf1e-metrics-certs podName:9ac2b8ef-ff60-4791-9e56-5ac64263bf1e nodeName:}" failed. No retries permitted until 2026-09-16 18:43:52.6344986 +0000 UTC m=+36.276214255 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9ac2b8ef-ff60-4791-9e56-5ac64263bf1e-metrics-certs") pod "router-default-7465b55b9d-njqmh" (UID: "9ac2b8ef-ff60-4791-9e56-5ac64263bf1e") : secret "router-metrics-certs-default" not found
Sep 16 18:43:51.673179 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.673149    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-cfz6r"]
Sep 16 18:43:51.679875 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:43:51.679849    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4d69278_94eb_4b53_8967_c787e4575efd.slice/crio-43fafb8de4d003c2daba6465ff2935275e9fe521a3c0e4795d90191b50ddc1e7 WatchSource:0}: Error finding container 43fafb8de4d003c2daba6465ff2935275e9fe521a3c0e4795d90191b50ddc1e7: Status 404 returned error can't find the container with id 43fafb8de4d003c2daba6465ff2935275e9fe521a3c0e4795d90191b50ddc1e7
Sep 16 18:43:51.734846 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.734810    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e040ac61-5af8-4092-9a00-ec184466deaf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-x6kvz\" (UID: \"e040ac61-5af8-4092-9a00-ec184466deaf\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-x6kvz"
Sep 16 18:43:51.735030 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:51.735010    2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 18:43:51.735123 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:51.735110    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e040ac61-5af8-4092-9a00-ec184466deaf-cluster-monitoring-operator-tls podName:e040ac61-5af8-4092-9a00-ec184466deaf nodeName:}" failed. No retries permitted until 2026-09-16 18:43:52.735086898 +0000 UTC m=+36.376802555 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e040ac61-5af8-4092-9a00-ec184466deaf-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-x6kvz" (UID: "e040ac61-5af8-4092-9a00-ec184466deaf") : secret "cluster-monitoring-operator-tls" not found
Sep 16 18:43:51.866938 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.866865    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g7q9m"
Sep 16 18:43:51.867080 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.866866    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tplxv"
Sep 16 18:43:51.867080 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.866866    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgl4n"
Sep 16 18:43:51.869904 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.869884    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:43:51.870051 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.869936    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mz62r\""
Sep 16 18:43:51.870051 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.869949    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:43:51.870051 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:51.869936    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fq6l6\""
Sep 16 18:43:52.051964 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:52.051921    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-czbqf" event={"ID":"18ccea05-a541-499e-abd4-68798bb34a45","Type":"ContainerStarted","Data":"25f0b7202085c39edbffb2d2ff32bdf69223bff06d054f4ae2d251058049f541"}
Sep 16 18:43:52.053869 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:52.053836    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-674c5bb56f-6xn2v" event={"ID":"ff403ffb-51c2-45d7-b90f-cf1ac24054d6","Type":"ContainerStarted","Data":"b6c69aa5b5c705d081e8ba00c01b79143a758ca70121cede27bfd3f114e550be"}
Sep 16 18:43:52.055502 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:52.055420    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wxss9" event={"ID":"d5d54719-5c41-4b38-a2a1-367dc4bd6801","Type":"ContainerStarted","Data":"ba31397305e8f03df798c2dbb9cfad52f9b5eb8a3fcdf040637c811b4d077399"}
Sep 16 18:43:52.058192 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:52.058136    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-nccjc" event={"ID":"c1f25728-6d57-4c30-9d73-d90c127bd06c","Type":"ContainerStarted","Data":"2c43755608b8cbb2a1682f3ef153d42a0df4350d79b9915f79ea9ab0f8b59713"}
Sep 16 18:43:52.060093 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:52.060027    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-cfz6r" event={"ID":"e4d69278-94eb-4b53-8967-c787e4575efd","Type":"ContainerStarted","Data":"43fafb8de4d003c2daba6465ff2935275e9fe521a3c0e4795d90191b50ddc1e7"}
Sep 16 18:43:52.061575 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:52.061512    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578df9684-v6zn8" event={"ID":"45d7cc49-fa21-4780-b9d5-a5d6a0c6b65d","Type":"ContainerStarted","Data":"94ee9a79d81491c5135d9d9fbab4b2925da101c6432489b299b5de1fc18222d6"}
Sep 16 18:43:52.063616 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:52.063556    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-7qwl2" event={"ID":"b6695f86-08c0-4836-87b1-f2a9d76084bd","Type":"ContainerStarted","Data":"637104c262941b75221755f4dbcd41a293c21f781d46e0d7946676f63acc0ed8"}
Sep 16 18:43:52.065335 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:52.065269    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5znf9" event={"ID":"fa83f1c9-b7b9-4c1e-b0c4-09328df33051","Type":"ContainerStarted","Data":"87567ce5710c94be0d975a9d699790c943297df9a237111b40a0ed58b2252498"}
Sep 16 18:43:52.076963 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:52.076063    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rkgjp" event={"ID":"c7890694-9f4a-4bf0-9647-1d84e8f43321","Type":"ContainerStarted","Data":"7e427f4b13c19e8288471a792058053a7b15e57d6bc8145ee8ba0390d2541327"}
Sep 16 18:43:52.079506 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:52.079437    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f4d45b76-rpnnc" event={"ID":"031b5959-191e-47b0-93c6-85b69794a6ba","Type":"ContainerStarted","Data":"383882bfe68427a4491735ca35039312f5e65246bacfac76b3c3e712ecbb0576"}
Sep 16 18:43:52.083917 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:52.083888    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-h28sc" event={"ID":"ed1b0dcc-4b23-4906-bc1d-4efa04e932cf","Type":"ContainerStarted","Data":"6bef060d919d70bb265d884e06346aec7649974f94f0930695caf6d4a7d10dc5"}
Sep 16 18:43:52.100393 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:52.100221    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-rkgjp" podStartSLOduration=7.463828114 podStartE2EDuration="35.100201862s" totalImagesPullingTime="27.636373748s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:17 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:52.097905143 +0000 UTC m=+35.739620824" watchObservedRunningTime="2026-09-16 18:43:52.100201862 +0000 UTC m=+35.741917527"
Sep 16 18:43:52.228495 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:52.228290    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pr4kj_bfffe95c-37f4-4aab-a7aa-19c64f66e810/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:52.321457 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:52.321415    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pr4kj_bfffe95c-37f4-4aab-a7aa-19c64f66e810/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:52.442568 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:52.441636    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fe6d9245-1609-46f1-9e0e-4fbdc31145d1-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-dvjf7\" (UID: \"fe6d9245-1609-46f1-9e0e-4fbdc31145d1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dvjf7"
Sep 16 18:43:52.442568 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:52.441785    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40a3ba8a-40a8-48eb-99df-da46c59ee997-registry-tls\") pod \"image-registry-79f6f8f56-gk2mv\" (UID: \"40a3ba8a-40a8-48eb-99df-da46c59ee997\") " pod="openshift-image-registry/image-registry-79f6f8f56-gk2mv"
Sep 16 18:43:52.442568 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:52.442014    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:43:52.442568 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:52.442037    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79f6f8f56-gk2mv: secret "image-registry-tls" not found
Sep 16 18:43:52.442568 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:52.442096    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40a3ba8a-40a8-48eb-99df-da46c59ee997-registry-tls podName:40a3ba8a-40a8-48eb-99df-da46c59ee997 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:54.442077796 +0000 UTC m=+38.083793452 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/40a3ba8a-40a8-48eb-99df-da46c59ee997-registry-tls") pod "image-registry-79f6f8f56-gk2mv" (UID: "40a3ba8a-40a8-48eb-99df-da46c59ee997") : secret "image-registry-tls" not found
Sep 16 18:43:52.442568 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:52.442485    2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:43:52.442568 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:52.442529    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe6d9245-1609-46f1-9e0e-4fbdc31145d1-samples-operator-tls podName:fe6d9245-1609-46f1-9e0e-4fbdc31145d1 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:54.442514822 +0000 UTC m=+38.084230469 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fe6d9245-1609-46f1-9e0e-4fbdc31145d1-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-dvjf7" (UID: "fe6d9245-1609-46f1-9e0e-4fbdc31145d1") : secret "samples-operator-tls" not found
Sep 16 18:43:52.542965 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:52.542927    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d76fc261-072a-4625-818d-e8faa5710cbd-metrics-tls\") pod \"dns-default-nw2m8\" (UID: \"d76fc261-072a-4625-818d-e8faa5710cbd\") " pod="openshift-dns/dns-default-nw2m8"
Sep 16 18:43:52.543145 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:52.543042    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ba5d0219-c94f-41a7-a6fc-86b14d2bccd2-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-l56nd\" (UID: \"ba5d0219-c94f-41a7-a6fc-86b14d2bccd2\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-l56nd"
Sep 16 18:43:52.543237 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:52.543211    2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 18:43:52.543295 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:52.543277    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba5d0219-c94f-41a7-a6fc-86b14d2bccd2-networking-console-plugin-cert podName:ba5d0219-c94f-41a7-a6fc-86b14d2bccd2 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:54.543258093 +0000 UTC m=+38.184973758 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ba5d0219-c94f-41a7-a6fc-86b14d2bccd2-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-l56nd" (UID: "ba5d0219-c94f-41a7-a6fc-86b14d2bccd2") : secret "networking-console-plugin-cert" not found
Sep 16 18:43:52.543710 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:52.543683    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:43:52.543858 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:52.543749    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d76fc261-072a-4625-818d-e8faa5710cbd-metrics-tls podName:d76fc261-072a-4625-818d-e8faa5710cbd nodeName:}" failed. No retries permitted until 2026-09-16 18:43:54.543733105 +0000 UTC m=+38.185448761 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d76fc261-072a-4625-818d-e8faa5710cbd-metrics-tls") pod "dns-default-nw2m8" (UID: "d76fc261-072a-4625-818d-e8faa5710cbd") : secret "dns-default-metrics-tls" not found
Sep 16 18:43:52.644441 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:52.644405    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ac2b8ef-ff60-4791-9e56-5ac64263bf1e-metrics-certs\") pod \"router-default-7465b55b9d-njqmh\" (UID: \"9ac2b8ef-ff60-4791-9e56-5ac64263bf1e\") " pod="openshift-ingress/router-default-7465b55b9d-njqmh"
Sep 16 18:43:52.644623 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:52.644463    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ac2b8ef-ff60-4791-9e56-5ac64263bf1e-service-ca-bundle\") pod \"router-default-7465b55b9d-njqmh\" (UID: \"9ac2b8ef-ff60-4791-9e56-5ac64263bf1e\") " pod="openshift-ingress/router-default-7465b55b9d-njqmh"
Sep 16 18:43:52.644685 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:52.644636    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9ac2b8ef-ff60-4791-9e56-5ac64263bf1e-service-ca-bundle podName:9ac2b8ef-ff60-4791-9e56-5ac64263bf1e nodeName:}" failed. No retries permitted until 2026-09-16 18:43:54.644614473 +0000 UTC m=+38.286330139 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9ac2b8ef-ff60-4791-9e56-5ac64263bf1e-service-ca-bundle") pod "router-default-7465b55b9d-njqmh" (UID: "9ac2b8ef-ff60-4791-9e56-5ac64263bf1e") : configmap references non-existent config key: service-ca.crt
Sep 16 18:43:52.645081 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:52.645059    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:43:52.645185 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:52.645114    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ac2b8ef-ff60-4791-9e56-5ac64263bf1e-metrics-certs podName:9ac2b8ef-ff60-4791-9e56-5ac64263bf1e nodeName:}" failed. No retries permitted until 2026-09-16 18:43:54.645100762 +0000 UTC m=+38.286816407 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9ac2b8ef-ff60-4791-9e56-5ac64263bf1e-metrics-certs") pod "router-default-7465b55b9d-njqmh" (UID: "9ac2b8ef-ff60-4791-9e56-5ac64263bf1e") : secret "router-metrics-certs-default" not found
Sep 16 18:43:52.745989 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:52.745953    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e040ac61-5af8-4092-9a00-ec184466deaf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-x6kvz\" (UID: \"e040ac61-5af8-4092-9a00-ec184466deaf\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-x6kvz"
Sep 16 18:43:52.746179 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:52.746122    2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 18:43:52.746240 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:52.746201    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e040ac61-5af8-4092-9a00-ec184466deaf-cluster-monitoring-operator-tls podName:e040ac61-5af8-4092-9a00-ec184466deaf nodeName:}" failed. No retries permitted until 2026-09-16 18:43:54.746180302 +0000 UTC m=+38.387895961 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e040ac61-5af8-4092-9a00-ec184466deaf-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-x6kvz" (UID: "e040ac61-5af8-4092-9a00-ec184466deaf") : secret "cluster-monitoring-operator-tls" not found
Sep 16 18:43:54.470106 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:54.469018    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fe6d9245-1609-46f1-9e0e-4fbdc31145d1-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-dvjf7\" (UID: \"fe6d9245-1609-46f1-9e0e-4fbdc31145d1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dvjf7"
Sep 16 18:43:54.470106 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:54.469351    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40a3ba8a-40a8-48eb-99df-da46c59ee997-registry-tls\") pod \"image-registry-79f6f8f56-gk2mv\" (UID: \"40a3ba8a-40a8-48eb-99df-da46c59ee997\") " pod="openshift-image-registry/image-registry-79f6f8f56-gk2mv"
Sep 16 18:43:54.470106 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:54.469505    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:43:54.470106 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:54.469518    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79f6f8f56-gk2mv: secret "image-registry-tls" not found
Sep 16 18:43:54.470106 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:54.469571    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40a3ba8a-40a8-48eb-99df-da46c59ee997-registry-tls podName:40a3ba8a-40a8-48eb-99df-da46c59ee997 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:58.469554037 +0000 UTC m=+42.111269680 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/40a3ba8a-40a8-48eb-99df-da46c59ee997-registry-tls") pod "image-registry-79f6f8f56-gk2mv" (UID: "40a3ba8a-40a8-48eb-99df-da46c59ee997") : secret "image-registry-tls" not found
Sep 16 18:43:54.470106 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:54.469969    2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:43:54.470106 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:54.470017    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe6d9245-1609-46f1-9e0e-4fbdc31145d1-samples-operator-tls podName:fe6d9245-1609-46f1-9e0e-4fbdc31145d1 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:58.470001583 +0000 UTC m=+42.111717242 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fe6d9245-1609-46f1-9e0e-4fbdc31145d1-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-dvjf7" (UID: "fe6d9245-1609-46f1-9e0e-4fbdc31145d1") : secret "samples-operator-tls" not found
Sep 16 18:43:54.571608 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:54.570697    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d76fc261-072a-4625-818d-e8faa5710cbd-metrics-tls\") pod \"dns-default-nw2m8\" (UID: \"d76fc261-072a-4625-818d-e8faa5710cbd\") " pod="openshift-dns/dns-default-nw2m8"
Sep 16 18:43:54.571608 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:54.570858    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ba5d0219-c94f-41a7-a6fc-86b14d2bccd2-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-l56nd\" (UID: \"ba5d0219-c94f-41a7-a6fc-86b14d2bccd2\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-l56nd"
Sep 16 18:43:54.571608 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:54.571069    2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 18:43:54.571608 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:54.571135    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba5d0219-c94f-41a7-a6fc-86b14d2bccd2-networking-console-plugin-cert podName:ba5d0219-c94f-41a7-a6fc-86b14d2bccd2 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:58.571117815 +0000 UTC m=+42.212833460 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ba5d0219-c94f-41a7-a6fc-86b14d2bccd2-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-l56nd" (UID: "ba5d0219-c94f-41a7-a6fc-86b14d2bccd2") : secret "networking-console-plugin-cert" not found
Sep 16 18:43:54.571608 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:54.571524    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:43:54.571608 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:54.571569    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d76fc261-072a-4625-818d-e8faa5710cbd-metrics-tls podName:d76fc261-072a-4625-818d-e8faa5710cbd nodeName:}" failed. No retries permitted until 2026-09-16 18:43:58.571553936 +0000 UTC m=+42.213269582 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d76fc261-072a-4625-818d-e8faa5710cbd-metrics-tls") pod "dns-default-nw2m8" (UID: "d76fc261-072a-4625-818d-e8faa5710cbd") : secret "dns-default-metrics-tls" not found
Sep 16 18:43:54.672865 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:54.671966    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ac2b8ef-ff60-4791-9e56-5ac64263bf1e-service-ca-bundle\") pod \"router-default-7465b55b9d-njqmh\" (UID: \"9ac2b8ef-ff60-4791-9e56-5ac64263bf1e\") " pod="openshift-ingress/router-default-7465b55b9d-njqmh"
Sep 16 18:43:54.672865 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:54.672192    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ac2b8ef-ff60-4791-9e56-5ac64263bf1e-metrics-certs\") pod \"router-default-7465b55b9d-njqmh\" (UID: \"9ac2b8ef-ff60-4791-9e56-5ac64263bf1e\") " pod="openshift-ingress/router-default-7465b55b9d-njqmh"
Sep 16 18:43:54.672865 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:54.672319    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:43:54.672865 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:54.672378    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ac2b8ef-ff60-4791-9e56-5ac64263bf1e-metrics-certs podName:9ac2b8ef-ff60-4791-9e56-5ac64263bf1e nodeName:}" failed. No retries permitted until 2026-09-16 18:43:58.672360499 +0000 UTC m=+42.314076145 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9ac2b8ef-ff60-4791-9e56-5ac64263bf1e-metrics-certs") pod "router-default-7465b55b9d-njqmh" (UID: "9ac2b8ef-ff60-4791-9e56-5ac64263bf1e") : secret "router-metrics-certs-default" not found
Sep 16 18:43:54.672865 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:54.672785    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9ac2b8ef-ff60-4791-9e56-5ac64263bf1e-service-ca-bundle podName:9ac2b8ef-ff60-4791-9e56-5ac64263bf1e nodeName:}" failed. No retries permitted until 2026-09-16 18:43:58.672768023 +0000 UTC m=+42.314483679 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9ac2b8ef-ff60-4791-9e56-5ac64263bf1e-service-ca-bundle") pod "router-default-7465b55b9d-njqmh" (UID: "9ac2b8ef-ff60-4791-9e56-5ac64263bf1e") : configmap references non-existent config key: service-ca.crt
Sep 16 18:43:54.773512 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:54.772899    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e040ac61-5af8-4092-9a00-ec184466deaf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-x6kvz\" (UID: \"e040ac61-5af8-4092-9a00-ec184466deaf\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-x6kvz"
Sep 16 18:43:54.773512 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:54.773106    2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 18:43:54.773512 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:54.773171    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e040ac61-5af8-4092-9a00-ec184466deaf-cluster-monitoring-operator-tls podName:e040ac61-5af8-4092-9a00-ec184466deaf nodeName:}" failed. No retries permitted until 2026-09-16 18:43:58.773152214 +0000 UTC m=+42.414867869 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e040ac61-5af8-4092-9a00-ec184466deaf-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-x6kvz" (UID: "e040ac61-5af8-4092-9a00-ec184466deaf") : secret "cluster-monitoring-operator-tls" not found
Sep 16 18:43:55.791853 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:55.791810    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pr4kj_bfffe95c-37f4-4aab-a7aa-19c64f66e810/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:55.892339 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:55.892302    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pr4kj_bfffe95c-37f4-4aab-a7aa-19c64f66e810/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:56.152506 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:56.152443    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pr4kj_bfffe95c-37f4-4aab-a7aa-19c64f66e810/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:58.512063 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:58.512023    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fe6d9245-1609-46f1-9e0e-4fbdc31145d1-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-dvjf7\" (UID: \"fe6d9245-1609-46f1-9e0e-4fbdc31145d1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dvjf7"
Sep 16 18:43:58.512414 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:58.512179    2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:43:58.512414 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:58.512240    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe6d9245-1609-46f1-9e0e-4fbdc31145d1-samples-operator-tls podName:fe6d9245-1609-46f1-9e0e-4fbdc31145d1 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:06.512223908 +0000 UTC m=+50.153939550 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fe6d9245-1609-46f1-9e0e-4fbdc31145d1-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-dvjf7" (UID: "fe6d9245-1609-46f1-9e0e-4fbdc31145d1") : secret "samples-operator-tls" not found
Sep 16 18:43:58.512414 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:58.512180    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40a3ba8a-40a8-48eb-99df-da46c59ee997-registry-tls\") pod \"image-registry-79f6f8f56-gk2mv\" (UID: \"40a3ba8a-40a8-48eb-99df-da46c59ee997\") " pod="openshift-image-registry/image-registry-79f6f8f56-gk2mv"
Sep 16 18:43:58.512414 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:58.512246    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:43:58.512414 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:58.512350    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79f6f8f56-gk2mv: secret "image-registry-tls" not found
Sep 16 18:43:58.512414 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:58.512406    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40a3ba8a-40a8-48eb-99df-da46c59ee997-registry-tls podName:40a3ba8a-40a8-48eb-99df-da46c59ee997 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:06.512391655 +0000 UTC m=+50.154107314 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/40a3ba8a-40a8-48eb-99df-da46c59ee997-registry-tls") pod "image-registry-79f6f8f56-gk2mv" (UID: "40a3ba8a-40a8-48eb-99df-da46c59ee997") : secret "image-registry-tls" not found
Sep 16 18:43:58.613138 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:58.613096    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d76fc261-072a-4625-818d-e8faa5710cbd-metrics-tls\") pod \"dns-default-nw2m8\" (UID: \"d76fc261-072a-4625-818d-e8faa5710cbd\") " pod="openshift-dns/dns-default-nw2m8"
Sep 16 18:43:58.613319 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:58.613202    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ba5d0219-c94f-41a7-a6fc-86b14d2bccd2-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-l56nd\" (UID: \"ba5d0219-c94f-41a7-a6fc-86b14d2bccd2\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-l56nd"
Sep 16 18:43:58.613319 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:58.613257    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:43:58.613437 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:58.613326    2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 18:43:58.613437 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:58.613339    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d76fc261-072a-4625-818d-e8faa5710cbd-metrics-tls podName:d76fc261-072a-4625-818d-e8faa5710cbd nodeName:}" failed. No retries permitted until 2026-09-16 18:44:06.613318333 +0000 UTC m=+50.255033980 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d76fc261-072a-4625-818d-e8faa5710cbd-metrics-tls") pod "dns-default-nw2m8" (UID: "d76fc261-072a-4625-818d-e8faa5710cbd") : secret "dns-default-metrics-tls" not found
Sep 16 18:43:58.613437 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:58.613380    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba5d0219-c94f-41a7-a6fc-86b14d2bccd2-networking-console-plugin-cert podName:ba5d0219-c94f-41a7-a6fc-86b14d2bccd2 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:06.613363692 +0000 UTC m=+50.255079334 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ba5d0219-c94f-41a7-a6fc-86b14d2bccd2-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-l56nd" (UID: "ba5d0219-c94f-41a7-a6fc-86b14d2bccd2") : secret "networking-console-plugin-cert" not found
Sep 16 18:43:58.714212 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:58.714161    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ac2b8ef-ff60-4791-9e56-5ac64263bf1e-metrics-certs\") pod \"router-default-7465b55b9d-njqmh\" (UID: \"9ac2b8ef-ff60-4791-9e56-5ac64263bf1e\") " pod="openshift-ingress/router-default-7465b55b9d-njqmh"
Sep 16 18:43:58.714393 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:58.714232    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ac2b8ef-ff60-4791-9e56-5ac64263bf1e-service-ca-bundle\") pod \"router-default-7465b55b9d-njqmh\" (UID: \"9ac2b8ef-ff60-4791-9e56-5ac64263bf1e\") " pod="openshift-ingress/router-default-7465b55b9d-njqmh"
Sep 16 18:43:58.714393 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:58.714327    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:43:58.714508 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:58.714408    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ac2b8ef-ff60-4791-9e56-5ac64263bf1e-metrics-certs podName:9ac2b8ef-ff60-4791-9e56-5ac64263bf1e nodeName:}" failed. No retries permitted until 2026-09-16 18:44:06.714388199 +0000 UTC m=+50.356103858 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9ac2b8ef-ff60-4791-9e56-5ac64263bf1e-metrics-certs") pod "router-default-7465b55b9d-njqmh" (UID: "9ac2b8ef-ff60-4791-9e56-5ac64263bf1e") : secret "router-metrics-certs-default" not found
Sep 16 18:43:58.714508 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:58.714432    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9ac2b8ef-ff60-4791-9e56-5ac64263bf1e-service-ca-bundle podName:9ac2b8ef-ff60-4791-9e56-5ac64263bf1e nodeName:}" failed. No retries permitted until 2026-09-16 18:44:06.714418641 +0000 UTC m=+50.356134284 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9ac2b8ef-ff60-4791-9e56-5ac64263bf1e-service-ca-bundle") pod "router-default-7465b55b9d-njqmh" (UID: "9ac2b8ef-ff60-4791-9e56-5ac64263bf1e") : configmap references non-existent config key: service-ca.crt
Sep 16 18:43:58.815709 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:43:58.815619    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e040ac61-5af8-4092-9a00-ec184466deaf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-x6kvz\" (UID: \"e040ac61-5af8-4092-9a00-ec184466deaf\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-x6kvz"
Sep 16 18:43:58.815884 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:58.815786    2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 18:43:58.815884 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:43:58.815850    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e040ac61-5af8-4092-9a00-ec184466deaf-cluster-monitoring-operator-tls podName:e040ac61-5af8-4092-9a00-ec184466deaf nodeName:}" failed. No retries permitted until 2026-09-16 18:44:06.81583423 +0000 UTC m=+50.457549872 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e040ac61-5af8-4092-9a00-ec184466deaf-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-x6kvz" (UID: "e040ac61-5af8-4092-9a00-ec184466deaf") : secret "cluster-monitoring-operator-tls" not found
Sep 16 18:44:06.586625 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:06.586501    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fe6d9245-1609-46f1-9e0e-4fbdc31145d1-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-dvjf7\" (UID: \"fe6d9245-1609-46f1-9e0e-4fbdc31145d1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dvjf7"
Sep 16 18:44:06.586625 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:06.586626    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40a3ba8a-40a8-48eb-99df-da46c59ee997-registry-tls\") pod \"image-registry-79f6f8f56-gk2mv\" (UID: \"40a3ba8a-40a8-48eb-99df-da46c59ee997\") " pod="openshift-image-registry/image-registry-79f6f8f56-gk2mv"
Sep 16 18:44:06.587149 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:44:06.586819    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:44:06.587149 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:44:06.586834    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79f6f8f56-gk2mv: secret "image-registry-tls" not found
Sep 16 18:44:06.587149 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:44:06.586891    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40a3ba8a-40a8-48eb-99df-da46c59ee997-registry-tls podName:40a3ba8a-40a8-48eb-99df-da46c59ee997 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:22.586871243 +0000 UTC m=+66.228586892 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/40a3ba8a-40a8-48eb-99df-da46c59ee997-registry-tls") pod "image-registry-79f6f8f56-gk2mv" (UID: "40a3ba8a-40a8-48eb-99df-da46c59ee997") : secret "image-registry-tls" not found
Sep 16 18:44:06.587768 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:44:06.587302    2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:44:06.587768 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:44:06.587358    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe6d9245-1609-46f1-9e0e-4fbdc31145d1-samples-operator-tls podName:fe6d9245-1609-46f1-9e0e-4fbdc31145d1 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:22.58734223 +0000 UTC m=+66.229057879 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fe6d9245-1609-46f1-9e0e-4fbdc31145d1-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-dvjf7" (UID: "fe6d9245-1609-46f1-9e0e-4fbdc31145d1") : secret "samples-operator-tls" not found
Sep 16 18:44:06.687160 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:06.687122    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ba5d0219-c94f-41a7-a6fc-86b14d2bccd2-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-l56nd\" (UID: \"ba5d0219-c94f-41a7-a6fc-86b14d2bccd2\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-l56nd"
Sep 16 18:44:06.687886 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:06.687258    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d76fc261-072a-4625-818d-e8faa5710cbd-metrics-tls\") pod \"dns-default-nw2m8\" (UID: \"d76fc261-072a-4625-818d-e8faa5710cbd\") " pod="openshift-dns/dns-default-nw2m8"
Sep 16 18:44:06.687886 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:44:06.687414    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:44:06.687886 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:44:06.687477    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d76fc261-072a-4625-818d-e8faa5710cbd-metrics-tls podName:d76fc261-072a-4625-818d-e8faa5710cbd nodeName:}" failed. No retries permitted until 2026-09-16 18:44:22.687457926 +0000 UTC m=+66.329173572 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d76fc261-072a-4625-818d-e8faa5710cbd-metrics-tls") pod "dns-default-nw2m8" (UID: "d76fc261-072a-4625-818d-e8faa5710cbd") : secret "dns-default-metrics-tls" not found
Sep 16 18:44:06.687886 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:44:06.687414    2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 18:44:06.687886 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:44:06.687859    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba5d0219-c94f-41a7-a6fc-86b14d2bccd2-networking-console-plugin-cert podName:ba5d0219-c94f-41a7-a6fc-86b14d2bccd2 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:22.687843501 +0000 UTC m=+66.329559150 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ba5d0219-c94f-41a7-a6fc-86b14d2bccd2-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-l56nd" (UID: "ba5d0219-c94f-41a7-a6fc-86b14d2bccd2") : secret "networking-console-plugin-cert" not found
Sep 16 18:44:06.788665 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:06.788024    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ac2b8ef-ff60-4791-9e56-5ac64263bf1e-metrics-certs\") pod \"router-default-7465b55b9d-njqmh\" (UID: \"9ac2b8ef-ff60-4791-9e56-5ac64263bf1e\") " pod="openshift-ingress/router-default-7465b55b9d-njqmh"
Sep 16 18:44:06.788665 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:06.788081    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ac2b8ef-ff60-4791-9e56-5ac64263bf1e-service-ca-bundle\") pod \"router-default-7465b55b9d-njqmh\" (UID: \"9ac2b8ef-ff60-4791-9e56-5ac64263bf1e\") " pod="openshift-ingress/router-default-7465b55b9d-njqmh"
Sep 16 18:44:06.788665 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:44:06.788221    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9ac2b8ef-ff60-4791-9e56-5ac64263bf1e-service-ca-bundle podName:9ac2b8ef-ff60-4791-9e56-5ac64263bf1e nodeName:}" failed. No retries permitted until 2026-09-16 18:44:22.788194405 +0000 UTC m=+66.429910052 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9ac2b8ef-ff60-4791-9e56-5ac64263bf1e-service-ca-bundle") pod "router-default-7465b55b9d-njqmh" (UID: "9ac2b8ef-ff60-4791-9e56-5ac64263bf1e") : configmap references non-existent config key: service-ca.crt
Sep 16 18:44:06.788665 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:44:06.788598    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:44:06.788665 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:44:06.788642    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ac2b8ef-ff60-4791-9e56-5ac64263bf1e-metrics-certs podName:9ac2b8ef-ff60-4791-9e56-5ac64263bf1e nodeName:}" failed. No retries permitted until 2026-09-16 18:44:22.788626781 +0000 UTC m=+66.430342431 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9ac2b8ef-ff60-4791-9e56-5ac64263bf1e-metrics-certs") pod "router-default-7465b55b9d-njqmh" (UID: "9ac2b8ef-ff60-4791-9e56-5ac64263bf1e") : secret "router-metrics-certs-default" not found
Sep 16 18:44:06.889007 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:06.888970    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e040ac61-5af8-4092-9a00-ec184466deaf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-x6kvz\" (UID: \"e040ac61-5af8-4092-9a00-ec184466deaf\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-x6kvz"
Sep 16 18:44:06.889130 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:44:06.889093    2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 18:44:06.889186 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:44:06.889149    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e040ac61-5af8-4092-9a00-ec184466deaf-cluster-monitoring-operator-tls podName:e040ac61-5af8-4092-9a00-ec184466deaf nodeName:}" failed. No retries permitted until 2026-09-16 18:44:22.88913109 +0000 UTC m=+66.530846740 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e040ac61-5af8-4092-9a00-ec184466deaf-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-x6kvz" (UID: "e040ac61-5af8-4092-9a00-ec184466deaf") : secret "cluster-monitoring-operator-tls" not found
Sep 16 18:44:07.154038 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:07.153992    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f4d45b76-rpnnc" event={"ID":"031b5959-191e-47b0-93c6-85b69794a6ba","Type":"ContainerStarted","Data":"fc55ab9d30e15a9e84c8b2a2cc4e87cc4ff62d10f930d803358994017259fab4"}
Sep 16 18:44:07.155895 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:07.155865    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-h28sc" event={"ID":"ed1b0dcc-4b23-4906-bc1d-4efa04e932cf","Type":"ContainerStarted","Data":"88023fa3e8dafbd7a0671d1961781602605d104f5e9e4b2e959b7c1223409fc7"}
Sep 16 18:44:07.157541 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:07.157522    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-czbqf" event={"ID":"18ccea05-a541-499e-abd4-68798bb34a45","Type":"ContainerStarted","Data":"d9f868353723cc1fdf0834ce936aadb8ac08287ae88a72f3ae55e1485c65d9d6"}
Sep 16 18:44:07.159923 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:07.159894    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-674c5bb56f-6xn2v" event={"ID":"ff403ffb-51c2-45d7-b90f-cf1ac24054d6","Type":"ContainerStarted","Data":"9a847b6bc74893a5654413b8273df6265a27009fcc780be4b349f348f789d83e"}
Sep 16 18:44:07.160056 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:07.159931    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-674c5bb56f-6xn2v" event={"ID":"ff403ffb-51c2-45d7-b90f-cf1ac24054d6","Type":"ContainerStarted","Data":"4bfda70523c9f5a6fd03594444010f9afe3cceac94c88f8c3859c271381dc974"}
Sep 16 18:44:07.160056 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:07.159947    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-674c5bb56f-6xn2v" event={"ID":"ff403ffb-51c2-45d7-b90f-cf1ac24054d6","Type":"ContainerStarted","Data":"53f16afaeb1df510b92522853fc974357ba6cfc483e56c67b9e8df17e291a938"}
Sep 16 18:44:07.161470 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:07.161447    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wxss9" event={"ID":"d5d54719-5c41-4b38-a2a1-367dc4bd6801","Type":"ContainerStarted","Data":"e64ee58a6a8e07ecf828bb6d056825e5de0e4abad424ae0a22748a3359690156"}
Sep 16 18:44:07.163242 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:07.163218    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-nccjc" event={"ID":"c1f25728-6d57-4c30-9d73-d90c127bd06c","Type":"ContainerStarted","Data":"4e666d7b7e14afffa9877f121d34c82aa66e3d344363b4459faa219fa9984b75"}
Sep 16 18:44:07.165263 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:07.165240    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-cfz6r" event={"ID":"e4d69278-94eb-4b53-8967-c787e4575efd","Type":"ContainerStarted","Data":"76ab1abede017c4e90029d22bb3399bee8dd1caa46884f4033f0963bfcb6ffbc"}
Sep 16 18:44:07.166990 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:07.166969    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578df9684-v6zn8" event={"ID":"45d7cc49-fa21-4780-b9d5-a5d6a0c6b65d","Type":"ContainerStarted","Data":"293146b832b019c702e49a45d459023fa380b86bd1dbdefb970f60be2e8982d1"}
Sep 16 18:44:07.167176 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:07.167136    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578df9684-v6zn8"
Sep 16 18:44:07.168638 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:07.168598    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578df9684-v6zn8"
Sep 16 18:44:07.171646 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:07.171594    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f4d45b76-rpnnc" podStartSLOduration=2.461230596 podStartE2EDuration="17.171544254s" totalImagesPullingTime="14.710313658s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:43:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:07.169113633 +0000 UTC m=+50.810829299" watchObservedRunningTime="2026-09-16 18:44:07.171544254 +0000 UTC m=+50.813259913"
Sep 16 18:44:07.176003 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:07.175985    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7qwl2_b6695f86-08c0-4836-87b1-f2a9d76084bd/console-operator/0.log"
Sep 16 18:44:07.176084 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:07.176019    2582 generic.go:356] "Generic (PLEG): container finished" podID="b6695f86-08c0-4836-87b1-f2a9d76084bd" containerID="f31485cee8a1a7f476d7be2e1d3aa9d4f880a07012a53e4f33d11fce059dc20b" exitCode=255
Sep 16 18:44:07.176084 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:07.176078    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-7qwl2" event={"ID":"b6695f86-08c0-4836-87b1-f2a9d76084bd","Type":"ContainerDied","Data":"f31485cee8a1a7f476d7be2e1d3aa9d4f880a07012a53e4f33d11fce059dc20b"}
Sep 16 18:44:07.176337 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:07.176318    2582 scope.go:117] "RemoveContainer" containerID="f31485cee8a1a7f476d7be2e1d3aa9d4f880a07012a53e4f33d11fce059dc20b"
Sep 16 18:44:07.177819 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:07.177797    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5znf9" event={"ID":"fa83f1c9-b7b9-4c1e-b0c4-09328df33051","Type":"ContainerStarted","Data":"3c3222678135bbb17bbd1315af43eccb59ad34728c2698c1595f372509eda7a1"}
Sep 16 18:44:07.186560 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:07.186515    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-nccjc" podStartSLOduration=27.490618389 podStartE2EDuration="42.186502736s" totalImagesPullingTime="14.695884347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:43:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:07.186267632 +0000 UTC m=+50.827983299" watchObservedRunningTime="2026-09-16 18:44:07.186502736 +0000 UTC m=+50.828218400"
Sep 16 18:44:07.208962 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:07.208356    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-674c5bb56f-6xn2v" podStartSLOduration=2.514251971 podStartE2EDuration="17.2083411s" totalImagesPullingTime="14.694089129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:43:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:07.207993427 +0000 UTC m=+50.849709103" watchObservedRunningTime="2026-09-16 18:44:07.2083411 +0000 UTC m=+50.850056764"
Sep 16 18:44:07.224837 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:07.224797    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-wxss9" podStartSLOduration=2.472405679 podStartE2EDuration="17.224780406s" totalImagesPullingTime="14.752374727s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:43:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:07.223646097 +0000 UTC m=+50.865361766" watchObservedRunningTime="2026-09-16 18:44:07.224780406 +0000 UTC m=+50.866496068"
Sep 16 18:44:07.240611 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:07.240556    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-czbqf" podStartSLOduration=27.606487929 podStartE2EDuration="42.240534618s" totalImagesPullingTime="14.634046689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:43:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:07.239274058 +0000 UTC m=+50.880989723" watchObservedRunningTime="2026-09-16 18:44:07.240534618 +0000 UTC m=+50.882250283"
Sep 16 18:44:07.271631 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:07.271221    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-cfz6r" podStartSLOduration=29.553836015 podStartE2EDuration="42.271196872s" totalImagesPullingTime="12.717360857s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:43:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:07.257014509 +0000 UTC m=+50.898730174" watchObservedRunningTime="2026-09-16 18:44:07.271196872 +0000 UTC m=+50.912912537"
Sep 16 18:44:07.311176 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:07.310192    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-h28sc" podStartSLOduration=28.123895704 podStartE2EDuration="42.310172587s" totalImagesPullingTime="14.186276883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:43:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:07.309796994 +0000 UTC m=+50.951512659" watchObservedRunningTime="2026-09-16 18:44:07.310172587 +0000 UTC m=+50.951888253"
Sep 16 18:44:07.311176 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:07.310939    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578df9684-v6zn8" podStartSLOduration=2.673247679 podStartE2EDuration="17.310927865s" totalImagesPullingTime="14.637680186s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:43:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:07.288282652 +0000 UTC m=+50.929998317" watchObservedRunningTime="2026-09-16 18:44:07.310927865 +0000 UTC m=+50.952643534"
Sep 16 18:44:07.352005 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:07.351858    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5znf9" podStartSLOduration=27.433057577 podStartE2EDuration="42.351834986s" totalImagesPullingTime="14.918777409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:43:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:07.350697704 +0000 UTC m=+50.992413369" watchObservedRunningTime="2026-09-16 18:44:07.351834986 +0000 UTC m=+50.993550651"
Sep 16 18:44:07.981840 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:07.981807    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-627pg"]
Sep 16 18:44:07.985167 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:07.985148    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-627pg"
Sep 16 18:44:07.987936 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:07.987896    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 18:44:07.988278 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:07.988255    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6r5qx\""
Sep 16 18:44:07.988373 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:07.988255    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 18:44:07.997936 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:07.997865    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-627pg"]
Sep 16 18:44:08.003651 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:08.003291    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbwxf\" (UniqueName: \"kubernetes.io/projected/e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339-kube-api-access-mbwxf\") pod \"insights-runtime-extractor-627pg\" (UID: \"e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339\") " pod="openshift-insights/insights-runtime-extractor-627pg"
Sep 16 18:44:08.003651 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:08.003377    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339-data-volume\") pod \"insights-runtime-extractor-627pg\" (UID: \"e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339\") " pod="openshift-insights/insights-runtime-extractor-627pg"
Sep 16 18:44:08.003651 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:08.003416    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-627pg\" (UID: \"e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339\") " pod="openshift-insights/insights-runtime-extractor-627pg"
Sep 16 18:44:08.003651 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:08.003523    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339-crio-socket\") pod \"insights-runtime-extractor-627pg\" (UID: \"e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339\") " pod="openshift-insights/insights-runtime-extractor-627pg"
Sep 16 18:44:08.003651 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:08.003547    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-627pg\" (UID: \"e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339\") " pod="openshift-insights/insights-runtime-extractor-627pg"
Sep 16 18:44:08.104246 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:08.104210    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339-crio-socket\") pod \"insights-runtime-extractor-627pg\" (UID: \"e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339\") " pod="openshift-insights/insights-runtime-extractor-627pg"
Sep 16 18:44:08.104246 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:08.104248    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-627pg\" (UID: \"e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339\") " pod="openshift-insights/insights-runtime-extractor-627pg"
Sep 16 18:44:08.104440 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:44:08.104356    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:44:08.104440 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:08.104408    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbwxf\" (UniqueName: \"kubernetes.io/projected/e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339-kube-api-access-mbwxf\") pod \"insights-runtime-extractor-627pg\" (UID: \"e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339\") " pod="openshift-insights/insights-runtime-extractor-627pg"
Sep 16 18:44:08.104532 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:08.104448    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339-crio-socket\") pod \"insights-runtime-extractor-627pg\" (UID: \"e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339\") " pod="openshift-insights/insights-runtime-extractor-627pg"
Sep 16 18:44:08.104532 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:44:08.104494    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339-insights-runtime-extractor-tls podName:e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:08.604472957 +0000 UTC m=+52.246188604 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339-insights-runtime-extractor-tls") pod "insights-runtime-extractor-627pg" (UID: "e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:44:08.104631 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:08.104565    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339-data-volume\") pod \"insights-runtime-extractor-627pg\" (UID: \"e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339\") " pod="openshift-insights/insights-runtime-extractor-627pg"
Sep 16 18:44:08.104631 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:08.104605    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-627pg\" (UID: \"e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339\") " pod="openshift-insights/insights-runtime-extractor-627pg"
Sep 16 18:44:08.104792 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:44:08.104771    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:44:08.104932 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:44:08.104829    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339-kube-rbac-proxy-cm podName:e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:08.604813255 +0000 UTC m=+52.246528897 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339-kube-rbac-proxy-cm") pod "insights-runtime-extractor-627pg" (UID: "e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339") : configmap "kube-rbac-proxy" not found
Sep 16 18:44:08.105009 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:08.104980    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339-data-volume\") pod \"insights-runtime-extractor-627pg\" (UID: \"e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339\") " pod="openshift-insights/insights-runtime-extractor-627pg"
Sep 16 18:44:08.114007 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:08.113982    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbwxf\" (UniqueName: \"kubernetes.io/projected/e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339-kube-api-access-mbwxf\") pod \"insights-runtime-extractor-627pg\" (UID: \"e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339\") " pod="openshift-insights/insights-runtime-extractor-627pg"
Sep 16 18:44:08.183076 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:08.183045    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7qwl2_b6695f86-08c0-4836-87b1-f2a9d76084bd/console-operator/1.log"
Sep 16 18:44:08.183523 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:08.183504    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7qwl2_b6695f86-08c0-4836-87b1-f2a9d76084bd/console-operator/0.log"
Sep 16 18:44:08.183580 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:08.183550    2582 generic.go:356] "Generic (PLEG): container finished" podID="b6695f86-08c0-4836-87b1-f2a9d76084bd" containerID="9ebe1b1efc8955fa4106d06023ef75113d1b296a56b16c1094e52ba3aee64222" exitCode=255
Sep 16 18:44:08.183685 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:08.183659    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-7qwl2" event={"ID":"b6695f86-08c0-4836-87b1-f2a9d76084bd","Type":"ContainerDied","Data":"9ebe1b1efc8955fa4106d06023ef75113d1b296a56b16c1094e52ba3aee64222"}
Sep 16 18:44:08.183794 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:08.183771    2582 scope.go:117] "RemoveContainer" containerID="f31485cee8a1a7f476d7be2e1d3aa9d4f880a07012a53e4f33d11fce059dc20b"
Sep 16 18:44:08.183927 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:08.183894    2582 scope.go:117] "RemoveContainer" containerID="9ebe1b1efc8955fa4106d06023ef75113d1b296a56b16c1094e52ba3aee64222"
Sep 16 18:44:08.184106 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:44:08.184073    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-7qwl2_openshift-console-operator(b6695f86-08c0-4836-87b1-f2a9d76084bd)\"" pod="openshift-console-operator/console-operator-696884fb9c-7qwl2" podUID="b6695f86-08c0-4836-87b1-f2a9d76084bd"
Sep 16 18:44:08.609882 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:08.609844    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-627pg\" (UID: \"e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339\") " pod="openshift-insights/insights-runtime-extractor-627pg"
Sep 16 18:44:08.610072 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:08.609940    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-627pg\" (UID: \"e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339\") " pod="openshift-insights/insights-runtime-extractor-627pg"
Sep 16 18:44:08.610072 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:44:08.609996    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:44:08.610184 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:44:08.610079    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339-kube-rbac-proxy-cm podName:e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:09.610063415 +0000 UTC m=+53.251779061 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339-kube-rbac-proxy-cm") pod "insights-runtime-extractor-627pg" (UID: "e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339") : configmap "kube-rbac-proxy" not found
Sep 16 18:44:08.610184 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:44:08.610132    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:44:08.610278 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:44:08.610207    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339-insights-runtime-extractor-tls podName:e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:09.610188369 +0000 UTC m=+53.251904026 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339-insights-runtime-extractor-tls") pod "insights-runtime-extractor-627pg" (UID: "e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:44:09.187945 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:09.187918    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7qwl2_b6695f86-08c0-4836-87b1-f2a9d76084bd/console-operator/1.log"
Sep 16 18:44:09.188459 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:09.188445    2582 scope.go:117] "RemoveContainer" containerID="9ebe1b1efc8955fa4106d06023ef75113d1b296a56b16c1094e52ba3aee64222"
Sep 16 18:44:09.188626 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:44:09.188607    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-7qwl2_openshift-console-operator(b6695f86-08c0-4836-87b1-f2a9d76084bd)\"" pod="openshift-console-operator/console-operator-696884fb9c-7qwl2" podUID="b6695f86-08c0-4836-87b1-f2a9d76084bd"
Sep 16 18:44:09.194905 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:09.194884    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-9xhbs"]
Sep 16 18:44:09.199209 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:09.199191    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-9xhbs"
Sep 16 18:44:09.202820 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:09.202787    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 16 18:44:09.202929 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:09.202911    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-pn2ww\""
Sep 16 18:44:09.203001 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:09.202967    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 16 18:44:09.208408 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:09.208388    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-9xhbs"]
Sep 16 18:44:09.216325 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:09.216295    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzdvv\" (UniqueName: \"kubernetes.io/projected/c1f943cd-af90-41fe-ae52-9e2a35ac33f5-kube-api-access-jzdvv\") pod \"migrator-8b4d697c5-9xhbs\" (UID: \"c1f943cd-af90-41fe-ae52-9e2a35ac33f5\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-9xhbs"
Sep 16 18:44:09.240633 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:09.240610    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wdq46_c60bca6d-3f8c-4725-94a2-eedd9a075586/dns-node-resolver/0.log"
Sep 16 18:44:09.317059 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:09.317027    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzdvv\" (UniqueName: \"kubernetes.io/projected/c1f943cd-af90-41fe-ae52-9e2a35ac33f5-kube-api-access-jzdvv\") pod \"migrator-8b4d697c5-9xhbs\" (UID: \"c1f943cd-af90-41fe-ae52-9e2a35ac33f5\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-9xhbs"
Sep 16 18:44:09.325123 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:09.325095    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzdvv\" (UniqueName: \"kubernetes.io/projected/c1f943cd-af90-41fe-ae52-9e2a35ac33f5-kube-api-access-jzdvv\") pod \"migrator-8b4d697c5-9xhbs\" (UID: \"c1f943cd-af90-41fe-ae52-9e2a35ac33f5\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-9xhbs"
Sep 16 18:44:09.509075 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:09.509046    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-9xhbs"
Sep 16 18:44:09.620017 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:09.619980    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-627pg\" (UID: \"e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339\") " pod="openshift-insights/insights-runtime-extractor-627pg"
Sep 16 18:44:09.620267 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:09.620123    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-627pg\" (UID: \"e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339\") " pod="openshift-insights/insights-runtime-extractor-627pg"
Sep 16 18:44:09.620267 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:44:09.620152    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:44:09.620267 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:44:09.620234    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:44:09.620267 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:44:09.620246    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339-insights-runtime-extractor-tls podName:e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:11.620226194 +0000 UTC m=+55.261941851 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339-insights-runtime-extractor-tls") pod "insights-runtime-extractor-627pg" (UID: "e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:44:09.620421 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:44:09.620282    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339-kube-rbac-proxy-cm podName:e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:11.620268244 +0000 UTC m=+55.261983886 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339-kube-rbac-proxy-cm") pod "insights-runtime-extractor-627pg" (UID: "e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339") : configmap "kube-rbac-proxy" not found
Sep 16 18:44:09.777954 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:09.777930    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-9xhbs"]
Sep 16 18:44:09.780364 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:44:09.780332    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1f943cd_af90_41fe_ae52_9e2a35ac33f5.slice/crio-dcced3b237d100afb6d725cfbeccab32eccd12a8f2c25b34714dd33e9f590d0f WatchSource:0}: Error finding container dcced3b237d100afb6d725cfbeccab32eccd12a8f2c25b34714dd33e9f590d0f: Status 404 returned error can't find the container with id dcced3b237d100afb6d725cfbeccab32eccd12a8f2c25b34714dd33e9f590d0f
Sep 16 18:44:10.191773 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:10.191686    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-9xhbs" event={"ID":"c1f943cd-af90-41fe-ae52-9e2a35ac33f5","Type":"ContainerStarted","Data":"dcced3b237d100afb6d725cfbeccab32eccd12a8f2c25b34714dd33e9f590d0f"}
Sep 16 18:44:10.439986 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:10.439960    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-w8t8m_870ff296-8673-44d5-9fc3-619486d607a1/node-ca/0.log"
Sep 16 18:44:10.837380 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:10.837357    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wxss9_d5d54719-5c41-4b38-a2a1-367dc4bd6801/serve-healthcheck-canary/0.log"
Sep 16 18:44:11.124455 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:11.124375    2582 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-7qwl2"
Sep 16 18:44:11.124455 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:11.124430    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-7qwl2"
Sep 16 18:44:11.124784 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:11.124771    2582 scope.go:117] "RemoveContainer" containerID="9ebe1b1efc8955fa4106d06023ef75113d1b296a56b16c1094e52ba3aee64222"
Sep 16 18:44:11.124940 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:44:11.124923    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-7qwl2_openshift-console-operator(b6695f86-08c0-4836-87b1-f2a9d76084bd)\"" pod="openshift-console-operator/console-operator-696884fb9c-7qwl2" podUID="b6695f86-08c0-4836-87b1-f2a9d76084bd"
Sep 16 18:44:11.197586 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:11.197552    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-9xhbs" event={"ID":"c1f943cd-af90-41fe-ae52-9e2a35ac33f5","Type":"ContainerStarted","Data":"4f58c7298429cc7f82fc233da7c9c5e2a5d623c0598cdaf8d46f0d51bd2b570a"}
Sep 16 18:44:11.198036 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:11.197605    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-9xhbs" event={"ID":"c1f943cd-af90-41fe-ae52-9e2a35ac33f5","Type":"ContainerStarted","Data":"38713b2b5606cbbf3ed84cbcc25af5640eb41820d3b4394f046e5bad69add75e"}
Sep 16 18:44:11.224797 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:11.224753    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-9xhbs" podStartSLOduration=1.119383734 podStartE2EDuration="2.224736039s" totalImagesPullingTime="1.105352305s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:11.223484064 +0000 UTC m=+54.865199739" watchObservedRunningTime="2026-09-16 18:44:11.224736039 +0000 UTC m=+54.866451700"
Sep 16 18:44:11.639755 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:11.639692    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-627pg\" (UID: \"e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339\") " pod="openshift-insights/insights-runtime-extractor-627pg"
Sep 16 18:44:11.639922 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:44:11.639854    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:44:11.639922 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:11.639879    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-627pg\" (UID: \"e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339\") " pod="openshift-insights/insights-runtime-extractor-627pg"
Sep 16 18:44:11.639990 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:44:11.639923    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339-insights-runtime-extractor-tls podName:e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:15.639903181 +0000 UTC m=+59.281618825 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339-insights-runtime-extractor-tls") pod "insights-runtime-extractor-627pg" (UID: "e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:44:11.639990 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:44:11.639950    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:44:11.640053 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:44:11.639990    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339-kube-rbac-proxy-cm podName:e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:15.639979088 +0000 UTC m=+59.281694742 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339-kube-rbac-proxy-cm") pod "insights-runtime-extractor-627pg" (UID: "e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339") : configmap "kube-rbac-proxy" not found
Sep 16 18:44:15.674670 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:15.674631    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-627pg\" (UID: \"e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339\") " pod="openshift-insights/insights-runtime-extractor-627pg"
Sep 16 18:44:15.675336 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:15.674750    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-627pg\" (UID: \"e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339\") " pod="openshift-insights/insights-runtime-extractor-627pg"
Sep 16 18:44:15.675336 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:44:15.674784    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:44:15.675336 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:44:15.674856    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339-kube-rbac-proxy-cm podName:e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:23.67483829 +0000 UTC m=+67.316553954 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339-kube-rbac-proxy-cm") pod "insights-runtime-extractor-627pg" (UID: "e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339") : configmap "kube-rbac-proxy" not found
Sep 16 18:44:15.675336 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:44:15.674895    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:44:15.675336 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:44:15.674951    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339-insights-runtime-extractor-tls podName:e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:23.674936723 +0000 UTC m=+67.316652368 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339-insights-runtime-extractor-tls") pod "insights-runtime-extractor-627pg" (UID: "e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:44:18.042294 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:18.042261    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pv8wf"
Sep 16 18:44:22.537881 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:22.537841    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de8ddfa3-fd54-49d7-8f90-035aa4439e98-metrics-certs\") pod \"network-metrics-daemon-g7q9m\" (UID: \"de8ddfa3-fd54-49d7-8f90-035aa4439e98\") " pod="openshift-multus/network-metrics-daemon-g7q9m"
Sep 16 18:44:22.541137 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:22.541116    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:44:22.548295 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:44:22.548275    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 18:44:22.548391 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:44:22.548346    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de8ddfa3-fd54-49d7-8f90-035aa4439e98-metrics-certs podName:de8ddfa3-fd54-49d7-8f90-035aa4439e98 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:26.548325352 +0000 UTC m=+130.190041011 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de8ddfa3-fd54-49d7-8f90-035aa4439e98-metrics-certs") pod "network-metrics-daemon-g7q9m" (UID: "de8ddfa3-fd54-49d7-8f90-035aa4439e98") : secret "metrics-daemon-secret" not found
Sep 16 18:44:22.638886 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:22.638835    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e6428035-ca16-4098-ac4e-1ca4931bb7a7-original-pull-secret\") pod \"global-pull-secret-syncer-tplxv\" (UID: \"e6428035-ca16-4098-ac4e-1ca4931bb7a7\") " pod="kube-system/global-pull-secret-syncer-tplxv"
Sep 16 18:44:22.639055 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:22.638913    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fe6d9245-1609-46f1-9e0e-4fbdc31145d1-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-dvjf7\" (UID: \"fe6d9245-1609-46f1-9e0e-4fbdc31145d1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dvjf7"
Sep 16 18:44:22.639055 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:22.639024    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40a3ba8a-40a8-48eb-99df-da46c59ee997-registry-tls\") pod \"image-registry-79f6f8f56-gk2mv\" (UID: \"40a3ba8a-40a8-48eb-99df-da46c59ee997\") " pod="openshift-image-registry/image-registry-79f6f8f56-gk2mv"
Sep 16 18:44:22.641545 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:22.641483    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40a3ba8a-40a8-48eb-99df-da46c59ee997-registry-tls\") pod \"image-registry-79f6f8f56-gk2mv\" (UID: \"40a3ba8a-40a8-48eb-99df-da46c59ee997\") " pod="openshift-image-registry/image-registry-79f6f8f56-gk2mv"
Sep 16 18:44:22.641545 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:22.641483    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fe6d9245-1609-46f1-9e0e-4fbdc31145d1-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-dvjf7\" (UID: \"fe6d9245-1609-46f1-9e0e-4fbdc31145d1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dvjf7"
Sep 16 18:44:22.641800 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:22.641785    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:44:22.650922 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:22.650900    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e6428035-ca16-4098-ac4e-1ca4931bb7a7-original-pull-secret\") pod \"global-pull-secret-syncer-tplxv\" (UID: \"e6428035-ca16-4098-ac4e-1ca4931bb7a7\") " pod="kube-system/global-pull-secret-syncer-tplxv"
Sep 16 18:44:22.740077 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:22.739985    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ba5d0219-c94f-41a7-a6fc-86b14d2bccd2-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-l56nd\" (UID: \"ba5d0219-c94f-41a7-a6fc-86b14d2bccd2\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-l56nd"
Sep 16 18:44:22.740077 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:22.740040    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmpfz\" (UniqueName: \"kubernetes.io/projected/f79ebb7c-f4b5-4e48-9506-81110189ba43-kube-api-access-pmpfz\") pod \"network-check-target-dgl4n\" (UID: \"f79ebb7c-f4b5-4e48-9506-81110189ba43\") " pod="openshift-network-diagnostics/network-check-target-dgl4n"
Sep 16 18:44:22.740266 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:44:22.740133    2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 18:44:22.740266 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:44:22.740233    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba5d0219-c94f-41a7-a6fc-86b14d2bccd2-networking-console-plugin-cert podName:ba5d0219-c94f-41a7-a6fc-86b14d2bccd2 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:54.740214069 +0000 UTC m=+98.381929711 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ba5d0219-c94f-41a7-a6fc-86b14d2bccd2-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-l56nd" (UID: "ba5d0219-c94f-41a7-a6fc-86b14d2bccd2") : secret "networking-console-plugin-cert" not found
Sep 16 18:44:22.740347 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:22.740275    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d76fc261-072a-4625-818d-e8faa5710cbd-metrics-tls\") pod \"dns-default-nw2m8\" (UID: \"d76fc261-072a-4625-818d-e8faa5710cbd\") " pod="openshift-dns/dns-default-nw2m8"
Sep 16 18:44:22.742471 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:22.742447    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmpfz\" (UniqueName: \"kubernetes.io/projected/f79ebb7c-f4b5-4e48-9506-81110189ba43-kube-api-access-pmpfz\") pod \"network-check-target-dgl4n\" (UID: \"f79ebb7c-f4b5-4e48-9506-81110189ba43\") " pod="openshift-network-diagnostics/network-check-target-dgl4n"
Sep 16 18:44:22.742572 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:22.742547    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d76fc261-072a-4625-818d-e8faa5710cbd-metrics-tls\") pod \"dns-default-nw2m8\" (UID: \"d76fc261-072a-4625-818d-e8faa5710cbd\") " pod="openshift-dns/dns-default-nw2m8"
Sep 16 18:44:22.786301 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:22.786264    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tplxv"
Sep 16 18:44:22.795212 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:22.795188    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fq6l6\""
Sep 16 18:44:22.803732 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:22.803696    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgl4n"
Sep 16 18:44:22.836979 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:22.836710    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4tghj\""
Sep 16 18:44:22.841305 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:22.840966    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ac2b8ef-ff60-4791-9e56-5ac64263bf1e-metrics-certs\") pod \"router-default-7465b55b9d-njqmh\" (UID: \"9ac2b8ef-ff60-4791-9e56-5ac64263bf1e\") " pod="openshift-ingress/router-default-7465b55b9d-njqmh"
Sep 16 18:44:22.841305 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:22.841016    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ac2b8ef-ff60-4791-9e56-5ac64263bf1e-service-ca-bundle\") pod \"router-default-7465b55b9d-njqmh\" (UID: \"9ac2b8ef-ff60-4791-9e56-5ac64263bf1e\") " pod="openshift-ingress/router-default-7465b55b9d-njqmh"
Sep 16 18:44:22.841676 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:22.841653    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ac2b8ef-ff60-4791-9e56-5ac64263bf1e-service-ca-bundle\") pod \"router-default-7465b55b9d-njqmh\" (UID: \"9ac2b8ef-ff60-4791-9e56-5ac64263bf1e\") " pod="openshift-ingress/router-default-7465b55b9d-njqmh"
Sep 16 18:44:22.844063 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:22.843996    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ac2b8ef-ff60-4791-9e56-5ac64263bf1e-metrics-certs\") pod \"router-default-7465b55b9d-njqmh\" (UID: \"9ac2b8ef-ff60-4791-9e56-5ac64263bf1e\") " pod="openshift-ingress/router-default-7465b55b9d-njqmh"
Sep 16 18:44:22.844184 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:22.844065    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79f6f8f56-gk2mv"
Sep 16 18:44:22.856239 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:22.856214    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-fhxx4\""
Sep 16 18:44:22.863833 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:22.863795    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dvjf7"
Sep 16 18:44:22.942318 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:22.942293    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e040ac61-5af8-4092-9a00-ec184466deaf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-x6kvz\" (UID: \"e040ac61-5af8-4092-9a00-ec184466deaf\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-x6kvz"
Sep 16 18:44:22.944632 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:22.944608    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e040ac61-5af8-4092-9a00-ec184466deaf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-x6kvz\" (UID: \"e040ac61-5af8-4092-9a00-ec184466deaf\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-x6kvz"
Sep 16 18:44:22.950843 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:22.950704    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xrh57\""
Sep 16 18:44:22.957515 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:22.957479    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nw2m8"
Sep 16 18:44:22.989174 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:22.989146    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-mq4xp\""
Sep 16 18:44:22.996765 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:22.996677    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7465b55b9d-njqmh"
Sep 16 18:44:23.066505 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:44:23.066475    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6428035_ca16_4098_ac4e_1ca4931bb7a7.slice/crio-a030f243892f09570a80e435ed7ede7a1b9d2cf4157ed3b7c9671d7107784fdd WatchSource:0}: Error finding container a030f243892f09570a80e435ed7ede7a1b9d2cf4157ed3b7c9671d7107784fdd: Status 404 returned error can't find the container with id a030f243892f09570a80e435ed7ede7a1b9d2cf4157ed3b7c9671d7107784fdd
Sep 16 18:44:23.068078 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:23.068051    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tplxv"]
Sep 16 18:44:23.085991 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:23.085965    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dgl4n"]
Sep 16 18:44:23.089186 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:44:23.089158    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf79ebb7c_f4b5_4e48_9506_81110189ba43.slice/crio-b26402f5a3dfee6b81fe0a93f61e362a75d8d9808d3822e1c705b44662ede745 WatchSource:0}: Error finding container b26402f5a3dfee6b81fe0a93f61e362a75d8d9808d3822e1c705b44662ede745: Status 404 returned error can't find the container with id b26402f5a3dfee6b81fe0a93f61e362a75d8d9808d3822e1c705b44662ede745
Sep 16 18:44:23.125858 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:23.125835    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-bw4mq\""
Sep 16 18:44:23.134668 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:23.134647    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-x6kvz"
Sep 16 18:44:23.139865 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:23.139843    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79f6f8f56-gk2mv"]
Sep 16 18:44:23.159108 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:44:23.159075    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40a3ba8a_40a8_48eb_99df_da46c59ee997.slice/crio-e5747062bf111283b3c6549c070118c00277e730cb6c16e6c6a0fb8d3444a2f4 WatchSource:0}: Error finding container e5747062bf111283b3c6549c070118c00277e730cb6c16e6c6a0fb8d3444a2f4: Status 404 returned error can't find the container with id e5747062bf111283b3c6549c070118c00277e730cb6c16e6c6a0fb8d3444a2f4
Sep 16 18:44:23.171322 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:23.171294    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dvjf7"]
Sep 16 18:44:23.237986 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:23.237954    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tplxv" event={"ID":"e6428035-ca16-4098-ac4e-1ca4931bb7a7","Type":"ContainerStarted","Data":"a030f243892f09570a80e435ed7ede7a1b9d2cf4157ed3b7c9671d7107784fdd"}
Sep 16 18:44:23.239152 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:23.239127    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dvjf7" event={"ID":"fe6d9245-1609-46f1-9e0e-4fbdc31145d1","Type":"ContainerStarted","Data":"4dcc1c3cc22541a1bcd756d63aa52f9da5c70a42bcb29b879da433d7d86b3c33"}
Sep 16 18:44:23.240625 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:23.240605    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79f6f8f56-gk2mv" event={"ID":"40a3ba8a-40a8-48eb-99df-da46c59ee997","Type":"ContainerStarted","Data":"b4a32a934952c08fe8599e9037c4f680895e9a4a17d2a6bdcd159669a8ede32d"}
Sep 16 18:44:23.240690 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:23.240641    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79f6f8f56-gk2mv" event={"ID":"40a3ba8a-40a8-48eb-99df-da46c59ee997","Type":"ContainerStarted","Data":"e5747062bf111283b3c6549c070118c00277e730cb6c16e6c6a0fb8d3444a2f4"}
Sep 16 18:44:23.240779 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:23.240736    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-79f6f8f56-gk2mv"
Sep 16 18:44:23.242339 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:23.242318    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dgl4n" event={"ID":"f79ebb7c-f4b5-4e48-9506-81110189ba43","Type":"ContainerStarted","Data":"53e8039a7bbfd75996e386181bb67ce854a0bdd6b3a5977023817d8b88db508d"}
Sep 16 18:44:23.242433 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:23.242350    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dgl4n" event={"ID":"f79ebb7c-f4b5-4e48-9506-81110189ba43","Type":"ContainerStarted","Data":"b26402f5a3dfee6b81fe0a93f61e362a75d8d9808d3822e1c705b44662ede745"}
Sep 16 18:44:23.242484 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:23.242432    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dgl4n"
Sep 16 18:44:23.250351 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:23.250291    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nw2m8"]
Sep 16 18:44:23.252940 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:44:23.252920    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd76fc261_072a_4625_818d_e8faa5710cbd.slice/crio-5499baff4d631ffd4a0b2a7cdad1fb47606e03c9446fd27dbfbb59f607f24983 WatchSource:0}: Error finding container 5499baff4d631ffd4a0b2a7cdad1fb47606e03c9446fd27dbfbb59f607f24983: Status 404 returned error can't find the container with id 5499baff4d631ffd4a0b2a7cdad1fb47606e03c9446fd27dbfbb59f607f24983
Sep 16 18:44:23.269140 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:23.268963    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-79f6f8f56-gk2mv" podStartSLOduration=66.268946026 podStartE2EDuration="1m6.268946026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:43:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:23.267950077 +0000 UTC m=+66.909665764" watchObservedRunningTime="2026-09-16 18:44:23.268946026 +0000 UTC m=+66.910661692"
Sep 16 18:44:23.283440 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:23.283415    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7465b55b9d-njqmh"]
Sep 16 18:44:23.286398 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:44:23.286362    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ac2b8ef_ff60_4791_9e56_5ac64263bf1e.slice/crio-e1d6ce8cc4786256bbf4c9eb2db473cb198fdc366f11edb7fe8176c5faf5032e WatchSource:0}: Error finding container e1d6ce8cc4786256bbf4c9eb2db473cb198fdc366f11edb7fe8176c5faf5032e: Status 404 returned error can't find the container with id e1d6ce8cc4786256bbf4c9eb2db473cb198fdc366f11edb7fe8176c5faf5032e
Sep 16 18:44:23.414787 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:23.414732    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-dgl4n" podStartSLOduration=66.414701381 podStartE2EDuration="1m6.414701381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:43:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:23.290983932 +0000 UTC m=+66.932699607" watchObservedRunningTime="2026-09-16 18:44:23.414701381 +0000 UTC m=+67.056417044"
Sep 16 18:44:23.416336 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:23.416312    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-x6kvz"]
Sep 16 18:44:23.419935 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:44:23.419903    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode040ac61_5af8_4092_9a00_ec184466deaf.slice/crio-be5c25bb394cf9eaa63d559c88bd7c580223a96dc95c73d04949516b406cbf80 WatchSource:0}: Error finding container be5c25bb394cf9eaa63d559c88bd7c580223a96dc95c73d04949516b406cbf80: Status 404 returned error can't find the container with id be5c25bb394cf9eaa63d559c88bd7c580223a96dc95c73d04949516b406cbf80
Sep 16 18:44:23.753187 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:23.753066    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-627pg\" (UID: \"e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339\") " pod="openshift-insights/insights-runtime-extractor-627pg"
Sep 16 18:44:23.753665 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:23.753205    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-627pg\" (UID: \"e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339\") " pod="openshift-insights/insights-runtime-extractor-627pg"
Sep 16 18:44:23.753665 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:44:23.753214    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:44:23.753665 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:44:23.753290    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339-insights-runtime-extractor-tls podName:e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:39.753269137 +0000 UTC m=+83.394984788 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339-insights-runtime-extractor-tls") pod "insights-runtime-extractor-627pg" (UID: "e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:44:23.753878 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:23.753821    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-627pg\" (UID: \"e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339\") " pod="openshift-insights/insights-runtime-extractor-627pg"
Sep 16 18:44:24.251314 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:24.251213    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-x6kvz" event={"ID":"e040ac61-5af8-4092-9a00-ec184466deaf","Type":"ContainerStarted","Data":"be5c25bb394cf9eaa63d559c88bd7c580223a96dc95c73d04949516b406cbf80"}
Sep 16 18:44:24.254295 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:24.254223    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nw2m8" event={"ID":"d76fc261-072a-4625-818d-e8faa5710cbd","Type":"ContainerStarted","Data":"5499baff4d631ffd4a0b2a7cdad1fb47606e03c9446fd27dbfbb59f607f24983"}
Sep 16 18:44:24.256887 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:24.256371    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7465b55b9d-njqmh" event={"ID":"9ac2b8ef-ff60-4791-9e56-5ac64263bf1e","Type":"ContainerStarted","Data":"d37e231d18c89b0b21d23cf67e8647291ac5eae420c50029bed544e31e6a63ec"}
Sep 16 18:44:24.256887 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:24.256421    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7465b55b9d-njqmh" event={"ID":"9ac2b8ef-ff60-4791-9e56-5ac64263bf1e","Type":"ContainerStarted","Data":"e1d6ce8cc4786256bbf4c9eb2db473cb198fdc366f11edb7fe8176c5faf5032e"}
Sep 16 18:44:24.281208 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:24.280668    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7465b55b9d-njqmh" podStartSLOduration=59.280645208 podStartE2EDuration="59.280645208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:43:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:24.278879013 +0000 UTC m=+67.920594678" watchObservedRunningTime="2026-09-16 18:44:24.280645208 +0000 UTC m=+67.922360873"
Sep 16 18:44:24.997785 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:24.997733    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7465b55b9d-njqmh"
Sep 16 18:44:25.001088 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:25.001063    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7465b55b9d-njqmh"
Sep 16 18:44:25.261291 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:25.261228    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7465b55b9d-njqmh"
Sep 16 18:44:25.262689 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:25.262660    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7465b55b9d-njqmh"
Sep 16 18:44:25.868056 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:25.868027    2582 scope.go:117] "RemoveContainer" containerID="9ebe1b1efc8955fa4106d06023ef75113d1b296a56b16c1094e52ba3aee64222"
Sep 16 18:44:28.276048 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:28.276000    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-x6kvz" event={"ID":"e040ac61-5af8-4092-9a00-ec184466deaf","Type":"ContainerStarted","Data":"017095574bcaa0f02ff2331405a8411d97abc70d2a27d922fd755c01364b88c9"}
Sep 16 18:44:28.278106 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:28.278064    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nw2m8" event={"ID":"d76fc261-072a-4625-818d-e8faa5710cbd","Type":"ContainerStarted","Data":"28fcdcd50e059f577d5f79839b4bb136f61b47710568bd068f0d35d4020de24d"}
Sep 16 18:44:28.278250 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:28.278118    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nw2m8" event={"ID":"d76fc261-072a-4625-818d-e8faa5710cbd","Type":"ContainerStarted","Data":"a39a5a13459b47221580658d30c285f500b4d00a807117c2cb72dd03b79312d0"}
Sep 16 18:44:28.278250 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:28.278147    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-nw2m8"
Sep 16 18:44:28.279879 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:28.279851    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tplxv" event={"ID":"e6428035-ca16-4098-ac4e-1ca4931bb7a7","Type":"ContainerStarted","Data":"f3444960605f2a7a13753d4c553e58ed04c32573bf7aad165429a5a4463583a2"}
Sep 16 18:44:28.282025 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:28.282003    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7qwl2_b6695f86-08c0-4836-87b1-f2a9d76084bd/console-operator/1.log"
Sep 16 18:44:28.282135 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:28.282116    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-7qwl2" event={"ID":"b6695f86-08c0-4836-87b1-f2a9d76084bd","Type":"ContainerStarted","Data":"e2d1588e77fe4cb2150702833e6f2af65bbfb58b3e219897ea519662ad457eb6"}
Sep 16 18:44:28.282558 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:28.282404    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-7qwl2"
Sep 16 18:44:28.287694 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:28.287667    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dvjf7" event={"ID":"fe6d9245-1609-46f1-9e0e-4fbdc31145d1","Type":"ContainerStarted","Data":"d32b251584b6b732c71018f2220d5c91c705e2140e4471aa8d67a246f29f066e"}
Sep 16 18:44:28.287812 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:28.287712    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dvjf7" event={"ID":"fe6d9245-1609-46f1-9e0e-4fbdc31145d1","Type":"ContainerStarted","Data":"4edddedfac3d3c5a8f5c0a8b044af5f25ddf4ae6c7f3dbecc3ab0c9f422deb25"}
Sep 16 18:44:28.294194 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:28.294156    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-x6kvz" podStartSLOduration=59.087433221 podStartE2EDuration="1m3.29414342s" totalImagesPullingTime="4.206710199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:43:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:28.293191862 +0000 UTC m=+71.934907562" watchObservedRunningTime="2026-09-16 18:44:28.29414342 +0000 UTC m=+71.935859145"
Sep 16 18:44:28.308444 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:28.308407    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-tplxv" podStartSLOduration=66.390177337 podStartE2EDuration="1m11.308394635s" totalImagesPullingTime="4.918217298s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:28.308247901 +0000 UTC m=+71.949963566" watchObservedRunningTime="2026-09-16 18:44:28.308394635 +0000 UTC m=+71.950110297"
Sep 16 18:44:28.327594 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:28.327535    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dvjf7" podStartSLOduration=58.922443052 podStartE2EDuration="1m3.327515262s" totalImagesPullingTime="4.40507221s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:43:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:28.325378187 +0000 UTC m=+71.967093855" watchObservedRunningTime="2026-09-16 18:44:28.327515262 +0000 UTC m=+71.969230926"
Sep 16 18:44:28.344437 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:28.344387    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-nw2m8" podStartSLOduration=33.976002564 podStartE2EDuration="38.344370576s" totalImagesPullingTime="4.368368012s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:43:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:28.342887521 +0000 UTC m=+71.984603184" watchObservedRunningTime="2026-09-16 18:44:28.344370576 +0000 UTC m=+71.986086239"
Sep 16 18:44:28.361253 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:28.361204    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-7qwl2" podStartSLOduration=48.580115003 podStartE2EDuration="1m3.36118877s" totalImagesPullingTime="14.781073767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:43:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:28.361047625 +0000 UTC m=+72.002763290" watchObservedRunningTime="2026-09-16 18:44:28.36118877 +0000 UTC m=+72.002904433"
Sep 16 18:44:28.790301 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:28.790267    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-7qwl2"
Sep 16 18:44:38.297554 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:38.297517    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-nw2m8"
Sep 16 18:44:39.795121 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:39.794988    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-627pg\" (UID: \"e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339\") " pod="openshift-insights/insights-runtime-extractor-627pg"
Sep 16 18:44:39.797254 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:39.797234    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-627pg\" (UID: \"e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339\") " pod="openshift-insights/insights-runtime-extractor-627pg"
Sep 16 18:44:40.098935 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:40.098849    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6r5qx\""
Sep 16 18:44:40.106484 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:40.106464    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-627pg"
Sep 16 18:44:40.410515 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:40.410434    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-627pg"]
Sep 16 18:44:40.413403 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:44:40.413377    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode603fb2e_bc3c_4c6e_9fa2_1c384bcc3339.slice/crio-1930e623fb796eee41bedfa66bf9b042175f8f160ec38e6e77179bc03c3777f4 WatchSource:0}: Error finding container 1930e623fb796eee41bedfa66bf9b042175f8f160ec38e6e77179bc03c3777f4: Status 404 returned error can't find the container with id 1930e623fb796eee41bedfa66bf9b042175f8f160ec38e6e77179bc03c3777f4
Sep 16 18:44:41.340576 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:41.340489    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-627pg" event={"ID":"e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339","Type":"ContainerStarted","Data":"429165b68c65669e13367843b11cd84295f8dbc5170d61e0f0d21299a9844ce2"}
Sep 16 18:44:41.340576 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:41.340546    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-627pg" event={"ID":"e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339","Type":"ContainerStarted","Data":"9e30a617dae622fbb6a8ce20e2d886fef64871f081c3b56bd167565c7a344957"}
Sep 16 18:44:41.340576 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:41.340566    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-627pg" event={"ID":"e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339","Type":"ContainerStarted","Data":"1930e623fb796eee41bedfa66bf9b042175f8f160ec38e6e77179bc03c3777f4"}
Sep 16 18:44:42.849303 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:42.849267    2582 patch_prober.go:28] interesting pod/image-registry-79f6f8f56-gk2mv container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 16 18:44:42.849671 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:42.849322    2582 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-79f6f8f56-gk2mv" podUID="40a3ba8a-40a8-48eb-99df-da46c59ee997" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:44:43.350364 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:43.350331    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-627pg" event={"ID":"e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339","Type":"ContainerStarted","Data":"cd451cee6b097ff9791f892fb9d7dfa368c14ef62c3ca59c4e12dbcd471258c4"}
Sep 16 18:44:43.372707 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:43.372657    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-627pg" podStartSLOduration=34.590293286 podStartE2EDuration="36.372643398s" totalImagesPullingTime="1.782350112s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:43.370381279 +0000 UTC m=+87.012096978" watchObservedRunningTime="2026-09-16 18:44:43.372643398 +0000 UTC m=+87.014359062"
Sep 16 18:44:44.261734 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:44.261699    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-79f6f8f56-gk2mv"
Sep 16 18:44:44.783504 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:44.783470    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-4b825"]
Sep 16 18:44:44.786932 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:44.786915    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4b825"
Sep 16 18:44:44.790057 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:44:44.790010    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"metrics-client-ca\" is forbidden: User \"system:node:ip-10-0-129-194.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-129-194.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" type="*v1.ConfigMap"
Sep 16 18:44:44.790184 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:44:44.790160    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"node-exporter-kube-rbac-proxy-config\" is forbidden: User \"system:node:ip-10-0-129-194.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-129-194.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" type="*v1.Secret"
Sep 16 18:44:44.790238 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:44:44.790195    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"node-exporter-accelerators-collector-config\" is forbidden: User \"system:node:ip-10-0-129-194.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-129-194.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" type="*v1.ConfigMap"
Sep 16 18:44:44.791051 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:44:44.791027    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"node-exporter-tls\" is forbidden: User \"system:node:ip-10-0-129-194.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-129-194.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" type="*v1.Secret"
Sep 16 18:44:44.793561 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:44:44.793533    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"node-exporter-4b825\" is forbidden: User \"system:node:ip-10-0-129-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-129-194.ec2.internal' and this object" podUID="ed2c35c3-6cab-4978-9ab5-958f4de177b6" pod="openshift-monitoring/node-exporter-4b825"
Sep 16 18:44:44.796778 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:44:44.796751    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"node-exporter-dockercfg-rh45j\" is forbidden: User \"system:node:ip-10-0-129-194.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-129-194.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rh45j\"" type="*v1.Secret"
Sep 16 18:44:44.938547 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:44.938515    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ed2c35c3-6cab-4978-9ab5-958f4de177b6-node-exporter-accelerators-collector-config\") pod \"node-exporter-4b825\" (UID: \"ed2c35c3-6cab-4978-9ab5-958f4de177b6\") " pod="openshift-monitoring/node-exporter-4b825"
Sep 16 18:44:44.938714 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:44.938554    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed2c35c3-6cab-4978-9ab5-958f4de177b6-metrics-client-ca\") pod \"node-exporter-4b825\" (UID: \"ed2c35c3-6cab-4978-9ab5-958f4de177b6\") " pod="openshift-monitoring/node-exporter-4b825"
Sep 16 18:44:44.938714 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:44.938576    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ed2c35c3-6cab-4978-9ab5-958f4de177b6-node-exporter-tls\") pod \"node-exporter-4b825\" (UID: \"ed2c35c3-6cab-4978-9ab5-958f4de177b6\") " pod="openshift-monitoring/node-exporter-4b825"
Sep 16 18:44:44.938714 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:44.938650    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ed2c35c3-6cab-4978-9ab5-958f4de177b6-node-exporter-wtmp\") pod \"node-exporter-4b825\" (UID: \"ed2c35c3-6cab-4978-9ab5-958f4de177b6\") " pod="openshift-monitoring/node-exporter-4b825"
Sep 16 18:44:44.938887 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:44.938739    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ed2c35c3-6cab-4978-9ab5-958f4de177b6-sys\") pod \"node-exporter-4b825\" (UID: \"ed2c35c3-6cab-4978-9ab5-958f4de177b6\") " pod="openshift-monitoring/node-exporter-4b825"
Sep 16 18:44:44.938887 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:44.938771    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ed2c35c3-6cab-4978-9ab5-958f4de177b6-node-exporter-textfile\") pod \"node-exporter-4b825\" (UID: \"ed2c35c3-6cab-4978-9ab5-958f4de177b6\") " pod="openshift-monitoring/node-exporter-4b825"
Sep 16 18:44:44.938887 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:44.938808    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ed2c35c3-6cab-4978-9ab5-958f4de177b6-root\") pod \"node-exporter-4b825\" (UID: \"ed2c35c3-6cab-4978-9ab5-958f4de177b6\") " pod="openshift-monitoring/node-exporter-4b825"
Sep 16 18:44:44.939030 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:44.938883    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqgzl\" (UniqueName: \"kubernetes.io/projected/ed2c35c3-6cab-4978-9ab5-958f4de177b6-kube-api-access-hqgzl\") pod \"node-exporter-4b825\" (UID: \"ed2c35c3-6cab-4978-9ab5-958f4de177b6\") " pod="openshift-monitoring/node-exporter-4b825"
Sep 16 18:44:44.939030 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:44.938949    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ed2c35c3-6cab-4978-9ab5-958f4de177b6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4b825\" (UID: \"ed2c35c3-6cab-4978-9ab5-958f4de177b6\") " pod="openshift-monitoring/node-exporter-4b825"
Sep 16 18:44:45.039910 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:45.039842    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqgzl\" (UniqueName: \"kubernetes.io/projected/ed2c35c3-6cab-4978-9ab5-958f4de177b6-kube-api-access-hqgzl\") pod \"node-exporter-4b825\" (UID: \"ed2c35c3-6cab-4978-9ab5-958f4de177b6\") " pod="openshift-monitoring/node-exporter-4b825"
Sep 16 18:44:45.039910 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:45.039883    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ed2c35c3-6cab-4978-9ab5-958f4de177b6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4b825\" (UID: \"ed2c35c3-6cab-4978-9ab5-958f4de177b6\") " pod="openshift-monitoring/node-exporter-4b825"
Sep 16 18:44:45.040108 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:45.039913    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ed2c35c3-6cab-4978-9ab5-958f4de177b6-node-exporter-accelerators-collector-config\") pod \"node-exporter-4b825\" (UID: \"ed2c35c3-6cab-4978-9ab5-958f4de177b6\") " pod="openshift-monitoring/node-exporter-4b825"
Sep 16 18:44:45.040108 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:45.039934    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed2c35c3-6cab-4978-9ab5-958f4de177b6-metrics-client-ca\") pod \"node-exporter-4b825\" (UID: \"ed2c35c3-6cab-4978-9ab5-958f4de177b6\") " pod="openshift-monitoring/node-exporter-4b825"
Sep 16 18:44:45.040108 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:45.039955    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ed2c35c3-6cab-4978-9ab5-958f4de177b6-node-exporter-tls\") pod \"node-exporter-4b825\" (UID: \"ed2c35c3-6cab-4978-9ab5-958f4de177b6\") " pod="openshift-monitoring/node-exporter-4b825"
Sep 16 18:44:45.040108 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:45.039994    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ed2c35c3-6cab-4978-9ab5-958f4de177b6-node-exporter-wtmp\") pod \"node-exporter-4b825\" (UID: \"ed2c35c3-6cab-4978-9ab5-958f4de177b6\") " pod="openshift-monitoring/node-exporter-4b825"
Sep 16 18:44:45.040108 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:45.040078    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ed2c35c3-6cab-4978-9ab5-958f4de177b6-sys\") pod \"node-exporter-4b825\" (UID: \"ed2c35c3-6cab-4978-9ab5-958f4de177b6\") " pod="openshift-monitoring/node-exporter-4b825"
Sep 16 18:44:45.040275 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:45.040127    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ed2c35c3-6cab-4978-9ab5-958f4de177b6-node-exporter-textfile\") pod \"node-exporter-4b825\" (UID: \"ed2c35c3-6cab-4978-9ab5-958f4de177b6\") " pod="openshift-monitoring/node-exporter-4b825"
Sep 16 18:44:45.040275 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:45.040127    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ed2c35c3-6cab-4978-9ab5-958f4de177b6-sys\") pod \"node-exporter-4b825\" (UID: \"ed2c35c3-6cab-4978-9ab5-958f4de177b6\") " pod="openshift-monitoring/node-exporter-4b825"
Sep 16 18:44:45.040275 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:45.040133    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ed2c35c3-6cab-4978-9ab5-958f4de177b6-node-exporter-wtmp\") pod \"node-exporter-4b825\" (UID: \"ed2c35c3-6cab-4978-9ab5-958f4de177b6\") " pod="openshift-monitoring/node-exporter-4b825"
Sep 16 18:44:45.040275 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:45.040193    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ed2c35c3-6cab-4978-9ab5-958f4de177b6-root\") pod \"node-exporter-4b825\" (UID: \"ed2c35c3-6cab-4978-9ab5-958f4de177b6\") " pod="openshift-monitoring/node-exporter-4b825"
Sep 16 18:44:45.040275 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:45.040217    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ed2c35c3-6cab-4978-9ab5-958f4de177b6-root\") pod \"node-exporter-4b825\" (UID: \"ed2c35c3-6cab-4978-9ab5-958f4de177b6\") " pod="openshift-monitoring/node-exporter-4b825"
Sep 16 18:44:45.040422 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:45.040384    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ed2c35c3-6cab-4978-9ab5-958f4de177b6-node-exporter-textfile\") pod \"node-exporter-4b825\" (UID: \"ed2c35c3-6cab-4978-9ab5-958f4de177b6\") " pod="openshift-monitoring/node-exporter-4b825"
Sep 16 18:44:45.051441 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:45.051416    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqgzl\" (UniqueName: \"kubernetes.io/projected/ed2c35c3-6cab-4978-9ab5-958f4de177b6-kube-api-access-hqgzl\") pod \"node-exporter-4b825\" (UID: \"ed2c35c3-6cab-4978-9ab5-958f4de177b6\") " pod="openshift-monitoring/node-exporter-4b825"
Sep 16 18:44:45.624511 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:45.624477    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 18:44:45.631302 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:45.631274    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed2c35c3-6cab-4978-9ab5-958f4de177b6-metrics-client-ca\") pod \"node-exporter-4b825\" (UID: \"ed2c35c3-6cab-4978-9ab5-958f4de177b6\") " pod="openshift-monitoring/node-exporter-4b825"
Sep 16 18:44:45.675119 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:45.675084    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 18:44:45.680664 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:45.680634    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ed2c35c3-6cab-4978-9ab5-958f4de177b6-node-exporter-accelerators-collector-config\") pod \"node-exporter-4b825\" (UID: \"ed2c35c3-6cab-4978-9ab5-958f4de177b6\") " pod="openshift-monitoring/node-exporter-4b825"
Sep 16 18:44:45.856632 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:45.856605    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 18:44:45.862823 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:45.862798    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ed2c35c3-6cab-4978-9ab5-958f4de177b6-node-exporter-tls\") pod \"node-exporter-4b825\" (UID: \"ed2c35c3-6cab-4978-9ab5-958f4de177b6\") " pod="openshift-monitoring/node-exporter-4b825"
Sep 16 18:44:46.041012 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:44:46.040974    2582 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-kube-rbac-proxy-config: failed to sync secret cache: timed out waiting for the condition
Sep 16 18:44:46.041165 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:44:46.041108    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed2c35c3-6cab-4978-9ab5-958f4de177b6-node-exporter-kube-rbac-proxy-config podName:ed2c35c3-6cab-4978-9ab5-958f4de177b6 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:46.541085378 +0000 UTC m=+90.182801022 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-kube-rbac-proxy-config" (UniqueName: "kubernetes.io/secret/ed2c35c3-6cab-4978-9ab5-958f4de177b6-node-exporter-kube-rbac-proxy-config") pod "node-exporter-4b825" (UID: "ed2c35c3-6cab-4978-9ab5-958f4de177b6") : failed to sync secret cache: timed out waiting for the condition
Sep 16 18:44:46.056929 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:46.056905    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 18:44:46.240460 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:46.240425    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rh45j\""
Sep 16 18:44:46.552446 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:46.552407    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ed2c35c3-6cab-4978-9ab5-958f4de177b6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4b825\" (UID: \"ed2c35c3-6cab-4978-9ab5-958f4de177b6\") " pod="openshift-monitoring/node-exporter-4b825"
Sep 16 18:44:46.554697 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:46.554676    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ed2c35c3-6cab-4978-9ab5-958f4de177b6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4b825\" (UID: \"ed2c35c3-6cab-4978-9ab5-958f4de177b6\") " pod="openshift-monitoring/node-exporter-4b825"
Sep 16 18:44:46.595732 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:46.595678    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4b825"
Sep 16 18:44:46.609406 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:44:46.609373    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded2c35c3_6cab_4978_9ab5_958f4de177b6.slice/crio-c29b342805957af240bf2e4fe43f6bde8c07668693bdf42be7d32df1e2d9bfb6 WatchSource:0}: Error finding container c29b342805957af240bf2e4fe43f6bde8c07668693bdf42be7d32df1e2d9bfb6: Status 404 returned error can't find the container with id c29b342805957af240bf2e4fe43f6bde8c07668693bdf42be7d32df1e2d9bfb6
Sep 16 18:44:47.367283 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:47.367233    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4b825" event={"ID":"ed2c35c3-6cab-4978-9ab5-958f4de177b6","Type":"ContainerStarted","Data":"c29b342805957af240bf2e4fe43f6bde8c07668693bdf42be7d32df1e2d9bfb6"}
Sep 16 18:44:48.372135 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:48.372101    2582 generic.go:356] "Generic (PLEG): container finished" podID="ed2c35c3-6cab-4978-9ab5-958f4de177b6" containerID="a671c879d10c1996b38eccf5ffc25d4ae333498a41e20c4f32e79b58525e81ac" exitCode=0
Sep 16 18:44:48.372526 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:48.372144    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4b825" event={"ID":"ed2c35c3-6cab-4978-9ab5-958f4de177b6","Type":"ContainerDied","Data":"a671c879d10c1996b38eccf5ffc25d4ae333498a41e20c4f32e79b58525e81ac"}
Sep 16 18:44:49.377912 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:49.377869    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4b825" event={"ID":"ed2c35c3-6cab-4978-9ab5-958f4de177b6","Type":"ContainerStarted","Data":"d483803e643018375d44825044ed2b74e34d877373a6eec57622cda5c7b11712"}
Sep 16 18:44:49.378382 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:49.377939    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4b825" event={"ID":"ed2c35c3-6cab-4978-9ab5-958f4de177b6","Type":"ContainerStarted","Data":"d312d9db9a5d48c34c90956e86b62942f52d8d2b20fb786df11bca5516820771"}
Sep 16 18:44:49.399325 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:49.399275    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-4b825" podStartSLOduration=4.431038203 podStartE2EDuration="5.399259316s" totalImagesPullingTime="904.770813ms" totalInitContainerRuntime="63.4503ms" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:49.398125023 +0000 UTC m=+93.039840688" watchObservedRunningTime="2026-09-16 18:44:49.399259316 +0000 UTC m=+93.040974979"
Sep 16 18:44:54.259178 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:54.259145    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-dgl4n"
Sep 16 18:44:54.818443 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:54.818398    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ba5d0219-c94f-41a7-a6fc-86b14d2bccd2-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-l56nd\" (UID: \"ba5d0219-c94f-41a7-a6fc-86b14d2bccd2\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-l56nd"
Sep 16 18:44:54.820915 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:54.820893    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ba5d0219-c94f-41a7-a6fc-86b14d2bccd2-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-l56nd\" (UID: \"ba5d0219-c94f-41a7-a6fc-86b14d2bccd2\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-l56nd"
Sep 16 18:44:55.004604 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:55.004570    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-vzfhx\""
Sep 16 18:44:55.012930 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:55.012908    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-l56nd"
Sep 16 18:44:55.284502 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:55.284475    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-l56nd"]
Sep 16 18:44:55.287141 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:44:55.287111    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba5d0219_c94f_41a7_a6fc_86b14d2bccd2.slice/crio-b5d14feb25e721ce07044d9813efb2b33dc60d768b6375bb663d56fe8142f40e WatchSource:0}: Error finding container b5d14feb25e721ce07044d9813efb2b33dc60d768b6375bb663d56fe8142f40e: Status 404 returned error can't find the container with id b5d14feb25e721ce07044d9813efb2b33dc60d768b6375bb663d56fe8142f40e
Sep 16 18:44:55.400990 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:55.400958    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-l56nd" event={"ID":"ba5d0219-c94f-41a7-a6fc-86b14d2bccd2","Type":"ContainerStarted","Data":"b5d14feb25e721ce07044d9813efb2b33dc60d768b6375bb663d56fe8142f40e"}
Sep 16 18:44:56.407179 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:56.407139    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-l56nd" event={"ID":"ba5d0219-c94f-41a7-a6fc-86b14d2bccd2","Type":"ContainerStarted","Data":"b50d75e0ad0095a5ad7fa0298234c0ae42dbb1c8a721f4ed457a048f8896c36e"}
Sep 16 18:44:56.434307 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:56.434242    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-l56nd" podStartSLOduration=90.41169295 podStartE2EDuration="1m31.434208382s" totalImagesPullingTime="1.022515432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:43:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:56.433843843 +0000 UTC m=+100.075559508" watchObservedRunningTime="2026-09-16 18:44:56.434208382 +0000 UTC m=+100.075924051"
Sep 16 18:44:58.831771 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:44:58.831731    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-79f6f8f56-gk2mv"]
Sep 16 18:45:13.469670 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:13.469635    2582 generic.go:356] "Generic (PLEG): container finished" podID="fa83f1c9-b7b9-4c1e-b0c4-09328df33051" containerID="3c3222678135bbb17bbd1315af43eccb59ad34728c2698c1595f372509eda7a1" exitCode=0
Sep 16 18:45:13.470097 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:13.469693    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5znf9" event={"ID":"fa83f1c9-b7b9-4c1e-b0c4-09328df33051","Type":"ContainerDied","Data":"3c3222678135bbb17bbd1315af43eccb59ad34728c2698c1595f372509eda7a1"}
Sep 16 18:45:13.470097 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:13.470034    2582 scope.go:117] "RemoveContainer" containerID="3c3222678135bbb17bbd1315af43eccb59ad34728c2698c1595f372509eda7a1"
Sep 16 18:45:14.474918 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:14.474884    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5znf9" event={"ID":"fa83f1c9-b7b9-4c1e-b0c4-09328df33051","Type":"ContainerStarted","Data":"dea505eb83abf41ec1f642d48775bcc43263c26d446306ae0cbdec7dedf5b92f"}
Sep 16 18:45:17.487352 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:17.487317    2582 generic.go:356] "Generic (PLEG): container finished" podID="ed1b0dcc-4b23-4906-bc1d-4efa04e932cf" containerID="88023fa3e8dafbd7a0671d1961781602605d104f5e9e4b2e959b7c1223409fc7" exitCode=0
Sep 16 18:45:17.487743 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:17.487376    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-h28sc" event={"ID":"ed1b0dcc-4b23-4906-bc1d-4efa04e932cf","Type":"ContainerDied","Data":"88023fa3e8dafbd7a0671d1961781602605d104f5e9e4b2e959b7c1223409fc7"}
Sep 16 18:45:17.487743 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:17.487713    2582 scope.go:117] "RemoveContainer" containerID="88023fa3e8dafbd7a0671d1961781602605d104f5e9e4b2e959b7c1223409fc7"
Sep 16 18:45:18.492745 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:18.492698    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-h28sc" event={"ID":"ed1b0dcc-4b23-4906-bc1d-4efa04e932cf","Type":"ContainerStarted","Data":"31096feeab496d2d5c2f94bc5d6e732ea6978a4318d6e8c5f699ff16101ce8da"}
Sep 16 18:45:21.268251 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:21.268213    2582 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-674c5bb56f-6xn2v" podUID="ff403ffb-51c2-45d7-b90f-cf1ac24054d6" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 16 18:45:23.851815 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:23.851771    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-79f6f8f56-gk2mv" podUID="40a3ba8a-40a8-48eb-99df-da46c59ee997" containerName="registry" containerID="cri-o://b4a32a934952c08fe8599e9037c4f680895e9a4a17d2a6bdcd159669a8ede32d" gracePeriod=30
Sep 16 18:45:24.097191 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:24.097166    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79f6f8f56-gk2mv"
Sep 16 18:45:24.180451 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:24.180358    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/40a3ba8a-40a8-48eb-99df-da46c59ee997-registry-certificates\") pod \"40a3ba8a-40a8-48eb-99df-da46c59ee997\" (UID: \"40a3ba8a-40a8-48eb-99df-da46c59ee997\") "
Sep 16 18:45:24.180451 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:24.180397    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/40a3ba8a-40a8-48eb-99df-da46c59ee997-installation-pull-secrets\") pod \"40a3ba8a-40a8-48eb-99df-da46c59ee997\" (UID: \"40a3ba8a-40a8-48eb-99df-da46c59ee997\") "
Sep 16 18:45:24.180451 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:24.180426    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vcdvf\" (UniqueName: \"kubernetes.io/projected/40a3ba8a-40a8-48eb-99df-da46c59ee997-kube-api-access-vcdvf\") pod \"40a3ba8a-40a8-48eb-99df-da46c59ee997\" (UID: \"40a3ba8a-40a8-48eb-99df-da46c59ee997\") "
Sep 16 18:45:24.180451 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:24.180443    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/40a3ba8a-40a8-48eb-99df-da46c59ee997-trusted-ca\") pod \"40a3ba8a-40a8-48eb-99df-da46c59ee997\" (UID: \"40a3ba8a-40a8-48eb-99df-da46c59ee997\") "
Sep 16 18:45:24.180757 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:24.180521    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/40a3ba8a-40a8-48eb-99df-da46c59ee997-bound-sa-token\") pod \"40a3ba8a-40a8-48eb-99df-da46c59ee997\" (UID: \"40a3ba8a-40a8-48eb-99df-da46c59ee997\") "
Sep 16 18:45:24.180757 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:24.180548    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/40a3ba8a-40a8-48eb-99df-da46c59ee997-image-registry-private-configuration\") pod \"40a3ba8a-40a8-48eb-99df-da46c59ee997\" (UID: \"40a3ba8a-40a8-48eb-99df-da46c59ee997\") "
Sep 16 18:45:24.180757 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:24.180585    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/40a3ba8a-40a8-48eb-99df-da46c59ee997-ca-trust-extracted\") pod \"40a3ba8a-40a8-48eb-99df-da46c59ee997\" (UID: \"40a3ba8a-40a8-48eb-99df-da46c59ee997\") "
Sep 16 18:45:24.180757 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:24.180619    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40a3ba8a-40a8-48eb-99df-da46c59ee997-registry-tls\") pod \"40a3ba8a-40a8-48eb-99df-da46c59ee997\" (UID: \"40a3ba8a-40a8-48eb-99df-da46c59ee997\") "
Sep 16 18:45:24.180962 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:24.180915    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40a3ba8a-40a8-48eb-99df-da46c59ee997-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "40a3ba8a-40a8-48eb-99df-da46c59ee997" (UID: "40a3ba8a-40a8-48eb-99df-da46c59ee997"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:45:24.181017 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:24.180968    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40a3ba8a-40a8-48eb-99df-da46c59ee997-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "40a3ba8a-40a8-48eb-99df-da46c59ee997" (UID: "40a3ba8a-40a8-48eb-99df-da46c59ee997"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:45:24.183126 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:24.183080    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40a3ba8a-40a8-48eb-99df-da46c59ee997-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "40a3ba8a-40a8-48eb-99df-da46c59ee997" (UID: "40a3ba8a-40a8-48eb-99df-da46c59ee997"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:45:24.183245 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:24.183166    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40a3ba8a-40a8-48eb-99df-da46c59ee997-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "40a3ba8a-40a8-48eb-99df-da46c59ee997" (UID: "40a3ba8a-40a8-48eb-99df-da46c59ee997"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:45:24.183316 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:24.183247    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40a3ba8a-40a8-48eb-99df-da46c59ee997-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "40a3ba8a-40a8-48eb-99df-da46c59ee997" (UID: "40a3ba8a-40a8-48eb-99df-da46c59ee997"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:45:24.183367 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:24.183321    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40a3ba8a-40a8-48eb-99df-da46c59ee997-kube-api-access-vcdvf" (OuterVolumeSpecName: "kube-api-access-vcdvf") pod "40a3ba8a-40a8-48eb-99df-da46c59ee997" (UID: "40a3ba8a-40a8-48eb-99df-da46c59ee997"). InnerVolumeSpecName "kube-api-access-vcdvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:45:24.183367 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:24.183351    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40a3ba8a-40a8-48eb-99df-da46c59ee997-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "40a3ba8a-40a8-48eb-99df-da46c59ee997" (UID: "40a3ba8a-40a8-48eb-99df-da46c59ee997"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:45:24.189076 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:24.189046    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40a3ba8a-40a8-48eb-99df-da46c59ee997-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "40a3ba8a-40a8-48eb-99df-da46c59ee997" (UID: "40a3ba8a-40a8-48eb-99df-da46c59ee997"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:45:24.281642 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:24.281600    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcdvf\" (UniqueName: \"kubernetes.io/projected/40a3ba8a-40a8-48eb-99df-da46c59ee997-kube-api-access-vcdvf\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 18:45:24.281642 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:24.281635    2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/40a3ba8a-40a8-48eb-99df-da46c59ee997-trusted-ca\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 18:45:24.281642 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:24.281646    2582 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/40a3ba8a-40a8-48eb-99df-da46c59ee997-bound-sa-token\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 18:45:24.281900 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:24.281656    2582 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/40a3ba8a-40a8-48eb-99df-da46c59ee997-image-registry-private-configuration\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 18:45:24.281900 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:24.281666    2582 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/40a3ba8a-40a8-48eb-99df-da46c59ee997-ca-trust-extracted\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 18:45:24.281900 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:24.281674    2582 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40a3ba8a-40a8-48eb-99df-da46c59ee997-registry-tls\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 18:45:24.281900 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:24.281682    2582 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/40a3ba8a-40a8-48eb-99df-da46c59ee997-registry-certificates\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 18:45:24.281900 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:24.281690    2582 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/40a3ba8a-40a8-48eb-99df-da46c59ee997-installation-pull-secrets\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 18:45:24.517402 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:24.517365    2582 generic.go:356] "Generic (PLEG): container finished" podID="40a3ba8a-40a8-48eb-99df-da46c59ee997" containerID="b4a32a934952c08fe8599e9037c4f680895e9a4a17d2a6bdcd159669a8ede32d" exitCode=0
Sep 16 18:45:24.517562 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:24.517419    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79f6f8f56-gk2mv"
Sep 16 18:45:24.517562 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:24.517454    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79f6f8f56-gk2mv" event={"ID":"40a3ba8a-40a8-48eb-99df-da46c59ee997","Type":"ContainerDied","Data":"b4a32a934952c08fe8599e9037c4f680895e9a4a17d2a6bdcd159669a8ede32d"}
Sep 16 18:45:24.517562 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:24.517508    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79f6f8f56-gk2mv" event={"ID":"40a3ba8a-40a8-48eb-99df-da46c59ee997","Type":"ContainerDied","Data":"e5747062bf111283b3c6549c070118c00277e730cb6c16e6c6a0fb8d3444a2f4"}
Sep 16 18:45:24.517562 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:24.517533    2582 scope.go:117] "RemoveContainer" containerID="b4a32a934952c08fe8599e9037c4f680895e9a4a17d2a6bdcd159669a8ede32d"
Sep 16 18:45:24.526157 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:24.526141    2582 scope.go:117] "RemoveContainer" containerID="b4a32a934952c08fe8599e9037c4f680895e9a4a17d2a6bdcd159669a8ede32d"
Sep 16 18:45:24.526470 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:45:24.526447    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4a32a934952c08fe8599e9037c4f680895e9a4a17d2a6bdcd159669a8ede32d\": container with ID starting with b4a32a934952c08fe8599e9037c4f680895e9a4a17d2a6bdcd159669a8ede32d not found: ID does not exist" containerID="b4a32a934952c08fe8599e9037c4f680895e9a4a17d2a6bdcd159669a8ede32d"
Sep 16 18:45:24.526575 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:24.526477    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4a32a934952c08fe8599e9037c4f680895e9a4a17d2a6bdcd159669a8ede32d"} err="failed to get container status \"b4a32a934952c08fe8599e9037c4f680895e9a4a17d2a6bdcd159669a8ede32d\": rpc error: code = NotFound desc = could not find container \"b4a32a934952c08fe8599e9037c4f680895e9a4a17d2a6bdcd159669a8ede32d\": container with ID starting with b4a32a934952c08fe8599e9037c4f680895e9a4a17d2a6bdcd159669a8ede32d not found: ID does not exist"
Sep 16 18:45:24.539773 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:24.539748    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-79f6f8f56-gk2mv"]
Sep 16 18:45:24.544630 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:24.544603    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-79f6f8f56-gk2mv"]
Sep 16 18:45:24.872461 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:24.872385    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40a3ba8a-40a8-48eb-99df-da46c59ee997" path="/var/lib/kubelet/pods/40a3ba8a-40a8-48eb-99df-da46c59ee997/volumes"
Sep 16 18:45:26.601608 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:26.601565    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de8ddfa3-fd54-49d7-8f90-035aa4439e98-metrics-certs\") pod \"network-metrics-daemon-g7q9m\" (UID: \"de8ddfa3-fd54-49d7-8f90-035aa4439e98\") " pod="openshift-multus/network-metrics-daemon-g7q9m"
Sep 16 18:45:26.603869 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:26.603848    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de8ddfa3-fd54-49d7-8f90-035aa4439e98-metrics-certs\") pod \"network-metrics-daemon-g7q9m\" (UID: \"de8ddfa3-fd54-49d7-8f90-035aa4439e98\") " pod="openshift-multus/network-metrics-daemon-g7q9m"
Sep 16 18:45:26.681333 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:26.681296    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mz62r\""
Sep 16 18:45:26.689101 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:26.689081    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g7q9m"
Sep 16 18:45:26.965345 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:26.965253    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-g7q9m"]
Sep 16 18:45:26.968319 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:45:26.968276    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde8ddfa3_fd54_49d7_8f90_035aa4439e98.slice/crio-692ee7a829153bfc70cb1ba62abdc40d2d19be5f3b65df1688ad4b3ff6181f7e WatchSource:0}: Error finding container 692ee7a829153bfc70cb1ba62abdc40d2d19be5f3b65df1688ad4b3ff6181f7e: Status 404 returned error can't find the container with id 692ee7a829153bfc70cb1ba62abdc40d2d19be5f3b65df1688ad4b3ff6181f7e
Sep 16 18:45:27.529767 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:27.529712    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g7q9m" event={"ID":"de8ddfa3-fd54-49d7-8f90-035aa4439e98","Type":"ContainerStarted","Data":"692ee7a829153bfc70cb1ba62abdc40d2d19be5f3b65df1688ad4b3ff6181f7e"}
Sep 16 18:45:28.536873 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:28.536838    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g7q9m" event={"ID":"de8ddfa3-fd54-49d7-8f90-035aa4439e98","Type":"ContainerStarted","Data":"0d83ec9f036372a9a41e6394d87ea82fb0bb6bc7b7c41cadc6ce6ede0fe9291c"}
Sep 16 18:45:28.537330 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:28.536899    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g7q9m" event={"ID":"de8ddfa3-fd54-49d7-8f90-035aa4439e98","Type":"ContainerStarted","Data":"421212fb2bb56eb0292d48b46b44c61158b491d30055a5f9eacd94637e32f6d9"}
Sep 16 18:45:28.557512 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:28.557449    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-g7q9m" podStartSLOduration=130.599011308 podStartE2EDuration="2m11.557426646s" totalImagesPullingTime="958.415338ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:43:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:28.555500274 +0000 UTC m=+132.197215939" watchObservedRunningTime="2026-09-16 18:45:28.557426646 +0000 UTC m=+132.199142311"
Sep 16 18:45:31.268641 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:31.268595    2582 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-674c5bb56f-6xn2v" podUID="ff403ffb-51c2-45d7-b90f-cf1ac24054d6" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 16 18:45:32.557111 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:32.557078    2582 generic.go:356] "Generic (PLEG): container finished" podID="c1f25728-6d57-4c30-9d73-d90c127bd06c" containerID="4e666d7b7e14afffa9877f121d34c82aa66e3d344363b4459faa219fa9984b75" exitCode=0
Sep 16 18:45:32.557444 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:32.557149    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-nccjc" event={"ID":"c1f25728-6d57-4c30-9d73-d90c127bd06c","Type":"ContainerDied","Data":"4e666d7b7e14afffa9877f121d34c82aa66e3d344363b4459faa219fa9984b75"}
Sep 16 18:45:32.557499 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:32.557467    2582 scope.go:117] "RemoveContainer" containerID="4e666d7b7e14afffa9877f121d34c82aa66e3d344363b4459faa219fa9984b75"
Sep 16 18:45:33.562715 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:33.562681    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-nccjc" event={"ID":"c1f25728-6d57-4c30-9d73-d90c127bd06c","Type":"ContainerStarted","Data":"11382538d63fa192fe41e62785da3db16b66bbf63123c932de1e90cda8d119c1"}
Sep 16 18:45:41.267967 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:41.267923    2582 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-674c5bb56f-6xn2v" podUID="ff403ffb-51c2-45d7-b90f-cf1ac24054d6" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 16 18:45:41.268341 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:41.267996    2582 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-674c5bb56f-6xn2v"
Sep 16 18:45:41.268468 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:41.268452    2582 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"9a847b6bc74893a5654413b8273df6265a27009fcc780be4b349f348f789d83e"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-674c5bb56f-6xn2v" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 16 18:45:41.268506 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:41.268487    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-674c5bb56f-6xn2v" podUID="ff403ffb-51c2-45d7-b90f-cf1ac24054d6" containerName="service-proxy" containerID="cri-o://9a847b6bc74893a5654413b8273df6265a27009fcc780be4b349f348f789d83e" gracePeriod=30
Sep 16 18:45:41.593698 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:41.593614    2582 generic.go:356] "Generic (PLEG): container finished" podID="ff403ffb-51c2-45d7-b90f-cf1ac24054d6" containerID="9a847b6bc74893a5654413b8273df6265a27009fcc780be4b349f348f789d83e" exitCode=2
Sep 16 18:45:41.593878 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:41.593689    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-674c5bb56f-6xn2v" event={"ID":"ff403ffb-51c2-45d7-b90f-cf1ac24054d6","Type":"ContainerDied","Data":"9a847b6bc74893a5654413b8273df6265a27009fcc780be4b349f348f789d83e"}
Sep 16 18:45:41.593878 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:45:41.593766    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-674c5bb56f-6xn2v" event={"ID":"ff403ffb-51c2-45d7-b90f-cf1ac24054d6","Type":"ContainerStarted","Data":"9af673ce7523af5e7ae993a275d0b34a73d7dd6b1dbf620f6c8b02bebbb934a0"}
Sep 16 18:48:16.774688 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:48:16.774656    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7qwl2_b6695f86-08c0-4836-87b1-f2a9d76084bd/console-operator/1.log"
Sep 16 18:48:16.781489 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:48:16.781460    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7qwl2_b6695f86-08c0-4836-87b1-f2a9d76084bd/console-operator/1.log"
Sep 16 18:48:16.786692 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:48:16.786672    2582 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 18:49:30.589242 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:30.589205    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt"]
Sep 16 18:49:30.589687 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:30.589618    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40a3ba8a-40a8-48eb-99df-da46c59ee997" containerName="registry"
Sep 16 18:49:30.589687 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:30.589636    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="40a3ba8a-40a8-48eb-99df-da46c59ee997" containerName="registry"
Sep 16 18:49:30.589786 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:30.589756    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="40a3ba8a-40a8-48eb-99df-da46c59ee997" containerName="registry"
Sep 16 18:49:30.592785 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:30.592764    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt"
Sep 16 18:49:30.597632 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:30.597606    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-ln62w\""
Sep 16 18:49:30.597770 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:30.597607    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 16 18:49:30.616729 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:30.616689    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt"]
Sep 16 18:49:30.699068 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:30.699033    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/949c2e66-950f-480a-96a3-16c9e4be467b-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt\" (UID: \"949c2e66-950f-480a-96a3-16c9e4be467b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt"
Sep 16 18:49:30.699068 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:30.699073    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/949c2e66-950f-480a-96a3-16c9e4be467b-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt\" (UID: \"949c2e66-950f-480a-96a3-16c9e4be467b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt"
Sep 16 18:49:30.699297 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:30.699094    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqz9b\" (UniqueName: \"kubernetes.io/projected/949c2e66-950f-480a-96a3-16c9e4be467b-kube-api-access-nqz9b\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt\" (UID: \"949c2e66-950f-480a-96a3-16c9e4be467b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt"
Sep 16 18:49:30.699297 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:30.699138    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/949c2e66-950f-480a-96a3-16c9e4be467b-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt\" (UID: \"949c2e66-950f-480a-96a3-16c9e4be467b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt"
Sep 16 18:49:30.699297 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:30.699156    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/949c2e66-950f-480a-96a3-16c9e4be467b-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt\" (UID: \"949c2e66-950f-480a-96a3-16c9e4be467b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt"
Sep 16 18:49:30.699297 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:30.699179    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/949c2e66-950f-480a-96a3-16c9e4be467b-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt\" (UID: \"949c2e66-950f-480a-96a3-16c9e4be467b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt"
Sep 16 18:49:30.699297 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:30.699255    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/949c2e66-950f-480a-96a3-16c9e4be467b-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt\" (UID: \"949c2e66-950f-480a-96a3-16c9e4be467b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt"
Sep 16 18:49:30.699297 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:30.699296    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/949c2e66-950f-480a-96a3-16c9e4be467b-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt\" (UID: \"949c2e66-950f-480a-96a3-16c9e4be467b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt"
Sep 16 18:49:30.699490 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:30.699321    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/949c2e66-950f-480a-96a3-16c9e4be467b-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt\" (UID: \"949c2e66-950f-480a-96a3-16c9e4be467b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt"
Sep 16 18:49:30.799705 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:30.799670    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/949c2e66-950f-480a-96a3-16c9e4be467b-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt\" (UID: \"949c2e66-950f-480a-96a3-16c9e4be467b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt"
Sep 16 18:49:30.799843 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:30.799710    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/949c2e66-950f-480a-96a3-16c9e4be467b-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt\" (UID: \"949c2e66-950f-480a-96a3-16c9e4be467b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt"
Sep 16 18:49:30.799843 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:30.799754    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqz9b\" (UniqueName: \"kubernetes.io/projected/949c2e66-950f-480a-96a3-16c9e4be467b-kube-api-access-nqz9b\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt\" (UID: \"949c2e66-950f-480a-96a3-16c9e4be467b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt"
Sep 16 18:49:30.799843 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:30.799789    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/949c2e66-950f-480a-96a3-16c9e4be467b-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt\" (UID: \"949c2e66-950f-480a-96a3-16c9e4be467b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt"
Sep 16 18:49:30.799843 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:30.799811    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/949c2e66-950f-480a-96a3-16c9e4be467b-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt\" (UID: \"949c2e66-950f-480a-96a3-16c9e4be467b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt"
Sep 16 18:49:30.799843 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:30.799836    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/949c2e66-950f-480a-96a3-16c9e4be467b-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt\" (UID: \"949c2e66-950f-480a-96a3-16c9e4be467b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt"
Sep 16 18:49:30.800105 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:30.799858    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/949c2e66-950f-480a-96a3-16c9e4be467b-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt\" (UID: \"949c2e66-950f-480a-96a3-16c9e4be467b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt"
Sep 16 18:49:30.800105 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:30.799905    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/949c2e66-950f-480a-96a3-16c9e4be467b-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt\" (UID: \"949c2e66-950f-480a-96a3-16c9e4be467b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt"
Sep 16 18:49:30.800105 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:30.799945    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/949c2e66-950f-480a-96a3-16c9e4be467b-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt\" (UID: \"949c2e66-950f-480a-96a3-16c9e4be467b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt"
Sep 16 18:49:30.800256 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:30.800115    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/949c2e66-950f-480a-96a3-16c9e4be467b-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt\" (UID: \"949c2e66-950f-480a-96a3-16c9e4be467b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt"
Sep 16 18:49:30.800256 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:30.800140    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/949c2e66-950f-480a-96a3-16c9e4be467b-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt\" (UID: \"949c2e66-950f-480a-96a3-16c9e4be467b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt"
Sep 16 18:49:30.800256 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:30.800216    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/949c2e66-950f-480a-96a3-16c9e4be467b-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt\" (UID: \"949c2e66-950f-480a-96a3-16c9e4be467b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt"
Sep 16 18:49:30.800418 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:30.800274    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/949c2e66-950f-480a-96a3-16c9e4be467b-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt\" (UID: \"949c2e66-950f-480a-96a3-16c9e4be467b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt"
Sep 16 18:49:30.800656 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:30.800633    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/949c2e66-950f-480a-96a3-16c9e4be467b-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt\" (UID: \"949c2e66-950f-480a-96a3-16c9e4be467b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt"
Sep 16 18:49:30.802055 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:30.802035    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/949c2e66-950f-480a-96a3-16c9e4be467b-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt\" (UID: \"949c2e66-950f-480a-96a3-16c9e4be467b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt"
Sep 16 18:49:30.802200 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:30.802183    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/949c2e66-950f-480a-96a3-16c9e4be467b-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt\" (UID: \"949c2e66-950f-480a-96a3-16c9e4be467b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt"
Sep 16 18:49:30.811026 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:30.811002    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/949c2e66-950f-480a-96a3-16c9e4be467b-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt\" (UID: \"949c2e66-950f-480a-96a3-16c9e4be467b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt"
Sep 16 18:49:30.811208 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:30.811189    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqz9b\" (UniqueName: \"kubernetes.io/projected/949c2e66-950f-480a-96a3-16c9e4be467b-kube-api-access-nqz9b\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt\" (UID: \"949c2e66-950f-480a-96a3-16c9e4be467b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt"
Sep 16 18:49:30.904106 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:30.904027    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt"
Sep 16 18:49:31.181935 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:31.181909    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt"]
Sep 16 18:49:31.184502 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:49:31.184470    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod949c2e66_950f_480a_96a3_16c9e4be467b.slice/crio-680d5cf678d24daedef3c63aa8b5dfd058c8a61d68925055dd6cdf2fd5a7116e WatchSource:0}: Error finding container 680d5cf678d24daedef3c63aa8b5dfd058c8a61d68925055dd6cdf2fd5a7116e: Status 404 returned error can't find the container with id 680d5cf678d24daedef3c63aa8b5dfd058c8a61d68925055dd6cdf2fd5a7116e
Sep 16 18:49:31.186617 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:31.186596    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:49:31.451205 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:31.451117    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt" event={"ID":"949c2e66-950f-480a-96a3-16c9e4be467b","Type":"ContainerStarted","Data":"680d5cf678d24daedef3c63aa8b5dfd058c8a61d68925055dd6cdf2fd5a7116e"}
Sep 16 18:49:37.152797 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:37.152753    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-77db69dd7c-4g645"]
Sep 16 18:49:37.157297 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:37.157271    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-77db69dd7c-4g645"
Sep 16 18:49:37.160785 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:37.160758    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-hjk7s\""
Sep 16 18:49:37.161916 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:37.161891    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 16 18:49:37.161916 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:37.161906    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:49:37.162150 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:37.161961    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 16 18:49:37.162150 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:37.161906    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 16 18:49:37.162150 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:37.161906    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 16 18:49:37.174164 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:37.174122    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-77db69dd7c-4g645"]
Sep 16 18:49:37.254631 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:37.254589    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5fdaf830-2e71-4373-aa18-0bcca69ea3e4-cert\") pod \"lws-controller-manager-77db69dd7c-4g645\" (UID: \"5fdaf830-2e71-4373-aa18-0bcca69ea3e4\") " pod="openshift-lws-operator/lws-controller-manager-77db69dd7c-4g645"
Sep 16 18:49:37.254826 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:37.254635    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5fdaf830-2e71-4373-aa18-0bcca69ea3e4-metrics-cert\") pod \"lws-controller-manager-77db69dd7c-4g645\" (UID: \"5fdaf830-2e71-4373-aa18-0bcca69ea3e4\") " pod="openshift-lws-operator/lws-controller-manager-77db69dd7c-4g645"
Sep 16 18:49:37.254826 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:37.254787    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5fdaf830-2e71-4373-aa18-0bcca69ea3e4-manager-config\") pod \"lws-controller-manager-77db69dd7c-4g645\" (UID: \"5fdaf830-2e71-4373-aa18-0bcca69ea3e4\") " pod="openshift-lws-operator/lws-controller-manager-77db69dd7c-4g645"
Sep 16 18:49:37.254826 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:37.254805    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z828z\" (UniqueName: \"kubernetes.io/projected/5fdaf830-2e71-4373-aa18-0bcca69ea3e4-kube-api-access-z828z\") pod \"lws-controller-manager-77db69dd7c-4g645\" (UID: \"5fdaf830-2e71-4373-aa18-0bcca69ea3e4\") " pod="openshift-lws-operator/lws-controller-manager-77db69dd7c-4g645"
Sep 16 18:49:37.355674 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:37.355634    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5fdaf830-2e71-4373-aa18-0bcca69ea3e4-metrics-cert\") pod \"lws-controller-manager-77db69dd7c-4g645\" (UID: \"5fdaf830-2e71-4373-aa18-0bcca69ea3e4\") " pod="openshift-lws-operator/lws-controller-manager-77db69dd7c-4g645"
Sep 16 18:49:37.355674 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:37.355679    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5fdaf830-2e71-4373-aa18-0bcca69ea3e4-manager-config\") pod \"lws-controller-manager-77db69dd7c-4g645\" (UID: \"5fdaf830-2e71-4373-aa18-0bcca69ea3e4\") " pod="openshift-lws-operator/lws-controller-manager-77db69dd7c-4g645"
Sep 16 18:49:37.355919 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:37.355699    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z828z\" (UniqueName: \"kubernetes.io/projected/5fdaf830-2e71-4373-aa18-0bcca69ea3e4-kube-api-access-z828z\") pod \"lws-controller-manager-77db69dd7c-4g645\" (UID: \"5fdaf830-2e71-4373-aa18-0bcca69ea3e4\") " pod="openshift-lws-operator/lws-controller-manager-77db69dd7c-4g645"
Sep 16 18:49:37.355919 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:37.355787    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5fdaf830-2e71-4373-aa18-0bcca69ea3e4-cert\") pod \"lws-controller-manager-77db69dd7c-4g645\" (UID: \"5fdaf830-2e71-4373-aa18-0bcca69ea3e4\") " pod="openshift-lws-operator/lws-controller-manager-77db69dd7c-4g645"
Sep 16 18:49:37.356426 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:37.356402    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5fdaf830-2e71-4373-aa18-0bcca69ea3e4-manager-config\") pod \"lws-controller-manager-77db69dd7c-4g645\" (UID: \"5fdaf830-2e71-4373-aa18-0bcca69ea3e4\") " pod="openshift-lws-operator/lws-controller-manager-77db69dd7c-4g645"
Sep 16 18:49:37.358338 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:37.358314    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5fdaf830-2e71-4373-aa18-0bcca69ea3e4-metrics-cert\") pod \"lws-controller-manager-77db69dd7c-4g645\" (UID: \"5fdaf830-2e71-4373-aa18-0bcca69ea3e4\") " pod="openshift-lws-operator/lws-controller-manager-77db69dd7c-4g645"
Sep 16 18:49:37.358424 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:37.358367    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5fdaf830-2e71-4373-aa18-0bcca69ea3e4-cert\") pod \"lws-controller-manager-77db69dd7c-4g645\" (UID: \"5fdaf830-2e71-4373-aa18-0bcca69ea3e4\") " pod="openshift-lws-operator/lws-controller-manager-77db69dd7c-4g645"
Sep 16 18:49:37.364903 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:37.364877    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z828z\" (UniqueName: \"kubernetes.io/projected/5fdaf830-2e71-4373-aa18-0bcca69ea3e4-kube-api-access-z828z\") pod \"lws-controller-manager-77db69dd7c-4g645\" (UID: \"5fdaf830-2e71-4373-aa18-0bcca69ea3e4\") " pod="openshift-lws-operator/lws-controller-manager-77db69dd7c-4g645"
Sep 16 18:49:37.467109 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:37.467006    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-77db69dd7c-4g645"
Sep 16 18:49:37.763509 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:37.763483    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-77db69dd7c-4g645"]
Sep 16 18:49:37.763966 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:49:37.763938    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fdaf830_2e71_4373_aa18_0bcca69ea3e4.slice/crio-a9b53ac540a93f0c3809343b882931166fab21b668519175834fff48de418ef8 WatchSource:0}: Error finding container a9b53ac540a93f0c3809343b882931166fab21b668519175834fff48de418ef8: Status 404 returned error can't find the container with id a9b53ac540a93f0c3809343b882931166fab21b668519175834fff48de418ef8
Sep 16 18:49:38.483872 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:38.483833    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-77db69dd7c-4g645" event={"ID":"5fdaf830-2e71-4373-aa18-0bcca69ea3e4","Type":"ContainerStarted","Data":"a9b53ac540a93f0c3809343b882931166fab21b668519175834fff48de418ef8"}
Sep 16 18:49:40.496620 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:40.496528    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-77db69dd7c-4g645" event={"ID":"5fdaf830-2e71-4373-aa18-0bcca69ea3e4","Type":"ContainerStarted","Data":"0217f975ebe32989e817d8e0df9cb7eaba3587c631941209cc1ec6d264ba7d5a"}
Sep 16 18:49:40.497019 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:40.496657    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-77db69dd7c-4g645"
Sep 16 18:49:40.536795 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:40.536747    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-77db69dd7c-4g645" podStartSLOduration=1.2537235629999999 podStartE2EDuration="3.536714693s" totalImagesPullingTime="2.28299113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:49:40.536104916 +0000 UTC m=+384.177820581" watchObservedRunningTime="2026-09-16 18:49:40.536714693 +0000 UTC m=+384.178430357"
Sep 16 18:49:44.580873 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:44.580826    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:49:44.581146 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:44.580906    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:49:44.581146 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:44.580934    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:49:45.522303 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:45.522268    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt" event={"ID":"949c2e66-950f-480a-96a3-16c9e4be467b","Type":"ContainerStarted","Data":"c7f227d8f2208951d8a93ef8000bb0367d191c9c120d98c07b667088b903fa67"}
Sep 16 18:49:45.549945 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:45.549898    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt" podStartSLOduration=2.156099401 podStartE2EDuration="15.549881899s" totalImagesPullingTime="13.393782498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:49:45.549189685 +0000 UTC m=+389.190905348" watchObservedRunningTime="2026-09-16 18:49:45.549881899 +0000 UTC m=+389.191597613"
Sep 16 18:49:45.904650 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:45.904551    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt"
Sep 16 18:49:45.904650 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:45.904627    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt"
Sep 16 18:49:45.909703 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:45.909678    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt"
Sep 16 18:49:46.527119 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:46.527088    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt"
Sep 16 18:49:51.502093 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:49:51.502062    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-77db69dd7c-4g645"
Sep 16 18:50:59.850839 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:50:59.850804    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w8w2f"]
Sep 16 18:50:59.853974 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:50:59.853955    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-w8w2f"
Sep 16 18:50:59.857570 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:50:59.857551    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 16 18:50:59.857844 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:50:59.857822    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-zj67w\""
Sep 16 18:50:59.858933 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:50:59.858914    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 16 18:50:59.859044 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:50:59.858914    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 16 18:50:59.870633 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:50:59.870606    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w8w2f"]
Sep 16 18:50:59.899768 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:50:59.899733    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w8w2f"]
Sep 16 18:50:59.917773 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:50:59.917714    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znplg\" (UniqueName: \"kubernetes.io/projected/9d2b060c-f303-4dca-945c-42b6ef7a2b09-kube-api-access-znplg\") pod \"limitador-limitador-84994f7ffc-w8w2f\" (UID: \"9d2b060c-f303-4dca-945c-42b6ef7a2b09\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-w8w2f"
Sep 16 18:50:59.917940 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:50:59.917810    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9d2b060c-f303-4dca-945c-42b6ef7a2b09-config-file\") pod \"limitador-limitador-84994f7ffc-w8w2f\" (UID: \"9d2b060c-f303-4dca-945c-42b6ef7a2b09\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-w8w2f"
Sep 16 18:51:00.019231 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:51:00.019168    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9d2b060c-f303-4dca-945c-42b6ef7a2b09-config-file\") pod \"limitador-limitador-84994f7ffc-w8w2f\" (UID: \"9d2b060c-f303-4dca-945c-42b6ef7a2b09\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-w8w2f"
Sep 16 18:51:00.019439 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:51:00.019309    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znplg\" (UniqueName: \"kubernetes.io/projected/9d2b060c-f303-4dca-945c-42b6ef7a2b09-kube-api-access-znplg\") pod \"limitador-limitador-84994f7ffc-w8w2f\" (UID: \"9d2b060c-f303-4dca-945c-42b6ef7a2b09\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-w8w2f"
Sep 16 18:51:00.019836 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:51:00.019814    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9d2b060c-f303-4dca-945c-42b6ef7a2b09-config-file\") pod \"limitador-limitador-84994f7ffc-w8w2f\" (UID: \"9d2b060c-f303-4dca-945c-42b6ef7a2b09\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-w8w2f"
Sep 16 18:51:00.030085 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:51:00.030057    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znplg\" (UniqueName: \"kubernetes.io/projected/9d2b060c-f303-4dca-945c-42b6ef7a2b09-kube-api-access-znplg\") pod \"limitador-limitador-84994f7ffc-w8w2f\" (UID: \"9d2b060c-f303-4dca-945c-42b6ef7a2b09\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-w8w2f"
Sep 16 18:51:00.164330 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:51:00.164227    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-w8w2f"
Sep 16 18:51:00.439906 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:51:00.439879    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w8w2f"]
Sep 16 18:51:00.442452 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:51:00.442420    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d2b060c_f303_4dca_945c_42b6ef7a2b09.slice/crio-2357531b38475fec7a01dad0d240a510869ce0e817490d5e1ef710b7aa3db57c WatchSource:0}: Error finding container 2357531b38475fec7a01dad0d240a510869ce0e817490d5e1ef710b7aa3db57c: Status 404 returned error can't find the container with id 2357531b38475fec7a01dad0d240a510869ce0e817490d5e1ef710b7aa3db57c
Sep 16 18:51:00.809465 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:51:00.809433    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-w8w2f" event={"ID":"9d2b060c-f303-4dca-945c-42b6ef7a2b09","Type":"ContainerStarted","Data":"2357531b38475fec7a01dad0d240a510869ce0e817490d5e1ef710b7aa3db57c"}
Sep 16 18:51:02.821032 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:51:02.820992    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-w8w2f" event={"ID":"9d2b060c-f303-4dca-945c-42b6ef7a2b09","Type":"ContainerStarted","Data":"55c51ae8a99a06caaa5c76ce51b28f6e8606b5f1ba4528ea8c39d60d3219a2a7"}
Sep 16 18:51:02.821542 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:51:02.821127    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-w8w2f"
Sep 16 18:51:13.826440 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:51:13.826410    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-w8w2f"
Sep 16 18:51:13.844425 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:51:13.844372    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-w8w2f" podStartSLOduration=12.613351459 podStartE2EDuration="14.8443549s" totalImagesPullingTime="2.231003441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:50:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:51:02.857669747 +0000 UTC m=+466.499385410" watchObservedRunningTime="2026-09-16 18:51:13.8443549 +0000 UTC m=+477.486070563"
Sep 16 18:53:16.802650 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:53:16.802619    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7qwl2_b6695f86-08c0-4836-87b1-f2a9d76084bd/console-operator/1.log"
Sep 16 18:53:16.808315 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:53:16.808292    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7qwl2_b6695f86-08c0-4836-87b1-f2a9d76084bd/console-operator/1.log"
Sep 16 18:53:33.026356 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:53:33.026321    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-rvnrb"]
Sep 16 18:53:33.029623 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:53:33.029601    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-rvnrb"
Sep 16 18:53:33.033209 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:53:33.033184    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 16 18:53:33.033330 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:53:33.033225    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-bkns9\""
Sep 16 18:53:33.034389 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:53:33.034373    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 18:53:33.034641 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:53:33.034624    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 18:53:33.034733 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:53:33.034653    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 16 18:53:33.045195 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:53:33.045165    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-rvnrb"]
Sep 16 18:53:33.076573 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:53:33.076531    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcvk7\" (UniqueName: \"kubernetes.io/projected/6054145c-eb8c-4d2e-a1a4-8f0f4c674573-kube-api-access-wcvk7\") pod \"s3-init-rvnrb\" (UID: \"6054145c-eb8c-4d2e-a1a4-8f0f4c674573\") " pod="kserve/s3-init-rvnrb"
Sep 16 18:53:33.076774 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:53:33.076594    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6054145c-eb8c-4d2e-a1a4-8f0f4c674573-model-cache\") pod \"s3-init-rvnrb\" (UID: \"6054145c-eb8c-4d2e-a1a4-8f0f4c674573\") " pod="kserve/s3-init-rvnrb"
Sep 16 18:53:33.177998 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:53:33.177959    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcvk7\" (UniqueName: \"kubernetes.io/projected/6054145c-eb8c-4d2e-a1a4-8f0f4c674573-kube-api-access-wcvk7\") pod \"s3-init-rvnrb\" (UID: \"6054145c-eb8c-4d2e-a1a4-8f0f4c674573\") " pod="kserve/s3-init-rvnrb"
Sep 16 18:53:33.178167 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:53:33.178023    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6054145c-eb8c-4d2e-a1a4-8f0f4c674573-model-cache\") pod \"s3-init-rvnrb\" (UID: \"6054145c-eb8c-4d2e-a1a4-8f0f4c674573\") " pod="kserve/s3-init-rvnrb"
Sep 16 18:53:33.178453 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:53:33.178434    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6054145c-eb8c-4d2e-a1a4-8f0f4c674573-model-cache\") pod \"s3-init-rvnrb\" (UID: \"6054145c-eb8c-4d2e-a1a4-8f0f4c674573\") " pod="kserve/s3-init-rvnrb"
Sep 16 18:53:33.191777 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:53:33.191741    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcvk7\" (UniqueName: \"kubernetes.io/projected/6054145c-eb8c-4d2e-a1a4-8f0f4c674573-kube-api-access-wcvk7\") pod \"s3-init-rvnrb\" (UID: \"6054145c-eb8c-4d2e-a1a4-8f0f4c674573\") " pod="kserve/s3-init-rvnrb"
Sep 16 18:53:33.339046 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:53:33.338958    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-rvnrb"
Sep 16 18:53:33.616552 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:53:33.616517    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-rvnrb"]
Sep 16 18:53:33.618735 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:53:33.618695    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6054145c_eb8c_4d2e_a1a4_8f0f4c674573.slice/crio-584242ad2483eaf3edeaac33e1921547c840211a7327b16c79171cac8a9c7d9f WatchSource:0}: Error finding container 584242ad2483eaf3edeaac33e1921547c840211a7327b16c79171cac8a9c7d9f: Status 404 returned error can't find the container with id 584242ad2483eaf3edeaac33e1921547c840211a7327b16c79171cac8a9c7d9f
Sep 16 18:53:34.416701 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:53:34.416652    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-rvnrb" event={"ID":"6054145c-eb8c-4d2e-a1a4-8f0f4c674573","Type":"ContainerStarted","Data":"584242ad2483eaf3edeaac33e1921547c840211a7327b16c79171cac8a9c7d9f"}
Sep 16 18:53:37.432268 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:53:37.432225    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-rvnrb" event={"ID":"6054145c-eb8c-4d2e-a1a4-8f0f4c674573","Type":"ContainerStarted","Data":"fe431d3f675f0e923db8eaeff105a6cc7e29907363a87afbef95492cece9e774"}
Sep 16 18:53:39.442500 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:53:39.442419    2582 generic.go:356] "Generic (PLEG): container finished" podID="6054145c-eb8c-4d2e-a1a4-8f0f4c674573" containerID="fe431d3f675f0e923db8eaeff105a6cc7e29907363a87afbef95492cece9e774" exitCode=0
Sep 16 18:53:39.442876 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:53:39.442491    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-rvnrb" event={"ID":"6054145c-eb8c-4d2e-a1a4-8f0f4c674573","Type":"ContainerDied","Data":"fe431d3f675f0e923db8eaeff105a6cc7e29907363a87afbef95492cece9e774"}
Sep 16 18:53:44.468118 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:53:44.468081    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-rvnrb" event={"ID":"6054145c-eb8c-4d2e-a1a4-8f0f4c674573","Type":"ContainerStarted","Data":"bc718e55050d438e77d320a9c803c55342dfad002860133be0008d7cf62bab01"}
Sep 16 18:53:44.487635 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:53:44.487585    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-rvnrb" podStartSLOduration=2.049833615 podStartE2EDuration="11.487568965s" totalImagesPullingTime="7.735803451s" totalInitContainerRuntime="1.701931899s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:53:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:53:44.485835794 +0000 UTC m=+628.127551471" watchObservedRunningTime="2026-09-16 18:53:44.487568965 +0000 UTC m=+628.129284629"
Sep 16 18:54:49.735302 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:54:49.735262    2582 generic.go:356] "Generic (PLEG): container finished" podID="6054145c-eb8c-4d2e-a1a4-8f0f4c674573" containerID="bc718e55050d438e77d320a9c803c55342dfad002860133be0008d7cf62bab01" exitCode=0
Sep 16 18:54:49.735705 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:54:49.735333    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-rvnrb" event={"ID":"6054145c-eb8c-4d2e-a1a4-8f0f4c674573","Type":"ContainerDied","Data":"bc718e55050d438e77d320a9c803c55342dfad002860133be0008d7cf62bab01"}
Sep 16 18:54:50.873531 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:54:50.873501    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-rvnrb"
Sep 16 18:54:51.016714 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:54:51.016680    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wcvk7\" (UniqueName: \"kubernetes.io/projected/6054145c-eb8c-4d2e-a1a4-8f0f4c674573-kube-api-access-wcvk7\") pod \"6054145c-eb8c-4d2e-a1a4-8f0f4c674573\" (UID: \"6054145c-eb8c-4d2e-a1a4-8f0f4c674573\") "
Sep 16 18:54:51.016901 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:54:51.016762    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6054145c-eb8c-4d2e-a1a4-8f0f4c674573-model-cache\") pod \"6054145c-eb8c-4d2e-a1a4-8f0f4c674573\" (UID: \"6054145c-eb8c-4d2e-a1a4-8f0f4c674573\") "
Sep 16 18:54:51.018955 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:54:51.018922    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6054145c-eb8c-4d2e-a1a4-8f0f4c674573-kube-api-access-wcvk7" (OuterVolumeSpecName: "kube-api-access-wcvk7") pod "6054145c-eb8c-4d2e-a1a4-8f0f4c674573" (UID: "6054145c-eb8c-4d2e-a1a4-8f0f4c674573"). InnerVolumeSpecName "kube-api-access-wcvk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:54:51.039599 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:54:51.039556    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6054145c-eb8c-4d2e-a1a4-8f0f4c674573-model-cache" (OuterVolumeSpecName: "model-cache") pod "6054145c-eb8c-4d2e-a1a4-8f0f4c674573" (UID: "6054145c-eb8c-4d2e-a1a4-8f0f4c674573"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:54:51.117406 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:54:51.117359    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcvk7\" (UniqueName: \"kubernetes.io/projected/6054145c-eb8c-4d2e-a1a4-8f0f4c674573-kube-api-access-wcvk7\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 18:54:51.117406 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:54:51.117404    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6054145c-eb8c-4d2e-a1a4-8f0f4c674573-model-cache\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 18:54:51.746343 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:54:51.746305    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-rvnrb" event={"ID":"6054145c-eb8c-4d2e-a1a4-8f0f4c674573","Type":"ContainerDied","Data":"584242ad2483eaf3edeaac33e1921547c840211a7327b16c79171cac8a9c7d9f"}
Sep 16 18:54:51.746343 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:54:51.746333    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-rvnrb"
Sep 16 18:54:51.746343 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:54:51.746343    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="584242ad2483eaf3edeaac33e1921547c840211a7327b16c79171cac8a9c7d9f"
Sep 16 18:55:22.708915 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:22.708876    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 16 18:55:22.709380 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:22.709217    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6054145c-eb8c-4d2e-a1a4-8f0f4c674573" containerName="download-hf-model"
Sep 16 18:55:22.709380 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:22.709228    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6054145c-eb8c-4d2e-a1a4-8f0f4c674573" containerName="download-hf-model"
Sep 16 18:55:22.709380 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:22.709238    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6054145c-eb8c-4d2e-a1a4-8f0f4c674573" containerName="s3-init"
Sep 16 18:55:22.709380 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:22.709245    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6054145c-eb8c-4d2e-a1a4-8f0f4c674573" containerName="s3-init"
Sep 16 18:55:22.709380 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:22.709298    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6054145c-eb8c-4d2e-a1a4-8f0f4c674573" containerName="s3-init"
Sep 16 18:55:22.712351 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:22.712329    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:55:22.715252 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:22.715227    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 16 18:55:22.716231 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:22.716208    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:55:22.716335 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:22.716277    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-v62w5\""
Sep 16 18:55:22.716335 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:22.716297    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 16 18:55:22.716439 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:22.716360    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-v9sp9\""
Sep 16 18:55:22.716439 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:22.716369    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 16 18:55:22.716530 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:22.716437    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 16 18:55:22.725613 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:22.725590    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 16 18:55:22.779072 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:22.779030    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/091c4903-c726-4ff5-b541-ddf988b96ffc-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"091c4903-c726-4ff5-b541-ddf988b96ffc\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:55:22.779266 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:22.779083    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/091c4903-c726-4ff5-b541-ddf988b96ffc-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"091c4903-c726-4ff5-b541-ddf988b96ffc\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:55:22.779266 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:22.779109    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxfr7\" (UniqueName: \"kubernetes.io/projected/091c4903-c726-4ff5-b541-ddf988b96ffc-kube-api-access-nxfr7\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"091c4903-c726-4ff5-b541-ddf988b96ffc\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:55:22.779266 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:22.779152    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/091c4903-c726-4ff5-b541-ddf988b96ffc-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"091c4903-c726-4ff5-b541-ddf988b96ffc\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:55:22.779266 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:22.779176    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/091c4903-c726-4ff5-b541-ddf988b96ffc-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"091c4903-c726-4ff5-b541-ddf988b96ffc\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:55:22.779266 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:22.779204    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/091c4903-c726-4ff5-b541-ddf988b96ffc-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"091c4903-c726-4ff5-b541-ddf988b96ffc\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:55:22.779266 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:22.779250    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/091c4903-c726-4ff5-b541-ddf988b96ffc-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"091c4903-c726-4ff5-b541-ddf988b96ffc\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:55:22.779495 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:22.779286    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/091c4903-c726-4ff5-b541-ddf988b96ffc-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"091c4903-c726-4ff5-b541-ddf988b96ffc\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:55:22.879780 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:22.879744    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/091c4903-c726-4ff5-b541-ddf988b96ffc-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"091c4903-c726-4ff5-b541-ddf988b96ffc\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:55:22.879973 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:22.879801    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/091c4903-c726-4ff5-b541-ddf988b96ffc-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"091c4903-c726-4ff5-b541-ddf988b96ffc\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:55:22.879973 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:22.879839    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/091c4903-c726-4ff5-b541-ddf988b96ffc-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"091c4903-c726-4ff5-b541-ddf988b96ffc\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:55:22.879973 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:22.879866    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxfr7\" (UniqueName: \"kubernetes.io/projected/091c4903-c726-4ff5-b541-ddf988b96ffc-kube-api-access-nxfr7\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"091c4903-c726-4ff5-b541-ddf988b96ffc\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:55:22.879973 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:22.879910    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/091c4903-c726-4ff5-b541-ddf988b96ffc-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"091c4903-c726-4ff5-b541-ddf988b96ffc\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:55:22.880187 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:22.880005    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/091c4903-c726-4ff5-b541-ddf988b96ffc-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"091c4903-c726-4ff5-b541-ddf988b96ffc\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:55:22.880187 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:22.880051    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/091c4903-c726-4ff5-b541-ddf988b96ffc-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"091c4903-c726-4ff5-b541-ddf988b96ffc\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:55:22.880187 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:22.880121    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/091c4903-c726-4ff5-b541-ddf988b96ffc-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"091c4903-c726-4ff5-b541-ddf988b96ffc\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:55:22.880187 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:22.880126    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/091c4903-c726-4ff5-b541-ddf988b96ffc-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"091c4903-c726-4ff5-b541-ddf988b96ffc\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:55:22.880385 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:22.880194    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/091c4903-c726-4ff5-b541-ddf988b96ffc-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"091c4903-c726-4ff5-b541-ddf988b96ffc\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:55:22.880385 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:22.880228    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/091c4903-c726-4ff5-b541-ddf988b96ffc-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"091c4903-c726-4ff5-b541-ddf988b96ffc\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:55:22.880385 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:22.880379    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/091c4903-c726-4ff5-b541-ddf988b96ffc-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"091c4903-c726-4ff5-b541-ddf988b96ffc\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:55:22.880614 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:22.880594    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/091c4903-c726-4ff5-b541-ddf988b96ffc-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"091c4903-c726-4ff5-b541-ddf988b96ffc\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:55:22.882240 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:22.882220    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/091c4903-c726-4ff5-b541-ddf988b96ffc-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"091c4903-c726-4ff5-b541-ddf988b96ffc\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:55:22.882364 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:22.882347    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/091c4903-c726-4ff5-b541-ddf988b96ffc-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"091c4903-c726-4ff5-b541-ddf988b96ffc\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:55:22.889165 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:22.889140    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxfr7\" (UniqueName: \"kubernetes.io/projected/091c4903-c726-4ff5-b541-ddf988b96ffc-kube-api-access-nxfr7\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"091c4903-c726-4ff5-b541-ddf988b96ffc\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:55:23.023327 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:23.023292    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:55:23.303216 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:23.303131    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 16 18:55:23.307365 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:55:23.307322    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod091c4903_c726_4ff5_b541_ddf988b96ffc.slice/crio-b50b6990311a0a27856893f3b106933afd9dccc252a949838126648c703613d8 WatchSource:0}: Error finding container b50b6990311a0a27856893f3b106933afd9dccc252a949838126648c703613d8: Status 404 returned error can't find the container with id b50b6990311a0a27856893f3b106933afd9dccc252a949838126648c703613d8
Sep 16 18:55:23.879559 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:23.879516    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"091c4903-c726-4ff5-b541-ddf988b96ffc","Type":"ContainerStarted","Data":"a2a61c9fb448fa32d0d2d493bc6be612754833e80a29afd65383f345f3815995"}
Sep 16 18:55:23.879967 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:23.879599    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"091c4903-c726-4ff5-b541-ddf988b96ffc","Type":"ContainerStarted","Data":"b50b6990311a0a27856893f3b106933afd9dccc252a949838126648c703613d8"}
Sep 16 18:55:26.907135 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:26.907099    2582 generic.go:356] "Generic (PLEG): container finished" podID="091c4903-c726-4ff5-b541-ddf988b96ffc" containerID="a2a61c9fb448fa32d0d2d493bc6be612754833e80a29afd65383f345f3815995" exitCode=0
Sep 16 18:55:26.907617 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:26.907157    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"091c4903-c726-4ff5-b541-ddf988b96ffc","Type":"ContainerDied","Data":"a2a61c9fb448fa32d0d2d493bc6be612754833e80a29afd65383f345f3815995"}
Sep 16 18:55:26.908753 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:26.908733    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:55:32.569111 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:32.568888    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.24:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 16 18:55:32.569111 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:32.568975    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt" podUID="949c2e66-950f-480a-96a3-16c9e4be467b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.24:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 18:55:33.961438 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:33.961355    2582 generic.go:356] "Generic (PLEG): container finished" podID="949c2e66-950f-480a-96a3-16c9e4be467b" containerID="c7f227d8f2208951d8a93ef8000bb0367d191c9c120d98c07b667088b903fa67" exitCode=0
Sep 16 18:55:33.962008 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:33.961437    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt" event={"ID":"949c2e66-950f-480a-96a3-16c9e4be467b","Type":"ContainerDied","Data":"c7f227d8f2208951d8a93ef8000bb0367d191c9c120d98c07b667088b903fa67"}
Sep 16 18:55:33.962008 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:33.961926    2582 scope.go:117] "RemoveContainer" containerID="c7f227d8f2208951d8a93ef8000bb0367d191c9c120d98c07b667088b903fa67"
Sep 16 18:55:38.432605 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:38.432541    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:55:38.433082 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:38.432649    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:55:38.433082 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:38.432688    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:55:41.004734 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:41.004675    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt" event={"ID":"949c2e66-950f-480a-96a3-16c9e4be467b","Type":"ContainerStarted","Data":"e097152cbe76179d7cf15ea23f5bbd37989a88178e632542bf16984a0756ff27"}
Sep 16 18:55:41.904525 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:41.904484    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt"
Sep 16 18:55:41.904824 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:41.904773    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.24:15021/healthz/ready\": dial tcp 10.132.0.24:15021: connect: connection refused" start-of-body=
Sep 16 18:55:41.904901 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:41.904835    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt" podUID="949c2e66-950f-480a-96a3-16c9e4be467b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.24:15021/healthz/ready\": dial tcp 10.132.0.24:15021: connect: connection refused"
Sep 16 18:55:42.904475 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:42.904437    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.24:15021/healthz/ready\": dial tcp 10.132.0.24:15021: connect: connection refused" start-of-body=
Sep 16 18:55:42.904950 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:42.904519    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt" podUID="949c2e66-950f-480a-96a3-16c9e4be467b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.24:15021/healthz/ready\": dial tcp 10.132.0.24:15021: connect: connection refused"
Sep 16 18:55:43.905545 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:43.905503    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.24:15021/healthz/ready\": dial tcp 10.132.0.24:15021: connect: connection refused" start-of-body=
Sep 16 18:55:43.905952 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:43.905596    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt" podUID="949c2e66-950f-480a-96a3-16c9e4be467b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.24:15021/healthz/ready\": dial tcp 10.132.0.24:15021: connect: connection refused"
Sep 16 18:55:44.905238 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:44.905203    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.24:15021/healthz/ready\": dial tcp 10.132.0.24:15021: connect: connection refused" start-of-body=
Sep 16 18:55:44.905421 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:44.905260    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt" podUID="949c2e66-950f-480a-96a3-16c9e4be467b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.24:15021/healthz/ready\": dial tcp 10.132.0.24:15021: connect: connection refused"
Sep 16 18:55:45.904888 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:45.904844    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.24:15021/healthz/ready\": dial tcp 10.132.0.24:15021: connect: connection refused" start-of-body=
Sep 16 18:55:45.905296 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:45.904919    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt" podUID="949c2e66-950f-480a-96a3-16c9e4be467b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.24:15021/healthz/ready\": dial tcp 10.132.0.24:15021: connect: connection refused"
Sep 16 18:55:46.526893 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:46.526848    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt"
Sep 16 18:55:46.910116 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:46.910025    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt"
Sep 16 18:55:47.037247 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:47.037217    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt"
Sep 16 18:55:58.103714 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:58.103678    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"091c4903-c726-4ff5-b541-ddf988b96ffc","Type":"ContainerStarted","Data":"eb6df4c8e3fc869fd974cc40b7130606c6a91156f48dd08b0cc24cbfc76663d1"}
Sep 16 18:55:58.130667 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:58.126802    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=3.40971356 podStartE2EDuration="36.126781271s" totalImagesPullingTime="30.199192319s" totalInitContainerRuntime="2.517875392s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:55:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:55:58.124777946 +0000 UTC m=+761.766493610" watchObservedRunningTime="2026-09-16 18:55:58.126781271 +0000 UTC m=+761.768496936"
Sep 16 18:55:59.024412 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:55:59.024377    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:56:03.023809 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:56:03.023769    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:56:03.025522 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:56:03.025493    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="091c4903-c726-4ff5-b541-ddf988b96ffc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/health\": dial tcp 10.132.0.28:8000: connect: connection refused"
Sep 16 18:56:13.024181 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:56:13.024127    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="091c4903-c726-4ff5-b541-ddf988b96ffc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/health\": dial tcp 10.132.0.28:8000: connect: connection refused"
Sep 16 18:56:23.024050 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:56:23.024001    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="091c4903-c726-4ff5-b541-ddf988b96ffc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/health\": dial tcp 10.132.0.28:8000: connect: connection refused"
Sep 16 18:56:33.024175 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:56:33.024131    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="091c4903-c726-4ff5-b541-ddf988b96ffc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/health\": dial tcp 10.132.0.28:8000: connect: connection refused"
Sep 16 18:56:43.024410 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:56:43.024312    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="091c4903-c726-4ff5-b541-ddf988b96ffc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/health\": dial tcp 10.132.0.28:8000: connect: connection refused"
Sep 16 18:56:53.024632 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:56:53.024587    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="091c4903-c726-4ff5-b541-ddf988b96ffc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/health\": dial tcp 10.132.0.28:8000: connect: connection refused"
Sep 16 18:57:03.024691 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:57:03.024642    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="091c4903-c726-4ff5-b541-ddf988b96ffc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/health\": dial tcp 10.132.0.28:8000: connect: connection refused"
Sep 16 18:57:13.023981 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:57:13.023933    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="091c4903-c726-4ff5-b541-ddf988b96ffc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/health\": dial tcp 10.132.0.28:8000: connect: connection refused"
Sep 16 18:57:23.024147 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:57:23.024102    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="091c4903-c726-4ff5-b541-ddf988b96ffc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/health\": dial tcp 10.132.0.28:8000: connect: connection refused"
Sep 16 18:57:33.033793 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:57:33.033748    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:57:33.041439 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:57:33.041411    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:57:39.875759 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:57:39.875711    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 16 18:57:54.898168 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:57:54.898100    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="091c4903-c726-4ff5-b541-ddf988b96ffc" containerName="main" containerID="cri-o://eb6df4c8e3fc869fd974cc40b7130606c6a91156f48dd08b0cc24cbfc76663d1" gracePeriod=45
Sep 16 18:57:55.034895 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:57:55.034856    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="091c4903-c726-4ff5-b541-ddf988b96ffc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/health\": dial tcp 10.132.0.28:8000: connect: connection refused"
Sep 16 18:57:55.640311 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:57:55.640286    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:57:55.656169 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:57:55.656136    2582 generic.go:356] "Generic (PLEG): container finished" podID="091c4903-c726-4ff5-b541-ddf988b96ffc" containerID="eb6df4c8e3fc869fd974cc40b7130606c6a91156f48dd08b0cc24cbfc76663d1" exitCode=0
Sep 16 18:57:55.656350 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:57:55.656185    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:57:55.656350 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:57:55.656201    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"091c4903-c726-4ff5-b541-ddf988b96ffc","Type":"ContainerDied","Data":"eb6df4c8e3fc869fd974cc40b7130606c6a91156f48dd08b0cc24cbfc76663d1"}
Sep 16 18:57:55.656350 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:57:55.656252    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"091c4903-c726-4ff5-b541-ddf988b96ffc","Type":"ContainerDied","Data":"b50b6990311a0a27856893f3b106933afd9dccc252a949838126648c703613d8"}
Sep 16 18:57:55.656350 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:57:55.656281    2582 scope.go:117] "RemoveContainer" containerID="eb6df4c8e3fc869fd974cc40b7130606c6a91156f48dd08b0cc24cbfc76663d1"
Sep 16 18:57:55.665690 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:57:55.665663    2582 scope.go:117] "RemoveContainer" containerID="a2a61c9fb448fa32d0d2d493bc6be612754833e80a29afd65383f345f3815995"
Sep 16 18:57:55.675223 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:57:55.675199    2582 scope.go:117] "RemoveContainer" containerID="eb6df4c8e3fc869fd974cc40b7130606c6a91156f48dd08b0cc24cbfc76663d1"
Sep 16 18:57:55.675501 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:57:55.675480    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb6df4c8e3fc869fd974cc40b7130606c6a91156f48dd08b0cc24cbfc76663d1\": container with ID starting with eb6df4c8e3fc869fd974cc40b7130606c6a91156f48dd08b0cc24cbfc76663d1 not found: ID does not exist" containerID="eb6df4c8e3fc869fd974cc40b7130606c6a91156f48dd08b0cc24cbfc76663d1"
Sep 16 18:57:55.675575 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:57:55.675508    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb6df4c8e3fc869fd974cc40b7130606c6a91156f48dd08b0cc24cbfc76663d1"} err="failed to get container status \"eb6df4c8e3fc869fd974cc40b7130606c6a91156f48dd08b0cc24cbfc76663d1\": rpc error: code = NotFound desc = could not find container \"eb6df4c8e3fc869fd974cc40b7130606c6a91156f48dd08b0cc24cbfc76663d1\": container with ID starting with eb6df4c8e3fc869fd974cc40b7130606c6a91156f48dd08b0cc24cbfc76663d1 not found: ID does not exist"
Sep 16 18:57:55.675575 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:57:55.675523    2582 scope.go:117] "RemoveContainer" containerID="a2a61c9fb448fa32d0d2d493bc6be612754833e80a29afd65383f345f3815995"
Sep 16 18:57:55.675774 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:57:55.675754    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2a61c9fb448fa32d0d2d493bc6be612754833e80a29afd65383f345f3815995\": container with ID starting with a2a61c9fb448fa32d0d2d493bc6be612754833e80a29afd65383f345f3815995 not found: ID does not exist" containerID="a2a61c9fb448fa32d0d2d493bc6be612754833e80a29afd65383f345f3815995"
Sep 16 18:57:55.675860 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:57:55.675778    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2a61c9fb448fa32d0d2d493bc6be612754833e80a29afd65383f345f3815995"} err="failed to get container status \"a2a61c9fb448fa32d0d2d493bc6be612754833e80a29afd65383f345f3815995\": rpc error: code = NotFound desc = could not find container \"a2a61c9fb448fa32d0d2d493bc6be612754833e80a29afd65383f345f3815995\": container with ID starting with a2a61c9fb448fa32d0d2d493bc6be612754833e80a29afd65383f345f3815995 not found: ID does not exist"
Sep 16 18:57:55.737175 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:57:55.737139    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/091c4903-c726-4ff5-b541-ddf988b96ffc-tls-certs\") pod \"091c4903-c726-4ff5-b541-ddf988b96ffc\" (UID: \"091c4903-c726-4ff5-b541-ddf988b96ffc\") "
Sep 16 18:57:55.737321 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:57:55.737190    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxfr7\" (UniqueName: \"kubernetes.io/projected/091c4903-c726-4ff5-b541-ddf988b96ffc-kube-api-access-nxfr7\") pod \"091c4903-c726-4ff5-b541-ddf988b96ffc\" (UID: \"091c4903-c726-4ff5-b541-ddf988b96ffc\") "
Sep 16 18:57:55.737321 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:57:55.737209    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/091c4903-c726-4ff5-b541-ddf988b96ffc-cabundle-cert\") pod \"091c4903-c726-4ff5-b541-ddf988b96ffc\" (UID: \"091c4903-c726-4ff5-b541-ddf988b96ffc\") "
Sep 16 18:57:55.737321 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:57:55.737229    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/091c4903-c726-4ff5-b541-ddf988b96ffc-home\") pod \"091c4903-c726-4ff5-b541-ddf988b96ffc\" (UID: \"091c4903-c726-4ff5-b541-ddf988b96ffc\") "
Sep 16 18:57:55.737321 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:57:55.737288    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/091c4903-c726-4ff5-b541-ddf988b96ffc-model-cache\") pod \"091c4903-c726-4ff5-b541-ddf988b96ffc\" (UID: \"091c4903-c726-4ff5-b541-ddf988b96ffc\") "
Sep 16 18:57:55.737321 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:57:55.737311    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/091c4903-c726-4ff5-b541-ddf988b96ffc-kserve-provision-location\") pod \"091c4903-c726-4ff5-b541-ddf988b96ffc\" (UID: \"091c4903-c726-4ff5-b541-ddf988b96ffc\") "
Sep 16 18:57:55.737568 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:57:55.737332    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/091c4903-c726-4ff5-b541-ddf988b96ffc-tmp-dir\") pod \"091c4903-c726-4ff5-b541-ddf988b96ffc\" (UID: \"091c4903-c726-4ff5-b541-ddf988b96ffc\") "
Sep 16 18:57:55.737568 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:57:55.737351    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/091c4903-c726-4ff5-b541-ddf988b96ffc-dshm\") pod \"091c4903-c726-4ff5-b541-ddf988b96ffc\" (UID: \"091c4903-c726-4ff5-b541-ddf988b96ffc\") "
Sep 16 18:57:55.737668 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:57:55.737587    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/091c4903-c726-4ff5-b541-ddf988b96ffc-model-cache" (OuterVolumeSpecName: "model-cache") pod "091c4903-c726-4ff5-b541-ddf988b96ffc" (UID: "091c4903-c726-4ff5-b541-ddf988b96ffc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:57:55.737668 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:57:55.737654    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/091c4903-c726-4ff5-b541-ddf988b96ffc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "091c4903-c726-4ff5-b541-ddf988b96ffc" (UID: "091c4903-c726-4ff5-b541-ddf988b96ffc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:57:55.737994 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:57:55.737967    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/091c4903-c726-4ff5-b541-ddf988b96ffc-home" (OuterVolumeSpecName: "home") pod "091c4903-c726-4ff5-b541-ddf988b96ffc" (UID: "091c4903-c726-4ff5-b541-ddf988b96ffc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:57:55.739561 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:57:55.739530    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/091c4903-c726-4ff5-b541-ddf988b96ffc-kube-api-access-nxfr7" (OuterVolumeSpecName: "kube-api-access-nxfr7") pod "091c4903-c726-4ff5-b541-ddf988b96ffc" (UID: "091c4903-c726-4ff5-b541-ddf988b96ffc"). InnerVolumeSpecName "kube-api-access-nxfr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:57:55.739710 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:57:55.739693    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/091c4903-c726-4ff5-b541-ddf988b96ffc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "091c4903-c726-4ff5-b541-ddf988b96ffc" (UID: "091c4903-c726-4ff5-b541-ddf988b96ffc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:57:55.740064 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:57:55.740046    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/091c4903-c726-4ff5-b541-ddf988b96ffc-dshm" (OuterVolumeSpecName: "dshm") pod "091c4903-c726-4ff5-b541-ddf988b96ffc" (UID: "091c4903-c726-4ff5-b541-ddf988b96ffc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:57:55.749577 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:57:55.749550    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/091c4903-c726-4ff5-b541-ddf988b96ffc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "091c4903-c726-4ff5-b541-ddf988b96ffc" (UID: "091c4903-c726-4ff5-b541-ddf988b96ffc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:57:55.757481 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:57:55.757450    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/091c4903-c726-4ff5-b541-ddf988b96ffc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "091c4903-c726-4ff5-b541-ddf988b96ffc" (UID: "091c4903-c726-4ff5-b541-ddf988b96ffc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:57:55.838872 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:57:55.838833    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/091c4903-c726-4ff5-b541-ddf988b96ffc-model-cache\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 18:57:55.838872 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:57:55.838865    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/091c4903-c726-4ff5-b541-ddf988b96ffc-kserve-provision-location\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 18:57:55.838872 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:57:55.838876    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/091c4903-c726-4ff5-b541-ddf988b96ffc-tmp-dir\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 18:57:55.839082 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:57:55.838888    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/091c4903-c726-4ff5-b541-ddf988b96ffc-dshm\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 18:57:55.839082 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:57:55.838896    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/091c4903-c726-4ff5-b541-ddf988b96ffc-tls-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 18:57:55.839082 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:57:55.838905    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxfr7\" (UniqueName: \"kubernetes.io/projected/091c4903-c726-4ff5-b541-ddf988b96ffc-kube-api-access-nxfr7\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 18:57:55.839082 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:57:55.838913    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/091c4903-c726-4ff5-b541-ddf988b96ffc-cabundle-cert\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 18:57:55.839082 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:57:55.838920    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/091c4903-c726-4ff5-b541-ddf988b96ffc-home\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 18:57:55.982489 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:57:55.982456    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 16 18:57:55.983334 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:57:55.983309    2582 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod091c4903_c726_4ff5_b541_ddf988b96ffc.slice\": RecentStats: unable to find data in memory cache]"
Sep 16 18:57:55.986115 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:57:55.986090    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 16 18:57:56.872379 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:57:56.872345    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="091c4903-c726-4ff5-b541-ddf988b96ffc" path="/var/lib/kubelet/pods/091c4903-c726-4ff5-b541-ddf988b96ffc/volumes"
Sep 16 18:58:16.834333 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:58:16.834300    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7qwl2_b6695f86-08c0-4836-87b1-f2a9d76084bd/console-operator/1.log"
Sep 16 18:58:16.834844 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:58:16.834425    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7qwl2_b6695f86-08c0-4836-87b1-f2a9d76084bd/console-operator/1.log"
Sep 16 18:59:21.615485 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:21.615454    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-cfsjc"]
Sep 16 18:59:21.615927 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:21.615771    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="091c4903-c726-4ff5-b541-ddf988b96ffc" containerName="main"
Sep 16 18:59:21.615927 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:21.615783    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="091c4903-c726-4ff5-b541-ddf988b96ffc" containerName="main"
Sep 16 18:59:21.615927 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:21.615795    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="091c4903-c726-4ff5-b541-ddf988b96ffc" containerName="storage-initializer"
Sep 16 18:59:21.615927 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:21.615802    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="091c4903-c726-4ff5-b541-ddf988b96ffc" containerName="storage-initializer"
Sep 16 18:59:21.615927 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:21.615854    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="091c4903-c726-4ff5-b541-ddf988b96ffc" containerName="main"
Sep 16 18:59:21.618724 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:21.618699    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-cfsjc"
Sep 16 18:59:21.621380 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:21.621357    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"default-dockercfg-w2mb2\""
Sep 16 18:59:21.622423 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:21.622405    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Sep 16 18:59:21.622533 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:21.622513    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Sep 16 18:59:21.622662 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:21.622645    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
Sep 16 18:59:21.633697 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:21.633672    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-cfsjc"]
Sep 16 18:59:21.762820 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:21.762783    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76a3e4a4-64d9-4297-9a4b-be02f9baca70-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-cfsjc\" (UID: \"76a3e4a4-64d9-4297-9a4b-be02f9baca70\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-cfsjc"
Sep 16 18:59:21.763011 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:21.762836    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76a3e4a4-64d9-4297-9a4b-be02f9baca70-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-cfsjc\" (UID: \"76a3e4a4-64d9-4297-9a4b-be02f9baca70\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-cfsjc"
Sep 16 18:59:21.763011 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:21.762898    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76a3e4a4-64d9-4297-9a4b-be02f9baca70-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-cfsjc\" (UID: \"76a3e4a4-64d9-4297-9a4b-be02f9baca70\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-cfsjc"
Sep 16 18:59:21.763011 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:21.762939    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76a3e4a4-64d9-4297-9a4b-be02f9baca70-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-cfsjc\" (UID: \"76a3e4a4-64d9-4297-9a4b-be02f9baca70\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-cfsjc"
Sep 16 18:59:21.763140 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:21.763017    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76a3e4a4-64d9-4297-9a4b-be02f9baca70-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-cfsjc\" (UID: \"76a3e4a4-64d9-4297-9a4b-be02f9baca70\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-cfsjc"
Sep 16 18:59:21.763140 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:21.763038    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sblfw\" (UniqueName: \"kubernetes.io/projected/76a3e4a4-64d9-4297-9a4b-be02f9baca70-kube-api-access-sblfw\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-cfsjc\" (UID: \"76a3e4a4-64d9-4297-9a4b-be02f9baca70\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-cfsjc"
Sep 16 18:59:21.863889 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:21.863852    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76a3e4a4-64d9-4297-9a4b-be02f9baca70-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-cfsjc\" (UID: \"76a3e4a4-64d9-4297-9a4b-be02f9baca70\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-cfsjc"
Sep 16 18:59:21.864088 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:21.863926    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76a3e4a4-64d9-4297-9a4b-be02f9baca70-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-cfsjc\" (UID: \"76a3e4a4-64d9-4297-9a4b-be02f9baca70\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-cfsjc"
Sep 16 18:59:21.864088 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:21.863945    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sblfw\" (UniqueName: \"kubernetes.io/projected/76a3e4a4-64d9-4297-9a4b-be02f9baca70-kube-api-access-sblfw\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-cfsjc\" (UID: \"76a3e4a4-64d9-4297-9a4b-be02f9baca70\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-cfsjc"
Sep 16 18:59:21.864088 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:21.863989    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76a3e4a4-64d9-4297-9a4b-be02f9baca70-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-cfsjc\" (UID: \"76a3e4a4-64d9-4297-9a4b-be02f9baca70\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-cfsjc"
Sep 16 18:59:21.864088 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:21.864013    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76a3e4a4-64d9-4297-9a4b-be02f9baca70-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-cfsjc\" (UID: \"76a3e4a4-64d9-4297-9a4b-be02f9baca70\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-cfsjc"
Sep 16 18:59:21.864088 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:21.864053    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76a3e4a4-64d9-4297-9a4b-be02f9baca70-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-cfsjc\" (UID: \"76a3e4a4-64d9-4297-9a4b-be02f9baca70\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-cfsjc"
Sep 16 18:59:21.864467 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:21.864428    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76a3e4a4-64d9-4297-9a4b-be02f9baca70-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-cfsjc\" (UID: \"76a3e4a4-64d9-4297-9a4b-be02f9baca70\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-cfsjc"
Sep 16 18:59:21.864580 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:21.864470    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76a3e4a4-64d9-4297-9a4b-be02f9baca70-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-cfsjc\" (UID: \"76a3e4a4-64d9-4297-9a4b-be02f9baca70\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-cfsjc"
Sep 16 18:59:21.864580 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:21.864493    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76a3e4a4-64d9-4297-9a4b-be02f9baca70-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-cfsjc\" (UID: \"76a3e4a4-64d9-4297-9a4b-be02f9baca70\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-cfsjc"
Sep 16 18:59:21.866375 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:21.866315    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76a3e4a4-64d9-4297-9a4b-be02f9baca70-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-cfsjc\" (UID: \"76a3e4a4-64d9-4297-9a4b-be02f9baca70\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-cfsjc"
Sep 16 18:59:21.866547 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:21.866530    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76a3e4a4-64d9-4297-9a4b-be02f9baca70-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-cfsjc\" (UID: \"76a3e4a4-64d9-4297-9a4b-be02f9baca70\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-cfsjc"
Sep 16 18:59:21.875357 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:21.875331    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sblfw\" (UniqueName: \"kubernetes.io/projected/76a3e4a4-64d9-4297-9a4b-be02f9baca70-kube-api-access-sblfw\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-cfsjc\" (UID: \"76a3e4a4-64d9-4297-9a4b-be02f9baca70\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-cfsjc"
Sep 16 18:59:21.928867 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:21.928828    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-cfsjc"
Sep 16 18:59:22.204855 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:22.204775    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-cfsjc"]
Sep 16 18:59:22.211467 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:59:22.211439    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76a3e4a4_64d9_4297_9a4b_be02f9baca70.slice/crio-e8defadd6d66ae540edd3eddb58f2bf356a66523e48cac328a6fd1041529d497 WatchSource:0}: Error finding container e8defadd6d66ae540edd3eddb58f2bf356a66523e48cac328a6fd1041529d497: Status 404 returned error can't find the container with id e8defadd6d66ae540edd3eddb58f2bf356a66523e48cac328a6fd1041529d497
Sep 16 18:59:23.023807 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:23.023767    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-cfsjc" event={"ID":"76a3e4a4-64d9-4297-9a4b-be02f9baca70","Type":"ContainerStarted","Data":"e8defadd6d66ae540edd3eddb58f2bf356a66523e48cac328a6fd1041529d497"}
Sep 16 18:59:24.029618 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:24.029579    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-cfsjc" event={"ID":"76a3e4a4-64d9-4297-9a4b-be02f9baca70","Type":"ContainerStarted","Data":"06c209ad6e2395617563f136df39fa7f9b1e80fcac20194dffa62c73fb064522"}
Sep 16 18:59:24.047635 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:24.047586    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-cfsjc" podStartSLOduration=1.6054905590000002 podStartE2EDuration="3.0475692s" totalImagesPullingTime="1.442078641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:59:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:59:24.046212324 +0000 UTC m=+967.687927985" watchObservedRunningTime="2026-09-16 18:59:24.0475692 +0000 UTC m=+967.689284909"
Sep 16 18:59:24.929430 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:24.929395    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-cfsjc"
Sep 16 18:59:28.089340 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:28.089305    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-cfsjc"]
Sep 16 18:59:28.098074 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:59:28.098040    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 18:59:28.098074 ip-10-0-129-194 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:59:28.098074 ip-10-0-129-194 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 18:59:28.098074 ip-10-0-129-194 kubenswrapper[2582]:  > containerID="06c209ad6e2395617563f136df39fa7f9b1e80fcac20194dffa62c73fb064522" cmd=["/bin/sleep","15"]
Sep 16 18:59:28.098295 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:28.098093    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-cfsjc" message="" err=<
Sep 16 18:59:28.098295 ip-10-0-129-194 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:59:28.098295 ip-10-0-129-194 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 18:59:28.098295 ip-10-0-129-194 kubenswrapper[2582]:  >
Sep 16 18:59:28.098295 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:59:28.098106    2582 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 18:59:28.098295 ip-10-0-129-194 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:59:28.098295 ip-10-0-129-194 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 18:59:28.098295 ip-10-0-129-194 kubenswrapper[2582]:  > pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-cfsjc" podUID="76a3e4a4-64d9-4297-9a4b-be02f9baca70" containerName="main" containerID="cri-o://06c209ad6e2395617563f136df39fa7f9b1e80fcac20194dffa62c73fb064522"
Sep 16 18:59:28.098295 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:28.098138    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-cfsjc" podUID="76a3e4a4-64d9-4297-9a4b-be02f9baca70" containerName="main" containerID="cri-o://06c209ad6e2395617563f136df39fa7f9b1e80fcac20194dffa62c73fb064522" gracePeriod=60
Sep 16 18:59:28.341080 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:28.341002    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-cfsjc"
Sep 16 18:59:28.521395 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:28.521362    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76a3e4a4-64d9-4297-9a4b-be02f9baca70-home\") pod \"76a3e4a4-64d9-4297-9a4b-be02f9baca70\" (UID: \"76a3e4a4-64d9-4297-9a4b-be02f9baca70\") "
Sep 16 18:59:28.521582 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:28.521414    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76a3e4a4-64d9-4297-9a4b-be02f9baca70-tmp-dir\") pod \"76a3e4a4-64d9-4297-9a4b-be02f9baca70\" (UID: \"76a3e4a4-64d9-4297-9a4b-be02f9baca70\") "
Sep 16 18:59:28.521582 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:28.521438    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sblfw\" (UniqueName: \"kubernetes.io/projected/76a3e4a4-64d9-4297-9a4b-be02f9baca70-kube-api-access-sblfw\") pod \"76a3e4a4-64d9-4297-9a4b-be02f9baca70\" (UID: \"76a3e4a4-64d9-4297-9a4b-be02f9baca70\") "
Sep 16 18:59:28.521582 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:28.521479    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76a3e4a4-64d9-4297-9a4b-be02f9baca70-model-cache\") pod \"76a3e4a4-64d9-4297-9a4b-be02f9baca70\" (UID: \"76a3e4a4-64d9-4297-9a4b-be02f9baca70\") "
Sep 16 18:59:28.521582 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:28.521514    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76a3e4a4-64d9-4297-9a4b-be02f9baca70-tls-certs\") pod \"76a3e4a4-64d9-4297-9a4b-be02f9baca70\" (UID: \"76a3e4a4-64d9-4297-9a4b-be02f9baca70\") "
Sep 16 18:59:28.521582 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:28.521548    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76a3e4a4-64d9-4297-9a4b-be02f9baca70-dshm\") pod \"76a3e4a4-64d9-4297-9a4b-be02f9baca70\" (UID: \"76a3e4a4-64d9-4297-9a4b-be02f9baca70\") "
Sep 16 18:59:28.521912 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:28.521690    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76a3e4a4-64d9-4297-9a4b-be02f9baca70-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "76a3e4a4-64d9-4297-9a4b-be02f9baca70" (UID: "76a3e4a4-64d9-4297-9a4b-be02f9baca70"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:59:28.521912 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:28.521706    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76a3e4a4-64d9-4297-9a4b-be02f9baca70-home" (OuterVolumeSpecName: "home") pod "76a3e4a4-64d9-4297-9a4b-be02f9baca70" (UID: "76a3e4a4-64d9-4297-9a4b-be02f9baca70"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:59:28.521912 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:28.521850    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76a3e4a4-64d9-4297-9a4b-be02f9baca70-model-cache" (OuterVolumeSpecName: "model-cache") pod "76a3e4a4-64d9-4297-9a4b-be02f9baca70" (UID: "76a3e4a4-64d9-4297-9a4b-be02f9baca70"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:59:28.523634 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:28.523609    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76a3e4a4-64d9-4297-9a4b-be02f9baca70-dshm" (OuterVolumeSpecName: "dshm") pod "76a3e4a4-64d9-4297-9a4b-be02f9baca70" (UID: "76a3e4a4-64d9-4297-9a4b-be02f9baca70"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:59:28.523634 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:28.523621    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76a3e4a4-64d9-4297-9a4b-be02f9baca70-kube-api-access-sblfw" (OuterVolumeSpecName: "kube-api-access-sblfw") pod "76a3e4a4-64d9-4297-9a4b-be02f9baca70" (UID: "76a3e4a4-64d9-4297-9a4b-be02f9baca70"). InnerVolumeSpecName "kube-api-access-sblfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:59:28.523745 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:28.523694    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76a3e4a4-64d9-4297-9a4b-be02f9baca70-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "76a3e4a4-64d9-4297-9a4b-be02f9baca70" (UID: "76a3e4a4-64d9-4297-9a4b-be02f9baca70"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:59:28.622300 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:28.622191    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76a3e4a4-64d9-4297-9a4b-be02f9baca70-home\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 18:59:28.622300 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:28.622237    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76a3e4a4-64d9-4297-9a4b-be02f9baca70-tmp-dir\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 18:59:28.622300 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:28.622252    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sblfw\" (UniqueName: \"kubernetes.io/projected/76a3e4a4-64d9-4297-9a4b-be02f9baca70-kube-api-access-sblfw\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 18:59:28.622300 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:28.622262    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76a3e4a4-64d9-4297-9a4b-be02f9baca70-model-cache\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 18:59:28.622300 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:28.622270    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76a3e4a4-64d9-4297-9a4b-be02f9baca70-tls-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 18:59:28.622300 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:28.622279    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76a3e4a4-64d9-4297-9a4b-be02f9baca70-dshm\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 18:59:29.052604 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:29.052566    2582 generic.go:356] "Generic (PLEG): container finished" podID="76a3e4a4-64d9-4297-9a4b-be02f9baca70" containerID="06c209ad6e2395617563f136df39fa7f9b1e80fcac20194dffa62c73fb064522" exitCode=0
Sep 16 18:59:29.052789 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:29.052615    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-cfsjc"
Sep 16 18:59:29.052789 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:29.052631    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-cfsjc" event={"ID":"76a3e4a4-64d9-4297-9a4b-be02f9baca70","Type":"ContainerDied","Data":"06c209ad6e2395617563f136df39fa7f9b1e80fcac20194dffa62c73fb064522"}
Sep 16 18:59:29.052789 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:29.052685    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-cfsjc" event={"ID":"76a3e4a4-64d9-4297-9a4b-be02f9baca70","Type":"ContainerDied","Data":"e8defadd6d66ae540edd3eddb58f2bf356a66523e48cac328a6fd1041529d497"}
Sep 16 18:59:29.052789 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:29.052714    2582 scope.go:117] "RemoveContainer" containerID="06c209ad6e2395617563f136df39fa7f9b1e80fcac20194dffa62c73fb064522"
Sep 16 18:59:29.061282 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:29.061254    2582 scope.go:117] "RemoveContainer" containerID="06c209ad6e2395617563f136df39fa7f9b1e80fcac20194dffa62c73fb064522"
Sep 16 18:59:29.061549 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:59:29.061525    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06c209ad6e2395617563f136df39fa7f9b1e80fcac20194dffa62c73fb064522\": container with ID starting with 06c209ad6e2395617563f136df39fa7f9b1e80fcac20194dffa62c73fb064522 not found: ID does not exist" containerID="06c209ad6e2395617563f136df39fa7f9b1e80fcac20194dffa62c73fb064522"
Sep 16 18:59:29.061622 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:29.061560    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06c209ad6e2395617563f136df39fa7f9b1e80fcac20194dffa62c73fb064522"} err="failed to get container status \"06c209ad6e2395617563f136df39fa7f9b1e80fcac20194dffa62c73fb064522\": rpc error: code = NotFound desc = could not find container \"06c209ad6e2395617563f136df39fa7f9b1e80fcac20194dffa62c73fb064522\": container with ID starting with 06c209ad6e2395617563f136df39fa7f9b1e80fcac20194dffa62c73fb064522 not found: ID does not exist"
Sep 16 18:59:29.073910 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:29.073881    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-cfsjc"]
Sep 16 18:59:29.078787 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:29.078763    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-cfsjc"]
Sep 16 18:59:30.873073 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:30.873033    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76a3e4a4-64d9-4297-9a4b-be02f9baca70" path="/var/lib/kubelet/pods/76a3e4a4-64d9-4297-9a4b-be02f9baca70/volumes"
Sep 16 18:59:43.811053 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:43.811012    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf75qzt"]
Sep 16 18:59:43.811560 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:43.811462    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76a3e4a4-64d9-4297-9a4b-be02f9baca70" containerName="main"
Sep 16 18:59:43.811560 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:43.811482    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="76a3e4a4-64d9-4297-9a4b-be02f9baca70" containerName="main"
Sep 16 18:59:43.811682 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:43.811586    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="76a3e4a4-64d9-4297-9a4b-be02f9baca70" containerName="main"
Sep 16 18:59:43.814524 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:43.814500    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf75qzt"
Sep 16 18:59:43.817123 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:43.817103    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Sep 16 18:59:43.818136 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:43.818121    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Sep 16 18:59:43.818209 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:43.818158    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"default-dockercfg-dxzfr\""
Sep 16 18:59:43.818209 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:43.818169    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\""
Sep 16 18:59:43.826094 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:43.826070    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf75qzt"]
Sep 16 18:59:43.830583 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:43.830555    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f4cdbb5-23d2-46f0-9c66-fee8872d8537-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf75qzt\" (UID: \"0f4cdbb5-23d2-46f0-9c66-fee8872d8537\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf75qzt"
Sep 16 18:59:43.830698 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:43.830613    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f4cdbb5-23d2-46f0-9c66-fee8872d8537-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf75qzt\" (UID: \"0f4cdbb5-23d2-46f0-9c66-fee8872d8537\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf75qzt"
Sep 16 18:59:43.830698 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:43.830677    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f4cdbb5-23d2-46f0-9c66-fee8872d8537-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf75qzt\" (UID: \"0f4cdbb5-23d2-46f0-9c66-fee8872d8537\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf75qzt"
Sep 16 18:59:43.830830 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:43.830754    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f4cdbb5-23d2-46f0-9c66-fee8872d8537-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf75qzt\" (UID: \"0f4cdbb5-23d2-46f0-9c66-fee8872d8537\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf75qzt"
Sep 16 18:59:43.830830 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:43.830794    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnvnw\" (UniqueName: \"kubernetes.io/projected/0f4cdbb5-23d2-46f0-9c66-fee8872d8537-kube-api-access-bnvnw\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf75qzt\" (UID: \"0f4cdbb5-23d2-46f0-9c66-fee8872d8537\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf75qzt"
Sep 16 18:59:43.830830 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:43.830828    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f4cdbb5-23d2-46f0-9c66-fee8872d8537-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf75qzt\" (UID: \"0f4cdbb5-23d2-46f0-9c66-fee8872d8537\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf75qzt"
Sep 16 18:59:43.931925 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:43.931879    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f4cdbb5-23d2-46f0-9c66-fee8872d8537-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf75qzt\" (UID: \"0f4cdbb5-23d2-46f0-9c66-fee8872d8537\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf75qzt"
Sep 16 18:59:43.932134 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:43.931935    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnvnw\" (UniqueName: \"kubernetes.io/projected/0f4cdbb5-23d2-46f0-9c66-fee8872d8537-kube-api-access-bnvnw\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf75qzt\" (UID: \"0f4cdbb5-23d2-46f0-9c66-fee8872d8537\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf75qzt"
Sep 16 18:59:43.932134 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:43.931985    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f4cdbb5-23d2-46f0-9c66-fee8872d8537-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf75qzt\" (UID: \"0f4cdbb5-23d2-46f0-9c66-fee8872d8537\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf75qzt"
Sep 16 18:59:43.932260 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:43.932191    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f4cdbb5-23d2-46f0-9c66-fee8872d8537-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf75qzt\" (UID: \"0f4cdbb5-23d2-46f0-9c66-fee8872d8537\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf75qzt"
Sep 16 18:59:43.932322 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:43.932281    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f4cdbb5-23d2-46f0-9c66-fee8872d8537-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf75qzt\" (UID: \"0f4cdbb5-23d2-46f0-9c66-fee8872d8537\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf75qzt"
Sep 16 18:59:43.932380 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:43.932324    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f4cdbb5-23d2-46f0-9c66-fee8872d8537-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf75qzt\" (UID: \"0f4cdbb5-23d2-46f0-9c66-fee8872d8537\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf75qzt"
Sep 16 18:59:43.932654 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:43.932631    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f4cdbb5-23d2-46f0-9c66-fee8872d8537-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf75qzt\" (UID: \"0f4cdbb5-23d2-46f0-9c66-fee8872d8537\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf75qzt"
Sep 16 18:59:43.932784 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:43.932658    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f4cdbb5-23d2-46f0-9c66-fee8872d8537-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf75qzt\" (UID: \"0f4cdbb5-23d2-46f0-9c66-fee8872d8537\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf75qzt"
Sep 16 18:59:43.932784 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:43.932706    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f4cdbb5-23d2-46f0-9c66-fee8872d8537-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf75qzt\" (UID: \"0f4cdbb5-23d2-46f0-9c66-fee8872d8537\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf75qzt"
Sep 16 18:59:43.934145 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:43.934128    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f4cdbb5-23d2-46f0-9c66-fee8872d8537-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf75qzt\" (UID: \"0f4cdbb5-23d2-46f0-9c66-fee8872d8537\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf75qzt"
Sep 16 18:59:43.934592 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:43.934577    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f4cdbb5-23d2-46f0-9c66-fee8872d8537-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf75qzt\" (UID: \"0f4cdbb5-23d2-46f0-9c66-fee8872d8537\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf75qzt"
Sep 16 18:59:43.940053 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:43.940028    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnvnw\" (UniqueName: \"kubernetes.io/projected/0f4cdbb5-23d2-46f0-9c66-fee8872d8537-kube-api-access-bnvnw\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf75qzt\" (UID: \"0f4cdbb5-23d2-46f0-9c66-fee8872d8537\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf75qzt"
Sep 16 18:59:44.125228 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:44.125143    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf75qzt"
Sep 16 18:59:44.394751 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:44.394656    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf75qzt"]
Sep 16 18:59:44.397581 ip-10-0-129-194 kubenswrapper[2582]: W0916 18:59:44.397549    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f4cdbb5_23d2_46f0_9c66_fee8872d8537.slice/crio-17281e49272a2c78778123ba5adac8abcbb78cd1a5e26883fe3ffc0a0050f186 WatchSource:0}: Error finding container 17281e49272a2c78778123ba5adac8abcbb78cd1a5e26883fe3ffc0a0050f186: Status 404 returned error can't find the container with id 17281e49272a2c78778123ba5adac8abcbb78cd1a5e26883fe3ffc0a0050f186
Sep 16 18:59:45.125488 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:45.125449    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf75qzt" event={"ID":"0f4cdbb5-23d2-46f0-9c66-fee8872d8537","Type":"ContainerStarted","Data":"dd258317b966a0a7d24668e3cf97e9ab7bb81e0e0f257efa9be24947c369ae84"}
Sep 16 18:59:45.125934 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:45.125505    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf75qzt" event={"ID":"0f4cdbb5-23d2-46f0-9c66-fee8872d8537","Type":"ContainerStarted","Data":"17281e49272a2c78778123ba5adac8abcbb78cd1a5e26883fe3ffc0a0050f186"}
Sep 16 18:59:45.145270 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:45.145220    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf75qzt" podStartSLOduration=2.145204492 podStartE2EDuration="2.145204492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:59:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:59:45.143206437 +0000 UTC m=+988.784922102" watchObservedRunningTime="2026-09-16 18:59:45.145204492 +0000 UTC m=+988.786920155"
Sep 16 18:59:46.125601 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:46.125558    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf75qzt"
Sep 16 18:59:46.824359 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:46.824324    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf75qzt"]
Sep 16 18:59:47.159166 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:59:47.159062    2582 secret.go:189] Couldn't get secret e2e-test-gateway-section-name-propagation-94799d44/gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs: secret "gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs" not found
Sep 16 18:59:47.159166 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:59:47.159144    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f4cdbb5-23d2-46f0-9c66-fee8872d8537-tls-certs podName:0f4cdbb5-23d2-46f0-9c66-fee8872d8537 nodeName:}" failed. No retries permitted until 2026-09-16 18:59:47.65912756 +0000 UTC m=+991.300843223 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/0f4cdbb5-23d2-46f0-9c66-fee8872d8537-tls-certs") pod "gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf75qzt" (UID: "0f4cdbb5-23d2-46f0-9c66-fee8872d8537") : secret "gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs" not found
Sep 16 18:59:47.663869 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:59:47.663833    2582 secret.go:189] Couldn't get secret e2e-test-gateway-section-name-propagation-94799d44/gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs: secret "gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs" not found
Sep 16 18:59:47.664069 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:59:47.663899    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f4cdbb5-23d2-46f0-9c66-fee8872d8537-tls-certs podName:0f4cdbb5-23d2-46f0-9c66-fee8872d8537 nodeName:}" failed. No retries permitted until 2026-09-16 18:59:48.663885374 +0000 UTC m=+992.305601016 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/0f4cdbb5-23d2-46f0-9c66-fee8872d8537-tls-certs") pod "gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf75qzt" (UID: "0f4cdbb5-23d2-46f0-9c66-fee8872d8537") : secret "gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs" not found
Sep 16 18:59:48.153509 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:59:48.153473    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 18:59:48.153509 ip-10-0-129-194 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:59:48.153509 ip-10-0-129-194 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 18:59:48.153509 ip-10-0-129-194 kubenswrapper[2582]:  > containerID="dd258317b966a0a7d24668e3cf97e9ab7bb81e0e0f257efa9be24947c369ae84" cmd=["/bin/sleep","15"]
Sep 16 18:59:48.153815 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:48.153520    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf75qzt" message="" err=<
Sep 16 18:59:48.153815 ip-10-0-129-194 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:59:48.153815 ip-10-0-129-194 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 18:59:48.153815 ip-10-0-129-194 kubenswrapper[2582]:  >
Sep 16 18:59:48.153815 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:59:48.153531    2582 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 18:59:48.153815 ip-10-0-129-194 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:59:48.153815 ip-10-0-129-194 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 18:59:48.153815 ip-10-0-129-194 kubenswrapper[2582]:  > pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf75qzt" podUID="0f4cdbb5-23d2-46f0-9c66-fee8872d8537" containerName="main" containerID="cri-o://dd258317b966a0a7d24668e3cf97e9ab7bb81e0e0f257efa9be24947c369ae84"
Sep 16 18:59:48.153815 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:48.153571    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf75qzt" podUID="0f4cdbb5-23d2-46f0-9c66-fee8872d8537" containerName="main" containerID="cri-o://dd258317b966a0a7d24668e3cf97e9ab7bb81e0e0f257efa9be24947c369ae84" gracePeriod=60
Sep 16 18:59:48.401569 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:48.401544    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf75qzt"
Sep 16 18:59:48.472334 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:48.472248    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f4cdbb5-23d2-46f0-9c66-fee8872d8537-tmp-dir\") pod \"0f4cdbb5-23d2-46f0-9c66-fee8872d8537\" (UID: \"0f4cdbb5-23d2-46f0-9c66-fee8872d8537\") "
Sep 16 18:59:48.472334 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:48.472320    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bnvnw\" (UniqueName: \"kubernetes.io/projected/0f4cdbb5-23d2-46f0-9c66-fee8872d8537-kube-api-access-bnvnw\") pod \"0f4cdbb5-23d2-46f0-9c66-fee8872d8537\" (UID: \"0f4cdbb5-23d2-46f0-9c66-fee8872d8537\") "
Sep 16 18:59:48.472512 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:48.472352    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f4cdbb5-23d2-46f0-9c66-fee8872d8537-home\") pod \"0f4cdbb5-23d2-46f0-9c66-fee8872d8537\" (UID: \"0f4cdbb5-23d2-46f0-9c66-fee8872d8537\") "
Sep 16 18:59:48.472512 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:48.472394    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f4cdbb5-23d2-46f0-9c66-fee8872d8537-model-cache\") pod \"0f4cdbb5-23d2-46f0-9c66-fee8872d8537\" (UID: \"0f4cdbb5-23d2-46f0-9c66-fee8872d8537\") "
Sep 16 18:59:48.472512 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:48.472423    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f4cdbb5-23d2-46f0-9c66-fee8872d8537-dshm\") pod \"0f4cdbb5-23d2-46f0-9c66-fee8872d8537\" (UID: \"0f4cdbb5-23d2-46f0-9c66-fee8872d8537\") "
Sep 16 18:59:48.472512 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:48.472497    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f4cdbb5-23d2-46f0-9c66-fee8872d8537-tls-certs\") pod \"0f4cdbb5-23d2-46f0-9c66-fee8872d8537\" (UID: \"0f4cdbb5-23d2-46f0-9c66-fee8872d8537\") "
Sep 16 18:59:48.472672 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:48.472618    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f4cdbb5-23d2-46f0-9c66-fee8872d8537-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0f4cdbb5-23d2-46f0-9c66-fee8872d8537" (UID: "0f4cdbb5-23d2-46f0-9c66-fee8872d8537"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:59:48.472672 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:48.472632    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f4cdbb5-23d2-46f0-9c66-fee8872d8537-home" (OuterVolumeSpecName: "home") pod "0f4cdbb5-23d2-46f0-9c66-fee8872d8537" (UID: "0f4cdbb5-23d2-46f0-9c66-fee8872d8537"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:59:48.472794 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:48.472767    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f4cdbb5-23d2-46f0-9c66-fee8872d8537-model-cache" (OuterVolumeSpecName: "model-cache") pod "0f4cdbb5-23d2-46f0-9c66-fee8872d8537" (UID: "0f4cdbb5-23d2-46f0-9c66-fee8872d8537"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:59:48.472847 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:48.472789    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f4cdbb5-23d2-46f0-9c66-fee8872d8537-home\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 18:59:48.472847 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:48.472807    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f4cdbb5-23d2-46f0-9c66-fee8872d8537-tmp-dir\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 18:59:48.474842 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:48.474813    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f4cdbb5-23d2-46f0-9c66-fee8872d8537-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0f4cdbb5-23d2-46f0-9c66-fee8872d8537" (UID: "0f4cdbb5-23d2-46f0-9c66-fee8872d8537"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:59:48.474842 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:48.474820    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f4cdbb5-23d2-46f0-9c66-fee8872d8537-kube-api-access-bnvnw" (OuterVolumeSpecName: "kube-api-access-bnvnw") pod "0f4cdbb5-23d2-46f0-9c66-fee8872d8537" (UID: "0f4cdbb5-23d2-46f0-9c66-fee8872d8537"). InnerVolumeSpecName "kube-api-access-bnvnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:59:48.475056 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:48.474866    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f4cdbb5-23d2-46f0-9c66-fee8872d8537-dshm" (OuterVolumeSpecName: "dshm") pod "0f4cdbb5-23d2-46f0-9c66-fee8872d8537" (UID: "0f4cdbb5-23d2-46f0-9c66-fee8872d8537"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:59:48.574242 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:48.574209    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f4cdbb5-23d2-46f0-9c66-fee8872d8537-model-cache\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 18:59:48.574242 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:48.574240    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f4cdbb5-23d2-46f0-9c66-fee8872d8537-dshm\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 18:59:48.574242 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:48.574248    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f4cdbb5-23d2-46f0-9c66-fee8872d8537-tls-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 18:59:48.574467 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:48.574257    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnvnw\" (UniqueName: \"kubernetes.io/projected/0f4cdbb5-23d2-46f0-9c66-fee8872d8537-kube-api-access-bnvnw\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 18:59:49.150222 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:49.150126    2582 generic.go:356] "Generic (PLEG): container finished" podID="0f4cdbb5-23d2-46f0-9c66-fee8872d8537" containerID="dd258317b966a0a7d24668e3cf97e9ab7bb81e0e0f257efa9be24947c369ae84" exitCode=0
Sep 16 18:59:49.150222 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:49.150170    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf75qzt"
Sep 16 18:59:49.150222 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:49.150210    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf75qzt" event={"ID":"0f4cdbb5-23d2-46f0-9c66-fee8872d8537","Type":"ContainerDied","Data":"dd258317b966a0a7d24668e3cf97e9ab7bb81e0e0f257efa9be24947c369ae84"}
Sep 16 18:59:49.150461 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:49.150261    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf75qzt" event={"ID":"0f4cdbb5-23d2-46f0-9c66-fee8872d8537","Type":"ContainerDied","Data":"17281e49272a2c78778123ba5adac8abcbb78cd1a5e26883fe3ffc0a0050f186"}
Sep 16 18:59:49.150461 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:49.150283    2582 scope.go:117] "RemoveContainer" containerID="dd258317b966a0a7d24668e3cf97e9ab7bb81e0e0f257efa9be24947c369ae84"
Sep 16 18:59:49.158877 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:49.158636    2582 scope.go:117] "RemoveContainer" containerID="dd258317b966a0a7d24668e3cf97e9ab7bb81e0e0f257efa9be24947c369ae84"
Sep 16 18:59:49.158977 ip-10-0-129-194 kubenswrapper[2582]: E0916 18:59:49.158953    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd258317b966a0a7d24668e3cf97e9ab7bb81e0e0f257efa9be24947c369ae84\": container with ID starting with dd258317b966a0a7d24668e3cf97e9ab7bb81e0e0f257efa9be24947c369ae84 not found: ID does not exist" containerID="dd258317b966a0a7d24668e3cf97e9ab7bb81e0e0f257efa9be24947c369ae84"
Sep 16 18:59:49.159021 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:49.158986    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd258317b966a0a7d24668e3cf97e9ab7bb81e0e0f257efa9be24947c369ae84"} err="failed to get container status \"dd258317b966a0a7d24668e3cf97e9ab7bb81e0e0f257efa9be24947c369ae84\": rpc error: code = NotFound desc = could not find container \"dd258317b966a0a7d24668e3cf97e9ab7bb81e0e0f257efa9be24947c369ae84\": container with ID starting with dd258317b966a0a7d24668e3cf97e9ab7bb81e0e0f257efa9be24947c369ae84 not found: ID does not exist"
Sep 16 18:59:49.171192 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:49.171168    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf75qzt"]
Sep 16 18:59:49.175595 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:49.175573    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf75qzt"]
Sep 16 18:59:50.872601 ip-10-0-129-194 kubenswrapper[2582]: I0916 18:59:50.872570    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f4cdbb5-23d2-46f0-9c66-fee8872d8537" path="/var/lib/kubelet/pods/0f4cdbb5-23d2-46f0-9c66-fee8872d8537/volumes"
Sep 16 19:00:05.395835 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:00:05.395799    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-c6cbd9685-zmcnv"]
Sep 16 19:00:05.396282 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:00:05.396254    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f4cdbb5-23d2-46f0-9c66-fee8872d8537" containerName="main"
Sep 16 19:00:05.396282 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:00:05.396272    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f4cdbb5-23d2-46f0-9c66-fee8872d8537" containerName="main"
Sep 16 19:00:05.396389 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:00:05.396363    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f4cdbb5-23d2-46f0-9c66-fee8872d8537" containerName="main"
Sep 16 19:00:05.404542 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:00:05.404516    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-c6cbd9685-zmcnv"
Sep 16 19:00:05.409523 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:00:05.409480    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"seaweedfs-s3-creds\""
Sep 16 19:00:05.409831 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:00:05.409808    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Sep 16 19:00:05.409915 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:00:05.409808    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Sep 16 19:00:05.409994 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:00:05.409972    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-hn45b\""
Sep 16 19:00:05.410068 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:00:05.410053    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:00:05.410596 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:00:05.410577    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Sep 16 19:00:05.417570 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:00:05.417549    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-c6cbd9685-zmcnv"]
Sep 16 19:00:05.513907 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:00:05.513869    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e84de93-b11b-4337-8e09-9de471ae46a0-cabundle-cert\") pod \"auth-enabled-test-kserve-c6cbd9685-zmcnv\" (UID: \"7e84de93-b11b-4337-8e09-9de471ae46a0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-c6cbd9685-zmcnv"
Sep 16 19:00:05.514053 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:00:05.513910    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e84de93-b11b-4337-8e09-9de471ae46a0-tmp-dir\") pod \"auth-enabled-test-kserve-c6cbd9685-zmcnv\" (UID: \"7e84de93-b11b-4337-8e09-9de471ae46a0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-c6cbd9685-zmcnv"
Sep 16 19:00:05.514053 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:00:05.514015    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e84de93-b11b-4337-8e09-9de471ae46a0-kserve-provision-location\") pod \"auth-enabled-test-kserve-c6cbd9685-zmcnv\" (UID: \"7e84de93-b11b-4337-8e09-9de471ae46a0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-c6cbd9685-zmcnv"
Sep 16 19:00:05.514144 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:00:05.514086    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e84de93-b11b-4337-8e09-9de471ae46a0-home\") pod \"auth-enabled-test-kserve-c6cbd9685-zmcnv\" (UID: \"7e84de93-b11b-4337-8e09-9de471ae46a0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-c6cbd9685-zmcnv"
Sep 16 19:00:05.514144 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:00:05.514116    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e84de93-b11b-4337-8e09-9de471ae46a0-tls-certs\") pod \"auth-enabled-test-kserve-c6cbd9685-zmcnv\" (UID: \"7e84de93-b11b-4337-8e09-9de471ae46a0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-c6cbd9685-zmcnv"
Sep 16 19:00:05.514217 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:00:05.514168    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e84de93-b11b-4337-8e09-9de471ae46a0-model-cache\") pod \"auth-enabled-test-kserve-c6cbd9685-zmcnv\" (UID: \"7e84de93-b11b-4337-8e09-9de471ae46a0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-c6cbd9685-zmcnv"
Sep 16 19:00:05.514217 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:00:05.514199    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e84de93-b11b-4337-8e09-9de471ae46a0-dshm\") pod \"auth-enabled-test-kserve-c6cbd9685-zmcnv\" (UID: \"7e84de93-b11b-4337-8e09-9de471ae46a0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-c6cbd9685-zmcnv"
Sep 16 19:00:05.514282 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:00:05.514272    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jdw8\" (UniqueName: \"kubernetes.io/projected/7e84de93-b11b-4337-8e09-9de471ae46a0-kube-api-access-2jdw8\") pod \"auth-enabled-test-kserve-c6cbd9685-zmcnv\" (UID: \"7e84de93-b11b-4337-8e09-9de471ae46a0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-c6cbd9685-zmcnv"
Sep 16 19:00:05.615534 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:00:05.615496    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jdw8\" (UniqueName: \"kubernetes.io/projected/7e84de93-b11b-4337-8e09-9de471ae46a0-kube-api-access-2jdw8\") pod \"auth-enabled-test-kserve-c6cbd9685-zmcnv\" (UID: \"7e84de93-b11b-4337-8e09-9de471ae46a0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-c6cbd9685-zmcnv"
Sep 16 19:00:05.615652 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:00:05.615543    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e84de93-b11b-4337-8e09-9de471ae46a0-cabundle-cert\") pod \"auth-enabled-test-kserve-c6cbd9685-zmcnv\" (UID: \"7e84de93-b11b-4337-8e09-9de471ae46a0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-c6cbd9685-zmcnv"
Sep 16 19:00:05.615652 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:00:05.615565    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e84de93-b11b-4337-8e09-9de471ae46a0-tmp-dir\") pod \"auth-enabled-test-kserve-c6cbd9685-zmcnv\" (UID: \"7e84de93-b11b-4337-8e09-9de471ae46a0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-c6cbd9685-zmcnv"
Sep 16 19:00:05.615652 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:00:05.615590    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e84de93-b11b-4337-8e09-9de471ae46a0-kserve-provision-location\") pod \"auth-enabled-test-kserve-c6cbd9685-zmcnv\" (UID: \"7e84de93-b11b-4337-8e09-9de471ae46a0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-c6cbd9685-zmcnv"
Sep 16 19:00:05.615652 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:00:05.615620    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e84de93-b11b-4337-8e09-9de471ae46a0-home\") pod \"auth-enabled-test-kserve-c6cbd9685-zmcnv\" (UID: \"7e84de93-b11b-4337-8e09-9de471ae46a0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-c6cbd9685-zmcnv"
Sep 16 19:00:05.615652 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:00:05.615635    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e84de93-b11b-4337-8e09-9de471ae46a0-tls-certs\") pod \"auth-enabled-test-kserve-c6cbd9685-zmcnv\" (UID: \"7e84de93-b11b-4337-8e09-9de471ae46a0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-c6cbd9685-zmcnv"
Sep 16 19:00:05.615961 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:00:05.615661    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e84de93-b11b-4337-8e09-9de471ae46a0-model-cache\") pod \"auth-enabled-test-kserve-c6cbd9685-zmcnv\" (UID: \"7e84de93-b11b-4337-8e09-9de471ae46a0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-c6cbd9685-zmcnv"
Sep 16 19:00:05.615961 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:00:05.615689    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e84de93-b11b-4337-8e09-9de471ae46a0-dshm\") pod \"auth-enabled-test-kserve-c6cbd9685-zmcnv\" (UID: \"7e84de93-b11b-4337-8e09-9de471ae46a0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-c6cbd9685-zmcnv"
Sep 16 19:00:05.616084 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:00:05.616058    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e84de93-b11b-4337-8e09-9de471ae46a0-tmp-dir\") pod \"auth-enabled-test-kserve-c6cbd9685-zmcnv\" (UID: \"7e84de93-b11b-4337-8e09-9de471ae46a0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-c6cbd9685-zmcnv"
Sep 16 19:00:05.616136 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:00:05.616077    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e84de93-b11b-4337-8e09-9de471ae46a0-home\") pod \"auth-enabled-test-kserve-c6cbd9685-zmcnv\" (UID: \"7e84de93-b11b-4337-8e09-9de471ae46a0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-c6cbd9685-zmcnv"
Sep 16 19:00:05.616189 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:00:05.616155    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e84de93-b11b-4337-8e09-9de471ae46a0-kserve-provision-location\") pod \"auth-enabled-test-kserve-c6cbd9685-zmcnv\" (UID: \"7e84de93-b11b-4337-8e09-9de471ae46a0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-c6cbd9685-zmcnv"
Sep 16 19:00:05.616189 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:00:05.616180    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e84de93-b11b-4337-8e09-9de471ae46a0-model-cache\") pod \"auth-enabled-test-kserve-c6cbd9685-zmcnv\" (UID: \"7e84de93-b11b-4337-8e09-9de471ae46a0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-c6cbd9685-zmcnv"
Sep 16 19:00:05.616444 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:00:05.616420    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e84de93-b11b-4337-8e09-9de471ae46a0-cabundle-cert\") pod \"auth-enabled-test-kserve-c6cbd9685-zmcnv\" (UID: \"7e84de93-b11b-4337-8e09-9de471ae46a0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-c6cbd9685-zmcnv"
Sep 16 19:00:05.618106 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:00:05.618078    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e84de93-b11b-4337-8e09-9de471ae46a0-dshm\") pod \"auth-enabled-test-kserve-c6cbd9685-zmcnv\" (UID: \"7e84de93-b11b-4337-8e09-9de471ae46a0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-c6cbd9685-zmcnv"
Sep 16 19:00:05.618358 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:00:05.618341    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e84de93-b11b-4337-8e09-9de471ae46a0-tls-certs\") pod \"auth-enabled-test-kserve-c6cbd9685-zmcnv\" (UID: \"7e84de93-b11b-4337-8e09-9de471ae46a0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-c6cbd9685-zmcnv"
Sep 16 19:00:05.624112 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:00:05.624089    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jdw8\" (UniqueName: \"kubernetes.io/projected/7e84de93-b11b-4337-8e09-9de471ae46a0-kube-api-access-2jdw8\") pod \"auth-enabled-test-kserve-c6cbd9685-zmcnv\" (UID: \"7e84de93-b11b-4337-8e09-9de471ae46a0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-c6cbd9685-zmcnv"
Sep 16 19:00:05.714397 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:00:05.714301    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-c6cbd9685-zmcnv"
Sep 16 19:00:05.997104 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:00:05.997074    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-c6cbd9685-zmcnv"]
Sep 16 19:00:05.999511 ip-10-0-129-194 kubenswrapper[2582]: W0916 19:00:05.999481    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e84de93_b11b_4337_8e09_9de471ae46a0.slice/crio-7531949a67eebbc065ed162482fd3fc09df9223d299d28ccb298b1a36373d1fe WatchSource:0}: Error finding container 7531949a67eebbc065ed162482fd3fc09df9223d299d28ccb298b1a36373d1fe: Status 404 returned error can't find the container with id 7531949a67eebbc065ed162482fd3fc09df9223d299d28ccb298b1a36373d1fe
Sep 16 19:00:06.231506 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:00:06.231469    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-c6cbd9685-zmcnv" event={"ID":"7e84de93-b11b-4337-8e09-9de471ae46a0","Type":"ContainerStarted","Data":"60f72fe9115501b33a892638ca67ab815dc82c67b06cf7f90256173f6837ad5f"}
Sep 16 19:00:06.231672 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:00:06.231528    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-c6cbd9685-zmcnv" event={"ID":"7e84de93-b11b-4337-8e09-9de471ae46a0","Type":"ContainerStarted","Data":"7531949a67eebbc065ed162482fd3fc09df9223d299d28ccb298b1a36373d1fe"}
Sep 16 19:00:09.244919 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:00:09.244883    2582 generic.go:356] "Generic (PLEG): container finished" podID="7e84de93-b11b-4337-8e09-9de471ae46a0" containerID="60f72fe9115501b33a892638ca67ab815dc82c67b06cf7f90256173f6837ad5f" exitCode=0
Sep 16 19:00:09.245387 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:00:09.244923    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-c6cbd9685-zmcnv" event={"ID":"7e84de93-b11b-4337-8e09-9de471ae46a0","Type":"ContainerDied","Data":"60f72fe9115501b33a892638ca67ab815dc82c67b06cf7f90256173f6837ad5f"}
Sep 16 19:00:10.251683 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:00:10.251644    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-c6cbd9685-zmcnv" event={"ID":"7e84de93-b11b-4337-8e09-9de471ae46a0","Type":"ContainerStarted","Data":"1310d1a6409e9496e86ccba8d0dbd02065dd8327308e5511e61c14a746bac598"}
Sep 16 19:00:10.298006 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:00:10.297937    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-c6cbd9685-zmcnv" podStartSLOduration=3.121170333 podStartE2EDuration="5.297911904s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.176741571s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:00:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:00:10.29532793 +0000 UTC m=+1013.937043596" watchObservedRunningTime="2026-09-16 19:00:10.297911904 +0000 UTC m=+1013.939627570"
Sep 16 19:00:10.714890 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:00:10.714835    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-c6cbd9685-zmcnv"
Sep 16 19:00:15.715438 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:00:15.715388    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-c6cbd9685-zmcnv"
Sep 16 19:00:15.717303 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:00:15.717267    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-c6cbd9685-zmcnv" podUID="7e84de93-b11b-4337-8e09-9de471ae46a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.31:8000/health\": dial tcp 10.132.0.31:8000: connect: connection refused"
Sep 16 19:00:25.715274 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:00:25.715226    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-c6cbd9685-zmcnv" podUID="7e84de93-b11b-4337-8e09-9de471ae46a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.31:8000/health\": dial tcp 10.132.0.31:8000: connect: connection refused"
Sep 16 19:00:35.715035 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:00:35.714985    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-c6cbd9685-zmcnv" podUID="7e84de93-b11b-4337-8e09-9de471ae46a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.31:8000/health\": dial tcp 10.132.0.31:8000: connect: connection refused"
Sep 16 19:00:45.715635 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:00:45.715596    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-c6cbd9685-zmcnv" podUID="7e84de93-b11b-4337-8e09-9de471ae46a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.31:8000/health\": dial tcp 10.132.0.31:8000: connect: connection refused"
Sep 16 19:00:55.715476 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:00:55.715431    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-c6cbd9685-zmcnv" podUID="7e84de93-b11b-4337-8e09-9de471ae46a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.31:8000/health\": dial tcp 10.132.0.31:8000: connect: connection refused"
Sep 16 19:01:05.714875 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:01:05.714825    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-c6cbd9685-zmcnv" podUID="7e84de93-b11b-4337-8e09-9de471ae46a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.31:8000/health\": dial tcp 10.132.0.31:8000: connect: connection refused"
Sep 16 19:01:15.715231 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:01:15.715184    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-c6cbd9685-zmcnv" podUID="7e84de93-b11b-4337-8e09-9de471ae46a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.31:8000/health\": dial tcp 10.132.0.31:8000: connect: connection refused"
Sep 16 19:01:25.715423 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:01:25.715374    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-c6cbd9685-zmcnv" podUID="7e84de93-b11b-4337-8e09-9de471ae46a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.31:8000/health\": dial tcp 10.132.0.31:8000: connect: connection refused"
Sep 16 19:01:35.715233 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:01:35.715186    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-c6cbd9685-zmcnv" podUID="7e84de93-b11b-4337-8e09-9de471ae46a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.31:8000/health\": dial tcp 10.132.0.31:8000: connect: connection refused"
Sep 16 19:01:45.715496 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:01:45.715452    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-c6cbd9685-zmcnv" podUID="7e84de93-b11b-4337-8e09-9de471ae46a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.31:8000/health\": dial tcp 10.132.0.31:8000: connect: connection refused"
Sep 16 19:01:55.715154 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:01:55.715113    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-c6cbd9685-zmcnv" podUID="7e84de93-b11b-4337-8e09-9de471ae46a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.31:8000/health\": dial tcp 10.132.0.31:8000: connect: connection refused"
Sep 16 19:02:05.724733 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:02:05.724696    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-c6cbd9685-zmcnv"
Sep 16 19:02:05.732910 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:02:05.732882    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-c6cbd9685-zmcnv"
Sep 16 19:02:10.827923 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:02:10.827889    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-c6cbd9685-zmcnv"]
Sep 16 19:02:25.850204 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:02:25.850127    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-c6cbd9685-zmcnv" podUID="7e84de93-b11b-4337-8e09-9de471ae46a0" containerName="main" containerID="cri-o://1310d1a6409e9496e86ccba8d0dbd02065dd8327308e5511e61c14a746bac598" gracePeriod=45
Sep 16 19:02:26.584412 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:02:26.584388    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-c6cbd9685-zmcnv"
Sep 16 19:02:26.630625 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:02:26.630535    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e84de93-b11b-4337-8e09-9de471ae46a0-tls-certs\") pod \"7e84de93-b11b-4337-8e09-9de471ae46a0\" (UID: \"7e84de93-b11b-4337-8e09-9de471ae46a0\") "
Sep 16 19:02:26.630625 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:02:26.630603    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e84de93-b11b-4337-8e09-9de471ae46a0-tmp-dir\") pod \"7e84de93-b11b-4337-8e09-9de471ae46a0\" (UID: \"7e84de93-b11b-4337-8e09-9de471ae46a0\") "
Sep 16 19:02:26.632744 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:02:26.632689    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e84de93-b11b-4337-8e09-9de471ae46a0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7e84de93-b11b-4337-8e09-9de471ae46a0" (UID: "7e84de93-b11b-4337-8e09-9de471ae46a0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:02:26.641291 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:02:26.641257    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e84de93-b11b-4337-8e09-9de471ae46a0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7e84de93-b11b-4337-8e09-9de471ae46a0" (UID: "7e84de93-b11b-4337-8e09-9de471ae46a0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:02:26.731098 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:02:26.731058    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2jdw8\" (UniqueName: \"kubernetes.io/projected/7e84de93-b11b-4337-8e09-9de471ae46a0-kube-api-access-2jdw8\") pod \"7e84de93-b11b-4337-8e09-9de471ae46a0\" (UID: \"7e84de93-b11b-4337-8e09-9de471ae46a0\") "
Sep 16 19:02:26.731273 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:02:26.731123    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e84de93-b11b-4337-8e09-9de471ae46a0-model-cache\") pod \"7e84de93-b11b-4337-8e09-9de471ae46a0\" (UID: \"7e84de93-b11b-4337-8e09-9de471ae46a0\") "
Sep 16 19:02:26.731273 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:02:26.731152    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e84de93-b11b-4337-8e09-9de471ae46a0-kserve-provision-location\") pod \"7e84de93-b11b-4337-8e09-9de471ae46a0\" (UID: \"7e84de93-b11b-4337-8e09-9de471ae46a0\") "
Sep 16 19:02:26.731273 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:02:26.731203    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e84de93-b11b-4337-8e09-9de471ae46a0-home\") pod \"7e84de93-b11b-4337-8e09-9de471ae46a0\" (UID: \"7e84de93-b11b-4337-8e09-9de471ae46a0\") "
Sep 16 19:02:26.731273 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:02:26.731230    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e84de93-b11b-4337-8e09-9de471ae46a0-cabundle-cert\") pod \"7e84de93-b11b-4337-8e09-9de471ae46a0\" (UID: \"7e84de93-b11b-4337-8e09-9de471ae46a0\") "
Sep 16 19:02:26.731273 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:02:26.731258    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e84de93-b11b-4337-8e09-9de471ae46a0-dshm\") pod \"7e84de93-b11b-4337-8e09-9de471ae46a0\" (UID: \"7e84de93-b11b-4337-8e09-9de471ae46a0\") "
Sep 16 19:02:26.731542 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:02:26.731429    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e84de93-b11b-4337-8e09-9de471ae46a0-model-cache" (OuterVolumeSpecName: "model-cache") pod "7e84de93-b11b-4337-8e09-9de471ae46a0" (UID: "7e84de93-b11b-4337-8e09-9de471ae46a0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:02:26.731542 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:02:26.731485    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e84de93-b11b-4337-8e09-9de471ae46a0-tls-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:02:26.731542 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:02:26.731503    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e84de93-b11b-4337-8e09-9de471ae46a0-tmp-dir\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:02:26.731700 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:02:26.731620    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e84de93-b11b-4337-8e09-9de471ae46a0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7e84de93-b11b-4337-8e09-9de471ae46a0" (UID: "7e84de93-b11b-4337-8e09-9de471ae46a0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:02:26.731896 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:02:26.731870    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e84de93-b11b-4337-8e09-9de471ae46a0-home" (OuterVolumeSpecName: "home") pod "7e84de93-b11b-4337-8e09-9de471ae46a0" (UID: "7e84de93-b11b-4337-8e09-9de471ae46a0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:02:26.733369 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:02:26.733336    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e84de93-b11b-4337-8e09-9de471ae46a0-dshm" (OuterVolumeSpecName: "dshm") pod "7e84de93-b11b-4337-8e09-9de471ae46a0" (UID: "7e84de93-b11b-4337-8e09-9de471ae46a0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:02:26.733369 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:02:26.733349    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e84de93-b11b-4337-8e09-9de471ae46a0-kube-api-access-2jdw8" (OuterVolumeSpecName: "kube-api-access-2jdw8") pod "7e84de93-b11b-4337-8e09-9de471ae46a0" (UID: "7e84de93-b11b-4337-8e09-9de471ae46a0"). InnerVolumeSpecName "kube-api-access-2jdw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:02:26.751959 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:02:26.751931    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e84de93-b11b-4337-8e09-9de471ae46a0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7e84de93-b11b-4337-8e09-9de471ae46a0" (UID: "7e84de93-b11b-4337-8e09-9de471ae46a0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:02:26.832536 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:02:26.832505    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e84de93-b11b-4337-8e09-9de471ae46a0-kserve-provision-location\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:02:26.832536 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:02:26.832530    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e84de93-b11b-4337-8e09-9de471ae46a0-home\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:02:26.832536 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:02:26.832539    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e84de93-b11b-4337-8e09-9de471ae46a0-cabundle-cert\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:02:26.832780 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:02:26.832549    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e84de93-b11b-4337-8e09-9de471ae46a0-dshm\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:02:26.832780 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:02:26.832557    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jdw8\" (UniqueName: \"kubernetes.io/projected/7e84de93-b11b-4337-8e09-9de471ae46a0-kube-api-access-2jdw8\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:02:26.832780 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:02:26.832564    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e84de93-b11b-4337-8e09-9de471ae46a0-model-cache\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:02:26.873395 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:02:26.873359    2582 generic.go:356] "Generic (PLEG): container finished" podID="7e84de93-b11b-4337-8e09-9de471ae46a0" containerID="1310d1a6409e9496e86ccba8d0dbd02065dd8327308e5511e61c14a746bac598" exitCode=0
Sep 16 19:02:26.873812 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:02:26.873495    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-c6cbd9685-zmcnv"
Sep 16 19:02:26.876918 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:02:26.876892    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-c6cbd9685-zmcnv" event={"ID":"7e84de93-b11b-4337-8e09-9de471ae46a0","Type":"ContainerDied","Data":"1310d1a6409e9496e86ccba8d0dbd02065dd8327308e5511e61c14a746bac598"}
Sep 16 19:02:26.877064 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:02:26.876941    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-c6cbd9685-zmcnv" event={"ID":"7e84de93-b11b-4337-8e09-9de471ae46a0","Type":"ContainerDied","Data":"7531949a67eebbc065ed162482fd3fc09df9223d299d28ccb298b1a36373d1fe"}
Sep 16 19:02:26.877064 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:02:26.876969    2582 scope.go:117] "RemoveContainer" containerID="1310d1a6409e9496e86ccba8d0dbd02065dd8327308e5511e61c14a746bac598"
Sep 16 19:02:26.886780 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:02:26.886753    2582 scope.go:117] "RemoveContainer" containerID="60f72fe9115501b33a892638ca67ab815dc82c67b06cf7f90256173f6837ad5f"
Sep 16 19:02:26.894406 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:02:26.894388    2582 scope.go:117] "RemoveContainer" containerID="1310d1a6409e9496e86ccba8d0dbd02065dd8327308e5511e61c14a746bac598"
Sep 16 19:02:26.894685 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:02:26.894662    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1310d1a6409e9496e86ccba8d0dbd02065dd8327308e5511e61c14a746bac598\": container with ID starting with 1310d1a6409e9496e86ccba8d0dbd02065dd8327308e5511e61c14a746bac598 not found: ID does not exist" containerID="1310d1a6409e9496e86ccba8d0dbd02065dd8327308e5511e61c14a746bac598"
Sep 16 19:02:26.894786 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:02:26.894697    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1310d1a6409e9496e86ccba8d0dbd02065dd8327308e5511e61c14a746bac598"} err="failed to get container status \"1310d1a6409e9496e86ccba8d0dbd02065dd8327308e5511e61c14a746bac598\": rpc error: code = NotFound desc = could not find container \"1310d1a6409e9496e86ccba8d0dbd02065dd8327308e5511e61c14a746bac598\": container with ID starting with 1310d1a6409e9496e86ccba8d0dbd02065dd8327308e5511e61c14a746bac598 not found: ID does not exist"
Sep 16 19:02:26.894786 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:02:26.894742    2582 scope.go:117] "RemoveContainer" containerID="60f72fe9115501b33a892638ca67ab815dc82c67b06cf7f90256173f6837ad5f"
Sep 16 19:02:26.894983 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:02:26.894966    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60f72fe9115501b33a892638ca67ab815dc82c67b06cf7f90256173f6837ad5f\": container with ID starting with 60f72fe9115501b33a892638ca67ab815dc82c67b06cf7f90256173f6837ad5f not found: ID does not exist" containerID="60f72fe9115501b33a892638ca67ab815dc82c67b06cf7f90256173f6837ad5f"
Sep 16 19:02:26.895023 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:02:26.894988    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60f72fe9115501b33a892638ca67ab815dc82c67b06cf7f90256173f6837ad5f"} err="failed to get container status \"60f72fe9115501b33a892638ca67ab815dc82c67b06cf7f90256173f6837ad5f\": rpc error: code = NotFound desc = could not find container \"60f72fe9115501b33a892638ca67ab815dc82c67b06cf7f90256173f6837ad5f\": container with ID starting with 60f72fe9115501b33a892638ca67ab815dc82c67b06cf7f90256173f6837ad5f not found: ID does not exist"
Sep 16 19:02:26.900401 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:02:26.900377    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-c6cbd9685-zmcnv"]
Sep 16 19:02:26.908037 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:02:26.908016    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-c6cbd9685-zmcnv"]
Sep 16 19:02:28.872330 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:02:28.872291    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e84de93-b11b-4337-8e09-9de471ae46a0" path="/var/lib/kubelet/pods/7e84de93-b11b-4337-8e09-9de471ae46a0/volumes"
Sep 16 19:03:16.860011 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:03:16.859975    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7qwl2_b6695f86-08c0-4836-87b1-f2a9d76084bd/console-operator/1.log"
Sep 16 19:03:16.861473 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:03:16.861446    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7qwl2_b6695f86-08c0-4836-87b1-f2a9d76084bd/console-operator/1.log"
Sep 16 19:03:52.205985 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:03:52.205947    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-grmv8"]
Sep 16 19:03:52.206582 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:03:52.206389    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e84de93-b11b-4337-8e09-9de471ae46a0" containerName="storage-initializer"
Sep 16 19:03:52.206582 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:03:52.206408    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e84de93-b11b-4337-8e09-9de471ae46a0" containerName="storage-initializer"
Sep 16 19:03:52.206582 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:03:52.206433    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e84de93-b11b-4337-8e09-9de471ae46a0" containerName="main"
Sep 16 19:03:52.206582 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:03:52.206442    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e84de93-b11b-4337-8e09-9de471ae46a0" containerName="main"
Sep 16 19:03:52.206582 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:03:52.206532    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e84de93-b11b-4337-8e09-9de471ae46a0" containerName="main"
Sep 16 19:03:52.209480 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:03:52.209457    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-grmv8"
Sep 16 19:03:52.212579 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:03:52.212547    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 16 19:03:52.213656 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:03:52.213638    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 16 19:03:52.213773 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:03:52.213637    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 16 19:03:52.213773 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:03:52.213638    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-hr6fv\""
Sep 16 19:03:52.220709 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:03:52.220685    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-grmv8"]
Sep 16 19:03:52.334205 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:03:52.334172    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f349e1f-bf45-46f9-857c-a0bf1de6943c-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-grmv8\" (UID: \"6f349e1f-bf45-46f9-857c-a0bf1de6943c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-grmv8"
Sep 16 19:03:52.334205 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:03:52.334211    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f349e1f-bf45-46f9-857c-a0bf1de6943c-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-grmv8\" (UID: \"6f349e1f-bf45-46f9-857c-a0bf1de6943c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-grmv8"
Sep 16 19:03:52.334412 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:03:52.334237    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f349e1f-bf45-46f9-857c-a0bf1de6943c-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-grmv8\" (UID: \"6f349e1f-bf45-46f9-857c-a0bf1de6943c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-grmv8"
Sep 16 19:03:52.334412 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:03:52.334263    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f349e1f-bf45-46f9-857c-a0bf1de6943c-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-grmv8\" (UID: \"6f349e1f-bf45-46f9-857c-a0bf1de6943c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-grmv8"
Sep 16 19:03:52.334412 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:03:52.334290    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f349e1f-bf45-46f9-857c-a0bf1de6943c-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-grmv8\" (UID: \"6f349e1f-bf45-46f9-857c-a0bf1de6943c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-grmv8"
Sep 16 19:03:52.334412 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:03:52.334374    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn8km\" (UniqueName: \"kubernetes.io/projected/6f349e1f-bf45-46f9-857c-a0bf1de6943c-kube-api-access-bn8km\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-grmv8\" (UID: \"6f349e1f-bf45-46f9-857c-a0bf1de6943c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-grmv8"
Sep 16 19:03:52.434943 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:03:52.434908    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn8km\" (UniqueName: \"kubernetes.io/projected/6f349e1f-bf45-46f9-857c-a0bf1de6943c-kube-api-access-bn8km\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-grmv8\" (UID: \"6f349e1f-bf45-46f9-857c-a0bf1de6943c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-grmv8"
Sep 16 19:03:52.435120 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:03:52.434953    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f349e1f-bf45-46f9-857c-a0bf1de6943c-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-grmv8\" (UID: \"6f349e1f-bf45-46f9-857c-a0bf1de6943c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-grmv8"
Sep 16 19:03:52.435120 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:03:52.434975    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f349e1f-bf45-46f9-857c-a0bf1de6943c-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-grmv8\" (UID: \"6f349e1f-bf45-46f9-857c-a0bf1de6943c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-grmv8"
Sep 16 19:03:52.435120 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:03:52.435002    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f349e1f-bf45-46f9-857c-a0bf1de6943c-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-grmv8\" (UID: \"6f349e1f-bf45-46f9-857c-a0bf1de6943c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-grmv8"
Sep 16 19:03:52.435120 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:03:52.435016    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f349e1f-bf45-46f9-857c-a0bf1de6943c-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-grmv8\" (UID: \"6f349e1f-bf45-46f9-857c-a0bf1de6943c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-grmv8"
Sep 16 19:03:52.435120 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:03:52.435042    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f349e1f-bf45-46f9-857c-a0bf1de6943c-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-grmv8\" (UID: \"6f349e1f-bf45-46f9-857c-a0bf1de6943c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-grmv8"
Sep 16 19:03:52.435378 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:03:52.435345    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f349e1f-bf45-46f9-857c-a0bf1de6943c-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-grmv8\" (UID: \"6f349e1f-bf45-46f9-857c-a0bf1de6943c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-grmv8"
Sep 16 19:03:52.435435 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:03:52.435420    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f349e1f-bf45-46f9-857c-a0bf1de6943c-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-grmv8\" (UID: \"6f349e1f-bf45-46f9-857c-a0bf1de6943c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-grmv8"
Sep 16 19:03:52.435488 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:03:52.435467    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f349e1f-bf45-46f9-857c-a0bf1de6943c-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-grmv8\" (UID: \"6f349e1f-bf45-46f9-857c-a0bf1de6943c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-grmv8"
Sep 16 19:03:52.437515 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:03:52.437491    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f349e1f-bf45-46f9-857c-a0bf1de6943c-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-grmv8\" (UID: \"6f349e1f-bf45-46f9-857c-a0bf1de6943c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-grmv8"
Sep 16 19:03:52.437790 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:03:52.437772    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f349e1f-bf45-46f9-857c-a0bf1de6943c-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-grmv8\" (UID: \"6f349e1f-bf45-46f9-857c-a0bf1de6943c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-grmv8"
Sep 16 19:03:52.444558 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:03:52.444533    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn8km\" (UniqueName: \"kubernetes.io/projected/6f349e1f-bf45-46f9-857c-a0bf1de6943c-kube-api-access-bn8km\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-grmv8\" (UID: \"6f349e1f-bf45-46f9-857c-a0bf1de6943c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-grmv8"
Sep 16 19:03:52.519738 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:03:52.519679    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-grmv8"
Sep 16 19:03:52.798979 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:03:52.798904    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-grmv8"]
Sep 16 19:03:52.801636 ip-10-0-129-194 kubenswrapper[2582]: W0916 19:03:52.801606    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f349e1f_bf45_46f9_857c_a0bf1de6943c.slice/crio-851ac833d84734a796b40c2575e19171c1b00b24c553c9b12b337d3273d83ff9 WatchSource:0}: Error finding container 851ac833d84734a796b40c2575e19171c1b00b24c553c9b12b337d3273d83ff9: Status 404 returned error can't find the container with id 851ac833d84734a796b40c2575e19171c1b00b24c553c9b12b337d3273d83ff9
Sep 16 19:03:53.231208 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:03:53.231111    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-grmv8" event={"ID":"6f349e1f-bf45-46f9-857c-a0bf1de6943c","Type":"ContainerStarted","Data":"64842e7ca48b9f7e1cda2350847b337ea1cfd4e53cbae271528313c16e9a0109"}
Sep 16 19:03:53.231208 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:03:53.231170    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-grmv8" event={"ID":"6f349e1f-bf45-46f9-857c-a0bf1de6943c","Type":"ContainerStarted","Data":"851ac833d84734a796b40c2575e19171c1b00b24c553c9b12b337d3273d83ff9"}
Sep 16 19:03:53.254189 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:03:53.254132    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-grmv8" podStartSLOduration=1.254115891 podStartE2EDuration="1.254115891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:03:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:03:53.252010327 +0000 UTC m=+1236.893725992" watchObservedRunningTime="2026-09-16 19:03:53.254115891 +0000 UTC m=+1236.895831555"
Sep 16 19:03:53.520547 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:03:53.520507    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-grmv8"
Sep 16 19:04:02.520359 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:04:02.520307    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-grmv8"
Sep 16 19:04:02.532850 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:04:02.532819    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-grmv8"
Sep 16 19:04:03.284140 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:04:03.284109    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-grmv8"
Sep 16 19:04:25.988510 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:04:25.988471    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-grmv8"]
Sep 16 19:04:25.997675 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:04:25.997632    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:04:25.997675 ip-10-0-129-194 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:04:25.997675 ip-10-0-129-194 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 19:04:25.997675 ip-10-0-129-194 kubenswrapper[2582]:  > containerID="64842e7ca48b9f7e1cda2350847b337ea1cfd4e53cbae271528313c16e9a0109" cmd=["/bin/sleep","15"]
Sep 16 19:04:25.997981 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:04:25.997697    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-grmv8" message="" err=<
Sep 16 19:04:25.997981 ip-10-0-129-194 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:04:25.997981 ip-10-0-129-194 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 19:04:25.997981 ip-10-0-129-194 kubenswrapper[2582]:  >
Sep 16 19:04:25.997981 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:04:25.997713    2582 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:04:25.997981 ip-10-0-129-194 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:04:25.997981 ip-10-0-129-194 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 19:04:25.997981 ip-10-0-129-194 kubenswrapper[2582]:  > pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-grmv8" podUID="6f349e1f-bf45-46f9-857c-a0bf1de6943c" containerName="main" containerID="cri-o://64842e7ca48b9f7e1cda2350847b337ea1cfd4e53cbae271528313c16e9a0109"
Sep 16 19:04:25.997981 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:04:25.997803    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-grmv8" podUID="6f349e1f-bf45-46f9-857c-a0bf1de6943c" containerName="main" containerID="cri-o://64842e7ca48b9f7e1cda2350847b337ea1cfd4e53cbae271528313c16e9a0109" gracePeriod=60
Sep 16 19:04:26.344392 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:04:26.344367    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-grmv8"
Sep 16 19:04:26.369685 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:04:26.369655    2582 generic.go:356] "Generic (PLEG): container finished" podID="6f349e1f-bf45-46f9-857c-a0bf1de6943c" containerID="64842e7ca48b9f7e1cda2350847b337ea1cfd4e53cbae271528313c16e9a0109" exitCode=0
Sep 16 19:04:26.369837 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:04:26.369702    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-grmv8"
Sep 16 19:04:26.369837 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:04:26.369710    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-grmv8" event={"ID":"6f349e1f-bf45-46f9-857c-a0bf1de6943c","Type":"ContainerDied","Data":"64842e7ca48b9f7e1cda2350847b337ea1cfd4e53cbae271528313c16e9a0109"}
Sep 16 19:04:26.369837 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:04:26.369799    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-grmv8" event={"ID":"6f349e1f-bf45-46f9-857c-a0bf1de6943c","Type":"ContainerDied","Data":"851ac833d84734a796b40c2575e19171c1b00b24c553c9b12b337d3273d83ff9"}
Sep 16 19:04:26.369837 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:04:26.369822    2582 scope.go:117] "RemoveContainer" containerID="64842e7ca48b9f7e1cda2350847b337ea1cfd4e53cbae271528313c16e9a0109"
Sep 16 19:04:26.377895 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:04:26.377875    2582 scope.go:117] "RemoveContainer" containerID="64842e7ca48b9f7e1cda2350847b337ea1cfd4e53cbae271528313c16e9a0109"
Sep 16 19:04:26.378140 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:04:26.378120    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64842e7ca48b9f7e1cda2350847b337ea1cfd4e53cbae271528313c16e9a0109\": container with ID starting with 64842e7ca48b9f7e1cda2350847b337ea1cfd4e53cbae271528313c16e9a0109 not found: ID does not exist" containerID="64842e7ca48b9f7e1cda2350847b337ea1cfd4e53cbae271528313c16e9a0109"
Sep 16 19:04:26.378206 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:04:26.378158    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64842e7ca48b9f7e1cda2350847b337ea1cfd4e53cbae271528313c16e9a0109"} err="failed to get container status \"64842e7ca48b9f7e1cda2350847b337ea1cfd4e53cbae271528313c16e9a0109\": rpc error: code = NotFound desc = could not find container \"64842e7ca48b9f7e1cda2350847b337ea1cfd4e53cbae271528313c16e9a0109\": container with ID starting with 64842e7ca48b9f7e1cda2350847b337ea1cfd4e53cbae271528313c16e9a0109 not found: ID does not exist"
Sep 16 19:04:26.523559 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:04:26.523464    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f349e1f-bf45-46f9-857c-a0bf1de6943c-tmp-dir\") pod \"6f349e1f-bf45-46f9-857c-a0bf1de6943c\" (UID: \"6f349e1f-bf45-46f9-857c-a0bf1de6943c\") "
Sep 16 19:04:26.523559 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:04:26.523509    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bn8km\" (UniqueName: \"kubernetes.io/projected/6f349e1f-bf45-46f9-857c-a0bf1de6943c-kube-api-access-bn8km\") pod \"6f349e1f-bf45-46f9-857c-a0bf1de6943c\" (UID: \"6f349e1f-bf45-46f9-857c-a0bf1de6943c\") "
Sep 16 19:04:26.523559 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:04:26.523545    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f349e1f-bf45-46f9-857c-a0bf1de6943c-home\") pod \"6f349e1f-bf45-46f9-857c-a0bf1de6943c\" (UID: \"6f349e1f-bf45-46f9-857c-a0bf1de6943c\") "
Sep 16 19:04:26.523883 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:04:26.523712    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f349e1f-bf45-46f9-857c-a0bf1de6943c-model-cache\") pod \"6f349e1f-bf45-46f9-857c-a0bf1de6943c\" (UID: \"6f349e1f-bf45-46f9-857c-a0bf1de6943c\") "
Sep 16 19:04:26.523883 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:04:26.523773    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f349e1f-bf45-46f9-857c-a0bf1de6943c-dshm\") pod \"6f349e1f-bf45-46f9-857c-a0bf1de6943c\" (UID: \"6f349e1f-bf45-46f9-857c-a0bf1de6943c\") "
Sep 16 19:04:26.523883 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:04:26.523805    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f349e1f-bf45-46f9-857c-a0bf1de6943c-tls-certs\") pod \"6f349e1f-bf45-46f9-857c-a0bf1de6943c\" (UID: \"6f349e1f-bf45-46f9-857c-a0bf1de6943c\") "
Sep 16 19:04:26.523883 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:04:26.523835    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f349e1f-bf45-46f9-857c-a0bf1de6943c-home" (OuterVolumeSpecName: "home") pod "6f349e1f-bf45-46f9-857c-a0bf1de6943c" (UID: "6f349e1f-bf45-46f9-857c-a0bf1de6943c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:04:26.523883 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:04:26.523854    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f349e1f-bf45-46f9-857c-a0bf1de6943c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6f349e1f-bf45-46f9-857c-a0bf1de6943c" (UID: "6f349e1f-bf45-46f9-857c-a0bf1de6943c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:04:26.524075 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:04:26.524003    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f349e1f-bf45-46f9-857c-a0bf1de6943c-model-cache" (OuterVolumeSpecName: "model-cache") pod "6f349e1f-bf45-46f9-857c-a0bf1de6943c" (UID: "6f349e1f-bf45-46f9-857c-a0bf1de6943c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:04:26.524114 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:04:26.524101    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f349e1f-bf45-46f9-857c-a0bf1de6943c-home\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:04:26.524148 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:04:26.524121    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f349e1f-bf45-46f9-857c-a0bf1de6943c-model-cache\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:04:26.524148 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:04:26.524137    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f349e1f-bf45-46f9-857c-a0bf1de6943c-tmp-dir\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:04:26.525916 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:04:26.525887    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f349e1f-bf45-46f9-857c-a0bf1de6943c-kube-api-access-bn8km" (OuterVolumeSpecName: "kube-api-access-bn8km") pod "6f349e1f-bf45-46f9-857c-a0bf1de6943c" (UID: "6f349e1f-bf45-46f9-857c-a0bf1de6943c"). InnerVolumeSpecName "kube-api-access-bn8km". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:04:26.525916 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:04:26.525904    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f349e1f-bf45-46f9-857c-a0bf1de6943c-dshm" (OuterVolumeSpecName: "dshm") pod "6f349e1f-bf45-46f9-857c-a0bf1de6943c" (UID: "6f349e1f-bf45-46f9-857c-a0bf1de6943c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:04:26.526063 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:04:26.525968    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f349e1f-bf45-46f9-857c-a0bf1de6943c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6f349e1f-bf45-46f9-857c-a0bf1de6943c" (UID: "6f349e1f-bf45-46f9-857c-a0bf1de6943c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:04:26.625038 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:04:26.625005    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn8km\" (UniqueName: \"kubernetes.io/projected/6f349e1f-bf45-46f9-857c-a0bf1de6943c-kube-api-access-bn8km\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:04:26.625038 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:04:26.625031    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f349e1f-bf45-46f9-857c-a0bf1de6943c-dshm\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:04:26.625038 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:04:26.625040    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f349e1f-bf45-46f9-857c-a0bf1de6943c-tls-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:04:26.693749 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:04:26.693705    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-grmv8"]
Sep 16 19:04:26.697589 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:04:26.697561    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-grmv8"]
Sep 16 19:04:26.872527 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:04:26.872453    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f349e1f-bf45-46f9-857c-a0bf1de6943c" path="/var/lib/kubelet/pods/6f349e1f-bf45-46f9-857c-a0bf1de6943c/volumes"
Sep 16 19:04:27.274378 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:04:27.274327    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-grmv8" podUID="6f349e1f-bf45-46f9-857c-a0bf1de6943c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": context deadline exceeded"
Sep 16 19:05:23.608146 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:23.608105    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4fdqx"]
Sep 16 19:05:23.608573 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:23.608427    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f349e1f-bf45-46f9-857c-a0bf1de6943c" containerName="main"
Sep 16 19:05:23.608573 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:23.608465    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f349e1f-bf45-46f9-857c-a0bf1de6943c" containerName="main"
Sep 16 19:05:23.608573 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:23.608523    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f349e1f-bf45-46f9-857c-a0bf1de6943c" containerName="main"
Sep 16 19:05:23.612658 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:23.612637    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4fdqx"
Sep 16 19:05:23.616353 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:23.616323    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Sep 16 19:05:23.616498 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:23.616364    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-vj2jl\""
Sep 16 19:05:23.616498 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:23.616422    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Sep 16 19:05:23.616498 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:23.616431    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Sep 16 19:05:23.621550 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:23.621526    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4fdqx"]
Sep 16 19:05:23.665570 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:23.665529    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd819ac2-8700-4f2a-b069-f1ab2fe57f13-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-4fdqx\" (UID: \"cd819ac2-8700-4f2a-b069-f1ab2fe57f13\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4fdqx"
Sep 16 19:05:23.665570 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:23.665578    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd819ac2-8700-4f2a-b069-f1ab2fe57f13-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-4fdqx\" (UID: \"cd819ac2-8700-4f2a-b069-f1ab2fe57f13\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4fdqx"
Sep 16 19:05:23.665850 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:23.665601    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd819ac2-8700-4f2a-b069-f1ab2fe57f13-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-4fdqx\" (UID: \"cd819ac2-8700-4f2a-b069-f1ab2fe57f13\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4fdqx"
Sep 16 19:05:23.665850 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:23.665618    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxpmz\" (UniqueName: \"kubernetes.io/projected/cd819ac2-8700-4f2a-b069-f1ab2fe57f13-kube-api-access-kxpmz\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-4fdqx\" (UID: \"cd819ac2-8700-4f2a-b069-f1ab2fe57f13\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4fdqx"
Sep 16 19:05:23.665850 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:23.665696    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd819ac2-8700-4f2a-b069-f1ab2fe57f13-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-4fdqx\" (UID: \"cd819ac2-8700-4f2a-b069-f1ab2fe57f13\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4fdqx"
Sep 16 19:05:23.665850 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:23.665758    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd819ac2-8700-4f2a-b069-f1ab2fe57f13-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-4fdqx\" (UID: \"cd819ac2-8700-4f2a-b069-f1ab2fe57f13\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4fdqx"
Sep 16 19:05:23.766580 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:23.766541    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd819ac2-8700-4f2a-b069-f1ab2fe57f13-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-4fdqx\" (UID: \"cd819ac2-8700-4f2a-b069-f1ab2fe57f13\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4fdqx"
Sep 16 19:05:23.766580 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:23.766584    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd819ac2-8700-4f2a-b069-f1ab2fe57f13-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-4fdqx\" (UID: \"cd819ac2-8700-4f2a-b069-f1ab2fe57f13\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4fdqx"
Sep 16 19:05:23.766853 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:23.766610    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd819ac2-8700-4f2a-b069-f1ab2fe57f13-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-4fdqx\" (UID: \"cd819ac2-8700-4f2a-b069-f1ab2fe57f13\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4fdqx"
Sep 16 19:05:23.766853 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:23.766754    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxpmz\" (UniqueName: \"kubernetes.io/projected/cd819ac2-8700-4f2a-b069-f1ab2fe57f13-kube-api-access-kxpmz\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-4fdqx\" (UID: \"cd819ac2-8700-4f2a-b069-f1ab2fe57f13\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4fdqx"
Sep 16 19:05:23.766853 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:23.766814    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd819ac2-8700-4f2a-b069-f1ab2fe57f13-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-4fdqx\" (UID: \"cd819ac2-8700-4f2a-b069-f1ab2fe57f13\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4fdqx"
Sep 16 19:05:23.766853 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:23.766840    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd819ac2-8700-4f2a-b069-f1ab2fe57f13-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-4fdqx\" (UID: \"cd819ac2-8700-4f2a-b069-f1ab2fe57f13\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4fdqx"
Sep 16 19:05:23.767066 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:23.766961    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd819ac2-8700-4f2a-b069-f1ab2fe57f13-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-4fdqx\" (UID: \"cd819ac2-8700-4f2a-b069-f1ab2fe57f13\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4fdqx"
Sep 16 19:05:23.767066 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:23.767042    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd819ac2-8700-4f2a-b069-f1ab2fe57f13-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-4fdqx\" (UID: \"cd819ac2-8700-4f2a-b069-f1ab2fe57f13\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4fdqx"
Sep 16 19:05:23.767143 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:23.767070    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd819ac2-8700-4f2a-b069-f1ab2fe57f13-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-4fdqx\" (UID: \"cd819ac2-8700-4f2a-b069-f1ab2fe57f13\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4fdqx"
Sep 16 19:05:23.769094 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:23.769063    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd819ac2-8700-4f2a-b069-f1ab2fe57f13-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-4fdqx\" (UID: \"cd819ac2-8700-4f2a-b069-f1ab2fe57f13\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4fdqx"
Sep 16 19:05:23.769348 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:23.769329    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd819ac2-8700-4f2a-b069-f1ab2fe57f13-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-4fdqx\" (UID: \"cd819ac2-8700-4f2a-b069-f1ab2fe57f13\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4fdqx"
Sep 16 19:05:23.775706 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:23.775678    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxpmz\" (UniqueName: \"kubernetes.io/projected/cd819ac2-8700-4f2a-b069-f1ab2fe57f13-kube-api-access-kxpmz\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-4fdqx\" (UID: \"cd819ac2-8700-4f2a-b069-f1ab2fe57f13\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4fdqx"
Sep 16 19:05:23.924169 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:23.924070    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4fdqx"
Sep 16 19:05:24.199012 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:24.198965    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4fdqx"]
Sep 16 19:05:24.201900 ip-10-0-129-194 kubenswrapper[2582]: W0916 19:05:24.201865    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd819ac2_8700_4f2a_b069_f1ab2fe57f13.slice/crio-0f3e9242bbbaae069fcfc2a563645dcf27ad9c7dcc50307d795326ed265d4146 WatchSource:0}: Error finding container 0f3e9242bbbaae069fcfc2a563645dcf27ad9c7dcc50307d795326ed265d4146: Status 404 returned error can't find the container with id 0f3e9242bbbaae069fcfc2a563645dcf27ad9c7dcc50307d795326ed265d4146
Sep 16 19:05:24.601264 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:24.601227    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4fdqx" event={"ID":"cd819ac2-8700-4f2a-b069-f1ab2fe57f13","Type":"ContainerStarted","Data":"03c7f9d5f2e0e4a12ec13b518ec231e7b0cf806b5b8c91e4d2ffbb9a12e8658a"}
Sep 16 19:05:24.601471 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:24.601327    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4fdqx" event={"ID":"cd819ac2-8700-4f2a-b069-f1ab2fe57f13","Type":"ContainerStarted","Data":"0f3e9242bbbaae069fcfc2a563645dcf27ad9c7dcc50307d795326ed265d4146"}
Sep 16 19:05:24.619842 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:24.619781    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4fdqx" podStartSLOduration=1.619763362 podStartE2EDuration="1.619763362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:05:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:24.618818225 +0000 UTC m=+1328.260533890" watchObservedRunningTime="2026-09-16 19:05:24.619763362 +0000 UTC m=+1328.261479026"
Sep 16 19:05:24.924617 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:24.924527    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4fdqx"
Sep 16 19:05:33.925188 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:33.925135    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4fdqx"
Sep 16 19:05:33.937805 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:33.937766    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4fdqx"
Sep 16 19:05:34.654085 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:34.654053    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4fdqx"
Sep 16 19:05:57.939729 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:57.939683    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4fdqx"]
Sep 16 19:05:57.948784 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:05:57.948749    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:05:57.948784 ip-10-0-129-194 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:05:57.948784 ip-10-0-129-194 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 19:05:57.948784 ip-10-0-129-194 kubenswrapper[2582]:  > containerID="03c7f9d5f2e0e4a12ec13b518ec231e7b0cf806b5b8c91e4d2ffbb9a12e8658a" cmd=["/bin/sleep","15"]
Sep 16 19:05:57.949048 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:57.948801    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4fdqx" message="" err=<
Sep 16 19:05:57.949048 ip-10-0-129-194 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:05:57.949048 ip-10-0-129-194 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 19:05:57.949048 ip-10-0-129-194 kubenswrapper[2582]:  >
Sep 16 19:05:57.949048 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:05:57.948814    2582 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:05:57.949048 ip-10-0-129-194 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:05:57.949048 ip-10-0-129-194 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 19:05:57.949048 ip-10-0-129-194 kubenswrapper[2582]:  > pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4fdqx" podUID="cd819ac2-8700-4f2a-b069-f1ab2fe57f13" containerName="main" containerID="cri-o://03c7f9d5f2e0e4a12ec13b518ec231e7b0cf806b5b8c91e4d2ffbb9a12e8658a"
Sep 16 19:05:57.949048 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:57.948859    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4fdqx" podUID="cd819ac2-8700-4f2a-b069-f1ab2fe57f13" containerName="main" containerID="cri-o://03c7f9d5f2e0e4a12ec13b518ec231e7b0cf806b5b8c91e4d2ffbb9a12e8658a" gracePeriod=60
Sep 16 19:05:58.290212 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:58.290184    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4fdqx"
Sep 16 19:05:58.379834 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:58.379797    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd819ac2-8700-4f2a-b069-f1ab2fe57f13-tmp-dir\") pod \"cd819ac2-8700-4f2a-b069-f1ab2fe57f13\" (UID: \"cd819ac2-8700-4f2a-b069-f1ab2fe57f13\") "
Sep 16 19:05:58.379834 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:58.379839    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd819ac2-8700-4f2a-b069-f1ab2fe57f13-home\") pod \"cd819ac2-8700-4f2a-b069-f1ab2fe57f13\" (UID: \"cd819ac2-8700-4f2a-b069-f1ab2fe57f13\") "
Sep 16 19:05:58.380064 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:58.379872    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd819ac2-8700-4f2a-b069-f1ab2fe57f13-dshm\") pod \"cd819ac2-8700-4f2a-b069-f1ab2fe57f13\" (UID: \"cd819ac2-8700-4f2a-b069-f1ab2fe57f13\") "
Sep 16 19:05:58.380064 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:58.379895    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kxpmz\" (UniqueName: \"kubernetes.io/projected/cd819ac2-8700-4f2a-b069-f1ab2fe57f13-kube-api-access-kxpmz\") pod \"cd819ac2-8700-4f2a-b069-f1ab2fe57f13\" (UID: \"cd819ac2-8700-4f2a-b069-f1ab2fe57f13\") "
Sep 16 19:05:58.380064 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:58.379931    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd819ac2-8700-4f2a-b069-f1ab2fe57f13-tls-certs\") pod \"cd819ac2-8700-4f2a-b069-f1ab2fe57f13\" (UID: \"cd819ac2-8700-4f2a-b069-f1ab2fe57f13\") "
Sep 16 19:05:58.380064 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:58.379976    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd819ac2-8700-4f2a-b069-f1ab2fe57f13-model-cache\") pod \"cd819ac2-8700-4f2a-b069-f1ab2fe57f13\" (UID: \"cd819ac2-8700-4f2a-b069-f1ab2fe57f13\") "
Sep 16 19:05:58.380275 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:58.380101    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd819ac2-8700-4f2a-b069-f1ab2fe57f13-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cd819ac2-8700-4f2a-b069-f1ab2fe57f13" (UID: "cd819ac2-8700-4f2a-b069-f1ab2fe57f13"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:05:58.380275 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:58.380227    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd819ac2-8700-4f2a-b069-f1ab2fe57f13-home" (OuterVolumeSpecName: "home") pod "cd819ac2-8700-4f2a-b069-f1ab2fe57f13" (UID: "cd819ac2-8700-4f2a-b069-f1ab2fe57f13"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:05:58.380275 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:58.380246    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd819ac2-8700-4f2a-b069-f1ab2fe57f13-tmp-dir\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:05:58.380391 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:58.380271    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd819ac2-8700-4f2a-b069-f1ab2fe57f13-model-cache" (OuterVolumeSpecName: "model-cache") pod "cd819ac2-8700-4f2a-b069-f1ab2fe57f13" (UID: "cd819ac2-8700-4f2a-b069-f1ab2fe57f13"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:05:58.382251 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:58.382223    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd819ac2-8700-4f2a-b069-f1ab2fe57f13-dshm" (OuterVolumeSpecName: "dshm") pod "cd819ac2-8700-4f2a-b069-f1ab2fe57f13" (UID: "cd819ac2-8700-4f2a-b069-f1ab2fe57f13"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:05:58.382368 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:58.382265    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd819ac2-8700-4f2a-b069-f1ab2fe57f13-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cd819ac2-8700-4f2a-b069-f1ab2fe57f13" (UID: "cd819ac2-8700-4f2a-b069-f1ab2fe57f13"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:05:58.382368 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:58.382340    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd819ac2-8700-4f2a-b069-f1ab2fe57f13-kube-api-access-kxpmz" (OuterVolumeSpecName: "kube-api-access-kxpmz") pod "cd819ac2-8700-4f2a-b069-f1ab2fe57f13" (UID: "cd819ac2-8700-4f2a-b069-f1ab2fe57f13"). InnerVolumeSpecName "kube-api-access-kxpmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:05:58.481040 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:58.480959    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd819ac2-8700-4f2a-b069-f1ab2fe57f13-home\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:05:58.481040 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:58.480987    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd819ac2-8700-4f2a-b069-f1ab2fe57f13-dshm\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:05:58.481040 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:58.480997    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxpmz\" (UniqueName: \"kubernetes.io/projected/cd819ac2-8700-4f2a-b069-f1ab2fe57f13-kube-api-access-kxpmz\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:05:58.481040 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:58.481007    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd819ac2-8700-4f2a-b069-f1ab2fe57f13-tls-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:05:58.481040 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:58.481016    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd819ac2-8700-4f2a-b069-f1ab2fe57f13-model-cache\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:05:58.747280 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:58.747198    2582 generic.go:356] "Generic (PLEG): container finished" podID="cd819ac2-8700-4f2a-b069-f1ab2fe57f13" containerID="03c7f9d5f2e0e4a12ec13b518ec231e7b0cf806b5b8c91e4d2ffbb9a12e8658a" exitCode=0
Sep 16 19:05:58.747280 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:58.747247    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4fdqx"
Sep 16 19:05:58.747456 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:58.747279    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4fdqx" event={"ID":"cd819ac2-8700-4f2a-b069-f1ab2fe57f13","Type":"ContainerDied","Data":"03c7f9d5f2e0e4a12ec13b518ec231e7b0cf806b5b8c91e4d2ffbb9a12e8658a"}
Sep 16 19:05:58.747456 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:58.747341    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4fdqx" event={"ID":"cd819ac2-8700-4f2a-b069-f1ab2fe57f13","Type":"ContainerDied","Data":"0f3e9242bbbaae069fcfc2a563645dcf27ad9c7dcc50307d795326ed265d4146"}
Sep 16 19:05:58.747456 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:58.747363    2582 scope.go:117] "RemoveContainer" containerID="03c7f9d5f2e0e4a12ec13b518ec231e7b0cf806b5b8c91e4d2ffbb9a12e8658a"
Sep 16 19:05:58.756352 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:58.756332    2582 scope.go:117] "RemoveContainer" containerID="03c7f9d5f2e0e4a12ec13b518ec231e7b0cf806b5b8c91e4d2ffbb9a12e8658a"
Sep 16 19:05:58.756668 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:05:58.756649    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03c7f9d5f2e0e4a12ec13b518ec231e7b0cf806b5b8c91e4d2ffbb9a12e8658a\": container with ID starting with 03c7f9d5f2e0e4a12ec13b518ec231e7b0cf806b5b8c91e4d2ffbb9a12e8658a not found: ID does not exist" containerID="03c7f9d5f2e0e4a12ec13b518ec231e7b0cf806b5b8c91e4d2ffbb9a12e8658a"
Sep 16 19:05:58.756833 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:58.756681    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03c7f9d5f2e0e4a12ec13b518ec231e7b0cf806b5b8c91e4d2ffbb9a12e8658a"} err="failed to get container status \"03c7f9d5f2e0e4a12ec13b518ec231e7b0cf806b5b8c91e4d2ffbb9a12e8658a\": rpc error: code = NotFound desc = could not find container \"03c7f9d5f2e0e4a12ec13b518ec231e7b0cf806b5b8c91e4d2ffbb9a12e8658a\": container with ID starting with 03c7f9d5f2e0e4a12ec13b518ec231e7b0cf806b5b8c91e4d2ffbb9a12e8658a not found: ID does not exist"
Sep 16 19:05:58.769705 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:58.769669    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4fdqx"]
Sep 16 19:05:58.771652 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:58.771626    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4fdqx"]
Sep 16 19:05:58.872809 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:05:58.872777    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd819ac2-8700-4f2a-b069-f1ab2fe57f13" path="/var/lib/kubelet/pods/cd819ac2-8700-4f2a-b069-f1ab2fe57f13/volumes"
Sep 16 19:07:25.254201 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:25.254167    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55"]
Sep 16 19:07:25.254631 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:25.254468    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd819ac2-8700-4f2a-b069-f1ab2fe57f13" containerName="main"
Sep 16 19:07:25.254631 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:25.254479    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd819ac2-8700-4f2a-b069-f1ab2fe57f13" containerName="main"
Sep 16 19:07:25.254631 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:25.254542    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd819ac2-8700-4f2a-b069-f1ab2fe57f13" containerName="main"
Sep 16 19:07:25.257772 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:25.257749    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55"
Sep 16 19:07:25.258478 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:25.258454    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-58pzt"]
Sep 16 19:07:25.261586 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:25.261394    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-pcphf\""
Sep 16 19:07:25.261586 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:25.261529    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 16 19:07:25.262209 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:25.261932    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-58pzt"
Sep 16 19:07:25.263100 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:25.262951    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 16 19:07:25.263100 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:25.263095    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 16 19:07:25.263442 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:25.262953    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-tqfd9\""
Sep 16 19:07:25.274973 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:25.274949    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55"]
Sep 16 19:07:25.279802 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:25.279775    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-58pzt"]
Sep 16 19:07:25.408636 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:25.408597    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d803842-cc6b-4f47-b71f-4d2ec8698a30-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-58pzt\" (UID: \"6d803842-cc6b-4f47-b71f-4d2ec8698a30\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-58pzt"
Sep 16 19:07:25.408636 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:25.408634    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d803842-cc6b-4f47-b71f-4d2ec8698a30-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-58pzt\" (UID: \"6d803842-cc6b-4f47-b71f-4d2ec8698a30\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-58pzt"
Sep 16 19:07:25.408902 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:25.408663    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxm8d\" (UniqueName: \"kubernetes.io/projected/3e8b83bf-bc1e-4dc1-82e6-5614861496a5-kube-api-access-rxm8d\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55\" (UID: \"3e8b83bf-bc1e-4dc1-82e6-5614861496a5\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55"
Sep 16 19:07:25.408902 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:25.408741    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d803842-cc6b-4f47-b71f-4d2ec8698a30-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-58pzt\" (UID: \"6d803842-cc6b-4f47-b71f-4d2ec8698a30\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-58pzt"
Sep 16 19:07:25.408902 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:25.408799    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e8b83bf-bc1e-4dc1-82e6-5614861496a5-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55\" (UID: \"3e8b83bf-bc1e-4dc1-82e6-5614861496a5\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55"
Sep 16 19:07:25.408902 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:25.408831    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d803842-cc6b-4f47-b71f-4d2ec8698a30-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-58pzt\" (UID: \"6d803842-cc6b-4f47-b71f-4d2ec8698a30\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-58pzt"
Sep 16 19:07:25.408902 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:25.408872    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e8b83bf-bc1e-4dc1-82e6-5614861496a5-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55\" (UID: \"3e8b83bf-bc1e-4dc1-82e6-5614861496a5\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55"
Sep 16 19:07:25.409077 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:25.408918    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d803842-cc6b-4f47-b71f-4d2ec8698a30-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-58pzt\" (UID: \"6d803842-cc6b-4f47-b71f-4d2ec8698a30\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-58pzt"
Sep 16 19:07:25.409077 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:25.408963    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9528z\" (UniqueName: \"kubernetes.io/projected/6d803842-cc6b-4f47-b71f-4d2ec8698a30-kube-api-access-9528z\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-58pzt\" (UID: \"6d803842-cc6b-4f47-b71f-4d2ec8698a30\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-58pzt"
Sep 16 19:07:25.409077 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:25.408995    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e8b83bf-bc1e-4dc1-82e6-5614861496a5-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55\" (UID: \"3e8b83bf-bc1e-4dc1-82e6-5614861496a5\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55"
Sep 16 19:07:25.409077 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:25.409027    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e8b83bf-bc1e-4dc1-82e6-5614861496a5-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55\" (UID: \"3e8b83bf-bc1e-4dc1-82e6-5614861496a5\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55"
Sep 16 19:07:25.409077 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:25.409043    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e8b83bf-bc1e-4dc1-82e6-5614861496a5-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55\" (UID: \"3e8b83bf-bc1e-4dc1-82e6-5614861496a5\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55"
Sep 16 19:07:25.509727 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:25.509686    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e8b83bf-bc1e-4dc1-82e6-5614861496a5-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55\" (UID: \"3e8b83bf-bc1e-4dc1-82e6-5614861496a5\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55"
Sep 16 19:07:25.509884 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:25.509776    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d803842-cc6b-4f47-b71f-4d2ec8698a30-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-58pzt\" (UID: \"6d803842-cc6b-4f47-b71f-4d2ec8698a30\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-58pzt"
Sep 16 19:07:25.509884 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:25.509797    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9528z\" (UniqueName: \"kubernetes.io/projected/6d803842-cc6b-4f47-b71f-4d2ec8698a30-kube-api-access-9528z\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-58pzt\" (UID: \"6d803842-cc6b-4f47-b71f-4d2ec8698a30\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-58pzt"
Sep 16 19:07:25.509884 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:25.509819    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e8b83bf-bc1e-4dc1-82e6-5614861496a5-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55\" (UID: \"3e8b83bf-bc1e-4dc1-82e6-5614861496a5\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55"
Sep 16 19:07:25.509884 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:25.509850    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e8b83bf-bc1e-4dc1-82e6-5614861496a5-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55\" (UID: \"3e8b83bf-bc1e-4dc1-82e6-5614861496a5\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55"
Sep 16 19:07:25.509884 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:25.509877    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e8b83bf-bc1e-4dc1-82e6-5614861496a5-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55\" (UID: \"3e8b83bf-bc1e-4dc1-82e6-5614861496a5\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55"
Sep 16 19:07:25.510146 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:25.509929    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d803842-cc6b-4f47-b71f-4d2ec8698a30-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-58pzt\" (UID: \"6d803842-cc6b-4f47-b71f-4d2ec8698a30\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-58pzt"
Sep 16 19:07:25.510146 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:25.509951    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d803842-cc6b-4f47-b71f-4d2ec8698a30-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-58pzt\" (UID: \"6d803842-cc6b-4f47-b71f-4d2ec8698a30\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-58pzt"
Sep 16 19:07:25.510146 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:25.509980    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxm8d\" (UniqueName: \"kubernetes.io/projected/3e8b83bf-bc1e-4dc1-82e6-5614861496a5-kube-api-access-rxm8d\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55\" (UID: \"3e8b83bf-bc1e-4dc1-82e6-5614861496a5\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55"
Sep 16 19:07:25.510146 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:25.510007    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d803842-cc6b-4f47-b71f-4d2ec8698a30-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-58pzt\" (UID: \"6d803842-cc6b-4f47-b71f-4d2ec8698a30\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-58pzt"
Sep 16 19:07:25.510146 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:25.510040    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e8b83bf-bc1e-4dc1-82e6-5614861496a5-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55\" (UID: \"3e8b83bf-bc1e-4dc1-82e6-5614861496a5\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55"
Sep 16 19:07:25.510146 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:25.510071    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d803842-cc6b-4f47-b71f-4d2ec8698a30-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-58pzt\" (UID: \"6d803842-cc6b-4f47-b71f-4d2ec8698a30\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-58pzt"
Sep 16 19:07:25.510451 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:25.510240    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d803842-cc6b-4f47-b71f-4d2ec8698a30-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-58pzt\" (UID: \"6d803842-cc6b-4f47-b71f-4d2ec8698a30\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-58pzt"
Sep 16 19:07:25.510451 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:25.510325    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d803842-cc6b-4f47-b71f-4d2ec8698a30-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-58pzt\" (UID: \"6d803842-cc6b-4f47-b71f-4d2ec8698a30\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-58pzt"
Sep 16 19:07:25.510451 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:25.510360    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e8b83bf-bc1e-4dc1-82e6-5614861496a5-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55\" (UID: \"3e8b83bf-bc1e-4dc1-82e6-5614861496a5\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55"
Sep 16 19:07:25.510451 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:25.510403    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d803842-cc6b-4f47-b71f-4d2ec8698a30-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-58pzt\" (UID: \"6d803842-cc6b-4f47-b71f-4d2ec8698a30\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-58pzt"
Sep 16 19:07:25.510451 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:25.510404    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e8b83bf-bc1e-4dc1-82e6-5614861496a5-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55\" (UID: \"3e8b83bf-bc1e-4dc1-82e6-5614861496a5\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55"
Sep 16 19:07:25.510912 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:25.510889    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e8b83bf-bc1e-4dc1-82e6-5614861496a5-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55\" (UID: \"3e8b83bf-bc1e-4dc1-82e6-5614861496a5\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55"
Sep 16 19:07:25.512666 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:25.512635    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d803842-cc6b-4f47-b71f-4d2ec8698a30-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-58pzt\" (UID: \"6d803842-cc6b-4f47-b71f-4d2ec8698a30\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-58pzt"
Sep 16 19:07:25.512858 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:25.512837    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e8b83bf-bc1e-4dc1-82e6-5614861496a5-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55\" (UID: \"3e8b83bf-bc1e-4dc1-82e6-5614861496a5\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55"
Sep 16 19:07:25.512915 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:25.512842    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d803842-cc6b-4f47-b71f-4d2ec8698a30-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-58pzt\" (UID: \"6d803842-cc6b-4f47-b71f-4d2ec8698a30\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-58pzt"
Sep 16 19:07:25.513008 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:25.512992    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e8b83bf-bc1e-4dc1-82e6-5614861496a5-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55\" (UID: \"3e8b83bf-bc1e-4dc1-82e6-5614861496a5\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55"
Sep 16 19:07:25.520666 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:25.520638    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxm8d\" (UniqueName: \"kubernetes.io/projected/3e8b83bf-bc1e-4dc1-82e6-5614861496a5-kube-api-access-rxm8d\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55\" (UID: \"3e8b83bf-bc1e-4dc1-82e6-5614861496a5\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55"
Sep 16 19:07:25.520962 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:25.520940    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9528z\" (UniqueName: \"kubernetes.io/projected/6d803842-cc6b-4f47-b71f-4d2ec8698a30-kube-api-access-9528z\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-58pzt\" (UID: \"6d803842-cc6b-4f47-b71f-4d2ec8698a30\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-58pzt"
Sep 16 19:07:25.572848 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:25.572810    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55"
Sep 16 19:07:25.580998 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:25.580977    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-58pzt"
Sep 16 19:07:25.856768 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:25.856742    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55"]
Sep 16 19:07:25.859487 ip-10-0-129-194 kubenswrapper[2582]: W0916 19:07:25.859457    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e8b83bf_bc1e_4dc1_82e6_5614861496a5.slice/crio-ee9cd6b9a87ba7c3bd0b3a29cb92d7c28897104a3d9235ee8cd752623296cb08 WatchSource:0}: Error finding container ee9cd6b9a87ba7c3bd0b3a29cb92d7c28897104a3d9235ee8cd752623296cb08: Status 404 returned error can't find the container with id ee9cd6b9a87ba7c3bd0b3a29cb92d7c28897104a3d9235ee8cd752623296cb08
Sep 16 19:07:25.861270 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:25.861178    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-58pzt"]
Sep 16 19:07:25.861749 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:25.861731    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:07:25.864079 ip-10-0-129-194 kubenswrapper[2582]: W0916 19:07:25.864055    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d803842_cc6b_4f47_b71f_4d2ec8698a30.slice/crio-e4fd48e8370988204c96f84d08ad406cd8ac2088219261542119af2f6d74d6c9 WatchSource:0}: Error finding container e4fd48e8370988204c96f84d08ad406cd8ac2088219261542119af2f6d74d6c9: Status 404 returned error can't find the container with id e4fd48e8370988204c96f84d08ad406cd8ac2088219261542119af2f6d74d6c9
Sep 16 19:07:26.104370 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:26.104326    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-58pzt" event={"ID":"6d803842-cc6b-4f47-b71f-4d2ec8698a30","Type":"ContainerStarted","Data":"4ddf449b342dd7de8d4485144647283f650490cce406fd2ff48f9a74bed0e1b5"}
Sep 16 19:07:26.104572 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:26.104385    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-58pzt" event={"ID":"6d803842-cc6b-4f47-b71f-4d2ec8698a30","Type":"ContainerStarted","Data":"e4fd48e8370988204c96f84d08ad406cd8ac2088219261542119af2f6d74d6c9"}
Sep 16 19:07:26.105538 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:26.105516    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55" event={"ID":"3e8b83bf-bc1e-4dc1-82e6-5614861496a5","Type":"ContainerStarted","Data":"ee9cd6b9a87ba7c3bd0b3a29cb92d7c28897104a3d9235ee8cd752623296cb08"}
Sep 16 19:07:26.122188 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:26.122082    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-58pzt" podStartSLOduration=1.122067363 podStartE2EDuration="1.122067363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:07:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:07:26.121034533 +0000 UTC m=+1449.762750198" watchObservedRunningTime="2026-09-16 19:07:26.122067363 +0000 UTC m=+1449.763783027"
Sep 16 19:07:26.581302 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:26.581263    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-58pzt"
Sep 16 19:07:29.120825 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:29.120790    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55" event={"ID":"3e8b83bf-bc1e-4dc1-82e6-5614861496a5","Type":"ContainerStarted","Data":"19cef258adca1f25786527408edd46c20a25f16a55cf784e66153ef24422a89a"}
Sep 16 19:07:29.121223 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:29.120911    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55"
Sep 16 19:07:30.127988 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:30.127953    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55" event={"ID":"3e8b83bf-bc1e-4dc1-82e6-5614861496a5","Type":"ContainerStarted","Data":"ba723f370b12039baa0c5f5d9f3612f450884656ca42dc318b5a975122f229c9"}
Sep 16 19:07:30.157356 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:30.157304    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55" podStartSLOduration=2.957387009 podStartE2EDuration="5.157288267s" totalImagesPullingTime="2.199901258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:07:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:07:30.154679185 +0000 UTC m=+1453.796394849" watchObservedRunningTime="2026-09-16 19:07:30.157288267 +0000 UTC m=+1453.799003931"
Sep 16 19:07:30.573581 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:30.573547    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55"
Sep 16 19:07:35.573242 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:35.573198    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55"
Sep 16 19:07:35.581878 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:35.581838    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-58pzt"
Sep 16 19:07:35.587107 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:35.587076    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55"
Sep 16 19:07:35.595396 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:35.595368    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-58pzt"
Sep 16 19:07:36.179291 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:36.179262    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-58pzt"
Sep 16 19:07:36.179475 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:36.179330    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55"
Sep 16 19:07:47.178695 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:07:47.178657    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55"
Sep 16 19:08:16.887169 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:08:16.887139    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7qwl2_b6695f86-08c0-4836-87b1-f2a9d76084bd/console-operator/1.log"
Sep 16 19:08:16.890149 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:08:16.890118    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7qwl2_b6695f86-08c0-4836-87b1-f2a9d76084bd/console-operator/1.log"
Sep 16 19:09:05.392737 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:05.392687    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55"]
Sep 16 19:09:05.402057 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:09:05.402016    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:09:05.402057 ip-10-0-129-194 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:09:05.402057 ip-10-0-129-194 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 19:09:05.402057 ip-10-0-129-194 kubenswrapper[2582]:  > containerID="ba723f370b12039baa0c5f5d9f3612f450884656ca42dc318b5a975122f229c9" cmd=["/bin/sleep","15"]
Sep 16 19:09:05.402320 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:05.402081    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55" message="" err=<
Sep 16 19:09:05.402320 ip-10-0-129-194 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:09:05.402320 ip-10-0-129-194 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 19:09:05.402320 ip-10-0-129-194 kubenswrapper[2582]:  >
Sep 16 19:09:05.402320 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:09:05.402094    2582 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:09:05.402320 ip-10-0-129-194 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:09:05.402320 ip-10-0-129-194 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 19:09:05.402320 ip-10-0-129-194 kubenswrapper[2582]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55" podUID="3e8b83bf-bc1e-4dc1-82e6-5614861496a5" containerName="main" containerID="cri-o://ba723f370b12039baa0c5f5d9f3612f450884656ca42dc318b5a975122f229c9"
Sep 16 19:09:05.402320 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:05.402138    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55" podUID="3e8b83bf-bc1e-4dc1-82e6-5614861496a5" containerName="main" containerID="cri-o://ba723f370b12039baa0c5f5d9f3612f450884656ca42dc318b5a975122f229c9" gracePeriod=60
Sep 16 19:09:05.404592 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:05.404568    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-58pzt"]
Sep 16 19:09:05.413106 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:09:05.413064    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:09:05.413106 ip-10-0-129-194 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:09:05.413106 ip-10-0-129-194 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 19:09:05.413106 ip-10-0-129-194 kubenswrapper[2582]:  > containerID="4ddf449b342dd7de8d4485144647283f650490cce406fd2ff48f9a74bed0e1b5" cmd=["/bin/sleep","15"]
Sep 16 19:09:05.413404 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:05.413126    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-58pzt" message="" err=<
Sep 16 19:09:05.413404 ip-10-0-129-194 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:09:05.413404 ip-10-0-129-194 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 19:09:05.413404 ip-10-0-129-194 kubenswrapper[2582]:  >
Sep 16 19:09:05.413404 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:09:05.413145    2582 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:09:05.413404 ip-10-0-129-194 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:09:05.413404 ip-10-0-129-194 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 19:09:05.413404 ip-10-0-129-194 kubenswrapper[2582]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-58pzt" podUID="6d803842-cc6b-4f47-b71f-4d2ec8698a30" containerName="main" containerID="cri-o://4ddf449b342dd7de8d4485144647283f650490cce406fd2ff48f9a74bed0e1b5"
Sep 16 19:09:05.413404 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:05.413192    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-58pzt" podUID="6d803842-cc6b-4f47-b71f-4d2ec8698a30" containerName="main" containerID="cri-o://4ddf449b342dd7de8d4485144647283f650490cce406fd2ff48f9a74bed0e1b5" gracePeriod=60
Sep 16 19:09:05.572514 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:05.572470    2582 generic.go:356] "Generic (PLEG): container finished" podID="3e8b83bf-bc1e-4dc1-82e6-5614861496a5" containerID="ba723f370b12039baa0c5f5d9f3612f450884656ca42dc318b5a975122f229c9" exitCode=0
Sep 16 19:09:05.572648 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:05.572512    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55" event={"ID":"3e8b83bf-bc1e-4dc1-82e6-5614861496a5","Type":"ContainerDied","Data":"ba723f370b12039baa0c5f5d9f3612f450884656ca42dc318b5a975122f229c9"}
Sep 16 19:09:05.761169 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:05.761137    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-58pzt"
Sep 16 19:09:05.802877 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:05.802841    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9528z\" (UniqueName: \"kubernetes.io/projected/6d803842-cc6b-4f47-b71f-4d2ec8698a30-kube-api-access-9528z\") pod \"6d803842-cc6b-4f47-b71f-4d2ec8698a30\" (UID: \"6d803842-cc6b-4f47-b71f-4d2ec8698a30\") "
Sep 16 19:09:05.803088 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:05.802886    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d803842-cc6b-4f47-b71f-4d2ec8698a30-home\") pod \"6d803842-cc6b-4f47-b71f-4d2ec8698a30\" (UID: \"6d803842-cc6b-4f47-b71f-4d2ec8698a30\") "
Sep 16 19:09:05.803088 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:05.802910    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d803842-cc6b-4f47-b71f-4d2ec8698a30-tls-certs\") pod \"6d803842-cc6b-4f47-b71f-4d2ec8698a30\" (UID: \"6d803842-cc6b-4f47-b71f-4d2ec8698a30\") "
Sep 16 19:09:05.803088 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:05.802934    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d803842-cc6b-4f47-b71f-4d2ec8698a30-tmp-dir\") pod \"6d803842-cc6b-4f47-b71f-4d2ec8698a30\" (UID: \"6d803842-cc6b-4f47-b71f-4d2ec8698a30\") "
Sep 16 19:09:05.803088 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:05.802961    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d803842-cc6b-4f47-b71f-4d2ec8698a30-model-cache\") pod \"6d803842-cc6b-4f47-b71f-4d2ec8698a30\" (UID: \"6d803842-cc6b-4f47-b71f-4d2ec8698a30\") "
Sep 16 19:09:05.803088 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:05.803001    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d803842-cc6b-4f47-b71f-4d2ec8698a30-dshm\") pod \"6d803842-cc6b-4f47-b71f-4d2ec8698a30\" (UID: \"6d803842-cc6b-4f47-b71f-4d2ec8698a30\") "
Sep 16 19:09:05.803350 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:05.803205    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d803842-cc6b-4f47-b71f-4d2ec8698a30-home" (OuterVolumeSpecName: "home") pod "6d803842-cc6b-4f47-b71f-4d2ec8698a30" (UID: "6d803842-cc6b-4f47-b71f-4d2ec8698a30"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:09:05.803350 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:05.803296    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d803842-cc6b-4f47-b71f-4d2ec8698a30-home\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:09:05.803350 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:05.803301    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d803842-cc6b-4f47-b71f-4d2ec8698a30-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6d803842-cc6b-4f47-b71f-4d2ec8698a30" (UID: "6d803842-cc6b-4f47-b71f-4d2ec8698a30"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:09:05.803477 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:05.803411    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d803842-cc6b-4f47-b71f-4d2ec8698a30-model-cache" (OuterVolumeSpecName: "model-cache") pod "6d803842-cc6b-4f47-b71f-4d2ec8698a30" (UID: "6d803842-cc6b-4f47-b71f-4d2ec8698a30"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:09:05.805396 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:05.805368    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d803842-cc6b-4f47-b71f-4d2ec8698a30-dshm" (OuterVolumeSpecName: "dshm") pod "6d803842-cc6b-4f47-b71f-4d2ec8698a30" (UID: "6d803842-cc6b-4f47-b71f-4d2ec8698a30"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:09:05.805641 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:05.805615    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d803842-cc6b-4f47-b71f-4d2ec8698a30-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6d803842-cc6b-4f47-b71f-4d2ec8698a30" (UID: "6d803842-cc6b-4f47-b71f-4d2ec8698a30"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:09:05.805759 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:05.805640    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d803842-cc6b-4f47-b71f-4d2ec8698a30-kube-api-access-9528z" (OuterVolumeSpecName: "kube-api-access-9528z") pod "6d803842-cc6b-4f47-b71f-4d2ec8698a30" (UID: "6d803842-cc6b-4f47-b71f-4d2ec8698a30"). InnerVolumeSpecName "kube-api-access-9528z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:09:05.903769 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:05.903680    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d803842-cc6b-4f47-b71f-4d2ec8698a30-dshm\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:09:05.903769 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:05.903764    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9528z\" (UniqueName: \"kubernetes.io/projected/6d803842-cc6b-4f47-b71f-4d2ec8698a30-kube-api-access-9528z\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:09:05.903769 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:05.903777    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d803842-cc6b-4f47-b71f-4d2ec8698a30-tls-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:09:05.903769 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:05.903787    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d803842-cc6b-4f47-b71f-4d2ec8698a30-tmp-dir\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:09:05.904048 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:05.903795    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d803842-cc6b-4f47-b71f-4d2ec8698a30-model-cache\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:09:06.015737 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:06.015688    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55" podUID="3e8b83bf-bc1e-4dc1-82e6-5614861496a5" containerName="llm-d-routing-sidecar" containerID="cri-o://19cef258adca1f25786527408edd46c20a25f16a55cf784e66153ef24422a89a" gracePeriod=60
Sep 16 19:09:06.162128 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:06.162084    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55" podUID="3e8b83bf-bc1e-4dc1-82e6-5614861496a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8001/health\": dial tcp 10.132.0.34:8001: connect: connection refused"
Sep 16 19:09:06.257858 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:06.257829    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55"
Sep 16 19:09:06.307583 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:06.307487    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e8b83bf-bc1e-4dc1-82e6-5614861496a5-home\") pod \"3e8b83bf-bc1e-4dc1-82e6-5614861496a5\" (UID: \"3e8b83bf-bc1e-4dc1-82e6-5614861496a5\") "
Sep 16 19:09:06.307583 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:06.307536    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e8b83bf-bc1e-4dc1-82e6-5614861496a5-tls-certs\") pod \"3e8b83bf-bc1e-4dc1-82e6-5614861496a5\" (UID: \"3e8b83bf-bc1e-4dc1-82e6-5614861496a5\") "
Sep 16 19:09:06.307583 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:06.307563    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e8b83bf-bc1e-4dc1-82e6-5614861496a5-model-cache\") pod \"3e8b83bf-bc1e-4dc1-82e6-5614861496a5\" (UID: \"3e8b83bf-bc1e-4dc1-82e6-5614861496a5\") "
Sep 16 19:09:06.307897 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:06.307612    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rxm8d\" (UniqueName: \"kubernetes.io/projected/3e8b83bf-bc1e-4dc1-82e6-5614861496a5-kube-api-access-rxm8d\") pod \"3e8b83bf-bc1e-4dc1-82e6-5614861496a5\" (UID: \"3e8b83bf-bc1e-4dc1-82e6-5614861496a5\") "
Sep 16 19:09:06.307897 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:06.307638    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e8b83bf-bc1e-4dc1-82e6-5614861496a5-dshm\") pod \"3e8b83bf-bc1e-4dc1-82e6-5614861496a5\" (UID: \"3e8b83bf-bc1e-4dc1-82e6-5614861496a5\") "
Sep 16 19:09:06.307897 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:06.307875    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8b83bf-bc1e-4dc1-82e6-5614861496a5-model-cache" (OuterVolumeSpecName: "model-cache") pod "3e8b83bf-bc1e-4dc1-82e6-5614861496a5" (UID: "3e8b83bf-bc1e-4dc1-82e6-5614861496a5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:09:06.308003 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:06.307875    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8b83bf-bc1e-4dc1-82e6-5614861496a5-home" (OuterVolumeSpecName: "home") pod "3e8b83bf-bc1e-4dc1-82e6-5614861496a5" (UID: "3e8b83bf-bc1e-4dc1-82e6-5614861496a5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:09:06.309861 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:06.309827    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e8b83bf-bc1e-4dc1-82e6-5614861496a5-kube-api-access-rxm8d" (OuterVolumeSpecName: "kube-api-access-rxm8d") pod "3e8b83bf-bc1e-4dc1-82e6-5614861496a5" (UID: "3e8b83bf-bc1e-4dc1-82e6-5614861496a5"). InnerVolumeSpecName "kube-api-access-rxm8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:09:06.309861 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:06.309844    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8b83bf-bc1e-4dc1-82e6-5614861496a5-dshm" (OuterVolumeSpecName: "dshm") pod "3e8b83bf-bc1e-4dc1-82e6-5614861496a5" (UID: "3e8b83bf-bc1e-4dc1-82e6-5614861496a5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:09:06.310009 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:06.309866    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e8b83bf-bc1e-4dc1-82e6-5614861496a5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3e8b83bf-bc1e-4dc1-82e6-5614861496a5" (UID: "3e8b83bf-bc1e-4dc1-82e6-5614861496a5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:09:06.408177 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:06.408141    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e8b83bf-bc1e-4dc1-82e6-5614861496a5-tmp-dir\") pod \"3e8b83bf-bc1e-4dc1-82e6-5614861496a5\" (UID: \"3e8b83bf-bc1e-4dc1-82e6-5614861496a5\") "
Sep 16 19:09:06.408548 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:06.408322    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e8b83bf-bc1e-4dc1-82e6-5614861496a5-home\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:09:06.408548 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:06.408335    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e8b83bf-bc1e-4dc1-82e6-5614861496a5-tls-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:09:06.408548 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:06.408346    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e8b83bf-bc1e-4dc1-82e6-5614861496a5-model-cache\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:09:06.408548 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:06.408354    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxm8d\" (UniqueName: \"kubernetes.io/projected/3e8b83bf-bc1e-4dc1-82e6-5614861496a5-kube-api-access-rxm8d\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:09:06.408548 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:06.408363    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e8b83bf-bc1e-4dc1-82e6-5614861496a5-dshm\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:09:06.408548 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:06.408409    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8b83bf-bc1e-4dc1-82e6-5614861496a5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3e8b83bf-bc1e-4dc1-82e6-5614861496a5" (UID: "3e8b83bf-bc1e-4dc1-82e6-5614861496a5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:09:06.509513 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:06.509477    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e8b83bf-bc1e-4dc1-82e6-5614861496a5-tmp-dir\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:09:06.578611 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:06.578511    2582 generic.go:356] "Generic (PLEG): container finished" podID="6d803842-cc6b-4f47-b71f-4d2ec8698a30" containerID="4ddf449b342dd7de8d4485144647283f650490cce406fd2ff48f9a74bed0e1b5" exitCode=0
Sep 16 19:09:06.578611 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:06.578573    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-58pzt"
Sep 16 19:09:06.578611 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:06.578596    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-58pzt" event={"ID":"6d803842-cc6b-4f47-b71f-4d2ec8698a30","Type":"ContainerDied","Data":"4ddf449b342dd7de8d4485144647283f650490cce406fd2ff48f9a74bed0e1b5"}
Sep 16 19:09:06.578925 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:06.578683    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-58pzt" event={"ID":"6d803842-cc6b-4f47-b71f-4d2ec8698a30","Type":"ContainerDied","Data":"e4fd48e8370988204c96f84d08ad406cd8ac2088219261542119af2f6d74d6c9"}
Sep 16 19:09:06.578925 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:06.578708    2582 scope.go:117] "RemoveContainer" containerID="4ddf449b342dd7de8d4485144647283f650490cce406fd2ff48f9a74bed0e1b5"
Sep 16 19:09:06.580382 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:06.580351    2582 generic.go:356] "Generic (PLEG): container finished" podID="3e8b83bf-bc1e-4dc1-82e6-5614861496a5" containerID="19cef258adca1f25786527408edd46c20a25f16a55cf784e66153ef24422a89a" exitCode=0
Sep 16 19:09:06.580511 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:06.580433    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55"
Sep 16 19:09:06.580511 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:06.580466    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55" event={"ID":"3e8b83bf-bc1e-4dc1-82e6-5614861496a5","Type":"ContainerDied","Data":"19cef258adca1f25786527408edd46c20a25f16a55cf784e66153ef24422a89a"}
Sep 16 19:09:06.580595 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:06.580515    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55" event={"ID":"3e8b83bf-bc1e-4dc1-82e6-5614861496a5","Type":"ContainerDied","Data":"ee9cd6b9a87ba7c3bd0b3a29cb92d7c28897104a3d9235ee8cd752623296cb08"}
Sep 16 19:09:06.590206 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:06.590178    2582 scope.go:117] "RemoveContainer" containerID="4ddf449b342dd7de8d4485144647283f650490cce406fd2ff48f9a74bed0e1b5"
Sep 16 19:09:06.590612 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:09:06.590584    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ddf449b342dd7de8d4485144647283f650490cce406fd2ff48f9a74bed0e1b5\": container with ID starting with 4ddf449b342dd7de8d4485144647283f650490cce406fd2ff48f9a74bed0e1b5 not found: ID does not exist" containerID="4ddf449b342dd7de8d4485144647283f650490cce406fd2ff48f9a74bed0e1b5"
Sep 16 19:09:06.590708 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:06.590619    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ddf449b342dd7de8d4485144647283f650490cce406fd2ff48f9a74bed0e1b5"} err="failed to get container status \"4ddf449b342dd7de8d4485144647283f650490cce406fd2ff48f9a74bed0e1b5\": rpc error: code = NotFound desc = could not find container \"4ddf449b342dd7de8d4485144647283f650490cce406fd2ff48f9a74bed0e1b5\": container with ID starting with 4ddf449b342dd7de8d4485144647283f650490cce406fd2ff48f9a74bed0e1b5 not found: ID does not exist"
Sep 16 19:09:06.590708 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:06.590643    2582 scope.go:117] "RemoveContainer" containerID="ba723f370b12039baa0c5f5d9f3612f450884656ca42dc318b5a975122f229c9"
Sep 16 19:09:06.602159 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:06.602127    2582 scope.go:117] "RemoveContainer" containerID="19cef258adca1f25786527408edd46c20a25f16a55cf784e66153ef24422a89a"
Sep 16 19:09:06.605847 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:06.605819    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-58pzt"]
Sep 16 19:09:06.610543 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:06.610515    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-58pzt"]
Sep 16 19:09:06.610854 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:06.610840    2582 scope.go:117] "RemoveContainer" containerID="ba723f370b12039baa0c5f5d9f3612f450884656ca42dc318b5a975122f229c9"
Sep 16 19:09:06.611154 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:09:06.611130    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba723f370b12039baa0c5f5d9f3612f450884656ca42dc318b5a975122f229c9\": container with ID starting with ba723f370b12039baa0c5f5d9f3612f450884656ca42dc318b5a975122f229c9 not found: ID does not exist" containerID="ba723f370b12039baa0c5f5d9f3612f450884656ca42dc318b5a975122f229c9"
Sep 16 19:09:06.611239 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:06.611166    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba723f370b12039baa0c5f5d9f3612f450884656ca42dc318b5a975122f229c9"} err="failed to get container status \"ba723f370b12039baa0c5f5d9f3612f450884656ca42dc318b5a975122f229c9\": rpc error: code = NotFound desc = could not find container \"ba723f370b12039baa0c5f5d9f3612f450884656ca42dc318b5a975122f229c9\": container with ID starting with ba723f370b12039baa0c5f5d9f3612f450884656ca42dc318b5a975122f229c9 not found: ID does not exist"
Sep 16 19:09:06.611239 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:06.611188    2582 scope.go:117] "RemoveContainer" containerID="19cef258adca1f25786527408edd46c20a25f16a55cf784e66153ef24422a89a"
Sep 16 19:09:06.611479 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:09:06.611463    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19cef258adca1f25786527408edd46c20a25f16a55cf784e66153ef24422a89a\": container with ID starting with 19cef258adca1f25786527408edd46c20a25f16a55cf784e66153ef24422a89a not found: ID does not exist" containerID="19cef258adca1f25786527408edd46c20a25f16a55cf784e66153ef24422a89a"
Sep 16 19:09:06.611530 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:06.611482    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19cef258adca1f25786527408edd46c20a25f16a55cf784e66153ef24422a89a"} err="failed to get container status \"19cef258adca1f25786527408edd46c20a25f16a55cf784e66153ef24422a89a\": rpc error: code = NotFound desc = could not find container \"19cef258adca1f25786527408edd46c20a25f16a55cf784e66153ef24422a89a\": container with ID starting with 19cef258adca1f25786527408edd46c20a25f16a55cf784e66153ef24422a89a not found: ID does not exist"
Sep 16 19:09:06.624975 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:06.624941    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55"]
Sep 16 19:09:06.626489 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:06.626464    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-cgs55"]
Sep 16 19:09:06.873113 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:06.873030    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e8b83bf-bc1e-4dc1-82e6-5614861496a5" path="/var/lib/kubelet/pods/3e8b83bf-bc1e-4dc1-82e6-5614861496a5/volumes"
Sep 16 19:09:06.873475 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:09:06.873462    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d803842-cc6b-4f47-b71f-4d2ec8698a30" path="/var/lib/kubelet/pods/6d803842-cc6b-4f47-b71f-4d2ec8698a30/volumes"
Sep 16 19:10:31.032075 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:10:31.032043    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d"]
Sep 16 19:10:31.032533 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:10:31.032345    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e8b83bf-bc1e-4dc1-82e6-5614861496a5" containerName="llm-d-routing-sidecar"
Sep 16 19:10:31.032533 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:10:31.032356    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e8b83bf-bc1e-4dc1-82e6-5614861496a5" containerName="llm-d-routing-sidecar"
Sep 16 19:10:31.032533 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:10:31.032370    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d803842-cc6b-4f47-b71f-4d2ec8698a30" containerName="main"
Sep 16 19:10:31.032533 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:10:31.032375    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d803842-cc6b-4f47-b71f-4d2ec8698a30" containerName="main"
Sep 16 19:10:31.032533 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:10:31.032381    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e8b83bf-bc1e-4dc1-82e6-5614861496a5" containerName="main"
Sep 16 19:10:31.032533 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:10:31.032387    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e8b83bf-bc1e-4dc1-82e6-5614861496a5" containerName="main"
Sep 16 19:10:31.032533 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:10:31.032441    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e8b83bf-bc1e-4dc1-82e6-5614861496a5" containerName="llm-d-routing-sidecar"
Sep 16 19:10:31.032533 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:10:31.032449    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e8b83bf-bc1e-4dc1-82e6-5614861496a5" containerName="main"
Sep 16 19:10:31.032533 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:10:31.032455    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d803842-cc6b-4f47-b71f-4d2ec8698a30" containerName="main"
Sep 16 19:10:31.035511 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:10:31.035483    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d"
Sep 16 19:10:31.043962 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:10:31.043934    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 16 19:10:31.043962 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:10:31.043992    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 16 19:10:31.044211 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:10:31.043934    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-ldrjs\""
Sep 16 19:10:31.045096 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:10:31.045075    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 16 19:10:31.045213 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:10:31.045185    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-dockercfg-lwlxb\""
Sep 16 19:10:31.063584 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:10:31.063555    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d"]
Sep 16 19:10:31.107627 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:10:31.107564    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67ce1208-ce2c-41ad-97d7-6f6a3b45776b-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d\" (UID: \"67ce1208-ce2c-41ad-97d7-6f6a3b45776b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d"
Sep 16 19:10:31.107627 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:10:31.107603    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt2vw\" (UniqueName: \"kubernetes.io/projected/67ce1208-ce2c-41ad-97d7-6f6a3b45776b-kube-api-access-jt2vw\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d\" (UID: \"67ce1208-ce2c-41ad-97d7-6f6a3b45776b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d"
Sep 16 19:10:31.107627 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:10:31.107633    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67ce1208-ce2c-41ad-97d7-6f6a3b45776b-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d\" (UID: \"67ce1208-ce2c-41ad-97d7-6f6a3b45776b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d"
Sep 16 19:10:31.107953 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:10:31.107764    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67ce1208-ce2c-41ad-97d7-6f6a3b45776b-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d\" (UID: \"67ce1208-ce2c-41ad-97d7-6f6a3b45776b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d"
Sep 16 19:10:31.107953 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:10:31.107828    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67ce1208-ce2c-41ad-97d7-6f6a3b45776b-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d\" (UID: \"67ce1208-ce2c-41ad-97d7-6f6a3b45776b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d"
Sep 16 19:10:31.107953 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:10:31.107864    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67ce1208-ce2c-41ad-97d7-6f6a3b45776b-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d\" (UID: \"67ce1208-ce2c-41ad-97d7-6f6a3b45776b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d"
Sep 16 19:10:31.208948 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:10:31.208910    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67ce1208-ce2c-41ad-97d7-6f6a3b45776b-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d\" (UID: \"67ce1208-ce2c-41ad-97d7-6f6a3b45776b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d"
Sep 16 19:10:31.208948 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:10:31.208950    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt2vw\" (UniqueName: \"kubernetes.io/projected/67ce1208-ce2c-41ad-97d7-6f6a3b45776b-kube-api-access-jt2vw\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d\" (UID: \"67ce1208-ce2c-41ad-97d7-6f6a3b45776b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d"
Sep 16 19:10:31.209177 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:10:31.209074    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67ce1208-ce2c-41ad-97d7-6f6a3b45776b-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d\" (UID: \"67ce1208-ce2c-41ad-97d7-6f6a3b45776b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d"
Sep 16 19:10:31.209177 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:10:31.209137    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67ce1208-ce2c-41ad-97d7-6f6a3b45776b-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d\" (UID: \"67ce1208-ce2c-41ad-97d7-6f6a3b45776b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d"
Sep 16 19:10:31.209275 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:10:31.209182    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67ce1208-ce2c-41ad-97d7-6f6a3b45776b-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d\" (UID: \"67ce1208-ce2c-41ad-97d7-6f6a3b45776b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d"
Sep 16 19:10:31.209275 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:10:31.209201    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67ce1208-ce2c-41ad-97d7-6f6a3b45776b-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d\" (UID: \"67ce1208-ce2c-41ad-97d7-6f6a3b45776b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d"
Sep 16 19:10:31.209385 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:10:31.209348    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67ce1208-ce2c-41ad-97d7-6f6a3b45776b-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d\" (UID: \"67ce1208-ce2c-41ad-97d7-6f6a3b45776b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d"
Sep 16 19:10:31.209484 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:10:31.209466    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67ce1208-ce2c-41ad-97d7-6f6a3b45776b-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d\" (UID: \"67ce1208-ce2c-41ad-97d7-6f6a3b45776b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d"
Sep 16 19:10:31.209547 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:10:31.209499    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67ce1208-ce2c-41ad-97d7-6f6a3b45776b-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d\" (UID: \"67ce1208-ce2c-41ad-97d7-6f6a3b45776b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d"
Sep 16 19:10:31.211394 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:10:31.211367    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67ce1208-ce2c-41ad-97d7-6f6a3b45776b-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d\" (UID: \"67ce1208-ce2c-41ad-97d7-6f6a3b45776b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d"
Sep 16 19:10:31.211542 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:10:31.211526    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67ce1208-ce2c-41ad-97d7-6f6a3b45776b-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d\" (UID: \"67ce1208-ce2c-41ad-97d7-6f6a3b45776b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d"
Sep 16 19:10:31.219090 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:10:31.219067    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt2vw\" (UniqueName: \"kubernetes.io/projected/67ce1208-ce2c-41ad-97d7-6f6a3b45776b-kube-api-access-jt2vw\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d\" (UID: \"67ce1208-ce2c-41ad-97d7-6f6a3b45776b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d"
Sep 16 19:10:31.346988 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:10:31.346897    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d"
Sep 16 19:10:31.619640 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:10:31.619613    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d"]
Sep 16 19:10:31.621683 ip-10-0-129-194 kubenswrapper[2582]: W0916 19:10:31.621659    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67ce1208_ce2c_41ad_97d7_6f6a3b45776b.slice/crio-eb06ee83f2a2790f1735b169435e1380a59c66ac8de7d5e1ec6bdbddea2fd1b1 WatchSource:0}: Error finding container eb06ee83f2a2790f1735b169435e1380a59c66ac8de7d5e1ec6bdbddea2fd1b1: Status 404 returned error can't find the container with id eb06ee83f2a2790f1735b169435e1380a59c66ac8de7d5e1ec6bdbddea2fd1b1
Sep 16 19:10:31.945104 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:10:31.945002    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d" event={"ID":"67ce1208-ce2c-41ad-97d7-6f6a3b45776b","Type":"ContainerStarted","Data":"e29e0dd0822429afdac95c317a974f972000b3f58d5a7e89f13035d30e5a46a8"}
Sep 16 19:10:31.945104 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:10:31.945056    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d" event={"ID":"67ce1208-ce2c-41ad-97d7-6f6a3b45776b","Type":"ContainerStarted","Data":"eb06ee83f2a2790f1735b169435e1380a59c66ac8de7d5e1ec6bdbddea2fd1b1"}
Sep 16 19:10:31.945315 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:10:31.945116    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d"
Sep 16 19:10:32.950991 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:10:32.950952    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d" event={"ID":"67ce1208-ce2c-41ad-97d7-6f6a3b45776b","Type":"ContainerStarted","Data":"906be88f825bf063ce4de1cb1149a9393f7bd2b235966d5285a618a058165d49"}
Sep 16 19:10:32.972792 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:10:32.972700    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d" podStartSLOduration=1.972685463 podStartE2EDuration="1.972685463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:10:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:10:32.971189198 +0000 UTC m=+1636.612904863" watchObservedRunningTime="2026-09-16 19:10:32.972685463 +0000 UTC m=+1636.614401126"
Sep 16 19:10:33.347533 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:10:33.347496    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d"
Sep 16 19:10:41.347798 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:10:41.347756    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d"
Sep 16 19:10:41.366632 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:10:41.366607    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d"
Sep 16 19:10:41.367556 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:10:41.367537    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d"
Sep 16 19:10:42.000087 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:10:42.000046    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d"
Sep 16 19:11:25.505772 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:11:25.505734    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d"]
Sep 16 19:11:25.515014 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:11:25.514980    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:11:25.515014 ip-10-0-129-194 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:11:25.515014 ip-10-0-129-194 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 19:11:25.515014 ip-10-0-129-194 kubenswrapper[2582]:  > containerID="906be88f825bf063ce4de1cb1149a9393f7bd2b235966d5285a618a058165d49" cmd=["/bin/sleep","15"]
Sep 16 19:11:25.515297 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:11:25.515039    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d" message="" err=<
Sep 16 19:11:25.515297 ip-10-0-129-194 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:11:25.515297 ip-10-0-129-194 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 19:11:25.515297 ip-10-0-129-194 kubenswrapper[2582]:  >
Sep 16 19:11:25.515297 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:11:25.515055    2582 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:11:25.515297 ip-10-0-129-194 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:11:25.515297 ip-10-0-129-194 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 19:11:25.515297 ip-10-0-129-194 kubenswrapper[2582]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d" podUID="67ce1208-ce2c-41ad-97d7-6f6a3b45776b" containerName="main" containerID="cri-o://906be88f825bf063ce4de1cb1149a9393f7bd2b235966d5285a618a058165d49"
Sep 16 19:11:25.515297 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:11:25.515103    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d" podUID="67ce1208-ce2c-41ad-97d7-6f6a3b45776b" containerName="main" containerID="cri-o://906be88f825bf063ce4de1cb1149a9393f7bd2b235966d5285a618a058165d49" gracePeriod=60
Sep 16 19:11:25.730695 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:11:25.730664    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d" podUID="67ce1208-ce2c-41ad-97d7-6f6a3b45776b" containerName="llm-d-routing-sidecar" containerID="cri-o://e29e0dd0822429afdac95c317a974f972000b3f58d5a7e89f13035d30e5a46a8" gracePeriod=60
Sep 16 19:11:25.981174 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:11:25.981146    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d"
Sep 16 19:11:26.048929 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:11:26.048897    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jt2vw\" (UniqueName: \"kubernetes.io/projected/67ce1208-ce2c-41ad-97d7-6f6a3b45776b-kube-api-access-jt2vw\") pod \"67ce1208-ce2c-41ad-97d7-6f6a3b45776b\" (UID: \"67ce1208-ce2c-41ad-97d7-6f6a3b45776b\") "
Sep 16 19:11:26.049108 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:11:26.048962    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67ce1208-ce2c-41ad-97d7-6f6a3b45776b-tmp-dir\") pod \"67ce1208-ce2c-41ad-97d7-6f6a3b45776b\" (UID: \"67ce1208-ce2c-41ad-97d7-6f6a3b45776b\") "
Sep 16 19:11:26.049108 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:11:26.048990    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67ce1208-ce2c-41ad-97d7-6f6a3b45776b-dshm\") pod \"67ce1208-ce2c-41ad-97d7-6f6a3b45776b\" (UID: \"67ce1208-ce2c-41ad-97d7-6f6a3b45776b\") "
Sep 16 19:11:26.049108 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:11:26.049011    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67ce1208-ce2c-41ad-97d7-6f6a3b45776b-tls-certs\") pod \"67ce1208-ce2c-41ad-97d7-6f6a3b45776b\" (UID: \"67ce1208-ce2c-41ad-97d7-6f6a3b45776b\") "
Sep 16 19:11:26.049108 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:11:26.049033    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67ce1208-ce2c-41ad-97d7-6f6a3b45776b-home\") pod \"67ce1208-ce2c-41ad-97d7-6f6a3b45776b\" (UID: \"67ce1208-ce2c-41ad-97d7-6f6a3b45776b\") "
Sep 16 19:11:26.049108 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:11:26.049076    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67ce1208-ce2c-41ad-97d7-6f6a3b45776b-model-cache\") pod \"67ce1208-ce2c-41ad-97d7-6f6a3b45776b\" (UID: \"67ce1208-ce2c-41ad-97d7-6f6a3b45776b\") "
Sep 16 19:11:26.049505 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:11:26.049214    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67ce1208-ce2c-41ad-97d7-6f6a3b45776b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "67ce1208-ce2c-41ad-97d7-6f6a3b45776b" (UID: "67ce1208-ce2c-41ad-97d7-6f6a3b45776b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:26.049505 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:11:26.049280    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67ce1208-ce2c-41ad-97d7-6f6a3b45776b-home" (OuterVolumeSpecName: "home") pod "67ce1208-ce2c-41ad-97d7-6f6a3b45776b" (UID: "67ce1208-ce2c-41ad-97d7-6f6a3b45776b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:26.049505 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:11:26.049322    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67ce1208-ce2c-41ad-97d7-6f6a3b45776b-tmp-dir\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:11:26.049505 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:11:26.049352    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67ce1208-ce2c-41ad-97d7-6f6a3b45776b-model-cache" (OuterVolumeSpecName: "model-cache") pod "67ce1208-ce2c-41ad-97d7-6f6a3b45776b" (UID: "67ce1208-ce2c-41ad-97d7-6f6a3b45776b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:26.051067 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:11:26.051045    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67ce1208-ce2c-41ad-97d7-6f6a3b45776b-dshm" (OuterVolumeSpecName: "dshm") pod "67ce1208-ce2c-41ad-97d7-6f6a3b45776b" (UID: "67ce1208-ce2c-41ad-97d7-6f6a3b45776b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:26.051171 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:11:26.051058    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67ce1208-ce2c-41ad-97d7-6f6a3b45776b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "67ce1208-ce2c-41ad-97d7-6f6a3b45776b" (UID: "67ce1208-ce2c-41ad-97d7-6f6a3b45776b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:11:26.051171 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:11:26.051111    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67ce1208-ce2c-41ad-97d7-6f6a3b45776b-kube-api-access-jt2vw" (OuterVolumeSpecName: "kube-api-access-jt2vw") pod "67ce1208-ce2c-41ad-97d7-6f6a3b45776b" (UID: "67ce1208-ce2c-41ad-97d7-6f6a3b45776b"). InnerVolumeSpecName "kube-api-access-jt2vw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:11:26.149870 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:11:26.149785    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67ce1208-ce2c-41ad-97d7-6f6a3b45776b-dshm\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:11:26.149870 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:11:26.149815    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67ce1208-ce2c-41ad-97d7-6f6a3b45776b-tls-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:11:26.149870 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:11:26.149825    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67ce1208-ce2c-41ad-97d7-6f6a3b45776b-home\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:11:26.149870 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:11:26.149835    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67ce1208-ce2c-41ad-97d7-6f6a3b45776b-model-cache\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:11:26.149870 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:11:26.149845    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jt2vw\" (UniqueName: \"kubernetes.io/projected/67ce1208-ce2c-41ad-97d7-6f6a3b45776b-kube-api-access-jt2vw\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:11:26.172677 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:11:26.172649    2582 generic.go:356] "Generic (PLEG): container finished" podID="67ce1208-ce2c-41ad-97d7-6f6a3b45776b" containerID="906be88f825bf063ce4de1cb1149a9393f7bd2b235966d5285a618a058165d49" exitCode=0
Sep 16 19:11:26.172677 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:11:26.172673    2582 generic.go:356] "Generic (PLEG): container finished" podID="67ce1208-ce2c-41ad-97d7-6f6a3b45776b" containerID="e29e0dd0822429afdac95c317a974f972000b3f58d5a7e89f13035d30e5a46a8" exitCode=0
Sep 16 19:11:26.172873 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:11:26.172701    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d"
Sep 16 19:11:26.172873 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:11:26.172750    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d" event={"ID":"67ce1208-ce2c-41ad-97d7-6f6a3b45776b","Type":"ContainerDied","Data":"906be88f825bf063ce4de1cb1149a9393f7bd2b235966d5285a618a058165d49"}
Sep 16 19:11:26.172873 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:11:26.172805    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d" event={"ID":"67ce1208-ce2c-41ad-97d7-6f6a3b45776b","Type":"ContainerDied","Data":"e29e0dd0822429afdac95c317a974f972000b3f58d5a7e89f13035d30e5a46a8"}
Sep 16 19:11:26.172873 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:11:26.172825    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d" event={"ID":"67ce1208-ce2c-41ad-97d7-6f6a3b45776b","Type":"ContainerDied","Data":"eb06ee83f2a2790f1735b169435e1380a59c66ac8de7d5e1ec6bdbddea2fd1b1"}
Sep 16 19:11:26.172873 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:11:26.172846    2582 scope.go:117] "RemoveContainer" containerID="906be88f825bf063ce4de1cb1149a9393f7bd2b235966d5285a618a058165d49"
Sep 16 19:11:26.181710 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:11:26.181694    2582 scope.go:117] "RemoveContainer" containerID="e29e0dd0822429afdac95c317a974f972000b3f58d5a7e89f13035d30e5a46a8"
Sep 16 19:11:26.191471 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:11:26.191453    2582 scope.go:117] "RemoveContainer" containerID="906be88f825bf063ce4de1cb1149a9393f7bd2b235966d5285a618a058165d49"
Sep 16 19:11:26.191763 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:11:26.191746    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"906be88f825bf063ce4de1cb1149a9393f7bd2b235966d5285a618a058165d49\": container with ID starting with 906be88f825bf063ce4de1cb1149a9393f7bd2b235966d5285a618a058165d49 not found: ID does not exist" containerID="906be88f825bf063ce4de1cb1149a9393f7bd2b235966d5285a618a058165d49"
Sep 16 19:11:26.191813 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:11:26.191770    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"906be88f825bf063ce4de1cb1149a9393f7bd2b235966d5285a618a058165d49"} err="failed to get container status \"906be88f825bf063ce4de1cb1149a9393f7bd2b235966d5285a618a058165d49\": rpc error: code = NotFound desc = could not find container \"906be88f825bf063ce4de1cb1149a9393f7bd2b235966d5285a618a058165d49\": container with ID starting with 906be88f825bf063ce4de1cb1149a9393f7bd2b235966d5285a618a058165d49 not found: ID does not exist"
Sep 16 19:11:26.191813 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:11:26.191784    2582 scope.go:117] "RemoveContainer" containerID="e29e0dd0822429afdac95c317a974f972000b3f58d5a7e89f13035d30e5a46a8"
Sep 16 19:11:26.192047 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:11:26.192026    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e29e0dd0822429afdac95c317a974f972000b3f58d5a7e89f13035d30e5a46a8\": container with ID starting with e29e0dd0822429afdac95c317a974f972000b3f58d5a7e89f13035d30e5a46a8 not found: ID does not exist" containerID="e29e0dd0822429afdac95c317a974f972000b3f58d5a7e89f13035d30e5a46a8"
Sep 16 19:11:26.192096 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:11:26.192052    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e29e0dd0822429afdac95c317a974f972000b3f58d5a7e89f13035d30e5a46a8"} err="failed to get container status \"e29e0dd0822429afdac95c317a974f972000b3f58d5a7e89f13035d30e5a46a8\": rpc error: code = NotFound desc = could not find container \"e29e0dd0822429afdac95c317a974f972000b3f58d5a7e89f13035d30e5a46a8\": container with ID starting with e29e0dd0822429afdac95c317a974f972000b3f58d5a7e89f13035d30e5a46a8 not found: ID does not exist"
Sep 16 19:11:26.192096 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:11:26.192067    2582 scope.go:117] "RemoveContainer" containerID="906be88f825bf063ce4de1cb1149a9393f7bd2b235966d5285a618a058165d49"
Sep 16 19:11:26.192283 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:11:26.192267    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"906be88f825bf063ce4de1cb1149a9393f7bd2b235966d5285a618a058165d49"} err="failed to get container status \"906be88f825bf063ce4de1cb1149a9393f7bd2b235966d5285a618a058165d49\": rpc error: code = NotFound desc = could not find container \"906be88f825bf063ce4de1cb1149a9393f7bd2b235966d5285a618a058165d49\": container with ID starting with 906be88f825bf063ce4de1cb1149a9393f7bd2b235966d5285a618a058165d49 not found: ID does not exist"
Sep 16 19:11:26.192322 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:11:26.192285    2582 scope.go:117] "RemoveContainer" containerID="e29e0dd0822429afdac95c317a974f972000b3f58d5a7e89f13035d30e5a46a8"
Sep 16 19:11:26.192510 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:11:26.192494    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e29e0dd0822429afdac95c317a974f972000b3f58d5a7e89f13035d30e5a46a8"} err="failed to get container status \"e29e0dd0822429afdac95c317a974f972000b3f58d5a7e89f13035d30e5a46a8\": rpc error: code = NotFound desc = could not find container \"e29e0dd0822429afdac95c317a974f972000b3f58d5a7e89f13035d30e5a46a8\": container with ID starting with e29e0dd0822429afdac95c317a974f972000b3f58d5a7e89f13035d30e5a46a8 not found: ID does not exist"
Sep 16 19:11:26.211036 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:11:26.211010    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d"]
Sep 16 19:11:26.222615 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:11:26.222592    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zrk2d"]
Sep 16 19:11:26.873759 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:11:26.873710    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67ce1208-ce2c-41ad-97d7-6f6a3b45776b" path="/var/lib/kubelet/pods/67ce1208-ce2c-41ad-97d7-6f6a3b45776b/volumes"
Sep 16 19:12:26.611945 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:12:26.611910    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk"]
Sep 16 19:12:26.612419 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:12:26.612252    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67ce1208-ce2c-41ad-97d7-6f6a3b45776b" containerName="llm-d-routing-sidecar"
Sep 16 19:12:26.612419 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:12:26.612264    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="67ce1208-ce2c-41ad-97d7-6f6a3b45776b" containerName="llm-d-routing-sidecar"
Sep 16 19:12:26.612419 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:12:26.612274    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67ce1208-ce2c-41ad-97d7-6f6a3b45776b" containerName="main"
Sep 16 19:12:26.612419 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:12:26.612281    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="67ce1208-ce2c-41ad-97d7-6f6a3b45776b" containerName="main"
Sep 16 19:12:26.612419 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:12:26.612342    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="67ce1208-ce2c-41ad-97d7-6f6a3b45776b" containerName="llm-d-routing-sidecar"
Sep 16 19:12:26.612419 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:12:26.612353    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="67ce1208-ce2c-41ad-97d7-6f6a3b45776b" containerName="main"
Sep 16 19:12:26.615293 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:12:26.615273    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk"
Sep 16 19:12:26.621043 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:12:26.621009    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 16 19:12:26.621195 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:12:26.621059    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 16 19:12:26.621195 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:12:26.621076    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:12:26.621674 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:12:26.621563    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 16 19:12:26.621798 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:12:26.621782    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-tzf24\""
Sep 16 19:12:26.622100 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:12:26.622083    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 16 19:12:26.627620 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:12:26.627599    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ce0d7a2-aae4-4054-ada7-568f8ce6f75f-home\") pod \"tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk\" (UID: \"6ce0d7a2-aae4-4054-ada7-568f8ce6f75f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk"
Sep 16 19:12:26.627691 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:12:26.627639    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ce0d7a2-aae4-4054-ada7-568f8ce6f75f-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk\" (UID: \"6ce0d7a2-aae4-4054-ada7-568f8ce6f75f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk"
Sep 16 19:12:26.627691 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:12:26.627657    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqjwk\" (UniqueName: \"kubernetes.io/projected/6ce0d7a2-aae4-4054-ada7-568f8ce6f75f-kube-api-access-lqjwk\") pod \"tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk\" (UID: \"6ce0d7a2-aae4-4054-ada7-568f8ce6f75f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk"
Sep 16 19:12:26.627783 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:12:26.627691    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ce0d7a2-aae4-4054-ada7-568f8ce6f75f-dshm\") pod \"tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk\" (UID: \"6ce0d7a2-aae4-4054-ada7-568f8ce6f75f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk"
Sep 16 19:12:26.627783 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:12:26.627711    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ce0d7a2-aae4-4054-ada7-568f8ce6f75f-tls-certs\") pod \"tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk\" (UID: \"6ce0d7a2-aae4-4054-ada7-568f8ce6f75f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk"
Sep 16 19:12:26.627783 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:12:26.627770    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ce0d7a2-aae4-4054-ada7-568f8ce6f75f-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk\" (UID: \"6ce0d7a2-aae4-4054-ada7-568f8ce6f75f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk"
Sep 16 19:12:26.627881 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:12:26.627812    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ce0d7a2-aae4-4054-ada7-568f8ce6f75f-model-cache\") pod \"tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk\" (UID: \"6ce0d7a2-aae4-4054-ada7-568f8ce6f75f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk"
Sep 16 19:12:26.627881 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:12:26.627830    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ce0d7a2-aae4-4054-ada7-568f8ce6f75f-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk\" (UID: \"6ce0d7a2-aae4-4054-ada7-568f8ce6f75f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk"
Sep 16 19:12:26.647054 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:12:26.647024    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk"]
Sep 16 19:12:26.728654 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:12:26.728611    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ce0d7a2-aae4-4054-ada7-568f8ce6f75f-model-cache\") pod \"tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk\" (UID: \"6ce0d7a2-aae4-4054-ada7-568f8ce6f75f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk"
Sep 16 19:12:26.728876 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:12:26.728660    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ce0d7a2-aae4-4054-ada7-568f8ce6f75f-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk\" (UID: \"6ce0d7a2-aae4-4054-ada7-568f8ce6f75f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk"
Sep 16 19:12:26.728876 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:12:26.728707    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ce0d7a2-aae4-4054-ada7-568f8ce6f75f-home\") pod \"tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk\" (UID: \"6ce0d7a2-aae4-4054-ada7-568f8ce6f75f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk"
Sep 16 19:12:26.728876 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:12:26.728779    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ce0d7a2-aae4-4054-ada7-568f8ce6f75f-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk\" (UID: \"6ce0d7a2-aae4-4054-ada7-568f8ce6f75f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk"
Sep 16 19:12:26.728876 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:12:26.728808    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqjwk\" (UniqueName: \"kubernetes.io/projected/6ce0d7a2-aae4-4054-ada7-568f8ce6f75f-kube-api-access-lqjwk\") pod \"tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk\" (UID: \"6ce0d7a2-aae4-4054-ada7-568f8ce6f75f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk"
Sep 16 19:12:26.728876 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:12:26.728832    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ce0d7a2-aae4-4054-ada7-568f8ce6f75f-dshm\") pod \"tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk\" (UID: \"6ce0d7a2-aae4-4054-ada7-568f8ce6f75f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk"
Sep 16 19:12:26.728876 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:12:26.728853    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ce0d7a2-aae4-4054-ada7-568f8ce6f75f-tls-certs\") pod \"tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk\" (UID: \"6ce0d7a2-aae4-4054-ada7-568f8ce6f75f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk"
Sep 16 19:12:26.729275 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:12:26.729131    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ce0d7a2-aae4-4054-ada7-568f8ce6f75f-model-cache\") pod \"tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk\" (UID: \"6ce0d7a2-aae4-4054-ada7-568f8ce6f75f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk"
Sep 16 19:12:26.729275 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:12:26.729148    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ce0d7a2-aae4-4054-ada7-568f8ce6f75f-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk\" (UID: \"6ce0d7a2-aae4-4054-ada7-568f8ce6f75f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk"
Sep 16 19:12:26.729275 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:12:26.729203    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ce0d7a2-aae4-4054-ada7-568f8ce6f75f-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk\" (UID: \"6ce0d7a2-aae4-4054-ada7-568f8ce6f75f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk"
Sep 16 19:12:26.729424 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:12:26.729309    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ce0d7a2-aae4-4054-ada7-568f8ce6f75f-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk\" (UID: \"6ce0d7a2-aae4-4054-ada7-568f8ce6f75f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk"
Sep 16 19:12:26.729424 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:12:26.729330    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ce0d7a2-aae4-4054-ada7-568f8ce6f75f-home\") pod \"tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk\" (UID: \"6ce0d7a2-aae4-4054-ada7-568f8ce6f75f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk"
Sep 16 19:12:26.729773 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:12:26.729757    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ce0d7a2-aae4-4054-ada7-568f8ce6f75f-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk\" (UID: \"6ce0d7a2-aae4-4054-ada7-568f8ce6f75f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk"
Sep 16 19:12:26.731229 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:12:26.731207    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ce0d7a2-aae4-4054-ada7-568f8ce6f75f-dshm\") pod \"tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk\" (UID: \"6ce0d7a2-aae4-4054-ada7-568f8ce6f75f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk"
Sep 16 19:12:26.731450 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:12:26.731432    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ce0d7a2-aae4-4054-ada7-568f8ce6f75f-tls-certs\") pod \"tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk\" (UID: \"6ce0d7a2-aae4-4054-ada7-568f8ce6f75f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk"
Sep 16 19:12:26.738525 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:12:26.738499    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqjwk\" (UniqueName: \"kubernetes.io/projected/6ce0d7a2-aae4-4054-ada7-568f8ce6f75f-kube-api-access-lqjwk\") pod \"tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk\" (UID: \"6ce0d7a2-aae4-4054-ada7-568f8ce6f75f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk"
Sep 16 19:12:26.925494 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:12:26.925399    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk"
Sep 16 19:12:27.210387 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:12:27.210357    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk"]
Sep 16 19:12:27.213081 ip-10-0-129-194 kubenswrapper[2582]: W0916 19:12:27.213053    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ce0d7a2_aae4_4054_ada7_568f8ce6f75f.slice/crio-6464b3ea59f59cc373a4992edc702246324b22b4bf3b1d17467b6313c0d66432 WatchSource:0}: Error finding container 6464b3ea59f59cc373a4992edc702246324b22b4bf3b1d17467b6313c0d66432: Status 404 returned error can't find the container with id 6464b3ea59f59cc373a4992edc702246324b22b4bf3b1d17467b6313c0d66432
Sep 16 19:12:27.428629 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:12:27.428596    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk" event={"ID":"6ce0d7a2-aae4-4054-ada7-568f8ce6f75f","Type":"ContainerStarted","Data":"2102d581fe49412cefbc122ffc4a03a98bd7fee8e170c6ed11dc8ee8fe3a9d89"}
Sep 16 19:12:27.428827 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:12:27.428646    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk" event={"ID":"6ce0d7a2-aae4-4054-ada7-568f8ce6f75f","Type":"ContainerStarted","Data":"6464b3ea59f59cc373a4992edc702246324b22b4bf3b1d17467b6313c0d66432"}
Sep 16 19:12:30.442449 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:12:30.442415    2582 generic.go:356] "Generic (PLEG): container finished" podID="6ce0d7a2-aae4-4054-ada7-568f8ce6f75f" containerID="2102d581fe49412cefbc122ffc4a03a98bd7fee8e170c6ed11dc8ee8fe3a9d89" exitCode=0
Sep 16 19:12:30.442833 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:12:30.442487    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk" event={"ID":"6ce0d7a2-aae4-4054-ada7-568f8ce6f75f","Type":"ContainerDied","Data":"2102d581fe49412cefbc122ffc4a03a98bd7fee8e170c6ed11dc8ee8fe3a9d89"}
Sep 16 19:12:31.448275 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:12:31.448237    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk" event={"ID":"6ce0d7a2-aae4-4054-ada7-568f8ce6f75f","Type":"ContainerStarted","Data":"0b6ee81c79803d69a9ca11795b49d79d6e7104d7703aa2565e0b35ed07e54d88"}
Sep 16 19:12:31.926321 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:12:31.926270    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk"
Sep 16 19:12:36.926540 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:12:36.926496    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk"
Sep 16 19:12:36.939110 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:12:36.939079    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk"
Sep 16 19:12:36.965239 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:12:36.965183    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk" podStartSLOduration=8.805029303 podStartE2EDuration="10.96516624s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.160136937s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:12:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:12:31.486783315 +0000 UTC m=+1755.128498980" watchObservedRunningTime="2026-09-16 19:12:36.96516624 +0000 UTC m=+1760.606881904"
Sep 16 19:12:37.483462 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:12:37.483433    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk"
Sep 16 19:13:16.914912 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:16.914880    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7qwl2_b6695f86-08c0-4836-87b1-f2a9d76084bd/console-operator/1.log"
Sep 16 19:13:16.921893 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:16.921858    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7qwl2_b6695f86-08c0-4836-87b1-f2a9d76084bd/console-operator/1.log"
Sep 16 19:13:20.293902 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:20.293863    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cnfzg"]
Sep 16 19:13:20.297085 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:20.297065    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cnfzg"
Sep 16 19:13:20.301788 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:20.301766    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"kube-root-ca.crt\""
Sep 16 19:13:20.302358 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:20.302332    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"test-div-v2-kserve-self-signed-certs\""
Sep 16 19:13:20.302476 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:20.302455    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"openshift-service-ca.crt\""
Sep 16 19:13:20.303654 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:20.303516    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"default-dockercfg-7vtq8\""
Sep 16 19:13:20.309653 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:20.309635    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cnfzg"]
Sep 16 19:13:20.393577 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:20.393544    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1421aed-67ea-42bd-bd57-829171062a5d-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-cnfzg\" (UID: \"e1421aed-67ea-42bd-bd57-829171062a5d\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cnfzg"
Sep 16 19:13:20.393577 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:20.393581    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1421aed-67ea-42bd-bd57-829171062a5d-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-cnfzg\" (UID: \"e1421aed-67ea-42bd-bd57-829171062a5d\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cnfzg"
Sep 16 19:13:20.393805 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:20.393657    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1421aed-67ea-42bd-bd57-829171062a5d-home\") pod \"test-div-v2-kserve-5f6fdb6b77-cnfzg\" (UID: \"e1421aed-67ea-42bd-bd57-829171062a5d\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cnfzg"
Sep 16 19:13:20.393805 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:20.393678    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v6rb\" (UniqueName: \"kubernetes.io/projected/e1421aed-67ea-42bd-bd57-829171062a5d-kube-api-access-8v6rb\") pod \"test-div-v2-kserve-5f6fdb6b77-cnfzg\" (UID: \"e1421aed-67ea-42bd-bd57-829171062a5d\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cnfzg"
Sep 16 19:13:20.393805 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:20.393765    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1421aed-67ea-42bd-bd57-829171062a5d-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-cnfzg\" (UID: \"e1421aed-67ea-42bd-bd57-829171062a5d\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cnfzg"
Sep 16 19:13:20.393908 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:20.393811    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1421aed-67ea-42bd-bd57-829171062a5d-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-cnfzg\" (UID: \"e1421aed-67ea-42bd-bd57-829171062a5d\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cnfzg"
Sep 16 19:13:20.494672 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:20.494621    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1421aed-67ea-42bd-bd57-829171062a5d-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-cnfzg\" (UID: \"e1421aed-67ea-42bd-bd57-829171062a5d\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cnfzg"
Sep 16 19:13:20.494672 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:20.494668    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1421aed-67ea-42bd-bd57-829171062a5d-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-cnfzg\" (UID: \"e1421aed-67ea-42bd-bd57-829171062a5d\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cnfzg"
Sep 16 19:13:20.494946 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:20.494702    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1421aed-67ea-42bd-bd57-829171062a5d-home\") pod \"test-div-v2-kserve-5f6fdb6b77-cnfzg\" (UID: \"e1421aed-67ea-42bd-bd57-829171062a5d\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cnfzg"
Sep 16 19:13:20.494946 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:20.494747    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v6rb\" (UniqueName: \"kubernetes.io/projected/e1421aed-67ea-42bd-bd57-829171062a5d-kube-api-access-8v6rb\") pod \"test-div-v2-kserve-5f6fdb6b77-cnfzg\" (UID: \"e1421aed-67ea-42bd-bd57-829171062a5d\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cnfzg"
Sep 16 19:13:20.494946 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:20.494809    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1421aed-67ea-42bd-bd57-829171062a5d-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-cnfzg\" (UID: \"e1421aed-67ea-42bd-bd57-829171062a5d\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cnfzg"
Sep 16 19:13:20.494946 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:20.494839    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1421aed-67ea-42bd-bd57-829171062a5d-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-cnfzg\" (UID: \"e1421aed-67ea-42bd-bd57-829171062a5d\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cnfzg"
Sep 16 19:13:20.495155 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:20.495102    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1421aed-67ea-42bd-bd57-829171062a5d-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-cnfzg\" (UID: \"e1421aed-67ea-42bd-bd57-829171062a5d\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cnfzg"
Sep 16 19:13:20.495205 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:20.495154    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1421aed-67ea-42bd-bd57-829171062a5d-home\") pod \"test-div-v2-kserve-5f6fdb6b77-cnfzg\" (UID: \"e1421aed-67ea-42bd-bd57-829171062a5d\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cnfzg"
Sep 16 19:13:20.495250 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:20.495233    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1421aed-67ea-42bd-bd57-829171062a5d-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-cnfzg\" (UID: \"e1421aed-67ea-42bd-bd57-829171062a5d\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cnfzg"
Sep 16 19:13:20.497043 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:20.497019    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1421aed-67ea-42bd-bd57-829171062a5d-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-cnfzg\" (UID: \"e1421aed-67ea-42bd-bd57-829171062a5d\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cnfzg"
Sep 16 19:13:20.497298 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:20.497283    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1421aed-67ea-42bd-bd57-829171062a5d-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-cnfzg\" (UID: \"e1421aed-67ea-42bd-bd57-829171062a5d\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cnfzg"
Sep 16 19:13:20.505519 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:20.505495    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v6rb\" (UniqueName: \"kubernetes.io/projected/e1421aed-67ea-42bd-bd57-829171062a5d-kube-api-access-8v6rb\") pod \"test-div-v2-kserve-5f6fdb6b77-cnfzg\" (UID: \"e1421aed-67ea-42bd-bd57-829171062a5d\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cnfzg"
Sep 16 19:13:20.607747 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:20.607634    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cnfzg"
Sep 16 19:13:20.890128 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:20.890097    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cnfzg"]
Sep 16 19:13:20.891882 ip-10-0-129-194 kubenswrapper[2582]: W0916 19:13:20.891849    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1421aed_67ea_42bd_bd57_829171062a5d.slice/crio-a623f971aedeba7d249c2cb04dc93b83381cb91b67c285b9b4e5d93bae018f8c WatchSource:0}: Error finding container a623f971aedeba7d249c2cb04dc93b83381cb91b67c285b9b4e5d93bae018f8c: Status 404 returned error can't find the container with id a623f971aedeba7d249c2cb04dc93b83381cb91b67c285b9b4e5d93bae018f8c
Sep 16 19:13:20.893783 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:20.893765    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:13:21.672905 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:21.672852    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cnfzg" event={"ID":"e1421aed-67ea-42bd-bd57-829171062a5d","Type":"ContainerStarted","Data":"a623f971aedeba7d249c2cb04dc93b83381cb91b67c285b9b4e5d93bae018f8c"}
Sep 16 19:13:22.678823 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:22.678785    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cnfzg" event={"ID":"e1421aed-67ea-42bd-bd57-829171062a5d","Type":"ContainerStarted","Data":"b0b8ff9abf6819a0e21478994bd742c325f3821bfe0d21a850842d03f607e05f"}
Sep 16 19:13:23.608223 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:23.608174    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cnfzg"
Sep 16 19:13:30.608458 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:30.608416    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cnfzg"
Sep 16 19:13:30.621117 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:30.621091    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cnfzg"
Sep 16 19:13:30.648664 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:30.648613    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cnfzg" podStartSLOduration=9.629081174 podStartE2EDuration="10.64859659s" totalImagesPullingTime="1.019515416s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:13:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:13:22.726862036 +0000 UTC m=+1806.368577699" watchObservedRunningTime="2026-09-16 19:13:30.64859659 +0000 UTC m=+1814.290312254"
Sep 16 19:13:30.724753 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:30.724703    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cnfzg"
Sep 16 19:13:35.709658 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:35.709622    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cnfzg"]
Sep 16 19:13:50.771436 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:50.771394    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cnfzg" podUID="e1421aed-67ea-42bd-bd57-829171062a5d" containerName="main" containerID="cri-o://b0b8ff9abf6819a0e21478994bd742c325f3821bfe0d21a850842d03f607e05f" gracePeriod=45
Sep 16 19:13:51.015574 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:51.015547    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cnfzg"
Sep 16 19:13:51.163681 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:51.163578    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1421aed-67ea-42bd-bd57-829171062a5d-home\") pod \"e1421aed-67ea-42bd-bd57-829171062a5d\" (UID: \"e1421aed-67ea-42bd-bd57-829171062a5d\") "
Sep 16 19:13:51.163681 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:51.163647    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8v6rb\" (UniqueName: \"kubernetes.io/projected/e1421aed-67ea-42bd-bd57-829171062a5d-kube-api-access-8v6rb\") pod \"e1421aed-67ea-42bd-bd57-829171062a5d\" (UID: \"e1421aed-67ea-42bd-bd57-829171062a5d\") "
Sep 16 19:13:51.163681 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:51.163672    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1421aed-67ea-42bd-bd57-829171062a5d-tmp-dir\") pod \"e1421aed-67ea-42bd-bd57-829171062a5d\" (UID: \"e1421aed-67ea-42bd-bd57-829171062a5d\") "
Sep 16 19:13:51.163982 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:51.163703    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1421aed-67ea-42bd-bd57-829171062a5d-dshm\") pod \"e1421aed-67ea-42bd-bd57-829171062a5d\" (UID: \"e1421aed-67ea-42bd-bd57-829171062a5d\") "
Sep 16 19:13:51.163982 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:51.163760    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1421aed-67ea-42bd-bd57-829171062a5d-tls-certs\") pod \"e1421aed-67ea-42bd-bd57-829171062a5d\" (UID: \"e1421aed-67ea-42bd-bd57-829171062a5d\") "
Sep 16 19:13:51.163982 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:51.163827    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1421aed-67ea-42bd-bd57-829171062a5d-model-cache\") pod \"e1421aed-67ea-42bd-bd57-829171062a5d\" (UID: \"e1421aed-67ea-42bd-bd57-829171062a5d\") "
Sep 16 19:13:51.163982 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:51.163898    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1421aed-67ea-42bd-bd57-829171062a5d-home" (OuterVolumeSpecName: "home") pod "e1421aed-67ea-42bd-bd57-829171062a5d" (UID: "e1421aed-67ea-42bd-bd57-829171062a5d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:51.164174 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:51.163977    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1421aed-67ea-42bd-bd57-829171062a5d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e1421aed-67ea-42bd-bd57-829171062a5d" (UID: "e1421aed-67ea-42bd-bd57-829171062a5d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:51.164174 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:51.164105    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1421aed-67ea-42bd-bd57-829171062a5d-home\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:13:51.164174 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:51.164125    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1421aed-67ea-42bd-bd57-829171062a5d-tmp-dir\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:13:51.164174 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:51.164151    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1421aed-67ea-42bd-bd57-829171062a5d-model-cache" (OuterVolumeSpecName: "model-cache") pod "e1421aed-67ea-42bd-bd57-829171062a5d" (UID: "e1421aed-67ea-42bd-bd57-829171062a5d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:51.166021 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:51.166000    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1421aed-67ea-42bd-bd57-829171062a5d-dshm" (OuterVolumeSpecName: "dshm") pod "e1421aed-67ea-42bd-bd57-829171062a5d" (UID: "e1421aed-67ea-42bd-bd57-829171062a5d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:51.166095 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:51.166064    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1421aed-67ea-42bd-bd57-829171062a5d-kube-api-access-8v6rb" (OuterVolumeSpecName: "kube-api-access-8v6rb") pod "e1421aed-67ea-42bd-bd57-829171062a5d" (UID: "e1421aed-67ea-42bd-bd57-829171062a5d"). InnerVolumeSpecName "kube-api-access-8v6rb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:13:51.166163 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:51.166134    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1421aed-67ea-42bd-bd57-829171062a5d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e1421aed-67ea-42bd-bd57-829171062a5d" (UID: "e1421aed-67ea-42bd-bd57-829171062a5d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:13:51.265053 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:51.265013    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1421aed-67ea-42bd-bd57-829171062a5d-model-cache\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:13:51.265053 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:51.265050    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8v6rb\" (UniqueName: \"kubernetes.io/projected/e1421aed-67ea-42bd-bd57-829171062a5d-kube-api-access-8v6rb\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:13:51.265265 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:51.265064    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1421aed-67ea-42bd-bd57-829171062a5d-dshm\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:13:51.265265 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:51.265078    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1421aed-67ea-42bd-bd57-829171062a5d-tls-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:13:51.804331 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:51.804296    2582 generic.go:356] "Generic (PLEG): container finished" podID="e1421aed-67ea-42bd-bd57-829171062a5d" containerID="b0b8ff9abf6819a0e21478994bd742c325f3821bfe0d21a850842d03f607e05f" exitCode=0
Sep 16 19:13:51.804823 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:51.804341    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cnfzg"
Sep 16 19:13:51.804823 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:51.804350    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cnfzg" event={"ID":"e1421aed-67ea-42bd-bd57-829171062a5d","Type":"ContainerDied","Data":"b0b8ff9abf6819a0e21478994bd742c325f3821bfe0d21a850842d03f607e05f"}
Sep 16 19:13:51.804823 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:51.804400    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cnfzg" event={"ID":"e1421aed-67ea-42bd-bd57-829171062a5d","Type":"ContainerDied","Data":"a623f971aedeba7d249c2cb04dc93b83381cb91b67c285b9b4e5d93bae018f8c"}
Sep 16 19:13:51.804823 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:51.804426    2582 scope.go:117] "RemoveContainer" containerID="b0b8ff9abf6819a0e21478994bd742c325f3821bfe0d21a850842d03f607e05f"
Sep 16 19:13:51.815126 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:51.815093    2582 scope.go:117] "RemoveContainer" containerID="b0b8ff9abf6819a0e21478994bd742c325f3821bfe0d21a850842d03f607e05f"
Sep 16 19:13:51.815490 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:13:51.815471    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0b8ff9abf6819a0e21478994bd742c325f3821bfe0d21a850842d03f607e05f\": container with ID starting with b0b8ff9abf6819a0e21478994bd742c325f3821bfe0d21a850842d03f607e05f not found: ID does not exist" containerID="b0b8ff9abf6819a0e21478994bd742c325f3821bfe0d21a850842d03f607e05f"
Sep 16 19:13:51.815536 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:51.815501    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0b8ff9abf6819a0e21478994bd742c325f3821bfe0d21a850842d03f607e05f"} err="failed to get container status \"b0b8ff9abf6819a0e21478994bd742c325f3821bfe0d21a850842d03f607e05f\": rpc error: code = NotFound desc = could not find container \"b0b8ff9abf6819a0e21478994bd742c325f3821bfe0d21a850842d03f607e05f\": container with ID starting with b0b8ff9abf6819a0e21478994bd742c325f3821bfe0d21a850842d03f607e05f not found: ID does not exist"
Sep 16 19:13:51.827627 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:51.827602    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cnfzg"]
Sep 16 19:13:51.836437 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:51.836414    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cnfzg"]
Sep 16 19:13:52.872530 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:13:52.872487    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1421aed-67ea-42bd-bd57-829171062a5d" path="/var/lib/kubelet/pods/e1421aed-67ea-42bd-bd57-829171062a5d/volumes"
Sep 16 19:14:18.613506 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:18.613467    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-wlmhh"]
Sep 16 19:14:18.613899 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:18.613789    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1421aed-67ea-42bd-bd57-829171062a5d" containerName="main"
Sep 16 19:14:18.613899 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:18.613801    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1421aed-67ea-42bd-bd57-829171062a5d" containerName="main"
Sep 16 19:14:18.613899 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:18.613865    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1421aed-67ea-42bd-bd57-829171062a5d" containerName="main"
Sep 16 19:14:18.619562 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:18.619544    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-wlmhh"
Sep 16 19:14:18.622575 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:18.622550    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"kube-root-ca.crt\""
Sep 16 19:14:18.623789 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:18.623771    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"default-dockercfg-5xpjk\""
Sep 16 19:14:18.623864 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:18.623771    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"openshift-service-ca.crt\""
Sep 16 19:14:18.623909 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:18.623864    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"test-div-v1-kserve-self-signed-certs\""
Sep 16 19:14:18.631917 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:18.631893    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-wlmhh"]
Sep 16 19:14:18.704141 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:18.704099    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6cn6\" (UniqueName: \"kubernetes.io/projected/ab3bcb65-931a-4193-ac82-051c7fe7faa0-kube-api-access-c6cn6\") pod \"test-div-v1-kserve-fb566d667-wlmhh\" (UID: \"ab3bcb65-931a-4193-ac82-051c7fe7faa0\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-wlmhh"
Sep 16 19:14:18.704318 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:18.704158    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab3bcb65-931a-4193-ac82-051c7fe7faa0-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-wlmhh\" (UID: \"ab3bcb65-931a-4193-ac82-051c7fe7faa0\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-wlmhh"
Sep 16 19:14:18.704318 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:18.704194    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab3bcb65-931a-4193-ac82-051c7fe7faa0-home\") pod \"test-div-v1-kserve-fb566d667-wlmhh\" (UID: \"ab3bcb65-931a-4193-ac82-051c7fe7faa0\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-wlmhh"
Sep 16 19:14:18.704318 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:18.704219    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab3bcb65-931a-4193-ac82-051c7fe7faa0-tls-certs\") pod \"test-div-v1-kserve-fb566d667-wlmhh\" (UID: \"ab3bcb65-931a-4193-ac82-051c7fe7faa0\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-wlmhh"
Sep 16 19:14:18.704318 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:18.704308    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab3bcb65-931a-4193-ac82-051c7fe7faa0-model-cache\") pod \"test-div-v1-kserve-fb566d667-wlmhh\" (UID: \"ab3bcb65-931a-4193-ac82-051c7fe7faa0\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-wlmhh"
Sep 16 19:14:18.704484 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:18.704338    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab3bcb65-931a-4193-ac82-051c7fe7faa0-dshm\") pod \"test-div-v1-kserve-fb566d667-wlmhh\" (UID: \"ab3bcb65-931a-4193-ac82-051c7fe7faa0\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-wlmhh"
Sep 16 19:14:18.805578 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:18.805543    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab3bcb65-931a-4193-ac82-051c7fe7faa0-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-wlmhh\" (UID: \"ab3bcb65-931a-4193-ac82-051c7fe7faa0\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-wlmhh"
Sep 16 19:14:18.805803 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:18.805587    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab3bcb65-931a-4193-ac82-051c7fe7faa0-home\") pod \"test-div-v1-kserve-fb566d667-wlmhh\" (UID: \"ab3bcb65-931a-4193-ac82-051c7fe7faa0\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-wlmhh"
Sep 16 19:14:18.805803 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:18.805607    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab3bcb65-931a-4193-ac82-051c7fe7faa0-tls-certs\") pod \"test-div-v1-kserve-fb566d667-wlmhh\" (UID: \"ab3bcb65-931a-4193-ac82-051c7fe7faa0\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-wlmhh"
Sep 16 19:14:18.805803 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:18.805751    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab3bcb65-931a-4193-ac82-051c7fe7faa0-model-cache\") pod \"test-div-v1-kserve-fb566d667-wlmhh\" (UID: \"ab3bcb65-931a-4193-ac82-051c7fe7faa0\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-wlmhh"
Sep 16 19:14:18.805803 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:18.805788    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab3bcb65-931a-4193-ac82-051c7fe7faa0-dshm\") pod \"test-div-v1-kserve-fb566d667-wlmhh\" (UID: \"ab3bcb65-931a-4193-ac82-051c7fe7faa0\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-wlmhh"
Sep 16 19:14:18.806044 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:18.805834    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6cn6\" (UniqueName: \"kubernetes.io/projected/ab3bcb65-931a-4193-ac82-051c7fe7faa0-kube-api-access-c6cn6\") pod \"test-div-v1-kserve-fb566d667-wlmhh\" (UID: \"ab3bcb65-931a-4193-ac82-051c7fe7faa0\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-wlmhh"
Sep 16 19:14:18.806102 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:18.806052    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab3bcb65-931a-4193-ac82-051c7fe7faa0-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-wlmhh\" (UID: \"ab3bcb65-931a-4193-ac82-051c7fe7faa0\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-wlmhh"
Sep 16 19:14:18.806102 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:18.806060    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab3bcb65-931a-4193-ac82-051c7fe7faa0-home\") pod \"test-div-v1-kserve-fb566d667-wlmhh\" (UID: \"ab3bcb65-931a-4193-ac82-051c7fe7faa0\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-wlmhh"
Sep 16 19:14:18.806204 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:18.806109    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab3bcb65-931a-4193-ac82-051c7fe7faa0-model-cache\") pod \"test-div-v1-kserve-fb566d667-wlmhh\" (UID: \"ab3bcb65-931a-4193-ac82-051c7fe7faa0\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-wlmhh"
Sep 16 19:14:18.808020 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:18.808000    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab3bcb65-931a-4193-ac82-051c7fe7faa0-dshm\") pod \"test-div-v1-kserve-fb566d667-wlmhh\" (UID: \"ab3bcb65-931a-4193-ac82-051c7fe7faa0\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-wlmhh"
Sep 16 19:14:18.808298 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:18.808281    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab3bcb65-931a-4193-ac82-051c7fe7faa0-tls-certs\") pod \"test-div-v1-kserve-fb566d667-wlmhh\" (UID: \"ab3bcb65-931a-4193-ac82-051c7fe7faa0\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-wlmhh"
Sep 16 19:14:18.815401 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:18.815375    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6cn6\" (UniqueName: \"kubernetes.io/projected/ab3bcb65-931a-4193-ac82-051c7fe7faa0-kube-api-access-c6cn6\") pod \"test-div-v1-kserve-fb566d667-wlmhh\" (UID: \"ab3bcb65-931a-4193-ac82-051c7fe7faa0\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-wlmhh"
Sep 16 19:14:18.929496 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:18.929413    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-wlmhh"
Sep 16 19:14:19.207496 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:19.207453    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-wlmhh"]
Sep 16 19:14:19.209244 ip-10-0-129-194 kubenswrapper[2582]: W0916 19:14:19.209217    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab3bcb65_931a_4193_ac82_051c7fe7faa0.slice/crio-79cb1ba1eedc1c389d3d8229e17176211bd438f1858c844430daddb256fecacc WatchSource:0}: Error finding container 79cb1ba1eedc1c389d3d8229e17176211bd438f1858c844430daddb256fecacc: Status 404 returned error can't find the container with id 79cb1ba1eedc1c389d3d8229e17176211bd438f1858c844430daddb256fecacc
Sep 16 19:14:19.933171 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:19.933134    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-wlmhh" event={"ID":"ab3bcb65-931a-4193-ac82-051c7fe7faa0","Type":"ContainerStarted","Data":"e2b199734c49ec04a8279f864ef5a2cb78e1fab2cd9f430e928421278f5a27fb"}
Sep 16 19:14:19.933574 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:19.933195    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-wlmhh" event={"ID":"ab3bcb65-931a-4193-ac82-051c7fe7faa0","Type":"ContainerStarted","Data":"79cb1ba1eedc1c389d3d8229e17176211bd438f1858c844430daddb256fecacc"}
Sep 16 19:14:19.962921 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:19.962865    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-wlmhh" podStartSLOduration=1.962846668 podStartE2EDuration="1.962846668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:14:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:14:19.961582729 +0000 UTC m=+1863.603298394" watchObservedRunningTime="2026-09-16 19:14:19.962846668 +0000 UTC m=+1863.604562332"
Sep 16 19:14:20.929523 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:20.929485    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-wlmhh"
Sep 16 19:14:28.930444 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:28.930409    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-wlmhh"
Sep 16 19:14:28.943361 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:28.943333    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-wlmhh"
Sep 16 19:14:28.985214 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:28.985183    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-wlmhh"
Sep 16 19:14:47.434549 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:47.434460    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk"]
Sep 16 19:14:47.443579 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:14:47.443534    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:14:47.443579 ip-10-0-129-194 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:14:47.443579 ip-10-0-129-194 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 19:14:47.443579 ip-10-0-129-194 kubenswrapper[2582]:  > containerID="0b6ee81c79803d69a9ca11795b49d79d6e7104d7703aa2565e0b35ed07e54d88" cmd=["/bin/sleep","15"]
Sep 16 19:14:47.443932 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:47.443606    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk" message="" err=<
Sep 16 19:14:47.443932 ip-10-0-129-194 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:14:47.443932 ip-10-0-129-194 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 19:14:47.443932 ip-10-0-129-194 kubenswrapper[2582]:  >
Sep 16 19:14:47.443932 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:14:47.443626    2582 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:14:47.443932 ip-10-0-129-194 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:14:47.443932 ip-10-0-129-194 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 19:14:47.443932 ip-10-0-129-194 kubenswrapper[2582]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk" podUID="6ce0d7a2-aae4-4054-ada7-568f8ce6f75f" containerName="main" containerID="cri-o://0b6ee81c79803d69a9ca11795b49d79d6e7104d7703aa2565e0b35ed07e54d88"
Sep 16 19:14:47.443932 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:47.443687    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk" podUID="6ce0d7a2-aae4-4054-ada7-568f8ce6f75f" containerName="main" containerID="cri-o://0b6ee81c79803d69a9ca11795b49d79d6e7104d7703aa2565e0b35ed07e54d88" gracePeriod=60
Sep 16 19:14:47.472775 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:47.472732    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk" podUID="6ce0d7a2-aae4-4054-ada7-568f8ce6f75f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
Sep 16 19:14:47.794884 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:47.794855    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk"
Sep 16 19:14:47.959375 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:47.959293    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ce0d7a2-aae4-4054-ada7-568f8ce6f75f-tls-certs\") pod \"6ce0d7a2-aae4-4054-ada7-568f8ce6f75f\" (UID: \"6ce0d7a2-aae4-4054-ada7-568f8ce6f75f\") "
Sep 16 19:14:47.959375 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:47.959367    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ce0d7a2-aae4-4054-ada7-568f8ce6f75f-tmp-dir\") pod \"6ce0d7a2-aae4-4054-ada7-568f8ce6f75f\" (UID: \"6ce0d7a2-aae4-4054-ada7-568f8ce6f75f\") "
Sep 16 19:14:47.959570 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:47.959393    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ce0d7a2-aae4-4054-ada7-568f8ce6f75f-cabundle-cert\") pod \"6ce0d7a2-aae4-4054-ada7-568f8ce6f75f\" (UID: \"6ce0d7a2-aae4-4054-ada7-568f8ce6f75f\") "
Sep 16 19:14:47.959570 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:47.959432    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ce0d7a2-aae4-4054-ada7-568f8ce6f75f-model-cache\") pod \"6ce0d7a2-aae4-4054-ada7-568f8ce6f75f\" (UID: \"6ce0d7a2-aae4-4054-ada7-568f8ce6f75f\") "
Sep 16 19:14:47.959570 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:47.959454    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ce0d7a2-aae4-4054-ada7-568f8ce6f75f-dshm\") pod \"6ce0d7a2-aae4-4054-ada7-568f8ce6f75f\" (UID: \"6ce0d7a2-aae4-4054-ada7-568f8ce6f75f\") "
Sep 16 19:14:47.959570 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:47.959501    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ce0d7a2-aae4-4054-ada7-568f8ce6f75f-home\") pod \"6ce0d7a2-aae4-4054-ada7-568f8ce6f75f\" (UID: \"6ce0d7a2-aae4-4054-ada7-568f8ce6f75f\") "
Sep 16 19:14:47.959570 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:47.959527    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqjwk\" (UniqueName: \"kubernetes.io/projected/6ce0d7a2-aae4-4054-ada7-568f8ce6f75f-kube-api-access-lqjwk\") pod \"6ce0d7a2-aae4-4054-ada7-568f8ce6f75f\" (UID: \"6ce0d7a2-aae4-4054-ada7-568f8ce6f75f\") "
Sep 16 19:14:47.959570 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:47.959563    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ce0d7a2-aae4-4054-ada7-568f8ce6f75f-kserve-provision-location\") pod \"6ce0d7a2-aae4-4054-ada7-568f8ce6f75f\" (UID: \"6ce0d7a2-aae4-4054-ada7-568f8ce6f75f\") "
Sep 16 19:14:47.959912 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:47.959651    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ce0d7a2-aae4-4054-ada7-568f8ce6f75f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6ce0d7a2-aae4-4054-ada7-568f8ce6f75f" (UID: "6ce0d7a2-aae4-4054-ada7-568f8ce6f75f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:47.959912 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:47.959688    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ce0d7a2-aae4-4054-ada7-568f8ce6f75f-model-cache" (OuterVolumeSpecName: "model-cache") pod "6ce0d7a2-aae4-4054-ada7-568f8ce6f75f" (UID: "6ce0d7a2-aae4-4054-ada7-568f8ce6f75f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:47.959912 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:47.959845    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ce0d7a2-aae4-4054-ada7-568f8ce6f75f-home" (OuterVolumeSpecName: "home") pod "6ce0d7a2-aae4-4054-ada7-568f8ce6f75f" (UID: "6ce0d7a2-aae4-4054-ada7-568f8ce6f75f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:47.959912 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:47.959876    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ce0d7a2-aae4-4054-ada7-568f8ce6f75f-tmp-dir\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:14:47.959912 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:47.959897    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ce0d7a2-aae4-4054-ada7-568f8ce6f75f-model-cache\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:14:47.959912 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:47.959875    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ce0d7a2-aae4-4054-ada7-568f8ce6f75f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6ce0d7a2-aae4-4054-ada7-568f8ce6f75f" (UID: "6ce0d7a2-aae4-4054-ada7-568f8ce6f75f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:14:47.961682 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:47.961656    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ce0d7a2-aae4-4054-ada7-568f8ce6f75f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6ce0d7a2-aae4-4054-ada7-568f8ce6f75f" (UID: "6ce0d7a2-aae4-4054-ada7-568f8ce6f75f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:14:47.961827 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:47.961696    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ce0d7a2-aae4-4054-ada7-568f8ce6f75f-kube-api-access-lqjwk" (OuterVolumeSpecName: "kube-api-access-lqjwk") pod "6ce0d7a2-aae4-4054-ada7-568f8ce6f75f" (UID: "6ce0d7a2-aae4-4054-ada7-568f8ce6f75f"). InnerVolumeSpecName "kube-api-access-lqjwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:14:47.961895 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:47.961856    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ce0d7a2-aae4-4054-ada7-568f8ce6f75f-dshm" (OuterVolumeSpecName: "dshm") pod "6ce0d7a2-aae4-4054-ada7-568f8ce6f75f" (UID: "6ce0d7a2-aae4-4054-ada7-568f8ce6f75f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:47.980644 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:47.980608    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ce0d7a2-aae4-4054-ada7-568f8ce6f75f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6ce0d7a2-aae4-4054-ada7-568f8ce6f75f" (UID: "6ce0d7a2-aae4-4054-ada7-568f8ce6f75f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:48.060513 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:48.060478    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ce0d7a2-aae4-4054-ada7-568f8ce6f75f-dshm\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:14:48.060513 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:48.060507    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ce0d7a2-aae4-4054-ada7-568f8ce6f75f-home\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:14:48.060513 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:48.060517    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqjwk\" (UniqueName: \"kubernetes.io/projected/6ce0d7a2-aae4-4054-ada7-568f8ce6f75f-kube-api-access-lqjwk\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:14:48.060782 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:48.060528    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ce0d7a2-aae4-4054-ada7-568f8ce6f75f-kserve-provision-location\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:14:48.060782 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:48.060544    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ce0d7a2-aae4-4054-ada7-568f8ce6f75f-tls-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:14:48.060782 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:48.060554    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ce0d7a2-aae4-4054-ada7-568f8ce6f75f-cabundle-cert\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:14:48.066706 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:48.066673    2582 generic.go:356] "Generic (PLEG): container finished" podID="6ce0d7a2-aae4-4054-ada7-568f8ce6f75f" containerID="0b6ee81c79803d69a9ca11795b49d79d6e7104d7703aa2565e0b35ed07e54d88" exitCode=0
Sep 16 19:14:48.066881 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:48.066760    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk"
Sep 16 19:14:48.066881 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:48.066757    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk" event={"ID":"6ce0d7a2-aae4-4054-ada7-568f8ce6f75f","Type":"ContainerDied","Data":"0b6ee81c79803d69a9ca11795b49d79d6e7104d7703aa2565e0b35ed07e54d88"}
Sep 16 19:14:48.066881 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:48.066869    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk" event={"ID":"6ce0d7a2-aae4-4054-ada7-568f8ce6f75f","Type":"ContainerDied","Data":"6464b3ea59f59cc373a4992edc702246324b22b4bf3b1d17467b6313c0d66432"}
Sep 16 19:14:48.066986 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:48.066900    2582 scope.go:117] "RemoveContainer" containerID="0b6ee81c79803d69a9ca11795b49d79d6e7104d7703aa2565e0b35ed07e54d88"
Sep 16 19:14:48.076895 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:48.076878    2582 scope.go:117] "RemoveContainer" containerID="2102d581fe49412cefbc122ffc4a03a98bd7fee8e170c6ed11dc8ee8fe3a9d89"
Sep 16 19:14:48.085072 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:48.085053    2582 scope.go:117] "RemoveContainer" containerID="0b6ee81c79803d69a9ca11795b49d79d6e7104d7703aa2565e0b35ed07e54d88"
Sep 16 19:14:48.085344 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:14:48.085326    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b6ee81c79803d69a9ca11795b49d79d6e7104d7703aa2565e0b35ed07e54d88\": container with ID starting with 0b6ee81c79803d69a9ca11795b49d79d6e7104d7703aa2565e0b35ed07e54d88 not found: ID does not exist" containerID="0b6ee81c79803d69a9ca11795b49d79d6e7104d7703aa2565e0b35ed07e54d88"
Sep 16 19:14:48.085392 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:48.085357    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b6ee81c79803d69a9ca11795b49d79d6e7104d7703aa2565e0b35ed07e54d88"} err="failed to get container status \"0b6ee81c79803d69a9ca11795b49d79d6e7104d7703aa2565e0b35ed07e54d88\": rpc error: code = NotFound desc = could not find container \"0b6ee81c79803d69a9ca11795b49d79d6e7104d7703aa2565e0b35ed07e54d88\": container with ID starting with 0b6ee81c79803d69a9ca11795b49d79d6e7104d7703aa2565e0b35ed07e54d88 not found: ID does not exist"
Sep 16 19:14:48.085392 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:48.085374    2582 scope.go:117] "RemoveContainer" containerID="2102d581fe49412cefbc122ffc4a03a98bd7fee8e170c6ed11dc8ee8fe3a9d89"
Sep 16 19:14:48.085645 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:14:48.085625    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2102d581fe49412cefbc122ffc4a03a98bd7fee8e170c6ed11dc8ee8fe3a9d89\": container with ID starting with 2102d581fe49412cefbc122ffc4a03a98bd7fee8e170c6ed11dc8ee8fe3a9d89 not found: ID does not exist" containerID="2102d581fe49412cefbc122ffc4a03a98bd7fee8e170c6ed11dc8ee8fe3a9d89"
Sep 16 19:14:48.085707 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:48.085655    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2102d581fe49412cefbc122ffc4a03a98bd7fee8e170c6ed11dc8ee8fe3a9d89"} err="failed to get container status \"2102d581fe49412cefbc122ffc4a03a98bd7fee8e170c6ed11dc8ee8fe3a9d89\": rpc error: code = NotFound desc = could not find container \"2102d581fe49412cefbc122ffc4a03a98bd7fee8e170c6ed11dc8ee8fe3a9d89\": container with ID starting with 2102d581fe49412cefbc122ffc4a03a98bd7fee8e170c6ed11dc8ee8fe3a9d89 not found: ID does not exist"
Sep 16 19:14:48.098499 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:48.098469    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk"]
Sep 16 19:14:48.102740 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:48.102703    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-bmsxk"]
Sep 16 19:14:48.872332 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:14:48.872286    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ce0d7a2-aae4-4054-ada7-568f8ce6f75f" path="/var/lib/kubelet/pods/6ce0d7a2-aae4-4054-ada7-568f8ce6f75f/volumes"
Sep 16 19:15:13.103224 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:15:13.103190    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-wlmhh"]
Sep 16 19:15:28.128945 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:15:28.128899    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-wlmhh" podUID="ab3bcb65-931a-4193-ac82-051c7fe7faa0" containerName="main" containerID="cri-o://e2b199734c49ec04a8279f864ef5a2cb78e1fab2cd9f430e928421278f5a27fb" gracePeriod=45
Sep 16 19:15:28.383625 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:15:28.383565    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-wlmhh"
Sep 16 19:15:28.466101 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:15:28.466070    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab3bcb65-931a-4193-ac82-051c7fe7faa0-dshm\") pod \"ab3bcb65-931a-4193-ac82-051c7fe7faa0\" (UID: \"ab3bcb65-931a-4193-ac82-051c7fe7faa0\") "
Sep 16 19:15:28.466251 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:15:28.466113    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab3bcb65-931a-4193-ac82-051c7fe7faa0-home\") pod \"ab3bcb65-931a-4193-ac82-051c7fe7faa0\" (UID: \"ab3bcb65-931a-4193-ac82-051c7fe7faa0\") "
Sep 16 19:15:28.466251 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:15:28.466169    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6cn6\" (UniqueName: \"kubernetes.io/projected/ab3bcb65-931a-4193-ac82-051c7fe7faa0-kube-api-access-c6cn6\") pod \"ab3bcb65-931a-4193-ac82-051c7fe7faa0\" (UID: \"ab3bcb65-931a-4193-ac82-051c7fe7faa0\") "
Sep 16 19:15:28.466251 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:15:28.466193    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab3bcb65-931a-4193-ac82-051c7fe7faa0-tls-certs\") pod \"ab3bcb65-931a-4193-ac82-051c7fe7faa0\" (UID: \"ab3bcb65-931a-4193-ac82-051c7fe7faa0\") "
Sep 16 19:15:28.466251 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:15:28.466212    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab3bcb65-931a-4193-ac82-051c7fe7faa0-model-cache\") pod \"ab3bcb65-931a-4193-ac82-051c7fe7faa0\" (UID: \"ab3bcb65-931a-4193-ac82-051c7fe7faa0\") "
Sep 16 19:15:28.466251 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:15:28.466250    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab3bcb65-931a-4193-ac82-051c7fe7faa0-tmp-dir\") pod \"ab3bcb65-931a-4193-ac82-051c7fe7faa0\" (UID: \"ab3bcb65-931a-4193-ac82-051c7fe7faa0\") "
Sep 16 19:15:28.466521 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:15:28.466461    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab3bcb65-931a-4193-ac82-051c7fe7faa0-home" (OuterVolumeSpecName: "home") pod "ab3bcb65-931a-4193-ac82-051c7fe7faa0" (UID: "ab3bcb65-931a-4193-ac82-051c7fe7faa0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:28.466521 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:15:28.466479    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab3bcb65-931a-4193-ac82-051c7fe7faa0-model-cache" (OuterVolumeSpecName: "model-cache") pod "ab3bcb65-931a-4193-ac82-051c7fe7faa0" (UID: "ab3bcb65-931a-4193-ac82-051c7fe7faa0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:28.466631 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:15:28.466585    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab3bcb65-931a-4193-ac82-051c7fe7faa0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ab3bcb65-931a-4193-ac82-051c7fe7faa0" (UID: "ab3bcb65-931a-4193-ac82-051c7fe7faa0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:28.468439 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:15:28.468415    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab3bcb65-931a-4193-ac82-051c7fe7faa0-dshm" (OuterVolumeSpecName: "dshm") pod "ab3bcb65-931a-4193-ac82-051c7fe7faa0" (UID: "ab3bcb65-931a-4193-ac82-051c7fe7faa0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:28.468529 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:15:28.468438    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab3bcb65-931a-4193-ac82-051c7fe7faa0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ab3bcb65-931a-4193-ac82-051c7fe7faa0" (UID: "ab3bcb65-931a-4193-ac82-051c7fe7faa0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:15:28.468529 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:15:28.468460    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab3bcb65-931a-4193-ac82-051c7fe7faa0-kube-api-access-c6cn6" (OuterVolumeSpecName: "kube-api-access-c6cn6") pod "ab3bcb65-931a-4193-ac82-051c7fe7faa0" (UID: "ab3bcb65-931a-4193-ac82-051c7fe7faa0"). InnerVolumeSpecName "kube-api-access-c6cn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:15:28.567041 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:15:28.567003    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab3bcb65-931a-4193-ac82-051c7fe7faa0-tmp-dir\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:15:28.567041 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:15:28.567040    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab3bcb65-931a-4193-ac82-051c7fe7faa0-dshm\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:15:28.567240 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:15:28.567054    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab3bcb65-931a-4193-ac82-051c7fe7faa0-home\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:15:28.567240 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:15:28.567064    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6cn6\" (UniqueName: \"kubernetes.io/projected/ab3bcb65-931a-4193-ac82-051c7fe7faa0-kube-api-access-c6cn6\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:15:28.567240 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:15:28.567075    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab3bcb65-931a-4193-ac82-051c7fe7faa0-tls-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:15:28.567240 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:15:28.567083    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab3bcb65-931a-4193-ac82-051c7fe7faa0-model-cache\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:15:29.248338 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:15:29.248236    2582 generic.go:356] "Generic (PLEG): container finished" podID="ab3bcb65-931a-4193-ac82-051c7fe7faa0" containerID="e2b199734c49ec04a8279f864ef5a2cb78e1fab2cd9f430e928421278f5a27fb" exitCode=0
Sep 16 19:15:29.248338 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:15:29.248293    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-wlmhh"
Sep 16 19:15:29.248908 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:15:29.248298    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-wlmhh" event={"ID":"ab3bcb65-931a-4193-ac82-051c7fe7faa0","Type":"ContainerDied","Data":"e2b199734c49ec04a8279f864ef5a2cb78e1fab2cd9f430e928421278f5a27fb"}
Sep 16 19:15:29.248908 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:15:29.248441    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-wlmhh" event={"ID":"ab3bcb65-931a-4193-ac82-051c7fe7faa0","Type":"ContainerDied","Data":"79cb1ba1eedc1c389d3d8229e17176211bd438f1858c844430daddb256fecacc"}
Sep 16 19:15:29.248908 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:15:29.248470    2582 scope.go:117] "RemoveContainer" containerID="e2b199734c49ec04a8279f864ef5a2cb78e1fab2cd9f430e928421278f5a27fb"
Sep 16 19:15:29.256946 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:15:29.256926    2582 scope.go:117] "RemoveContainer" containerID="e2b199734c49ec04a8279f864ef5a2cb78e1fab2cd9f430e928421278f5a27fb"
Sep 16 19:15:29.257216 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:15:29.257197    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2b199734c49ec04a8279f864ef5a2cb78e1fab2cd9f430e928421278f5a27fb\": container with ID starting with e2b199734c49ec04a8279f864ef5a2cb78e1fab2cd9f430e928421278f5a27fb not found: ID does not exist" containerID="e2b199734c49ec04a8279f864ef5a2cb78e1fab2cd9f430e928421278f5a27fb"
Sep 16 19:15:29.257293 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:15:29.257223    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2b199734c49ec04a8279f864ef5a2cb78e1fab2cd9f430e928421278f5a27fb"} err="failed to get container status \"e2b199734c49ec04a8279f864ef5a2cb78e1fab2cd9f430e928421278f5a27fb\": rpc error: code = NotFound desc = could not find container \"e2b199734c49ec04a8279f864ef5a2cb78e1fab2cd9f430e928421278f5a27fb\": container with ID starting with e2b199734c49ec04a8279f864ef5a2cb78e1fab2cd9f430e928421278f5a27fb not found: ID does not exist"
Sep 16 19:15:29.266183 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:15:29.266160    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-wlmhh"]
Sep 16 19:15:29.269555 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:15:29.269529    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-wlmhh"]
Sep 16 19:15:30.872789 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:15:30.872751    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab3bcb65-931a-4193-ac82-051c7fe7faa0" path="/var/lib/kubelet/pods/ab3bcb65-931a-4193-ac82-051c7fe7faa0/volumes"
Sep 16 19:16:18.599388 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:16:18.599354    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-7vb66"]
Sep 16 19:16:18.599898 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:16:18.599692    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab3bcb65-931a-4193-ac82-051c7fe7faa0" containerName="main"
Sep 16 19:16:18.599898 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:16:18.599706    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab3bcb65-931a-4193-ac82-051c7fe7faa0" containerName="main"
Sep 16 19:16:18.599898 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:16:18.599733    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ce0d7a2-aae4-4054-ada7-568f8ce6f75f" containerName="storage-initializer"
Sep 16 19:16:18.599898 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:16:18.599742    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ce0d7a2-aae4-4054-ada7-568f8ce6f75f" containerName="storage-initializer"
Sep 16 19:16:18.599898 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:16:18.599761    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ce0d7a2-aae4-4054-ada7-568f8ce6f75f" containerName="main"
Sep 16 19:16:18.599898 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:16:18.599767    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ce0d7a2-aae4-4054-ada7-568f8ce6f75f" containerName="main"
Sep 16 19:16:18.599898 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:16:18.599832    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab3bcb65-931a-4193-ac82-051c7fe7faa0" containerName="main"
Sep 16 19:16:18.599898 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:16:18.599841    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ce0d7a2-aae4-4054-ada7-568f8ce6f75f" containerName="main"
Sep 16 19:16:18.604072 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:16:18.604050    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-7vb66"
Sep 16 19:16:18.606749 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:16:18.606706    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 16 19:16:18.606879 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:16:18.606760    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 16 19:16:18.607080 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:16:18.607062    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:16:18.607207 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:16:18.607117    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-85x4z\""
Sep 16 19:16:18.607882 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:16:18.607859    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 16 19:16:18.608348 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:16:18.608327    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 16 19:16:18.619323 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:16:18.619292    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-7vb66"]
Sep 16 19:16:18.677208 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:16:18.677173    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e375e012-acb2-4f8a-8aa5-587939d882d7-dshm\") pod \"tokenizer-migration-path-test-kserve-774757bb8d-7vb66\" (UID: \"e375e012-acb2-4f8a-8aa5-587939d882d7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-7vb66"
Sep 16 19:16:18.677208 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:16:18.677215    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e375e012-acb2-4f8a-8aa5-587939d882d7-home\") pod \"tokenizer-migration-path-test-kserve-774757bb8d-7vb66\" (UID: \"e375e012-acb2-4f8a-8aa5-587939d882d7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-7vb66"
Sep 16 19:16:18.677436 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:16:18.677255    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e375e012-acb2-4f8a-8aa5-587939d882d7-tls-certs\") pod \"tokenizer-migration-path-test-kserve-774757bb8d-7vb66\" (UID: \"e375e012-acb2-4f8a-8aa5-587939d882d7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-7vb66"
Sep 16 19:16:18.677436 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:16:18.677282    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e375e012-acb2-4f8a-8aa5-587939d882d7-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-774757bb8d-7vb66\" (UID: \"e375e012-acb2-4f8a-8aa5-587939d882d7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-7vb66"
Sep 16 19:16:18.677436 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:16:18.677309    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e375e012-acb2-4f8a-8aa5-587939d882d7-model-cache\") pod \"tokenizer-migration-path-test-kserve-774757bb8d-7vb66\" (UID: \"e375e012-acb2-4f8a-8aa5-587939d882d7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-7vb66"
Sep 16 19:16:18.677436 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:16:18.677361    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e375e012-acb2-4f8a-8aa5-587939d882d7-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-774757bb8d-7vb66\" (UID: \"e375e012-acb2-4f8a-8aa5-587939d882d7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-7vb66"
Sep 16 19:16:18.677436 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:16:18.677389    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g565x\" (UniqueName: \"kubernetes.io/projected/e375e012-acb2-4f8a-8aa5-587939d882d7-kube-api-access-g565x\") pod \"tokenizer-migration-path-test-kserve-774757bb8d-7vb66\" (UID: \"e375e012-acb2-4f8a-8aa5-587939d882d7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-7vb66"
Sep 16 19:16:18.677436 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:16:18.677422    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e375e012-acb2-4f8a-8aa5-587939d882d7-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-774757bb8d-7vb66\" (UID: \"e375e012-acb2-4f8a-8aa5-587939d882d7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-7vb66"
Sep 16 19:16:18.778833 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:16:18.778797    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g565x\" (UniqueName: \"kubernetes.io/projected/e375e012-acb2-4f8a-8aa5-587939d882d7-kube-api-access-g565x\") pod \"tokenizer-migration-path-test-kserve-774757bb8d-7vb66\" (UID: \"e375e012-acb2-4f8a-8aa5-587939d882d7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-7vb66"
Sep 16 19:16:18.779021 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:16:18.778840    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e375e012-acb2-4f8a-8aa5-587939d882d7-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-774757bb8d-7vb66\" (UID: \"e375e012-acb2-4f8a-8aa5-587939d882d7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-7vb66"
Sep 16 19:16:18.779021 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:16:18.778917    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e375e012-acb2-4f8a-8aa5-587939d882d7-dshm\") pod \"tokenizer-migration-path-test-kserve-774757bb8d-7vb66\" (UID: \"e375e012-acb2-4f8a-8aa5-587939d882d7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-7vb66"
Sep 16 19:16:18.779021 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:16:18.778978    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e375e012-acb2-4f8a-8aa5-587939d882d7-home\") pod \"tokenizer-migration-path-test-kserve-774757bb8d-7vb66\" (UID: \"e375e012-acb2-4f8a-8aa5-587939d882d7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-7vb66"
Sep 16 19:16:18.779021 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:16:18.779019    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e375e012-acb2-4f8a-8aa5-587939d882d7-tls-certs\") pod \"tokenizer-migration-path-test-kserve-774757bb8d-7vb66\" (UID: \"e375e012-acb2-4f8a-8aa5-587939d882d7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-7vb66"
Sep 16 19:16:18.779232 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:16:18.779059    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e375e012-acb2-4f8a-8aa5-587939d882d7-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-774757bb8d-7vb66\" (UID: \"e375e012-acb2-4f8a-8aa5-587939d882d7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-7vb66"
Sep 16 19:16:18.779232 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:16:18.779099    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e375e012-acb2-4f8a-8aa5-587939d882d7-model-cache\") pod \"tokenizer-migration-path-test-kserve-774757bb8d-7vb66\" (UID: \"e375e012-acb2-4f8a-8aa5-587939d882d7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-7vb66"
Sep 16 19:16:18.779232 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:16:18.779143    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e375e012-acb2-4f8a-8aa5-587939d882d7-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-774757bb8d-7vb66\" (UID: \"e375e012-acb2-4f8a-8aa5-587939d882d7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-7vb66"
Sep 16 19:16:18.779387 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:16:18.779241    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e375e012-acb2-4f8a-8aa5-587939d882d7-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-774757bb8d-7vb66\" (UID: \"e375e012-acb2-4f8a-8aa5-587939d882d7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-7vb66"
Sep 16 19:16:18.779387 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:16:18.779366    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e375e012-acb2-4f8a-8aa5-587939d882d7-home\") pod \"tokenizer-migration-path-test-kserve-774757bb8d-7vb66\" (UID: \"e375e012-acb2-4f8a-8aa5-587939d882d7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-7vb66"
Sep 16 19:16:18.779503 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:16:18.779457    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e375e012-acb2-4f8a-8aa5-587939d882d7-model-cache\") pod \"tokenizer-migration-path-test-kserve-774757bb8d-7vb66\" (UID: \"e375e012-acb2-4f8a-8aa5-587939d882d7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-7vb66"
Sep 16 19:16:18.779503 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:16:18.779494    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e375e012-acb2-4f8a-8aa5-587939d882d7-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-774757bb8d-7vb66\" (UID: \"e375e012-acb2-4f8a-8aa5-587939d882d7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-7vb66"
Sep 16 19:16:18.779796 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:16:18.779771    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e375e012-acb2-4f8a-8aa5-587939d882d7-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-774757bb8d-7vb66\" (UID: \"e375e012-acb2-4f8a-8aa5-587939d882d7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-7vb66"
Sep 16 19:16:18.781236 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:16:18.781219    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e375e012-acb2-4f8a-8aa5-587939d882d7-dshm\") pod \"tokenizer-migration-path-test-kserve-774757bb8d-7vb66\" (UID: \"e375e012-acb2-4f8a-8aa5-587939d882d7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-7vb66"
Sep 16 19:16:18.781532 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:16:18.781514    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e375e012-acb2-4f8a-8aa5-587939d882d7-tls-certs\") pod \"tokenizer-migration-path-test-kserve-774757bb8d-7vb66\" (UID: \"e375e012-acb2-4f8a-8aa5-587939d882d7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-7vb66"
Sep 16 19:16:18.788330 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:16:18.788306    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g565x\" (UniqueName: \"kubernetes.io/projected/e375e012-acb2-4f8a-8aa5-587939d882d7-kube-api-access-g565x\") pod \"tokenizer-migration-path-test-kserve-774757bb8d-7vb66\" (UID: \"e375e012-acb2-4f8a-8aa5-587939d882d7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-7vb66"
Sep 16 19:16:18.915827 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:16:18.915738    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-7vb66"
Sep 16 19:16:19.195681 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:16:19.195371    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-7vb66"]
Sep 16 19:16:19.198580 ip-10-0-129-194 kubenswrapper[2582]: W0916 19:16:19.198550    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode375e012_acb2_4f8a_8aa5_587939d882d7.slice/crio-a1f48331bbd24efdeae425916ec3c706ff7487f325266051ea9244e3016963b8 WatchSource:0}: Error finding container a1f48331bbd24efdeae425916ec3c706ff7487f325266051ea9244e3016963b8: Status 404 returned error can't find the container with id a1f48331bbd24efdeae425916ec3c706ff7487f325266051ea9244e3016963b8
Sep 16 19:16:19.459109 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:16:19.459003    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-7vb66" event={"ID":"e375e012-acb2-4f8a-8aa5-587939d882d7","Type":"ContainerStarted","Data":"d104fd07bc998384e678bbecd19566582ee8c8aa9922d2a84b87e842b2dfb62a"}
Sep 16 19:16:19.459109 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:16:19.459068    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-7vb66" event={"ID":"e375e012-acb2-4f8a-8aa5-587939d882d7","Type":"ContainerStarted","Data":"a1f48331bbd24efdeae425916ec3c706ff7487f325266051ea9244e3016963b8"}
Sep 16 19:16:21.477254 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:16:21.477223    2582 generic.go:356] "Generic (PLEG): container finished" podID="e375e012-acb2-4f8a-8aa5-587939d882d7" containerID="d104fd07bc998384e678bbecd19566582ee8c8aa9922d2a84b87e842b2dfb62a" exitCode=0
Sep 16 19:16:21.477584 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:16:21.477290    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-7vb66" event={"ID":"e375e012-acb2-4f8a-8aa5-587939d882d7","Type":"ContainerDied","Data":"d104fd07bc998384e678bbecd19566582ee8c8aa9922d2a84b87e842b2dfb62a"}
Sep 16 19:16:22.483643 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:16:22.483610    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-7vb66" event={"ID":"e375e012-acb2-4f8a-8aa5-587939d882d7","Type":"ContainerStarted","Data":"0274f36b80735206ad6eedde4ad5238a3796140264ed33451f2e00af96b4466b"}
Sep 16 19:16:22.508468 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:16:22.508417    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-7vb66" podStartSLOduration=2.318623249 podStartE2EDuration="4.508399981s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.189776732s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:16:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:16:22.507905502 +0000 UTC m=+1986.149621168" watchObservedRunningTime="2026-09-16 19:16:22.508399981 +0000 UTC m=+1986.150115644"
Sep 16 19:16:22.916698 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:16:22.916650    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-7vb66"
Sep 16 19:16:28.916527 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:16:28.916489    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-7vb66"
Sep 16 19:16:28.929403 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:16:28.929369    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-7vb66"
Sep 16 19:16:29.526058 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:16:29.526024    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-7vb66"
Sep 16 19:17:18.768693 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:18.768658    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xb8gn"]
Sep 16 19:17:18.772107 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:18.772090    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xb8gn"
Sep 16 19:17:18.776844 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:18.776814    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"default-dockercfg-fldhq\""
Sep 16 19:17:18.776844 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:18.776814    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"openshift-service-ca.crt\""
Sep 16 19:17:18.777035 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:18.776869    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"decom-v1-kserve-self-signed-certs\""
Sep 16 19:17:18.777035 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:18.776935    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"kube-root-ca.crt\""
Sep 16 19:17:18.779955 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:18.779934    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe95f14a-a9c4-4974-b7bc-e7e390a022fd-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-xb8gn\" (UID: \"fe95f14a-a9c4-4974-b7bc-e7e390a022fd\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xb8gn"
Sep 16 19:17:18.780059 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:18.779962    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe95f14a-a9c4-4974-b7bc-e7e390a022fd-home\") pod \"decom-v1-kserve-6cb797fb75-xb8gn\" (UID: \"fe95f14a-a9c4-4974-b7bc-e7e390a022fd\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xb8gn"
Sep 16 19:17:18.780059 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:18.779980    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe95f14a-a9c4-4974-b7bc-e7e390a022fd-dshm\") pod \"decom-v1-kserve-6cb797fb75-xb8gn\" (UID: \"fe95f14a-a9c4-4974-b7bc-e7e390a022fd\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xb8gn"
Sep 16 19:17:18.780059 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:18.780010    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx9lf\" (UniqueName: \"kubernetes.io/projected/fe95f14a-a9c4-4974-b7bc-e7e390a022fd-kube-api-access-zx9lf\") pod \"decom-v1-kserve-6cb797fb75-xb8gn\" (UID: \"fe95f14a-a9c4-4974-b7bc-e7e390a022fd\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xb8gn"
Sep 16 19:17:18.780193 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:18.780110    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe95f14a-a9c4-4974-b7bc-e7e390a022fd-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-xb8gn\" (UID: \"fe95f14a-a9c4-4974-b7bc-e7e390a022fd\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xb8gn"
Sep 16 19:17:18.780193 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:18.780180    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe95f14a-a9c4-4974-b7bc-e7e390a022fd-model-cache\") pod \"decom-v1-kserve-6cb797fb75-xb8gn\" (UID: \"fe95f14a-a9c4-4974-b7bc-e7e390a022fd\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xb8gn"
Sep 16 19:17:18.817826 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:18.817793    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xb8gn"]
Sep 16 19:17:18.880877 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:18.880848    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe95f14a-a9c4-4974-b7bc-e7e390a022fd-home\") pod \"decom-v1-kserve-6cb797fb75-xb8gn\" (UID: \"fe95f14a-a9c4-4974-b7bc-e7e390a022fd\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xb8gn"
Sep 16 19:17:18.880877 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:18.880880    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe95f14a-a9c4-4974-b7bc-e7e390a022fd-dshm\") pod \"decom-v1-kserve-6cb797fb75-xb8gn\" (UID: \"fe95f14a-a9c4-4974-b7bc-e7e390a022fd\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xb8gn"
Sep 16 19:17:18.881136 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:18.880912    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx9lf\" (UniqueName: \"kubernetes.io/projected/fe95f14a-a9c4-4974-b7bc-e7e390a022fd-kube-api-access-zx9lf\") pod \"decom-v1-kserve-6cb797fb75-xb8gn\" (UID: \"fe95f14a-a9c4-4974-b7bc-e7e390a022fd\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xb8gn"
Sep 16 19:17:18.881136 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:18.880949    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe95f14a-a9c4-4974-b7bc-e7e390a022fd-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-xb8gn\" (UID: \"fe95f14a-a9c4-4974-b7bc-e7e390a022fd\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xb8gn"
Sep 16 19:17:18.881136 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:18.880992    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe95f14a-a9c4-4974-b7bc-e7e390a022fd-model-cache\") pod \"decom-v1-kserve-6cb797fb75-xb8gn\" (UID: \"fe95f14a-a9c4-4974-b7bc-e7e390a022fd\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xb8gn"
Sep 16 19:17:18.881136 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:18.881051    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe95f14a-a9c4-4974-b7bc-e7e390a022fd-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-xb8gn\" (UID: \"fe95f14a-a9c4-4974-b7bc-e7e390a022fd\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xb8gn"
Sep 16 19:17:18.881376 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:18.881322    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe95f14a-a9c4-4974-b7bc-e7e390a022fd-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-xb8gn\" (UID: \"fe95f14a-a9c4-4974-b7bc-e7e390a022fd\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xb8gn"
Sep 16 19:17:18.881436 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:18.881372    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe95f14a-a9c4-4974-b7bc-e7e390a022fd-home\") pod \"decom-v1-kserve-6cb797fb75-xb8gn\" (UID: \"fe95f14a-a9c4-4974-b7bc-e7e390a022fd\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xb8gn"
Sep 16 19:17:18.881436 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:18.881412    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe95f14a-a9c4-4974-b7bc-e7e390a022fd-model-cache\") pod \"decom-v1-kserve-6cb797fb75-xb8gn\" (UID: \"fe95f14a-a9c4-4974-b7bc-e7e390a022fd\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xb8gn"
Sep 16 19:17:18.883273 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:18.883253    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe95f14a-a9c4-4974-b7bc-e7e390a022fd-dshm\") pod \"decom-v1-kserve-6cb797fb75-xb8gn\" (UID: \"fe95f14a-a9c4-4974-b7bc-e7e390a022fd\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xb8gn"
Sep 16 19:17:18.883500 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:18.883482    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe95f14a-a9c4-4974-b7bc-e7e390a022fd-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-xb8gn\" (UID: \"fe95f14a-a9c4-4974-b7bc-e7e390a022fd\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xb8gn"
Sep 16 19:17:18.900329 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:18.900300    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx9lf\" (UniqueName: \"kubernetes.io/projected/fe95f14a-a9c4-4974-b7bc-e7e390a022fd-kube-api-access-zx9lf\") pod \"decom-v1-kserve-6cb797fb75-xb8gn\" (UID: \"fe95f14a-a9c4-4974-b7bc-e7e390a022fd\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xb8gn"
Sep 16 19:17:19.082862 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:19.082760    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xb8gn"
Sep 16 19:17:19.354600 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:19.354511    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xb8gn"]
Sep 16 19:17:19.357371 ip-10-0-129-194 kubenswrapper[2582]: W0916 19:17:19.357334    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe95f14a_a9c4_4974_b7bc_e7e390a022fd.slice/crio-278b3b37ac6d1a212ad720094aa2dc5ccc653a3dd581a7dba668be17a76854e2 WatchSource:0}: Error finding container 278b3b37ac6d1a212ad720094aa2dc5ccc653a3dd581a7dba668be17a76854e2: Status 404 returned error can't find the container with id 278b3b37ac6d1a212ad720094aa2dc5ccc653a3dd581a7dba668be17a76854e2
Sep 16 19:17:19.729862 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:19.729765    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xb8gn" event={"ID":"fe95f14a-a9c4-4974-b7bc-e7e390a022fd","Type":"ContainerStarted","Data":"c7a83217adb5c0da767012b885500ead2d7f7f8207349cae2b373c820043b6ee"}
Sep 16 19:17:19.729862 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:19.729828    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xb8gn" event={"ID":"fe95f14a-a9c4-4974-b7bc-e7e390a022fd","Type":"ContainerStarted","Data":"278b3b37ac6d1a212ad720094aa2dc5ccc653a3dd581a7dba668be17a76854e2"}
Sep 16 19:17:19.747784 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:19.747713    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xb8gn" podStartSLOduration=1.747697616 podStartE2EDuration="1.747697616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:17:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:17:19.74684222 +0000 UTC m=+2043.388557885" watchObservedRunningTime="2026-09-16 19:17:19.747697616 +0000 UTC m=+2043.389413280"
Sep 16 19:17:20.083478 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:20.083443    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xb8gn"
Sep 16 19:17:29.083891 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:29.083847    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xb8gn"
Sep 16 19:17:29.096309 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:29.096276    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xb8gn"
Sep 16 19:17:29.789849 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:29.789822    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xb8gn"
Sep 16 19:17:47.178521 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:47.178432    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-7vb66"]
Sep 16 19:17:47.187555 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:17:47.187506    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:17:47.187555 ip-10-0-129-194 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:17:47.187555 ip-10-0-129-194 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 19:17:47.187555 ip-10-0-129-194 kubenswrapper[2582]:  > containerID="0274f36b80735206ad6eedde4ad5238a3796140264ed33451f2e00af96b4466b" cmd=["/bin/sleep","15"]
Sep 16 19:17:47.187838 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:47.187581    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-7vb66" message="" err=<
Sep 16 19:17:47.187838 ip-10-0-129-194 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:17:47.187838 ip-10-0-129-194 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 19:17:47.187838 ip-10-0-129-194 kubenswrapper[2582]:  >
Sep 16 19:17:47.187838 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:17:47.187601    2582 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:17:47.187838 ip-10-0-129-194 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:17:47.187838 ip-10-0-129-194 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 19:17:47.187838 ip-10-0-129-194 kubenswrapper[2582]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-7vb66" podUID="e375e012-acb2-4f8a-8aa5-587939d882d7" containerName="main" containerID="cri-o://0274f36b80735206ad6eedde4ad5238a3796140264ed33451f2e00af96b4466b"
Sep 16 19:17:47.187838 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:47.187664    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-7vb66" podUID="e375e012-acb2-4f8a-8aa5-587939d882d7" containerName="main" containerID="cri-o://0274f36b80735206ad6eedde4ad5238a3796140264ed33451f2e00af96b4466b" gracePeriod=60
Sep 16 19:17:47.515086 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:47.515051    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-7vb66" podUID="e375e012-acb2-4f8a-8aa5-587939d882d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
Sep 16 19:17:47.866448 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:47.866412    2582 generic.go:356] "Generic (PLEG): container finished" podID="e375e012-acb2-4f8a-8aa5-587939d882d7" containerID="0274f36b80735206ad6eedde4ad5238a3796140264ed33451f2e00af96b4466b" exitCode=0
Sep 16 19:17:47.866589 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:47.866452    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-7vb66" event={"ID":"e375e012-acb2-4f8a-8aa5-587939d882d7","Type":"ContainerDied","Data":"0274f36b80735206ad6eedde4ad5238a3796140264ed33451f2e00af96b4466b"}
Sep 16 19:17:47.936774 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:47.936748    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-7vb66"
Sep 16 19:17:48.026648 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:48.026612    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e375e012-acb2-4f8a-8aa5-587939d882d7-tmp-dir\") pod \"e375e012-acb2-4f8a-8aa5-587939d882d7\" (UID: \"e375e012-acb2-4f8a-8aa5-587939d882d7\") "
Sep 16 19:17:48.026648 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:48.026647    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e375e012-acb2-4f8a-8aa5-587939d882d7-kserve-provision-location\") pod \"e375e012-acb2-4f8a-8aa5-587939d882d7\" (UID: \"e375e012-acb2-4f8a-8aa5-587939d882d7\") "
Sep 16 19:17:48.026870 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:48.026696    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e375e012-acb2-4f8a-8aa5-587939d882d7-dshm\") pod \"e375e012-acb2-4f8a-8aa5-587939d882d7\" (UID: \"e375e012-acb2-4f8a-8aa5-587939d882d7\") "
Sep 16 19:17:48.026870 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:48.026736    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e375e012-acb2-4f8a-8aa5-587939d882d7-tls-certs\") pod \"e375e012-acb2-4f8a-8aa5-587939d882d7\" (UID: \"e375e012-acb2-4f8a-8aa5-587939d882d7\") "
Sep 16 19:17:48.026870 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:48.026771    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e375e012-acb2-4f8a-8aa5-587939d882d7-model-cache\") pod \"e375e012-acb2-4f8a-8aa5-587939d882d7\" (UID: \"e375e012-acb2-4f8a-8aa5-587939d882d7\") "
Sep 16 19:17:48.026991 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:48.026956    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e375e012-acb2-4f8a-8aa5-587939d882d7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e375e012-acb2-4f8a-8aa5-587939d882d7" (UID: "e375e012-acb2-4f8a-8aa5-587939d882d7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:48.027041 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:48.026980    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e375e012-acb2-4f8a-8aa5-587939d882d7-cabundle-cert\") pod \"e375e012-acb2-4f8a-8aa5-587939d882d7\" (UID: \"e375e012-acb2-4f8a-8aa5-587939d882d7\") "
Sep 16 19:17:48.027091 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:48.027030    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e375e012-acb2-4f8a-8aa5-587939d882d7-model-cache" (OuterVolumeSpecName: "model-cache") pod "e375e012-acb2-4f8a-8aa5-587939d882d7" (UID: "e375e012-acb2-4f8a-8aa5-587939d882d7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:48.027091 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:48.027076    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g565x\" (UniqueName: \"kubernetes.io/projected/e375e012-acb2-4f8a-8aa5-587939d882d7-kube-api-access-g565x\") pod \"e375e012-acb2-4f8a-8aa5-587939d882d7\" (UID: \"e375e012-acb2-4f8a-8aa5-587939d882d7\") "
Sep 16 19:17:48.027172 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:48.027123    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e375e012-acb2-4f8a-8aa5-587939d882d7-home\") pod \"e375e012-acb2-4f8a-8aa5-587939d882d7\" (UID: \"e375e012-acb2-4f8a-8aa5-587939d882d7\") "
Sep 16 19:17:48.027324 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:48.027288    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e375e012-acb2-4f8a-8aa5-587939d882d7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e375e012-acb2-4f8a-8aa5-587939d882d7" (UID: "e375e012-acb2-4f8a-8aa5-587939d882d7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:17:48.027552 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:48.027431    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e375e012-acb2-4f8a-8aa5-587939d882d7-home" (OuterVolumeSpecName: "home") pod "e375e012-acb2-4f8a-8aa5-587939d882d7" (UID: "e375e012-acb2-4f8a-8aa5-587939d882d7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:48.027552 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:48.027457    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e375e012-acb2-4f8a-8aa5-587939d882d7-cabundle-cert\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:17:48.027552 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:48.027474    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e375e012-acb2-4f8a-8aa5-587939d882d7-tmp-dir\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:17:48.027552 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:48.027485    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e375e012-acb2-4f8a-8aa5-587939d882d7-model-cache\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:17:48.029003 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:48.028982    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e375e012-acb2-4f8a-8aa5-587939d882d7-dshm" (OuterVolumeSpecName: "dshm") pod "e375e012-acb2-4f8a-8aa5-587939d882d7" (UID: "e375e012-acb2-4f8a-8aa5-587939d882d7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:48.029485 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:48.029464    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e375e012-acb2-4f8a-8aa5-587939d882d7-kube-api-access-g565x" (OuterVolumeSpecName: "kube-api-access-g565x") pod "e375e012-acb2-4f8a-8aa5-587939d882d7" (UID: "e375e012-acb2-4f8a-8aa5-587939d882d7"). InnerVolumeSpecName "kube-api-access-g565x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:17:48.029559 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:48.029465    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e375e012-acb2-4f8a-8aa5-587939d882d7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e375e012-acb2-4f8a-8aa5-587939d882d7" (UID: "e375e012-acb2-4f8a-8aa5-587939d882d7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:17:48.046857 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:48.046830    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e375e012-acb2-4f8a-8aa5-587939d882d7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e375e012-acb2-4f8a-8aa5-587939d882d7" (UID: "e375e012-acb2-4f8a-8aa5-587939d882d7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:48.128176 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:48.128090    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e375e012-acb2-4f8a-8aa5-587939d882d7-kserve-provision-location\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:17:48.128176 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:48.128121    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e375e012-acb2-4f8a-8aa5-587939d882d7-dshm\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:17:48.128176 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:48.128130    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e375e012-acb2-4f8a-8aa5-587939d882d7-tls-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:17:48.128176 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:48.128141    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g565x\" (UniqueName: \"kubernetes.io/projected/e375e012-acb2-4f8a-8aa5-587939d882d7-kube-api-access-g565x\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:17:48.128176 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:48.128149    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e375e012-acb2-4f8a-8aa5-587939d882d7-home\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:17:48.872940 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:48.872913    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-7vb66"
Sep 16 19:17:48.873434 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:48.872991    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-7vb66" event={"ID":"e375e012-acb2-4f8a-8aa5-587939d882d7","Type":"ContainerDied","Data":"a1f48331bbd24efdeae425916ec3c706ff7487f325266051ea9244e3016963b8"}
Sep 16 19:17:48.873434 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:48.873043    2582 scope.go:117] "RemoveContainer" containerID="0274f36b80735206ad6eedde4ad5238a3796140264ed33451f2e00af96b4466b"
Sep 16 19:17:48.882185 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:48.882166    2582 scope.go:117] "RemoveContainer" containerID="d104fd07bc998384e678bbecd19566582ee8c8aa9922d2a84b87e842b2dfb62a"
Sep 16 19:17:48.901544 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:48.901515    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-7vb66"]
Sep 16 19:17:48.909436 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:48.909415    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-7vb66"]
Sep 16 19:17:50.873347 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:17:50.873314    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e375e012-acb2-4f8a-8aa5-587939d882d7" path="/var/lib/kubelet/pods/e375e012-acb2-4f8a-8aa5-587939d882d7/volumes"
Sep 16 19:18:16.951500 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:18:16.951467    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7qwl2_b6695f86-08c0-4836-87b1-f2a9d76084bd/console-operator/1.log"
Sep 16 19:18:16.957576 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:18:16.957551    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7qwl2_b6695f86-08c0-4836-87b1-f2a9d76084bd/console-operator/1.log"
Sep 16 19:18:30.147908 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:18:30.147871    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xb8gn"]
Sep 16 19:18:45.166695 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:18:45.166647    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xb8gn" podUID="fe95f14a-a9c4-4974-b7bc-e7e390a022fd" containerName="main" containerID="cri-o://c7a83217adb5c0da767012b885500ead2d7f7f8207349cae2b373c820043b6ee" gracePeriod=45
Sep 16 19:18:45.417767 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:18:45.417690    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xb8gn"
Sep 16 19:18:45.503868 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:18:45.503837    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe95f14a-a9c4-4974-b7bc-e7e390a022fd-model-cache\") pod \"fe95f14a-a9c4-4974-b7bc-e7e390a022fd\" (UID: \"fe95f14a-a9c4-4974-b7bc-e7e390a022fd\") "
Sep 16 19:18:45.504048 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:18:45.503876    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe95f14a-a9c4-4974-b7bc-e7e390a022fd-home\") pod \"fe95f14a-a9c4-4974-b7bc-e7e390a022fd\" (UID: \"fe95f14a-a9c4-4974-b7bc-e7e390a022fd\") "
Sep 16 19:18:45.504048 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:18:45.503900    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe95f14a-a9c4-4974-b7bc-e7e390a022fd-tmp-dir\") pod \"fe95f14a-a9c4-4974-b7bc-e7e390a022fd\" (UID: \"fe95f14a-a9c4-4974-b7bc-e7e390a022fd\") "
Sep 16 19:18:45.504048 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:18:45.503932    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe95f14a-a9c4-4974-b7bc-e7e390a022fd-tls-certs\") pod \"fe95f14a-a9c4-4974-b7bc-e7e390a022fd\" (UID: \"fe95f14a-a9c4-4974-b7bc-e7e390a022fd\") "
Sep 16 19:18:45.504048 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:18:45.503962    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zx9lf\" (UniqueName: \"kubernetes.io/projected/fe95f14a-a9c4-4974-b7bc-e7e390a022fd-kube-api-access-zx9lf\") pod \"fe95f14a-a9c4-4974-b7bc-e7e390a022fd\" (UID: \"fe95f14a-a9c4-4974-b7bc-e7e390a022fd\") "
Sep 16 19:18:45.504048 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:18:45.504033    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe95f14a-a9c4-4974-b7bc-e7e390a022fd-dshm\") pod \"fe95f14a-a9c4-4974-b7bc-e7e390a022fd\" (UID: \"fe95f14a-a9c4-4974-b7bc-e7e390a022fd\") "
Sep 16 19:18:45.504307 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:18:45.504146    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe95f14a-a9c4-4974-b7bc-e7e390a022fd-home" (OuterVolumeSpecName: "home") pod "fe95f14a-a9c4-4974-b7bc-e7e390a022fd" (UID: "fe95f14a-a9c4-4974-b7bc-e7e390a022fd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:18:45.504307 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:18:45.504181    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe95f14a-a9c4-4974-b7bc-e7e390a022fd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fe95f14a-a9c4-4974-b7bc-e7e390a022fd" (UID: "fe95f14a-a9c4-4974-b7bc-e7e390a022fd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:18:45.504307 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:18:45.504151    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe95f14a-a9c4-4974-b7bc-e7e390a022fd-model-cache" (OuterVolumeSpecName: "model-cache") pod "fe95f14a-a9c4-4974-b7bc-e7e390a022fd" (UID: "fe95f14a-a9c4-4974-b7bc-e7e390a022fd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:18:45.504307 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:18:45.504288    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe95f14a-a9c4-4974-b7bc-e7e390a022fd-model-cache\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:18:45.504307 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:18:45.504306    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe95f14a-a9c4-4974-b7bc-e7e390a022fd-home\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:18:45.504517 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:18:45.504319    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe95f14a-a9c4-4974-b7bc-e7e390a022fd-tmp-dir\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:18:45.506216 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:18:45.506189    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe95f14a-a9c4-4974-b7bc-e7e390a022fd-dshm" (OuterVolumeSpecName: "dshm") pod "fe95f14a-a9c4-4974-b7bc-e7e390a022fd" (UID: "fe95f14a-a9c4-4974-b7bc-e7e390a022fd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:18:45.506353 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:18:45.506226    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe95f14a-a9c4-4974-b7bc-e7e390a022fd-kube-api-access-zx9lf" (OuterVolumeSpecName: "kube-api-access-zx9lf") pod "fe95f14a-a9c4-4974-b7bc-e7e390a022fd" (UID: "fe95f14a-a9c4-4974-b7bc-e7e390a022fd"). InnerVolumeSpecName "kube-api-access-zx9lf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:18:45.506353 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:18:45.506287    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe95f14a-a9c4-4974-b7bc-e7e390a022fd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fe95f14a-a9c4-4974-b7bc-e7e390a022fd" (UID: "fe95f14a-a9c4-4974-b7bc-e7e390a022fd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:18:45.605633 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:18:45.605582    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe95f14a-a9c4-4974-b7bc-e7e390a022fd-dshm\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:18:45.605633 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:18:45.605624    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe95f14a-a9c4-4974-b7bc-e7e390a022fd-tls-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:18:45.605633 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:18:45.605636    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx9lf\" (UniqueName: \"kubernetes.io/projected/fe95f14a-a9c4-4974-b7bc-e7e390a022fd-kube-api-access-zx9lf\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:18:46.127609 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:18:46.127573    2582 generic.go:356] "Generic (PLEG): container finished" podID="fe95f14a-a9c4-4974-b7bc-e7e390a022fd" containerID="c7a83217adb5c0da767012b885500ead2d7f7f8207349cae2b373c820043b6ee" exitCode=0
Sep 16 19:18:46.127840 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:18:46.127640    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xb8gn"
Sep 16 19:18:46.127840 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:18:46.127661    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xb8gn" event={"ID":"fe95f14a-a9c4-4974-b7bc-e7e390a022fd","Type":"ContainerDied","Data":"c7a83217adb5c0da767012b885500ead2d7f7f8207349cae2b373c820043b6ee"}
Sep 16 19:18:46.127840 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:18:46.127715    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xb8gn" event={"ID":"fe95f14a-a9c4-4974-b7bc-e7e390a022fd","Type":"ContainerDied","Data":"278b3b37ac6d1a212ad720094aa2dc5ccc653a3dd581a7dba668be17a76854e2"}
Sep 16 19:18:46.127840 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:18:46.127750    2582 scope.go:117] "RemoveContainer" containerID="c7a83217adb5c0da767012b885500ead2d7f7f8207349cae2b373c820043b6ee"
Sep 16 19:18:46.139414 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:18:46.139393    2582 scope.go:117] "RemoveContainer" containerID="c7a83217adb5c0da767012b885500ead2d7f7f8207349cae2b373c820043b6ee"
Sep 16 19:18:46.139882 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:18:46.139858    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7a83217adb5c0da767012b885500ead2d7f7f8207349cae2b373c820043b6ee\": container with ID starting with c7a83217adb5c0da767012b885500ead2d7f7f8207349cae2b373c820043b6ee not found: ID does not exist" containerID="c7a83217adb5c0da767012b885500ead2d7f7f8207349cae2b373c820043b6ee"
Sep 16 19:18:46.139981 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:18:46.139890    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7a83217adb5c0da767012b885500ead2d7f7f8207349cae2b373c820043b6ee"} err="failed to get container status \"c7a83217adb5c0da767012b885500ead2d7f7f8207349cae2b373c820043b6ee\": rpc error: code = NotFound desc = could not find container \"c7a83217adb5c0da767012b885500ead2d7f7f8207349cae2b373c820043b6ee\": container with ID starting with c7a83217adb5c0da767012b885500ead2d7f7f8207349cae2b373c820043b6ee not found: ID does not exist"
Sep 16 19:18:46.153710 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:18:46.153685    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xb8gn"]
Sep 16 19:18:46.155351 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:18:46.155331    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xb8gn"]
Sep 16 19:18:46.872806 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:18:46.872771    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe95f14a-a9c4-4974-b7bc-e7e390a022fd" path="/var/lib/kubelet/pods/fe95f14a-a9c4-4974-b7bc-e7e390a022fd/volumes"
Sep 16 19:20:05.866795 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:05.866758    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-vg699"]
Sep 16 19:20:05.867258 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:05.867062    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e375e012-acb2-4f8a-8aa5-587939d882d7" containerName="storage-initializer"
Sep 16 19:20:05.867258 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:05.867073    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e375e012-acb2-4f8a-8aa5-587939d882d7" containerName="storage-initializer"
Sep 16 19:20:05.867258 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:05.867089    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe95f14a-a9c4-4974-b7bc-e7e390a022fd" containerName="main"
Sep 16 19:20:05.867258 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:05.867094    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe95f14a-a9c4-4974-b7bc-e7e390a022fd" containerName="main"
Sep 16 19:20:05.867258 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:05.867106    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e375e012-acb2-4f8a-8aa5-587939d882d7" containerName="main"
Sep 16 19:20:05.867258 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:05.867113    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e375e012-acb2-4f8a-8aa5-587939d882d7" containerName="main"
Sep 16 19:20:05.867258 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:05.867165    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe95f14a-a9c4-4974-b7bc-e7e390a022fd" containerName="main"
Sep 16 19:20:05.867258 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:05.867175    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e375e012-acb2-4f8a-8aa5-587939d882d7" containerName="main"
Sep 16 19:20:05.870095 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:05.870076    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-vg699"
Sep 16 19:20:05.884475 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:05.884443    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 16 19:20:05.884635 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:05.884502    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:20:05.886291 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:05.886255    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 16 19:20:05.886291 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:05.886287    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-c4ct4\""
Sep 16 19:20:05.886493 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:05.886272    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 16 19:20:05.886493 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:05.886278    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 16 19:20:05.903204 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:05.903177    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-vg699"]
Sep 16 19:20:05.969778 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:05.969745    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69-tls-certs\") pod \"tokenizer-template-args-test-kserve-656f6fb7b8-vg699\" (UID: \"1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-vg699"
Sep 16 19:20:05.969961 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:05.969825    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69-dshm\") pod \"tokenizer-template-args-test-kserve-656f6fb7b8-vg699\" (UID: \"1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-vg699"
Sep 16 19:20:05.969961 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:05.969849    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69-model-cache\") pod \"tokenizer-template-args-test-kserve-656f6fb7b8-vg699\" (UID: \"1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-vg699"
Sep 16 19:20:05.969961 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:05.969865    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69-tmp-dir\") pod \"tokenizer-template-args-test-kserve-656f6fb7b8-vg699\" (UID: \"1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-vg699"
Sep 16 19:20:05.969961 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:05.969913    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-656f6fb7b8-vg699\" (UID: \"1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-vg699"
Sep 16 19:20:05.969961 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:05.969936    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69-home\") pod \"tokenizer-template-args-test-kserve-656f6fb7b8-vg699\" (UID: \"1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-vg699"
Sep 16 19:20:05.970169 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:05.969970    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25zzt\" (UniqueName: \"kubernetes.io/projected/1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69-kube-api-access-25zzt\") pod \"tokenizer-template-args-test-kserve-656f6fb7b8-vg699\" (UID: \"1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-vg699"
Sep 16 19:20:05.970169 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:05.969994    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-656f6fb7b8-vg699\" (UID: \"1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-vg699"
Sep 16 19:20:06.070508 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:06.070466    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69-dshm\") pod \"tokenizer-template-args-test-kserve-656f6fb7b8-vg699\" (UID: \"1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-vg699"
Sep 16 19:20:06.070705 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:06.070519    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69-model-cache\") pod \"tokenizer-template-args-test-kserve-656f6fb7b8-vg699\" (UID: \"1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-vg699"
Sep 16 19:20:06.070705 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:06.070639    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69-tmp-dir\") pod \"tokenizer-template-args-test-kserve-656f6fb7b8-vg699\" (UID: \"1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-vg699"
Sep 16 19:20:06.070705 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:06.070698    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-656f6fb7b8-vg699\" (UID: \"1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-vg699"
Sep 16 19:20:06.070917 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:06.070745    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69-home\") pod \"tokenizer-template-args-test-kserve-656f6fb7b8-vg699\" (UID: \"1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-vg699"
Sep 16 19:20:06.070917 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:06.070776    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25zzt\" (UniqueName: \"kubernetes.io/projected/1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69-kube-api-access-25zzt\") pod \"tokenizer-template-args-test-kserve-656f6fb7b8-vg699\" (UID: \"1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-vg699"
Sep 16 19:20:06.070917 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:06.070812    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-656f6fb7b8-vg699\" (UID: \"1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-vg699"
Sep 16 19:20:06.070917 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:06.070860    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69-tls-certs\") pod \"tokenizer-template-args-test-kserve-656f6fb7b8-vg699\" (UID: \"1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-vg699"
Sep 16 19:20:06.071138 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:06.070975    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69-model-cache\") pod \"tokenizer-template-args-test-kserve-656f6fb7b8-vg699\" (UID: \"1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-vg699"
Sep 16 19:20:06.071138 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:06.071046    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-656f6fb7b8-vg699\" (UID: \"1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-vg699"
Sep 16 19:20:06.071138 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:06.071087    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69-tmp-dir\") pod \"tokenizer-template-args-test-kserve-656f6fb7b8-vg699\" (UID: \"1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-vg699"
Sep 16 19:20:06.071307 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:06.071175    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69-home\") pod \"tokenizer-template-args-test-kserve-656f6fb7b8-vg699\" (UID: \"1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-vg699"
Sep 16 19:20:06.071587 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:06.071564    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-656f6fb7b8-vg699\" (UID: \"1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-vg699"
Sep 16 19:20:06.072933 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:06.072914    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69-dshm\") pod \"tokenizer-template-args-test-kserve-656f6fb7b8-vg699\" (UID: \"1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-vg699"
Sep 16 19:20:06.073297 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:06.073279    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69-tls-certs\") pod \"tokenizer-template-args-test-kserve-656f6fb7b8-vg699\" (UID: \"1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-vg699"
Sep 16 19:20:06.079703 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:06.079679    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25zzt\" (UniqueName: \"kubernetes.io/projected/1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69-kube-api-access-25zzt\") pod \"tokenizer-template-args-test-kserve-656f6fb7b8-vg699\" (UID: \"1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-vg699"
Sep 16 19:20:06.179889 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:06.179799    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-vg699"
Sep 16 19:20:06.467957 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:06.467930    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-vg699"]
Sep 16 19:20:06.470063 ip-10-0-129-194 kubenswrapper[2582]: W0916 19:20:06.470036    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b0e2b68_bd2a_46bc_a4a4_7dd198f9dc69.slice/crio-dba01892b76f7791bab0b4e186c52895055788d039e3b60efd34735cfe0072f3 WatchSource:0}: Error finding container dba01892b76f7791bab0b4e186c52895055788d039e3b60efd34735cfe0072f3: Status 404 returned error can't find the container with id dba01892b76f7791bab0b4e186c52895055788d039e3b60efd34735cfe0072f3
Sep 16 19:20:07.456388 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:07.456348    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-vg699" event={"ID":"1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69","Type":"ContainerStarted","Data":"34a50c8c852a6b74c6152ad0a89986ba51d586b1a3882331efbbc68d9a616669"}
Sep 16 19:20:07.456793 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:07.456403    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-vg699" event={"ID":"1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69","Type":"ContainerStarted","Data":"dba01892b76f7791bab0b4e186c52895055788d039e3b60efd34735cfe0072f3"}
Sep 16 19:20:09.465558 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:09.465524    2582 generic.go:356] "Generic (PLEG): container finished" podID="1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69" containerID="34a50c8c852a6b74c6152ad0a89986ba51d586b1a3882331efbbc68d9a616669" exitCode=0
Sep 16 19:20:09.465970 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:09.465600    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-vg699" event={"ID":"1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69","Type":"ContainerDied","Data":"34a50c8c852a6b74c6152ad0a89986ba51d586b1a3882331efbbc68d9a616669"}
Sep 16 19:20:10.471931 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:10.471892    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-vg699" event={"ID":"1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69","Type":"ContainerStarted","Data":"6e52559eaa6aeb2fac793d14d3dbe404190a3e9823bbea452d441deb707486a4"}
Sep 16 19:20:10.513773 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:10.513693    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-vg699" podStartSLOduration=3.234710811 podStartE2EDuration="5.513678914s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.278968103s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:20:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:20:10.513112729 +0000 UTC m=+2214.154828393" watchObservedRunningTime="2026-09-16 19:20:10.513678914 +0000 UTC m=+2214.155394578"
Sep 16 19:20:11.180525 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:11.180471    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-vg699"
Sep 16 19:20:16.180384 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:16.180342    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-vg699"
Sep 16 19:20:16.192973 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:16.192945    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-vg699"
Sep 16 19:20:16.508186 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:16.508153    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-vg699"
Sep 16 19:20:45.881666 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:45.881575    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-vg699"]
Sep 16 19:20:45.890467 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:20:45.890424    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:20:45.890467 ip-10-0-129-194 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:20:45.890467 ip-10-0-129-194 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 19:20:45.890467 ip-10-0-129-194 kubenswrapper[2582]:  > containerID="6e52559eaa6aeb2fac793d14d3dbe404190a3e9823bbea452d441deb707486a4" cmd=["/bin/sleep","15"]
Sep 16 19:20:45.890823 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:45.890484    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-vg699" message="" err=<
Sep 16 19:20:45.890823 ip-10-0-129-194 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:20:45.890823 ip-10-0-129-194 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 19:20:45.890823 ip-10-0-129-194 kubenswrapper[2582]:  >
Sep 16 19:20:45.890823 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:20:45.890496    2582 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:20:45.890823 ip-10-0-129-194 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:20:45.890823 ip-10-0-129-194 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 19:20:45.890823 ip-10-0-129-194 kubenswrapper[2582]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-vg699" podUID="1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69" containerName="main" containerID="cri-o://6e52559eaa6aeb2fac793d14d3dbe404190a3e9823bbea452d441deb707486a4"
Sep 16 19:20:45.890823 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:45.890544    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-vg699" podUID="1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69" containerName="main" containerID="cri-o://6e52559eaa6aeb2fac793d14d3dbe404190a3e9823bbea452d441deb707486a4" gracePeriod=60
Sep 16 19:20:46.247761 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:46.247735    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-vg699"
Sep 16 19:20:46.292471 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:46.292435    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69-home\") pod \"1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69\" (UID: \"1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69\") "
Sep 16 19:20:46.292662 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:46.292484    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69-cabundle-cert\") pod \"1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69\" (UID: \"1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69\") "
Sep 16 19:20:46.292662 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:46.292525    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69-tmp-dir\") pod \"1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69\" (UID: \"1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69\") "
Sep 16 19:20:46.292662 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:46.292550    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-25zzt\" (UniqueName: \"kubernetes.io/projected/1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69-kube-api-access-25zzt\") pod \"1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69\" (UID: \"1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69\") "
Sep 16 19:20:46.292662 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:46.292616    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69-kserve-provision-location\") pod \"1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69\" (UID: \"1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69\") "
Sep 16 19:20:46.292662 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:46.292652    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69-model-cache\") pod \"1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69\" (UID: \"1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69\") "
Sep 16 19:20:46.292956 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:46.292685    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69-dshm\") pod \"1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69\" (UID: \"1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69\") "
Sep 16 19:20:46.292956 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:46.292761    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69-tls-certs\") pod \"1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69\" (UID: \"1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69\") "
Sep 16 19:20:46.292956 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:46.292792    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69-home" (OuterVolumeSpecName: "home") pod "1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69" (UID: "1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:20:46.292956 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:46.292850    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69" (UID: "1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:20:46.293172 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:46.292966    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69" (UID: "1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:20:46.293172 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:46.293094    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69-model-cache" (OuterVolumeSpecName: "model-cache") pod "1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69" (UID: "1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:20:46.293172 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:46.293107    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69-home\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:20:46.293172 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:46.293141    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69-cabundle-cert\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:20:46.293172 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:46.293156    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69-tmp-dir\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:20:46.295141 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:46.295110    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69-kube-api-access-25zzt" (OuterVolumeSpecName: "kube-api-access-25zzt") pod "1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69" (UID: "1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69"). InnerVolumeSpecName "kube-api-access-25zzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:20:46.295416 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:46.295386    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69-dshm" (OuterVolumeSpecName: "dshm") pod "1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69" (UID: "1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:20:46.295513 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:46.295473    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69" (UID: "1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:20:46.319534 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:46.319499    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69" (UID: "1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:20:46.393856 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:46.393824    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69-model-cache\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:20:46.393856 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:46.393850    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69-dshm\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:20:46.393856 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:46.393860    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69-tls-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:20:46.394086 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:46.393870    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25zzt\" (UniqueName: \"kubernetes.io/projected/1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69-kube-api-access-25zzt\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:20:46.394086 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:46.393881    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69-kserve-provision-location\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:20:46.628574 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:46.628535    2582 generic.go:356] "Generic (PLEG): container finished" podID="1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69" containerID="6e52559eaa6aeb2fac793d14d3dbe404190a3e9823bbea452d441deb707486a4" exitCode=0
Sep 16 19:20:46.628765 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:46.628607    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-vg699"
Sep 16 19:20:46.628765 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:46.628617    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-vg699" event={"ID":"1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69","Type":"ContainerDied","Data":"6e52559eaa6aeb2fac793d14d3dbe404190a3e9823bbea452d441deb707486a4"}
Sep 16 19:20:46.628765 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:46.628710    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-vg699" event={"ID":"1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69","Type":"ContainerDied","Data":"dba01892b76f7791bab0b4e186c52895055788d039e3b60efd34735cfe0072f3"}
Sep 16 19:20:46.628765 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:46.628759    2582 scope.go:117] "RemoveContainer" containerID="6e52559eaa6aeb2fac793d14d3dbe404190a3e9823bbea452d441deb707486a4"
Sep 16 19:20:46.638265 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:46.638241    2582 scope.go:117] "RemoveContainer" containerID="34a50c8c852a6b74c6152ad0a89986ba51d586b1a3882331efbbc68d9a616669"
Sep 16 19:20:46.646099 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:46.646077    2582 scope.go:117] "RemoveContainer" containerID="6e52559eaa6aeb2fac793d14d3dbe404190a3e9823bbea452d441deb707486a4"
Sep 16 19:20:46.646355 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:20:46.646337    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e52559eaa6aeb2fac793d14d3dbe404190a3e9823bbea452d441deb707486a4\": container with ID starting with 6e52559eaa6aeb2fac793d14d3dbe404190a3e9823bbea452d441deb707486a4 not found: ID does not exist" containerID="6e52559eaa6aeb2fac793d14d3dbe404190a3e9823bbea452d441deb707486a4"
Sep 16 19:20:46.646403 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:46.646364    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e52559eaa6aeb2fac793d14d3dbe404190a3e9823bbea452d441deb707486a4"} err="failed to get container status \"6e52559eaa6aeb2fac793d14d3dbe404190a3e9823bbea452d441deb707486a4\": rpc error: code = NotFound desc = could not find container \"6e52559eaa6aeb2fac793d14d3dbe404190a3e9823bbea452d441deb707486a4\": container with ID starting with 6e52559eaa6aeb2fac793d14d3dbe404190a3e9823bbea452d441deb707486a4 not found: ID does not exist"
Sep 16 19:20:46.646403 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:46.646380    2582 scope.go:117] "RemoveContainer" containerID="34a50c8c852a6b74c6152ad0a89986ba51d586b1a3882331efbbc68d9a616669"
Sep 16 19:20:46.646631 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:20:46.646614    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34a50c8c852a6b74c6152ad0a89986ba51d586b1a3882331efbbc68d9a616669\": container with ID starting with 34a50c8c852a6b74c6152ad0a89986ba51d586b1a3882331efbbc68d9a616669 not found: ID does not exist" containerID="34a50c8c852a6b74c6152ad0a89986ba51d586b1a3882331efbbc68d9a616669"
Sep 16 19:20:46.646682 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:46.646635    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34a50c8c852a6b74c6152ad0a89986ba51d586b1a3882331efbbc68d9a616669"} err="failed to get container status \"34a50c8c852a6b74c6152ad0a89986ba51d586b1a3882331efbbc68d9a616669\": rpc error: code = NotFound desc = could not find container \"34a50c8c852a6b74c6152ad0a89986ba51d586b1a3882331efbbc68d9a616669\": container with ID starting with 34a50c8c852a6b74c6152ad0a89986ba51d586b1a3882331efbbc68d9a616669 not found: ID does not exist"
Sep 16 19:20:46.662781 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:46.662756    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-vg699"]
Sep 16 19:20:46.670509 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:46.670484    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-vg699"]
Sep 16 19:20:46.872196 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:20:46.872164    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69" path="/var/lib/kubelet/pods/1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69/volumes"
Sep 16 19:21:07.416418 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:21:07.416378    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-jt2j2"]
Sep 16 19:21:07.416974 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:21:07.416757    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69" containerName="main"
Sep 16 19:21:07.416974 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:21:07.416770    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69" containerName="main"
Sep 16 19:21:07.416974 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:21:07.416786    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69" containerName="storage-initializer"
Sep 16 19:21:07.416974 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:21:07.416793    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69" containerName="storage-initializer"
Sep 16 19:21:07.416974 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:21:07.416864    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b0e2b68-bd2a-46bc-a4a4-7dd198f9dc69" containerName="main"
Sep 16 19:21:07.420095 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:21:07.420075    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-jt2j2"
Sep 16 19:21:07.422952 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:21:07.422928    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"kube-root-ca.crt\""
Sep 16 19:21:07.422952 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:21:07.422947    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"openshift-service-ca.crt\""
Sep 16 19:21:07.424390 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:21:07.424371    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"default-dockercfg-l4nkv\""
Sep 16 19:21:07.424472 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:21:07.424402    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v1-kserve-self-signed-certs\""
Sep 16 19:21:07.431049 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:21:07.431026    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-jt2j2"]
Sep 16 19:21:07.572073 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:21:07.572034    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/493d2602-2259-4157-989a-0eb8d11c1917-dshm\") pod \"tri-v1-kserve-79f8f98bc5-jt2j2\" (UID: \"493d2602-2259-4157-989a-0eb8d11c1917\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-jt2j2"
Sep 16 19:21:07.572073 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:21:07.572079    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/493d2602-2259-4157-989a-0eb8d11c1917-home\") pod \"tri-v1-kserve-79f8f98bc5-jt2j2\" (UID: \"493d2602-2259-4157-989a-0eb8d11c1917\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-jt2j2"
Sep 16 19:21:07.572285 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:21:07.572095    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/493d2602-2259-4157-989a-0eb8d11c1917-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-jt2j2\" (UID: \"493d2602-2259-4157-989a-0eb8d11c1917\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-jt2j2"
Sep 16 19:21:07.572285 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:21:07.572115    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/493d2602-2259-4157-989a-0eb8d11c1917-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-jt2j2\" (UID: \"493d2602-2259-4157-989a-0eb8d11c1917\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-jt2j2"
Sep 16 19:21:07.572285 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:21:07.572179    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f7zv\" (UniqueName: \"kubernetes.io/projected/493d2602-2259-4157-989a-0eb8d11c1917-kube-api-access-4f7zv\") pod \"tri-v1-kserve-79f8f98bc5-jt2j2\" (UID: \"493d2602-2259-4157-989a-0eb8d11c1917\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-jt2j2"
Sep 16 19:21:07.572285 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:21:07.572225    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/493d2602-2259-4157-989a-0eb8d11c1917-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-jt2j2\" (UID: \"493d2602-2259-4157-989a-0eb8d11c1917\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-jt2j2"
Sep 16 19:21:07.672988 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:21:07.672830    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f7zv\" (UniqueName: \"kubernetes.io/projected/493d2602-2259-4157-989a-0eb8d11c1917-kube-api-access-4f7zv\") pod \"tri-v1-kserve-79f8f98bc5-jt2j2\" (UID: \"493d2602-2259-4157-989a-0eb8d11c1917\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-jt2j2"
Sep 16 19:21:07.672988 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:21:07.672880    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/493d2602-2259-4157-989a-0eb8d11c1917-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-jt2j2\" (UID: \"493d2602-2259-4157-989a-0eb8d11c1917\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-jt2j2"
Sep 16 19:21:07.672988 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:21:07.672946    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/493d2602-2259-4157-989a-0eb8d11c1917-dshm\") pod \"tri-v1-kserve-79f8f98bc5-jt2j2\" (UID: \"493d2602-2259-4157-989a-0eb8d11c1917\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-jt2j2"
Sep 16 19:21:07.672988 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:21:07.672984    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/493d2602-2259-4157-989a-0eb8d11c1917-home\") pod \"tri-v1-kserve-79f8f98bc5-jt2j2\" (UID: \"493d2602-2259-4157-989a-0eb8d11c1917\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-jt2j2"
Sep 16 19:21:07.673390 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:21:07.673007    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/493d2602-2259-4157-989a-0eb8d11c1917-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-jt2j2\" (UID: \"493d2602-2259-4157-989a-0eb8d11c1917\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-jt2j2"
Sep 16 19:21:07.673390 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:21:07.673031    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/493d2602-2259-4157-989a-0eb8d11c1917-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-jt2j2\" (UID: \"493d2602-2259-4157-989a-0eb8d11c1917\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-jt2j2"
Sep 16 19:21:07.673499 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:21:07.673436    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/493d2602-2259-4157-989a-0eb8d11c1917-home\") pod \"tri-v1-kserve-79f8f98bc5-jt2j2\" (UID: \"493d2602-2259-4157-989a-0eb8d11c1917\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-jt2j2"
Sep 16 19:21:07.673584 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:21:07.673565    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/493d2602-2259-4157-989a-0eb8d11c1917-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-jt2j2\" (UID: \"493d2602-2259-4157-989a-0eb8d11c1917\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-jt2j2"
Sep 16 19:21:07.673737 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:21:07.673704    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/493d2602-2259-4157-989a-0eb8d11c1917-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-jt2j2\" (UID: \"493d2602-2259-4157-989a-0eb8d11c1917\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-jt2j2"
Sep 16 19:21:07.675361 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:21:07.675341    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/493d2602-2259-4157-989a-0eb8d11c1917-dshm\") pod \"tri-v1-kserve-79f8f98bc5-jt2j2\" (UID: \"493d2602-2259-4157-989a-0eb8d11c1917\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-jt2j2"
Sep 16 19:21:07.675497 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:21:07.675480    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/493d2602-2259-4157-989a-0eb8d11c1917-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-jt2j2\" (UID: \"493d2602-2259-4157-989a-0eb8d11c1917\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-jt2j2"
Sep 16 19:21:07.684291 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:21:07.684265    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f7zv\" (UniqueName: \"kubernetes.io/projected/493d2602-2259-4157-989a-0eb8d11c1917-kube-api-access-4f7zv\") pod \"tri-v1-kserve-79f8f98bc5-jt2j2\" (UID: \"493d2602-2259-4157-989a-0eb8d11c1917\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-jt2j2"
Sep 16 19:21:07.730657 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:21:07.730622    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-jt2j2"
Sep 16 19:21:08.010283 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:21:08.010258    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-jt2j2"]
Sep 16 19:21:08.012964 ip-10-0-129-194 kubenswrapper[2582]: W0916 19:21:08.012938    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod493d2602_2259_4157_989a_0eb8d11c1917.slice/crio-8c06828e7fa7ed8fe64d63109e3afbba25c47ee9288661c685c742ecf6747604 WatchSource:0}: Error finding container 8c06828e7fa7ed8fe64d63109e3afbba25c47ee9288661c685c742ecf6747604: Status 404 returned error can't find the container with id 8c06828e7fa7ed8fe64d63109e3afbba25c47ee9288661c685c742ecf6747604
Sep 16 19:21:08.720323 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:21:08.720283    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-jt2j2" event={"ID":"493d2602-2259-4157-989a-0eb8d11c1917","Type":"ContainerStarted","Data":"7e1e27e46bacf027fd998354e6d1c8f7c34420f13754315f6185a72011cfc3f1"}
Sep 16 19:21:08.720835 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:21:08.720348    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-jt2j2" event={"ID":"493d2602-2259-4157-989a-0eb8d11c1917","Type":"ContainerStarted","Data":"8c06828e7fa7ed8fe64d63109e3afbba25c47ee9288661c685c742ecf6747604"}
Sep 16 19:21:08.731180 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:21:08.731152    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-jt2j2"
Sep 16 19:21:08.740574 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:21:08.740528    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-jt2j2" podStartSLOduration=1.740512116 podStartE2EDuration="1.740512116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:21:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:21:08.739152427 +0000 UTC m=+2272.380868094" watchObservedRunningTime="2026-09-16 19:21:08.740512116 +0000 UTC m=+2272.382227774"
Sep 16 19:21:17.730955 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:21:17.730916    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-jt2j2"
Sep 16 19:21:17.743076 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:21:17.743050    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-jt2j2"
Sep 16 19:21:17.771403 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:21:17.771375    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-jt2j2"
Sep 16 19:23:16.977382 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:23:16.977352    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7qwl2_b6695f86-08c0-4836-87b1-f2a9d76084bd/console-operator/1.log"
Sep 16 19:23:16.984433 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:23:16.984410    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7qwl2_b6695f86-08c0-4836-87b1-f2a9d76084bd/console-operator/1.log"
Sep 16 19:23:41.921864 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:23:41.921785    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-jt2j2"]
Sep 16 19:23:56.940428 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:23:56.940378    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-jt2j2" podUID="493d2602-2259-4157-989a-0eb8d11c1917" containerName="main" containerID="cri-o://7e1e27e46bacf027fd998354e6d1c8f7c34420f13754315f6185a72011cfc3f1" gracePeriod=45
Sep 16 19:23:57.191167 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:23:57.191105    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-jt2j2"
Sep 16 19:23:57.306205 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:23:57.306151    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4f7zv\" (UniqueName: \"kubernetes.io/projected/493d2602-2259-4157-989a-0eb8d11c1917-kube-api-access-4f7zv\") pod \"493d2602-2259-4157-989a-0eb8d11c1917\" (UID: \"493d2602-2259-4157-989a-0eb8d11c1917\") "
Sep 16 19:23:57.306205 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:23:57.306215    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/493d2602-2259-4157-989a-0eb8d11c1917-home\") pod \"493d2602-2259-4157-989a-0eb8d11c1917\" (UID: \"493d2602-2259-4157-989a-0eb8d11c1917\") "
Sep 16 19:23:57.306425 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:23:57.306323    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/493d2602-2259-4157-989a-0eb8d11c1917-tmp-dir\") pod \"493d2602-2259-4157-989a-0eb8d11c1917\" (UID: \"493d2602-2259-4157-989a-0eb8d11c1917\") "
Sep 16 19:23:57.306425 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:23:57.306376    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/493d2602-2259-4157-989a-0eb8d11c1917-model-cache\") pod \"493d2602-2259-4157-989a-0eb8d11c1917\" (UID: \"493d2602-2259-4157-989a-0eb8d11c1917\") "
Sep 16 19:23:57.306425 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:23:57.306406    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/493d2602-2259-4157-989a-0eb8d11c1917-tls-certs\") pod \"493d2602-2259-4157-989a-0eb8d11c1917\" (UID: \"493d2602-2259-4157-989a-0eb8d11c1917\") "
Sep 16 19:23:57.306551 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:23:57.306455    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/493d2602-2259-4157-989a-0eb8d11c1917-dshm\") pod \"493d2602-2259-4157-989a-0eb8d11c1917\" (UID: \"493d2602-2259-4157-989a-0eb8d11c1917\") "
Sep 16 19:23:57.306551 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:23:57.306494    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/493d2602-2259-4157-989a-0eb8d11c1917-home" (OuterVolumeSpecName: "home") pod "493d2602-2259-4157-989a-0eb8d11c1917" (UID: "493d2602-2259-4157-989a-0eb8d11c1917"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:23:57.306697 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:23:57.306660    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/493d2602-2259-4157-989a-0eb8d11c1917-model-cache" (OuterVolumeSpecName: "model-cache") pod "493d2602-2259-4157-989a-0eb8d11c1917" (UID: "493d2602-2259-4157-989a-0eb8d11c1917"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:23:57.306697 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:23:57.306680    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/493d2602-2259-4157-989a-0eb8d11c1917-home\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:23:57.307024 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:23:57.306996    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/493d2602-2259-4157-989a-0eb8d11c1917-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "493d2602-2259-4157-989a-0eb8d11c1917" (UID: "493d2602-2259-4157-989a-0eb8d11c1917"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:23:57.308527 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:23:57.308500    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/493d2602-2259-4157-989a-0eb8d11c1917-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "493d2602-2259-4157-989a-0eb8d11c1917" (UID: "493d2602-2259-4157-989a-0eb8d11c1917"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:23:57.308620 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:23:57.308591    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/493d2602-2259-4157-989a-0eb8d11c1917-kube-api-access-4f7zv" (OuterVolumeSpecName: "kube-api-access-4f7zv") pod "493d2602-2259-4157-989a-0eb8d11c1917" (UID: "493d2602-2259-4157-989a-0eb8d11c1917"). InnerVolumeSpecName "kube-api-access-4f7zv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:23:57.308670 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:23:57.308632    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/493d2602-2259-4157-989a-0eb8d11c1917-dshm" (OuterVolumeSpecName: "dshm") pod "493d2602-2259-4157-989a-0eb8d11c1917" (UID: "493d2602-2259-4157-989a-0eb8d11c1917"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:23:57.408077 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:23:57.408028    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/493d2602-2259-4157-989a-0eb8d11c1917-model-cache\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:23:57.408077 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:23:57.408078    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/493d2602-2259-4157-989a-0eb8d11c1917-tls-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:23:57.408279 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:23:57.408093    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/493d2602-2259-4157-989a-0eb8d11c1917-dshm\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:23:57.408279 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:23:57.408109    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f7zv\" (UniqueName: \"kubernetes.io/projected/493d2602-2259-4157-989a-0eb8d11c1917-kube-api-access-4f7zv\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:23:57.408279 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:23:57.408124    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/493d2602-2259-4157-989a-0eb8d11c1917-tmp-dir\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:23:57.428707 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:23:57.428672    2582 generic.go:356] "Generic (PLEG): container finished" podID="493d2602-2259-4157-989a-0eb8d11c1917" containerID="7e1e27e46bacf027fd998354e6d1c8f7c34420f13754315f6185a72011cfc3f1" exitCode=0
Sep 16 19:23:57.428879 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:23:57.428730    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-jt2j2"
Sep 16 19:23:57.428879 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:23:57.428759    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-jt2j2" event={"ID":"493d2602-2259-4157-989a-0eb8d11c1917","Type":"ContainerDied","Data":"7e1e27e46bacf027fd998354e6d1c8f7c34420f13754315f6185a72011cfc3f1"}
Sep 16 19:23:57.428879 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:23:57.428812    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-jt2j2" event={"ID":"493d2602-2259-4157-989a-0eb8d11c1917","Type":"ContainerDied","Data":"8c06828e7fa7ed8fe64d63109e3afbba25c47ee9288661c685c742ecf6747604"}
Sep 16 19:23:57.428879 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:23:57.428835    2582 scope.go:117] "RemoveContainer" containerID="7e1e27e46bacf027fd998354e6d1c8f7c34420f13754315f6185a72011cfc3f1"
Sep 16 19:23:57.437558 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:23:57.437539    2582 scope.go:117] "RemoveContainer" containerID="7e1e27e46bacf027fd998354e6d1c8f7c34420f13754315f6185a72011cfc3f1"
Sep 16 19:23:57.437859 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:23:57.437832    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e1e27e46bacf027fd998354e6d1c8f7c34420f13754315f6185a72011cfc3f1\": container with ID starting with 7e1e27e46bacf027fd998354e6d1c8f7c34420f13754315f6185a72011cfc3f1 not found: ID does not exist" containerID="7e1e27e46bacf027fd998354e6d1c8f7c34420f13754315f6185a72011cfc3f1"
Sep 16 19:23:57.437927 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:23:57.437871    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e1e27e46bacf027fd998354e6d1c8f7c34420f13754315f6185a72011cfc3f1"} err="failed to get container status \"7e1e27e46bacf027fd998354e6d1c8f7c34420f13754315f6185a72011cfc3f1\": rpc error: code = NotFound desc = could not find container \"7e1e27e46bacf027fd998354e6d1c8f7c34420f13754315f6185a72011cfc3f1\": container with ID starting with 7e1e27e46bacf027fd998354e6d1c8f7c34420f13754315f6185a72011cfc3f1 not found: ID does not exist"
Sep 16 19:23:57.455413 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:23:57.455327    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-jt2j2"]
Sep 16 19:23:57.459713 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:23:57.459690    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-jt2j2"]
Sep 16 19:23:58.872419 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:23:58.872384    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="493d2602-2259-4157-989a-0eb8d11c1917" path="/var/lib/kubelet/pods/493d2602-2259-4157-989a-0eb8d11c1917/volumes"
Sep 16 19:24:45.717417 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:24:45.717380    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zzx4j"]
Sep 16 19:24:45.717927 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:24:45.717871    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="493d2602-2259-4157-989a-0eb8d11c1917" containerName="main"
Sep 16 19:24:45.717927 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:24:45.717893    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="493d2602-2259-4157-989a-0eb8d11c1917" containerName="main"
Sep 16 19:24:45.718044 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:24:45.717975    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="493d2602-2259-4157-989a-0eb8d11c1917" containerName="main"
Sep 16 19:24:45.720813 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:24:45.720790    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zzx4j"
Sep 16 19:24:45.729623 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:24:45.729602    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"openshift-service-ca.crt\""
Sep 16 19:24:45.730976 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:24:45.730958    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"late-v2-kserve-self-signed-certs\""
Sep 16 19:24:45.731086 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:24:45.731046    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"default-dockercfg-d6lsx\""
Sep 16 19:24:45.731195 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:24:45.731157    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"kube-root-ca.crt\""
Sep 16 19:24:45.742604 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:24:45.742575    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zzx4j"]
Sep 16 19:24:45.784486 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:24:45.784451    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06ca8aec-e256-4d21-a8b2-d8a7435755b1-model-cache\") pod \"late-v2-kserve-86766546f-zzx4j\" (UID: \"06ca8aec-e256-4d21-a8b2-d8a7435755b1\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zzx4j"
Sep 16 19:24:45.784755 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:24:45.784501    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06ca8aec-e256-4d21-a8b2-d8a7435755b1-home\") pod \"late-v2-kserve-86766546f-zzx4j\" (UID: \"06ca8aec-e256-4d21-a8b2-d8a7435755b1\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zzx4j"
Sep 16 19:24:45.784755 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:24:45.784566    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgvq6\" (UniqueName: \"kubernetes.io/projected/06ca8aec-e256-4d21-a8b2-d8a7435755b1-kube-api-access-vgvq6\") pod \"late-v2-kserve-86766546f-zzx4j\" (UID: \"06ca8aec-e256-4d21-a8b2-d8a7435755b1\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zzx4j"
Sep 16 19:24:45.784755 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:24:45.784602    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06ca8aec-e256-4d21-a8b2-d8a7435755b1-tls-certs\") pod \"late-v2-kserve-86766546f-zzx4j\" (UID: \"06ca8aec-e256-4d21-a8b2-d8a7435755b1\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zzx4j"
Sep 16 19:24:45.784755 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:24:45.784639    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06ca8aec-e256-4d21-a8b2-d8a7435755b1-dshm\") pod \"late-v2-kserve-86766546f-zzx4j\" (UID: \"06ca8aec-e256-4d21-a8b2-d8a7435755b1\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zzx4j"
Sep 16 19:24:45.784755 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:24:45.784655    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06ca8aec-e256-4d21-a8b2-d8a7435755b1-tmp-dir\") pod \"late-v2-kserve-86766546f-zzx4j\" (UID: \"06ca8aec-e256-4d21-a8b2-d8a7435755b1\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zzx4j"
Sep 16 19:24:45.885472 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:24:45.885435    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06ca8aec-e256-4d21-a8b2-d8a7435755b1-model-cache\") pod \"late-v2-kserve-86766546f-zzx4j\" (UID: \"06ca8aec-e256-4d21-a8b2-d8a7435755b1\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zzx4j"
Sep 16 19:24:45.885670 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:24:45.885483    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06ca8aec-e256-4d21-a8b2-d8a7435755b1-home\") pod \"late-v2-kserve-86766546f-zzx4j\" (UID: \"06ca8aec-e256-4d21-a8b2-d8a7435755b1\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zzx4j"
Sep 16 19:24:45.885670 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:24:45.885505    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgvq6\" (UniqueName: \"kubernetes.io/projected/06ca8aec-e256-4d21-a8b2-d8a7435755b1-kube-api-access-vgvq6\") pod \"late-v2-kserve-86766546f-zzx4j\" (UID: \"06ca8aec-e256-4d21-a8b2-d8a7435755b1\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zzx4j"
Sep 16 19:24:45.885670 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:24:45.885616    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06ca8aec-e256-4d21-a8b2-d8a7435755b1-tls-certs\") pod \"late-v2-kserve-86766546f-zzx4j\" (UID: \"06ca8aec-e256-4d21-a8b2-d8a7435755b1\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zzx4j"
Sep 16 19:24:45.885670 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:24:45.885662    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06ca8aec-e256-4d21-a8b2-d8a7435755b1-dshm\") pod \"late-v2-kserve-86766546f-zzx4j\" (UID: \"06ca8aec-e256-4d21-a8b2-d8a7435755b1\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zzx4j"
Sep 16 19:24:45.885956 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:24:45.885678    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06ca8aec-e256-4d21-a8b2-d8a7435755b1-tmp-dir\") pod \"late-v2-kserve-86766546f-zzx4j\" (UID: \"06ca8aec-e256-4d21-a8b2-d8a7435755b1\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zzx4j"
Sep 16 19:24:45.885956 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:24:45.885872    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06ca8aec-e256-4d21-a8b2-d8a7435755b1-model-cache\") pod \"late-v2-kserve-86766546f-zzx4j\" (UID: \"06ca8aec-e256-4d21-a8b2-d8a7435755b1\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zzx4j"
Sep 16 19:24:45.885956 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:24:45.885931    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06ca8aec-e256-4d21-a8b2-d8a7435755b1-home\") pod \"late-v2-kserve-86766546f-zzx4j\" (UID: \"06ca8aec-e256-4d21-a8b2-d8a7435755b1\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zzx4j"
Sep 16 19:24:45.886122 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:24:45.886094    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06ca8aec-e256-4d21-a8b2-d8a7435755b1-tmp-dir\") pod \"late-v2-kserve-86766546f-zzx4j\" (UID: \"06ca8aec-e256-4d21-a8b2-d8a7435755b1\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zzx4j"
Sep 16 19:24:45.887873 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:24:45.887854    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06ca8aec-e256-4d21-a8b2-d8a7435755b1-dshm\") pod \"late-v2-kserve-86766546f-zzx4j\" (UID: \"06ca8aec-e256-4d21-a8b2-d8a7435755b1\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zzx4j"
Sep 16 19:24:45.888202 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:24:45.888184    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06ca8aec-e256-4d21-a8b2-d8a7435755b1-tls-certs\") pod \"late-v2-kserve-86766546f-zzx4j\" (UID: \"06ca8aec-e256-4d21-a8b2-d8a7435755b1\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zzx4j"
Sep 16 19:24:45.895099 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:24:45.895078    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgvq6\" (UniqueName: \"kubernetes.io/projected/06ca8aec-e256-4d21-a8b2-d8a7435755b1-kube-api-access-vgvq6\") pod \"late-v2-kserve-86766546f-zzx4j\" (UID: \"06ca8aec-e256-4d21-a8b2-d8a7435755b1\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zzx4j"
Sep 16 19:24:46.030878 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:24:46.030843    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zzx4j"
Sep 16 19:24:46.305062 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:24:46.305022    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zzx4j"]
Sep 16 19:24:46.307474 ip-10-0-129-194 kubenswrapper[2582]: W0916 19:24:46.307445    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06ca8aec_e256_4d21_a8b2_d8a7435755b1.slice/crio-ac87ee45f81471fd677fb95848542d5e91793ca3b7cd7c530ff0758919d77ca0 WatchSource:0}: Error finding container ac87ee45f81471fd677fb95848542d5e91793ca3b7cd7c530ff0758919d77ca0: Status 404 returned error can't find the container with id ac87ee45f81471fd677fb95848542d5e91793ca3b7cd7c530ff0758919d77ca0
Sep 16 19:24:46.626916 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:24:46.626822    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zzx4j" event={"ID":"06ca8aec-e256-4d21-a8b2-d8a7435755b1","Type":"ContainerStarted","Data":"067bf28e27610018e878517f52c1833fd67c1c8325e42bbc2e165844b02b6973"}
Sep 16 19:24:46.626916 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:24:46.626904    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zzx4j" event={"ID":"06ca8aec-e256-4d21-a8b2-d8a7435755b1","Type":"ContainerStarted","Data":"ac87ee45f81471fd677fb95848542d5e91793ca3b7cd7c530ff0758919d77ca0"}
Sep 16 19:24:46.648562 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:24:46.648515    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zzx4j" podStartSLOduration=1.6485007390000002 podStartE2EDuration="1.648500739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:24:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:24:46.647825223 +0000 UTC m=+2490.289540900" watchObservedRunningTime="2026-09-16 19:24:46.648500739 +0000 UTC m=+2490.290216403"
Sep 16 19:24:47.031312 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:24:47.031278    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zzx4j"
Sep 16 19:24:56.031054 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:24:56.031021    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zzx4j"
Sep 16 19:24:56.043531 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:24:56.043505    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zzx4j"
Sep 16 19:24:56.682402 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:24:56.682372    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zzx4j"
Sep 16 19:25:23.522391 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:25:23.522350    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wt8lb"]
Sep 16 19:25:23.526259 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:25:23.526241    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wt8lb"
Sep 16 19:25:23.530690 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:25:23.530659    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 16 19:25:23.530690 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:25:23.530678    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-cjx6p\""
Sep 16 19:25:23.530887 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:25:23.530680    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 16 19:25:23.530887 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:25:23.530680    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Sep 16 19:25:23.544961 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:25:23.544929    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wt8lb"]
Sep 16 19:25:23.710298 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:25:23.710264    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d6f59bd-dd0b-4253-89e5-aa9f784b64b0-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wt8lb\" (UID: \"3d6f59bd-dd0b-4253-89e5-aa9f784b64b0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wt8lb"
Sep 16 19:25:23.710298 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:25:23.710302    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d6f59bd-dd0b-4253-89e5-aa9f784b64b0-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wt8lb\" (UID: \"3d6f59bd-dd0b-4253-89e5-aa9f784b64b0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wt8lb"
Sep 16 19:25:23.710526 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:25:23.710329    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d6f59bd-dd0b-4253-89e5-aa9f784b64b0-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wt8lb\" (UID: \"3d6f59bd-dd0b-4253-89e5-aa9f784b64b0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wt8lb"
Sep 16 19:25:23.710526 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:25:23.710408    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d6f59bd-dd0b-4253-89e5-aa9f784b64b0-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wt8lb\" (UID: \"3d6f59bd-dd0b-4253-89e5-aa9f784b64b0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wt8lb"
Sep 16 19:25:23.710526 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:25:23.710455    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzxhs\" (UniqueName: \"kubernetes.io/projected/3d6f59bd-dd0b-4253-89e5-aa9f784b64b0-kube-api-access-fzxhs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wt8lb\" (UID: \"3d6f59bd-dd0b-4253-89e5-aa9f784b64b0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wt8lb"
Sep 16 19:25:23.710526 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:25:23.710485    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d6f59bd-dd0b-4253-89e5-aa9f784b64b0-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wt8lb\" (UID: \"3d6f59bd-dd0b-4253-89e5-aa9f784b64b0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wt8lb"
Sep 16 19:25:23.811610 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:25:23.811512    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d6f59bd-dd0b-4253-89e5-aa9f784b64b0-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wt8lb\" (UID: \"3d6f59bd-dd0b-4253-89e5-aa9f784b64b0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wt8lb"
Sep 16 19:25:23.811610 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:25:23.811556    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d6f59bd-dd0b-4253-89e5-aa9f784b64b0-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wt8lb\" (UID: \"3d6f59bd-dd0b-4253-89e5-aa9f784b64b0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wt8lb"
Sep 16 19:25:23.811610 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:25:23.811590    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d6f59bd-dd0b-4253-89e5-aa9f784b64b0-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wt8lb\" (UID: \"3d6f59bd-dd0b-4253-89e5-aa9f784b64b0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wt8lb"
Sep 16 19:25:23.811925 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:25:23.811632    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d6f59bd-dd0b-4253-89e5-aa9f784b64b0-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wt8lb\" (UID: \"3d6f59bd-dd0b-4253-89e5-aa9f784b64b0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wt8lb"
Sep 16 19:25:23.811925 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:25:23.811675    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzxhs\" (UniqueName: \"kubernetes.io/projected/3d6f59bd-dd0b-4253-89e5-aa9f784b64b0-kube-api-access-fzxhs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wt8lb\" (UID: \"3d6f59bd-dd0b-4253-89e5-aa9f784b64b0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wt8lb"
Sep 16 19:25:23.811925 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:25:23.811699    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d6f59bd-dd0b-4253-89e5-aa9f784b64b0-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wt8lb\" (UID: \"3d6f59bd-dd0b-4253-89e5-aa9f784b64b0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wt8lb"
Sep 16 19:25:23.812088 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:25:23.811980    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d6f59bd-dd0b-4253-89e5-aa9f784b64b0-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wt8lb\" (UID: \"3d6f59bd-dd0b-4253-89e5-aa9f784b64b0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wt8lb"
Sep 16 19:25:23.812088 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:25:23.812060    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d6f59bd-dd0b-4253-89e5-aa9f784b64b0-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wt8lb\" (UID: \"3d6f59bd-dd0b-4253-89e5-aa9f784b64b0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wt8lb"
Sep 16 19:25:23.812203 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:25:23.812153    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d6f59bd-dd0b-4253-89e5-aa9f784b64b0-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wt8lb\" (UID: \"3d6f59bd-dd0b-4253-89e5-aa9f784b64b0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wt8lb"
Sep 16 19:25:23.813949 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:25:23.813928    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d6f59bd-dd0b-4253-89e5-aa9f784b64b0-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wt8lb\" (UID: \"3d6f59bd-dd0b-4253-89e5-aa9f784b64b0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wt8lb"
Sep 16 19:25:23.814095 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:25:23.814078    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d6f59bd-dd0b-4253-89e5-aa9f784b64b0-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wt8lb\" (UID: \"3d6f59bd-dd0b-4253-89e5-aa9f784b64b0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wt8lb"
Sep 16 19:25:23.820468 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:25:23.820439    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzxhs\" (UniqueName: \"kubernetes.io/projected/3d6f59bd-dd0b-4253-89e5-aa9f784b64b0-kube-api-access-fzxhs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wt8lb\" (UID: \"3d6f59bd-dd0b-4253-89e5-aa9f784b64b0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wt8lb"
Sep 16 19:25:23.836286 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:25:23.836265    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wt8lb"
Sep 16 19:25:24.113400 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:25:24.113208    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wt8lb"]
Sep 16 19:25:24.116076 ip-10-0-129-194 kubenswrapper[2582]: W0916 19:25:24.116041    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d6f59bd_dd0b_4253_89e5_aa9f784b64b0.slice/crio-aac110e61c26cc58e10d5e38af71d92b2f2218733ece4154d3b7f69e95883b34 WatchSource:0}: Error finding container aac110e61c26cc58e10d5e38af71d92b2f2218733ece4154d3b7f69e95883b34: Status 404 returned error can't find the container with id aac110e61c26cc58e10d5e38af71d92b2f2218733ece4154d3b7f69e95883b34
Sep 16 19:25:24.787855 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:25:24.787821    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wt8lb" event={"ID":"3d6f59bd-dd0b-4253-89e5-aa9f784b64b0","Type":"ContainerStarted","Data":"ac1ef61b3c2cd3e477705b7e4a2dda733153e90b8712fa64b258d1d6af7b688b"}
Sep 16 19:25:24.788345 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:25:24.787873    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wt8lb" event={"ID":"3d6f59bd-dd0b-4253-89e5-aa9f784b64b0","Type":"ContainerStarted","Data":"aac110e61c26cc58e10d5e38af71d92b2f2218733ece4154d3b7f69e95883b34"}
Sep 16 19:25:24.816977 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:25:24.816925    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wt8lb" podStartSLOduration=1.8169102499999998 podStartE2EDuration="1.81691025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:25:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:25:24.816518698 +0000 UTC m=+2528.458234362" watchObservedRunningTime="2026-09-16 19:25:24.81691025 +0000 UTC m=+2528.458625914"
Sep 16 19:25:24.836789 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:25:24.836754    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wt8lb"
Sep 16 19:25:33.836887 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:25:33.836849    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wt8lb"
Sep 16 19:25:33.849466 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:25:33.849431    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wt8lb"
Sep 16 19:25:34.845385 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:25:34.845354    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wt8lb"
Sep 16 19:26:00.587475 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:00.587433    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wt8lb"]
Sep 16 19:26:00.596832 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:26:00.596739    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:26:00.596832 ip-10-0-129-194 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:26:00.596832 ip-10-0-129-194 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 19:26:00.596832 ip-10-0-129-194 kubenswrapper[2582]:  > containerID="ac1ef61b3c2cd3e477705b7e4a2dda733153e90b8712fa64b258d1d6af7b688b" cmd=["/bin/sleep","15"]
Sep 16 19:26:00.596832 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:00.596802    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wt8lb" message="" err=<
Sep 16 19:26:00.596832 ip-10-0-129-194 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:26:00.596832 ip-10-0-129-194 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 19:26:00.596832 ip-10-0-129-194 kubenswrapper[2582]:  >
Sep 16 19:26:00.596832 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:26:00.596817    2582 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:26:00.596832 ip-10-0-129-194 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:26:00.596832 ip-10-0-129-194 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 19:26:00.596832 ip-10-0-129-194 kubenswrapper[2582]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wt8lb" podUID="3d6f59bd-dd0b-4253-89e5-aa9f784b64b0" containerName="main" containerID="cri-o://ac1ef61b3c2cd3e477705b7e4a2dda733153e90b8712fa64b258d1d6af7b688b"
Sep 16 19:26:00.597366 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:00.596864    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wt8lb" podUID="3d6f59bd-dd0b-4253-89e5-aa9f784b64b0" containerName="main" containerID="cri-o://ac1ef61b3c2cd3e477705b7e4a2dda733153e90b8712fa64b258d1d6af7b688b" gracePeriod=60
Sep 16 19:26:00.834897 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:00.834864    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wt8lb" podUID="3d6f59bd-dd0b-4253-89e5-aa9f784b64b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Sep 16 19:26:00.944102 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:00.944078    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wt8lb"
Sep 16 19:26:00.946172 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:00.946145    2582 generic.go:356] "Generic (PLEG): container finished" podID="3d6f59bd-dd0b-4253-89e5-aa9f784b64b0" containerID="ac1ef61b3c2cd3e477705b7e4a2dda733153e90b8712fa64b258d1d6af7b688b" exitCode=0
Sep 16 19:26:00.946283 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:00.946178    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wt8lb"
Sep 16 19:26:00.946283 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:00.946178    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wt8lb" event={"ID":"3d6f59bd-dd0b-4253-89e5-aa9f784b64b0","Type":"ContainerDied","Data":"ac1ef61b3c2cd3e477705b7e4a2dda733153e90b8712fa64b258d1d6af7b688b"}
Sep 16 19:26:00.946283 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:00.946232    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wt8lb" event={"ID":"3d6f59bd-dd0b-4253-89e5-aa9f784b64b0","Type":"ContainerDied","Data":"aac110e61c26cc58e10d5e38af71d92b2f2218733ece4154d3b7f69e95883b34"}
Sep 16 19:26:00.946283 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:00.946259    2582 scope.go:117] "RemoveContainer" containerID="ac1ef61b3c2cd3e477705b7e4a2dda733153e90b8712fa64b258d1d6af7b688b"
Sep 16 19:26:00.954551 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:00.954533    2582 scope.go:117] "RemoveContainer" containerID="ac1ef61b3c2cd3e477705b7e4a2dda733153e90b8712fa64b258d1d6af7b688b"
Sep 16 19:26:00.954840 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:26:00.954820    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac1ef61b3c2cd3e477705b7e4a2dda733153e90b8712fa64b258d1d6af7b688b\": container with ID starting with ac1ef61b3c2cd3e477705b7e4a2dda733153e90b8712fa64b258d1d6af7b688b not found: ID does not exist" containerID="ac1ef61b3c2cd3e477705b7e4a2dda733153e90b8712fa64b258d1d6af7b688b"
Sep 16 19:26:00.954894 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:00.954848    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac1ef61b3c2cd3e477705b7e4a2dda733153e90b8712fa64b258d1d6af7b688b"} err="failed to get container status \"ac1ef61b3c2cd3e477705b7e4a2dda733153e90b8712fa64b258d1d6af7b688b\": rpc error: code = NotFound desc = could not find container \"ac1ef61b3c2cd3e477705b7e4a2dda733153e90b8712fa64b258d1d6af7b688b\": container with ID starting with ac1ef61b3c2cd3e477705b7e4a2dda733153e90b8712fa64b258d1d6af7b688b not found: ID does not exist"
Sep 16 19:26:01.022575 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:01.022541    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d6f59bd-dd0b-4253-89e5-aa9f784b64b0-home\") pod \"3d6f59bd-dd0b-4253-89e5-aa9f784b64b0\" (UID: \"3d6f59bd-dd0b-4253-89e5-aa9f784b64b0\") "
Sep 16 19:26:01.022575 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:01.022576    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d6f59bd-dd0b-4253-89e5-aa9f784b64b0-dshm\") pod \"3d6f59bd-dd0b-4253-89e5-aa9f784b64b0\" (UID: \"3d6f59bd-dd0b-4253-89e5-aa9f784b64b0\") "
Sep 16 19:26:01.022854 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:01.022599    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d6f59bd-dd0b-4253-89e5-aa9f784b64b0-model-cache\") pod \"3d6f59bd-dd0b-4253-89e5-aa9f784b64b0\" (UID: \"3d6f59bd-dd0b-4253-89e5-aa9f784b64b0\") "
Sep 16 19:26:01.022854 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:01.022643    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d6f59bd-dd0b-4253-89e5-aa9f784b64b0-tmp-dir\") pod \"3d6f59bd-dd0b-4253-89e5-aa9f784b64b0\" (UID: \"3d6f59bd-dd0b-4253-89e5-aa9f784b64b0\") "
Sep 16 19:26:01.022854 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:01.022688    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d6f59bd-dd0b-4253-89e5-aa9f784b64b0-tls-certs\") pod \"3d6f59bd-dd0b-4253-89e5-aa9f784b64b0\" (UID: \"3d6f59bd-dd0b-4253-89e5-aa9f784b64b0\") "
Sep 16 19:26:01.022854 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:01.022744    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fzxhs\" (UniqueName: \"kubernetes.io/projected/3d6f59bd-dd0b-4253-89e5-aa9f784b64b0-kube-api-access-fzxhs\") pod \"3d6f59bd-dd0b-4253-89e5-aa9f784b64b0\" (UID: \"3d6f59bd-dd0b-4253-89e5-aa9f784b64b0\") "
Sep 16 19:26:01.023064 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:01.022921    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d6f59bd-dd0b-4253-89e5-aa9f784b64b0-home" (OuterVolumeSpecName: "home") pod "3d6f59bd-dd0b-4253-89e5-aa9f784b64b0" (UID: "3d6f59bd-dd0b-4253-89e5-aa9f784b64b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:26:01.023064 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:01.022985    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d6f59bd-dd0b-4253-89e5-aa9f784b64b0-model-cache" (OuterVolumeSpecName: "model-cache") pod "3d6f59bd-dd0b-4253-89e5-aa9f784b64b0" (UID: "3d6f59bd-dd0b-4253-89e5-aa9f784b64b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:26:01.023064 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:01.022997    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d6f59bd-dd0b-4253-89e5-aa9f784b64b0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3d6f59bd-dd0b-4253-89e5-aa9f784b64b0" (UID: "3d6f59bd-dd0b-4253-89e5-aa9f784b64b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:26:01.023175 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:01.023074    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d6f59bd-dd0b-4253-89e5-aa9f784b64b0-home\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:26:01.024978 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:01.024954    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d6f59bd-dd0b-4253-89e5-aa9f784b64b0-dshm" (OuterVolumeSpecName: "dshm") pod "3d6f59bd-dd0b-4253-89e5-aa9f784b64b0" (UID: "3d6f59bd-dd0b-4253-89e5-aa9f784b64b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:26:01.025097 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:01.024982    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d6f59bd-dd0b-4253-89e5-aa9f784b64b0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3d6f59bd-dd0b-4253-89e5-aa9f784b64b0" (UID: "3d6f59bd-dd0b-4253-89e5-aa9f784b64b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:26:01.025097 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:01.024993    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d6f59bd-dd0b-4253-89e5-aa9f784b64b0-kube-api-access-fzxhs" (OuterVolumeSpecName: "kube-api-access-fzxhs") pod "3d6f59bd-dd0b-4253-89e5-aa9f784b64b0" (UID: "3d6f59bd-dd0b-4253-89e5-aa9f784b64b0"). InnerVolumeSpecName "kube-api-access-fzxhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:26:01.124133 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:01.124027    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d6f59bd-dd0b-4253-89e5-aa9f784b64b0-dshm\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:26:01.124133 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:01.124066    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d6f59bd-dd0b-4253-89e5-aa9f784b64b0-model-cache\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:26:01.124133 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:01.124080    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d6f59bd-dd0b-4253-89e5-aa9f784b64b0-tmp-dir\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:26:01.124133 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:01.124090    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d6f59bd-dd0b-4253-89e5-aa9f784b64b0-tls-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:26:01.124133 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:01.124102    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzxhs\" (UniqueName: \"kubernetes.io/projected/3d6f59bd-dd0b-4253-89e5-aa9f784b64b0-kube-api-access-fzxhs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:26:01.272801 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:01.272770    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wt8lb"]
Sep 16 19:26:01.275618 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:01.275591    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wt8lb"]
Sep 16 19:26:02.872509 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:02.872473    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d6f59bd-dd0b-4253-89e5-aa9f784b64b0" path="/var/lib/kubelet/pods/3d6f59bd-dd0b-4253-89e5-aa9f784b64b0/volumes"
Sep 16 19:26:20.311605 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:20.311570    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zzx4j"]
Sep 16 19:26:35.330508 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:35.330460    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zzx4j" podUID="06ca8aec-e256-4d21-a8b2-d8a7435755b1" containerName="main" containerID="cri-o://067bf28e27610018e878517f52c1833fd67c1c8325e42bbc2e165844b02b6973" gracePeriod=45
Sep 16 19:26:35.579788 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:35.579752    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zzx4j"
Sep 16 19:26:35.604465 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:35.604379    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06ca8aec-e256-4d21-a8b2-d8a7435755b1-dshm\") pod \"06ca8aec-e256-4d21-a8b2-d8a7435755b1\" (UID: \"06ca8aec-e256-4d21-a8b2-d8a7435755b1\") "
Sep 16 19:26:35.604465 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:35.604418    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vgvq6\" (UniqueName: \"kubernetes.io/projected/06ca8aec-e256-4d21-a8b2-d8a7435755b1-kube-api-access-vgvq6\") pod \"06ca8aec-e256-4d21-a8b2-d8a7435755b1\" (UID: \"06ca8aec-e256-4d21-a8b2-d8a7435755b1\") "
Sep 16 19:26:35.604465 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:35.604459    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06ca8aec-e256-4d21-a8b2-d8a7435755b1-home\") pod \"06ca8aec-e256-4d21-a8b2-d8a7435755b1\" (UID: \"06ca8aec-e256-4d21-a8b2-d8a7435755b1\") "
Sep 16 19:26:35.604747 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:35.604501    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06ca8aec-e256-4d21-a8b2-d8a7435755b1-model-cache\") pod \"06ca8aec-e256-4d21-a8b2-d8a7435755b1\" (UID: \"06ca8aec-e256-4d21-a8b2-d8a7435755b1\") "
Sep 16 19:26:35.604747 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:35.604527    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06ca8aec-e256-4d21-a8b2-d8a7435755b1-tmp-dir\") pod \"06ca8aec-e256-4d21-a8b2-d8a7435755b1\" (UID: \"06ca8aec-e256-4d21-a8b2-d8a7435755b1\") "
Sep 16 19:26:35.604747 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:35.604565    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06ca8aec-e256-4d21-a8b2-d8a7435755b1-tls-certs\") pod \"06ca8aec-e256-4d21-a8b2-d8a7435755b1\" (UID: \"06ca8aec-e256-4d21-a8b2-d8a7435755b1\") "
Sep 16 19:26:35.605232 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:35.605196    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06ca8aec-e256-4d21-a8b2-d8a7435755b1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "06ca8aec-e256-4d21-a8b2-d8a7435755b1" (UID: "06ca8aec-e256-4d21-a8b2-d8a7435755b1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:26:35.605232 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:35.605222    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06ca8aec-e256-4d21-a8b2-d8a7435755b1-model-cache" (OuterVolumeSpecName: "model-cache") pod "06ca8aec-e256-4d21-a8b2-d8a7435755b1" (UID: "06ca8aec-e256-4d21-a8b2-d8a7435755b1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:26:35.605433 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:35.605288    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06ca8aec-e256-4d21-a8b2-d8a7435755b1-home" (OuterVolumeSpecName: "home") pod "06ca8aec-e256-4d21-a8b2-d8a7435755b1" (UID: "06ca8aec-e256-4d21-a8b2-d8a7435755b1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:26:35.607174 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:35.607144    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06ca8aec-e256-4d21-a8b2-d8a7435755b1-dshm" (OuterVolumeSpecName: "dshm") pod "06ca8aec-e256-4d21-a8b2-d8a7435755b1" (UID: "06ca8aec-e256-4d21-a8b2-d8a7435755b1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:26:35.607613 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:35.607581    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06ca8aec-e256-4d21-a8b2-d8a7435755b1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "06ca8aec-e256-4d21-a8b2-d8a7435755b1" (UID: "06ca8aec-e256-4d21-a8b2-d8a7435755b1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:26:35.607705 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:35.607629    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06ca8aec-e256-4d21-a8b2-d8a7435755b1-kube-api-access-vgvq6" (OuterVolumeSpecName: "kube-api-access-vgvq6") pod "06ca8aec-e256-4d21-a8b2-d8a7435755b1" (UID: "06ca8aec-e256-4d21-a8b2-d8a7435755b1"). InnerVolumeSpecName "kube-api-access-vgvq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:26:35.705152 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:35.705098    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06ca8aec-e256-4d21-a8b2-d8a7435755b1-dshm\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:26:35.705152 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:35.705148    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgvq6\" (UniqueName: \"kubernetes.io/projected/06ca8aec-e256-4d21-a8b2-d8a7435755b1-kube-api-access-vgvq6\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:26:35.705152 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:35.705166    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06ca8aec-e256-4d21-a8b2-d8a7435755b1-home\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:26:35.705414 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:35.705180    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06ca8aec-e256-4d21-a8b2-d8a7435755b1-model-cache\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:26:35.705414 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:35.705216    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06ca8aec-e256-4d21-a8b2-d8a7435755b1-tmp-dir\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:26:35.705414 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:35.705229    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06ca8aec-e256-4d21-a8b2-d8a7435755b1-tls-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:26:36.094205 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:36.094165    2582 generic.go:356] "Generic (PLEG): container finished" podID="06ca8aec-e256-4d21-a8b2-d8a7435755b1" containerID="067bf28e27610018e878517f52c1833fd67c1c8325e42bbc2e165844b02b6973" exitCode=0
Sep 16 19:26:36.094403 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:36.094208    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zzx4j"
Sep 16 19:26:36.094403 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:36.094223    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zzx4j" event={"ID":"06ca8aec-e256-4d21-a8b2-d8a7435755b1","Type":"ContainerDied","Data":"067bf28e27610018e878517f52c1833fd67c1c8325e42bbc2e165844b02b6973"}
Sep 16 19:26:36.094403 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:36.094271    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zzx4j" event={"ID":"06ca8aec-e256-4d21-a8b2-d8a7435755b1","Type":"ContainerDied","Data":"ac87ee45f81471fd677fb95848542d5e91793ca3b7cd7c530ff0758919d77ca0"}
Sep 16 19:26:36.094403 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:36.094296    2582 scope.go:117] "RemoveContainer" containerID="067bf28e27610018e878517f52c1833fd67c1c8325e42bbc2e165844b02b6973"
Sep 16 19:26:36.102899 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:36.102879    2582 scope.go:117] "RemoveContainer" containerID="067bf28e27610018e878517f52c1833fd67c1c8325e42bbc2e165844b02b6973"
Sep 16 19:26:36.103154 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:26:36.103135    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"067bf28e27610018e878517f52c1833fd67c1c8325e42bbc2e165844b02b6973\": container with ID starting with 067bf28e27610018e878517f52c1833fd67c1c8325e42bbc2e165844b02b6973 not found: ID does not exist" containerID="067bf28e27610018e878517f52c1833fd67c1c8325e42bbc2e165844b02b6973"
Sep 16 19:26:36.103203 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:36.103163    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"067bf28e27610018e878517f52c1833fd67c1c8325e42bbc2e165844b02b6973"} err="failed to get container status \"067bf28e27610018e878517f52c1833fd67c1c8325e42bbc2e165844b02b6973\": rpc error: code = NotFound desc = could not find container \"067bf28e27610018e878517f52c1833fd67c1c8325e42bbc2e165844b02b6973\": container with ID starting with 067bf28e27610018e878517f52c1833fd67c1c8325e42bbc2e165844b02b6973 not found: ID does not exist"
Sep 16 19:26:36.132675 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:36.132645    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zzx4j"]
Sep 16 19:26:36.139995 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:36.139969    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zzx4j"]
Sep 16 19:26:36.872562 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:36.872528    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06ca8aec-e256-4d21-a8b2-d8a7435755b1" path="/var/lib/kubelet/pods/06ca8aec-e256-4d21-a8b2-d8a7435755b1/volumes"
Sep 16 19:26:58.807642 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:58.807606    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-9nd9h"]
Sep 16 19:26:58.808178 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:58.808074    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06ca8aec-e256-4d21-a8b2-d8a7435755b1" containerName="main"
Sep 16 19:26:58.808178 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:58.808095    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="06ca8aec-e256-4d21-a8b2-d8a7435755b1" containerName="main"
Sep 16 19:26:58.808178 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:58.808120    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d6f59bd-dd0b-4253-89e5-aa9f784b64b0" containerName="main"
Sep 16 19:26:58.808178 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:58.808129    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d6f59bd-dd0b-4253-89e5-aa9f784b64b0" containerName="main"
Sep 16 19:26:58.808390 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:58.808231    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="06ca8aec-e256-4d21-a8b2-d8a7435755b1" containerName="main"
Sep 16 19:26:58.808390 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:58.808243    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d6f59bd-dd0b-4253-89e5-aa9f784b64b0" containerName="main"
Sep 16 19:26:58.811180 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:58.811158    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-9nd9h"
Sep 16 19:26:58.814459 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:58.814433    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"openshift-service-ca.crt\""
Sep 16 19:26:58.815663 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:58.815635    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"kube-root-ca.crt\""
Sep 16 19:26:58.815830 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:58.815636    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"default-dockercfg-62x7b\""
Sep 16 19:26:58.815946 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:58.815932    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"delw-v2-kserve-self-signed-certs\""
Sep 16 19:26:58.824668 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:58.824646    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-9nd9h"]
Sep 16 19:26:58.889646 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:58.889603    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2170a01f-bae6-4ee8-b91b-33f6e465c42e-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-9nd9h\" (UID: \"2170a01f-bae6-4ee8-b91b-33f6e465c42e\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-9nd9h"
Sep 16 19:26:58.889878 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:58.889664    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2170a01f-bae6-4ee8-b91b-33f6e465c42e-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-9nd9h\" (UID: \"2170a01f-bae6-4ee8-b91b-33f6e465c42e\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-9nd9h"
Sep 16 19:26:58.889878 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:58.889701    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2170a01f-bae6-4ee8-b91b-33f6e465c42e-home\") pod \"delw-v2-kserve-5b84cc7ddf-9nd9h\" (UID: \"2170a01f-bae6-4ee8-b91b-33f6e465c42e\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-9nd9h"
Sep 16 19:26:58.889878 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:58.889749    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2170a01f-bae6-4ee8-b91b-33f6e465c42e-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-9nd9h\" (UID: \"2170a01f-bae6-4ee8-b91b-33f6e465c42e\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-9nd9h"
Sep 16 19:26:58.889878 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:58.889815    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgv8m\" (UniqueName: \"kubernetes.io/projected/2170a01f-bae6-4ee8-b91b-33f6e465c42e-kube-api-access-kgv8m\") pod \"delw-v2-kserve-5b84cc7ddf-9nd9h\" (UID: \"2170a01f-bae6-4ee8-b91b-33f6e465c42e\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-9nd9h"
Sep 16 19:26:58.889878 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:58.889865    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2170a01f-bae6-4ee8-b91b-33f6e465c42e-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-9nd9h\" (UID: \"2170a01f-bae6-4ee8-b91b-33f6e465c42e\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-9nd9h"
Sep 16 19:26:58.990954 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:58.990910    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2170a01f-bae6-4ee8-b91b-33f6e465c42e-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-9nd9h\" (UID: \"2170a01f-bae6-4ee8-b91b-33f6e465c42e\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-9nd9h"
Sep 16 19:26:58.990954 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:58.990960    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2170a01f-bae6-4ee8-b91b-33f6e465c42e-home\") pod \"delw-v2-kserve-5b84cc7ddf-9nd9h\" (UID: \"2170a01f-bae6-4ee8-b91b-33f6e465c42e\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-9nd9h"
Sep 16 19:26:58.991214 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:58.991087    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2170a01f-bae6-4ee8-b91b-33f6e465c42e-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-9nd9h\" (UID: \"2170a01f-bae6-4ee8-b91b-33f6e465c42e\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-9nd9h"
Sep 16 19:26:58.991214 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:58.991179    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgv8m\" (UniqueName: \"kubernetes.io/projected/2170a01f-bae6-4ee8-b91b-33f6e465c42e-kube-api-access-kgv8m\") pod \"delw-v2-kserve-5b84cc7ddf-9nd9h\" (UID: \"2170a01f-bae6-4ee8-b91b-33f6e465c42e\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-9nd9h"
Sep 16 19:26:58.991335 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:58.991223    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2170a01f-bae6-4ee8-b91b-33f6e465c42e-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-9nd9h\" (UID: \"2170a01f-bae6-4ee8-b91b-33f6e465c42e\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-9nd9h"
Sep 16 19:26:58.991335 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:58.991285    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2170a01f-bae6-4ee8-b91b-33f6e465c42e-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-9nd9h\" (UID: \"2170a01f-bae6-4ee8-b91b-33f6e465c42e\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-9nd9h"
Sep 16 19:26:58.991442 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:58.991397    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2170a01f-bae6-4ee8-b91b-33f6e465c42e-home\") pod \"delw-v2-kserve-5b84cc7ddf-9nd9h\" (UID: \"2170a01f-bae6-4ee8-b91b-33f6e465c42e\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-9nd9h"
Sep 16 19:26:58.991483 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:58.991461    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2170a01f-bae6-4ee8-b91b-33f6e465c42e-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-9nd9h\" (UID: \"2170a01f-bae6-4ee8-b91b-33f6e465c42e\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-9nd9h"
Sep 16 19:26:58.991635 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:58.991613    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2170a01f-bae6-4ee8-b91b-33f6e465c42e-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-9nd9h\" (UID: \"2170a01f-bae6-4ee8-b91b-33f6e465c42e\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-9nd9h"
Sep 16 19:26:58.993430 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:58.993400    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2170a01f-bae6-4ee8-b91b-33f6e465c42e-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-9nd9h\" (UID: \"2170a01f-bae6-4ee8-b91b-33f6e465c42e\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-9nd9h"
Sep 16 19:26:58.993638 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:58.993620    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2170a01f-bae6-4ee8-b91b-33f6e465c42e-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-9nd9h\" (UID: \"2170a01f-bae6-4ee8-b91b-33f6e465c42e\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-9nd9h"
Sep 16 19:26:58.999893 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:58.999871    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgv8m\" (UniqueName: \"kubernetes.io/projected/2170a01f-bae6-4ee8-b91b-33f6e465c42e-kube-api-access-kgv8m\") pod \"delw-v2-kserve-5b84cc7ddf-9nd9h\" (UID: \"2170a01f-bae6-4ee8-b91b-33f6e465c42e\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-9nd9h"
Sep 16 19:26:59.122606 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:59.122496    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-9nd9h"
Sep 16 19:26:59.399879 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:26:59.399854    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-9nd9h"]
Sep 16 19:26:59.401876 ip-10-0-129-194 kubenswrapper[2582]: W0916 19:26:59.401848    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2170a01f_bae6_4ee8_b91b_33f6e465c42e.slice/crio-845050d080955661148c953f9b3bfdeb444da52aa76282873912826914487ed7 WatchSource:0}: Error finding container 845050d080955661148c953f9b3bfdeb444da52aa76282873912826914487ed7: Status 404 returned error can't find the container with id 845050d080955661148c953f9b3bfdeb444da52aa76282873912826914487ed7
Sep 16 19:27:00.198463 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:27:00.198426    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-9nd9h" event={"ID":"2170a01f-bae6-4ee8-b91b-33f6e465c42e","Type":"ContainerStarted","Data":"3bce245259da5fc8e8b4d6acf9a167eabde4ce2725925a6bb0e12c53f118e315"}
Sep 16 19:27:00.198878 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:27:00.198479    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-9nd9h" event={"ID":"2170a01f-bae6-4ee8-b91b-33f6e465c42e","Type":"ContainerStarted","Data":"845050d080955661148c953f9b3bfdeb444da52aa76282873912826914487ed7"}
Sep 16 19:27:00.220874 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:27:00.220816    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-9nd9h" podStartSLOduration=2.220796412 podStartE2EDuration="2.220796412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:26:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:27:00.219713551 +0000 UTC m=+2623.861429211" watchObservedRunningTime="2026-09-16 19:27:00.220796412 +0000 UTC m=+2623.862512113"
Sep 16 19:27:01.123334 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:27:01.123293    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-9nd9h"
Sep 16 19:27:09.123001 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:27:09.122962    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-9nd9h"
Sep 16 19:27:09.135644 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:27:09.135618    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-9nd9h"
Sep 16 19:27:09.248832 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:27:09.248794    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-9nd9h"
Sep 16 19:27:52.895595 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:27:52.895562    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-9nd9h"]
Sep 16 19:28:07.914472 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:07.914409    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-9nd9h" podUID="2170a01f-bae6-4ee8-b91b-33f6e465c42e" containerName="main" containerID="cri-o://3bce245259da5fc8e8b4d6acf9a167eabde4ce2725925a6bb0e12c53f118e315" gracePeriod=45
Sep 16 19:28:08.162582 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:08.162553    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-9nd9h"
Sep 16 19:28:08.275307 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:08.275271    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2170a01f-bae6-4ee8-b91b-33f6e465c42e-home\") pod \"2170a01f-bae6-4ee8-b91b-33f6e465c42e\" (UID: \"2170a01f-bae6-4ee8-b91b-33f6e465c42e\") "
Sep 16 19:28:08.275513 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:08.275324    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2170a01f-bae6-4ee8-b91b-33f6e465c42e-dshm\") pod \"2170a01f-bae6-4ee8-b91b-33f6e465c42e\" (UID: \"2170a01f-bae6-4ee8-b91b-33f6e465c42e\") "
Sep 16 19:28:08.275513 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:08.275351    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kgv8m\" (UniqueName: \"kubernetes.io/projected/2170a01f-bae6-4ee8-b91b-33f6e465c42e-kube-api-access-kgv8m\") pod \"2170a01f-bae6-4ee8-b91b-33f6e465c42e\" (UID: \"2170a01f-bae6-4ee8-b91b-33f6e465c42e\") "
Sep 16 19:28:08.275513 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:08.275389    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2170a01f-bae6-4ee8-b91b-33f6e465c42e-tls-certs\") pod \"2170a01f-bae6-4ee8-b91b-33f6e465c42e\" (UID: \"2170a01f-bae6-4ee8-b91b-33f6e465c42e\") "
Sep 16 19:28:08.275513 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:08.275412    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2170a01f-bae6-4ee8-b91b-33f6e465c42e-tmp-dir\") pod \"2170a01f-bae6-4ee8-b91b-33f6e465c42e\" (UID: \"2170a01f-bae6-4ee8-b91b-33f6e465c42e\") "
Sep 16 19:28:08.275513 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:08.275440    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2170a01f-bae6-4ee8-b91b-33f6e465c42e-model-cache\") pod \"2170a01f-bae6-4ee8-b91b-33f6e465c42e\" (UID: \"2170a01f-bae6-4ee8-b91b-33f6e465c42e\") "
Sep 16 19:28:08.275847 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:08.275619    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2170a01f-bae6-4ee8-b91b-33f6e465c42e-home" (OuterVolumeSpecName: "home") pod "2170a01f-bae6-4ee8-b91b-33f6e465c42e" (UID: "2170a01f-bae6-4ee8-b91b-33f6e465c42e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:28:08.275908 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:08.275849    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2170a01f-bae6-4ee8-b91b-33f6e465c42e-model-cache" (OuterVolumeSpecName: "model-cache") pod "2170a01f-bae6-4ee8-b91b-33f6e465c42e" (UID: "2170a01f-bae6-4ee8-b91b-33f6e465c42e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:28:08.275908 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:08.275864    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2170a01f-bae6-4ee8-b91b-33f6e465c42e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2170a01f-bae6-4ee8-b91b-33f6e465c42e" (UID: "2170a01f-bae6-4ee8-b91b-33f6e465c42e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:28:08.277678 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:08.277655    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2170a01f-bae6-4ee8-b91b-33f6e465c42e-dshm" (OuterVolumeSpecName: "dshm") pod "2170a01f-bae6-4ee8-b91b-33f6e465c42e" (UID: "2170a01f-bae6-4ee8-b91b-33f6e465c42e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:28:08.277800 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:08.277705    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2170a01f-bae6-4ee8-b91b-33f6e465c42e-kube-api-access-kgv8m" (OuterVolumeSpecName: "kube-api-access-kgv8m") pod "2170a01f-bae6-4ee8-b91b-33f6e465c42e" (UID: "2170a01f-bae6-4ee8-b91b-33f6e465c42e"). InnerVolumeSpecName "kube-api-access-kgv8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:28:08.277800 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:08.277731    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2170a01f-bae6-4ee8-b91b-33f6e465c42e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2170a01f-bae6-4ee8-b91b-33f6e465c42e" (UID: "2170a01f-bae6-4ee8-b91b-33f6e465c42e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:28:08.376324 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:08.376283    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2170a01f-bae6-4ee8-b91b-33f6e465c42e-tmp-dir\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:28:08.376324 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:08.376320    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2170a01f-bae6-4ee8-b91b-33f6e465c42e-model-cache\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:28:08.376324 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:08.376330    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2170a01f-bae6-4ee8-b91b-33f6e465c42e-home\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:28:08.376324 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:08.376337    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2170a01f-bae6-4ee8-b91b-33f6e465c42e-dshm\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:28:08.376604 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:08.376346    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgv8m\" (UniqueName: \"kubernetes.io/projected/2170a01f-bae6-4ee8-b91b-33f6e465c42e-kube-api-access-kgv8m\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:28:08.376604 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:08.376354    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2170a01f-bae6-4ee8-b91b-33f6e465c42e-tls-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:28:08.500164 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:08.500126    2582 generic.go:356] "Generic (PLEG): container finished" podID="2170a01f-bae6-4ee8-b91b-33f6e465c42e" containerID="3bce245259da5fc8e8b4d6acf9a167eabde4ce2725925a6bb0e12c53f118e315" exitCode=0
Sep 16 19:28:08.500314 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:08.500176    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-9nd9h"
Sep 16 19:28:08.500314 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:08.500208    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-9nd9h" event={"ID":"2170a01f-bae6-4ee8-b91b-33f6e465c42e","Type":"ContainerDied","Data":"3bce245259da5fc8e8b4d6acf9a167eabde4ce2725925a6bb0e12c53f118e315"}
Sep 16 19:28:08.500314 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:08.500262    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-9nd9h" event={"ID":"2170a01f-bae6-4ee8-b91b-33f6e465c42e","Type":"ContainerDied","Data":"845050d080955661148c953f9b3bfdeb444da52aa76282873912826914487ed7"}
Sep 16 19:28:08.500314 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:08.500284    2582 scope.go:117] "RemoveContainer" containerID="3bce245259da5fc8e8b4d6acf9a167eabde4ce2725925a6bb0e12c53f118e315"
Sep 16 19:28:08.508897 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:08.508879    2582 scope.go:117] "RemoveContainer" containerID="3bce245259da5fc8e8b4d6acf9a167eabde4ce2725925a6bb0e12c53f118e315"
Sep 16 19:28:08.509162 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:28:08.509142    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bce245259da5fc8e8b4d6acf9a167eabde4ce2725925a6bb0e12c53f118e315\": container with ID starting with 3bce245259da5fc8e8b4d6acf9a167eabde4ce2725925a6bb0e12c53f118e315 not found: ID does not exist" containerID="3bce245259da5fc8e8b4d6acf9a167eabde4ce2725925a6bb0e12c53f118e315"
Sep 16 19:28:08.509225 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:08.509169    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bce245259da5fc8e8b4d6acf9a167eabde4ce2725925a6bb0e12c53f118e315"} err="failed to get container status \"3bce245259da5fc8e8b4d6acf9a167eabde4ce2725925a6bb0e12c53f118e315\": rpc error: code = NotFound desc = could not find container \"3bce245259da5fc8e8b4d6acf9a167eabde4ce2725925a6bb0e12c53f118e315\": container with ID starting with 3bce245259da5fc8e8b4d6acf9a167eabde4ce2725925a6bb0e12c53f118e315 not found: ID does not exist"
Sep 16 19:28:08.524297 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:08.524271    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-9nd9h"]
Sep 16 19:28:08.528961 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:08.528901    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-9nd9h"]
Sep 16 19:28:08.872390 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:08.872306    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2170a01f-bae6-4ee8-b91b-33f6e465c42e" path="/var/lib/kubelet/pods/2170a01f-bae6-4ee8-b91b-33f6e465c42e/volumes"
Sep 16 19:28:17.003910 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:17.003877    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7qwl2_b6695f86-08c0-4836-87b1-f2a9d76084bd/console-operator/1.log"
Sep 16 19:28:17.010557 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:17.010530    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7qwl2_b6695f86-08c0-4836-87b1-f2a9d76084bd/console-operator/1.log"
Sep 16 19:28:26.178760 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:26.178706    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27hpt"]
Sep 16 19:28:26.179161 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:26.179082    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2170a01f-bae6-4ee8-b91b-33f6e465c42e" containerName="main"
Sep 16 19:28:26.179161 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:26.179095    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2170a01f-bae6-4ee8-b91b-33f6e465c42e" containerName="main"
Sep 16 19:28:26.179161 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:26.179154    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2170a01f-bae6-4ee8-b91b-33f6e465c42e" containerName="main"
Sep 16 19:28:26.181959 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:26.181939    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27hpt"
Sep 16 19:28:26.189327 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:26.189303    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"roll-v1-kserve-self-signed-certs\""
Sep 16 19:28:26.189327 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:26.189318    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"default-dockercfg-mmtkh\""
Sep 16 19:28:26.189493 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:26.189302    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"openshift-service-ca.crt\""
Sep 16 19:28:26.189493 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:26.189302    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"kube-root-ca.crt\""
Sep 16 19:28:26.209502 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:26.209471    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27hpt"]
Sep 16 19:28:26.323959 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:26.323920    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85f990bd-5573-47e8-bb26-58ede0d2e38d-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-27hpt\" (UID: \"85f990bd-5573-47e8-bb26-58ede0d2e38d\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27hpt"
Sep 16 19:28:26.323959 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:26.323959    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85f990bd-5573-47e8-bb26-58ede0d2e38d-home\") pod \"roll-v1-kserve-69cf8bcfc-27hpt\" (UID: \"85f990bd-5573-47e8-bb26-58ede0d2e38d\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27hpt"
Sep 16 19:28:26.324166 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:26.323987    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85f990bd-5573-47e8-bb26-58ede0d2e38d-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-27hpt\" (UID: \"85f990bd-5573-47e8-bb26-58ede0d2e38d\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27hpt"
Sep 16 19:28:26.324166 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:26.324049    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz4s5\" (UniqueName: \"kubernetes.io/projected/85f990bd-5573-47e8-bb26-58ede0d2e38d-kube-api-access-jz4s5\") pod \"roll-v1-kserve-69cf8bcfc-27hpt\" (UID: \"85f990bd-5573-47e8-bb26-58ede0d2e38d\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27hpt"
Sep 16 19:28:26.324166 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:26.324127    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85f990bd-5573-47e8-bb26-58ede0d2e38d-dshm\") pod \"roll-v1-kserve-69cf8bcfc-27hpt\" (UID: \"85f990bd-5573-47e8-bb26-58ede0d2e38d\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27hpt"
Sep 16 19:28:26.324166 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:26.324152    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85f990bd-5573-47e8-bb26-58ede0d2e38d-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-27hpt\" (UID: \"85f990bd-5573-47e8-bb26-58ede0d2e38d\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27hpt"
Sep 16 19:28:26.425405 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:26.425365    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85f990bd-5573-47e8-bb26-58ede0d2e38d-dshm\") pod \"roll-v1-kserve-69cf8bcfc-27hpt\" (UID: \"85f990bd-5573-47e8-bb26-58ede0d2e38d\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27hpt"
Sep 16 19:28:26.425595 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:26.425413    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85f990bd-5573-47e8-bb26-58ede0d2e38d-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-27hpt\" (UID: \"85f990bd-5573-47e8-bb26-58ede0d2e38d\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27hpt"
Sep 16 19:28:26.425595 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:26.425455    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85f990bd-5573-47e8-bb26-58ede0d2e38d-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-27hpt\" (UID: \"85f990bd-5573-47e8-bb26-58ede0d2e38d\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27hpt"
Sep 16 19:28:26.425595 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:26.425481    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85f990bd-5573-47e8-bb26-58ede0d2e38d-home\") pod \"roll-v1-kserve-69cf8bcfc-27hpt\" (UID: \"85f990bd-5573-47e8-bb26-58ede0d2e38d\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27hpt"
Sep 16 19:28:26.425595 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:26.425520    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85f990bd-5573-47e8-bb26-58ede0d2e38d-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-27hpt\" (UID: \"85f990bd-5573-47e8-bb26-58ede0d2e38d\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27hpt"
Sep 16 19:28:26.425595 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:26.425551    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz4s5\" (UniqueName: \"kubernetes.io/projected/85f990bd-5573-47e8-bb26-58ede0d2e38d-kube-api-access-jz4s5\") pod \"roll-v1-kserve-69cf8bcfc-27hpt\" (UID: \"85f990bd-5573-47e8-bb26-58ede0d2e38d\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27hpt"
Sep 16 19:28:26.425916 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:26.425866    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85f990bd-5573-47e8-bb26-58ede0d2e38d-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-27hpt\" (UID: \"85f990bd-5573-47e8-bb26-58ede0d2e38d\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27hpt"
Sep 16 19:28:26.425964 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:26.425926    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85f990bd-5573-47e8-bb26-58ede0d2e38d-home\") pod \"roll-v1-kserve-69cf8bcfc-27hpt\" (UID: \"85f990bd-5573-47e8-bb26-58ede0d2e38d\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27hpt"
Sep 16 19:28:26.425964 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:26.425950    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85f990bd-5573-47e8-bb26-58ede0d2e38d-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-27hpt\" (UID: \"85f990bd-5573-47e8-bb26-58ede0d2e38d\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27hpt"
Sep 16 19:28:26.427857 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:26.427829    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85f990bd-5573-47e8-bb26-58ede0d2e38d-dshm\") pod \"roll-v1-kserve-69cf8bcfc-27hpt\" (UID: \"85f990bd-5573-47e8-bb26-58ede0d2e38d\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27hpt"
Sep 16 19:28:26.428016 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:26.427997    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85f990bd-5573-47e8-bb26-58ede0d2e38d-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-27hpt\" (UID: \"85f990bd-5573-47e8-bb26-58ede0d2e38d\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27hpt"
Sep 16 19:28:26.434532 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:26.434472    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz4s5\" (UniqueName: \"kubernetes.io/projected/85f990bd-5573-47e8-bb26-58ede0d2e38d-kube-api-access-jz4s5\") pod \"roll-v1-kserve-69cf8bcfc-27hpt\" (UID: \"85f990bd-5573-47e8-bb26-58ede0d2e38d\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27hpt"
Sep 16 19:28:26.491527 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:26.491478    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27hpt"
Sep 16 19:28:26.769571 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:26.769545    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27hpt"]
Sep 16 19:28:26.772155 ip-10-0-129-194 kubenswrapper[2582]: W0916 19:28:26.772127    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85f990bd_5573_47e8_bb26_58ede0d2e38d.slice/crio-c9370671fc305b7c8b44322e478157387af3789f5c2d01c8eac26cca8b14c900 WatchSource:0}: Error finding container c9370671fc305b7c8b44322e478157387af3789f5c2d01c8eac26cca8b14c900: Status 404 returned error can't find the container with id c9370671fc305b7c8b44322e478157387af3789f5c2d01c8eac26cca8b14c900
Sep 16 19:28:27.587266 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:27.587226    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27hpt" event={"ID":"85f990bd-5573-47e8-bb26-58ede0d2e38d","Type":"ContainerStarted","Data":"74aa5c01bf39491054fba27ce59c326c0ee55bdb1e27b4c0606946cf2f4f416c"}
Sep 16 19:28:27.587732 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:27.587291    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27hpt" event={"ID":"85f990bd-5573-47e8-bb26-58ede0d2e38d","Type":"ContainerStarted","Data":"c9370671fc305b7c8b44322e478157387af3789f5c2d01c8eac26cca8b14c900"}
Sep 16 19:28:27.609807 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:27.609757    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27hpt" podStartSLOduration=1.609737157 podStartE2EDuration="1.609737157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:28:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:28:27.60736286 +0000 UTC m=+2711.249078536" watchObservedRunningTime="2026-09-16 19:28:27.609737157 +0000 UTC m=+2711.251452814"
Sep 16 19:28:28.492088 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:28.492054    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27hpt"
Sep 16 19:28:36.492637 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:36.492594    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27hpt"
Sep 16 19:28:36.505482 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:36.505453    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27hpt"
Sep 16 19:28:36.634194 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:36.634157    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27hpt"
Sep 16 19:28:42.601049 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:42.601011    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-56wxp"]
Sep 16 19:28:42.604372 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:42.604349    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-56wxp"
Sep 16 19:28:42.607106 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:42.607083    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"roll-v2-kserve-self-signed-certs\""
Sep 16 19:28:42.614808 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:42.614781    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-56wxp"]
Sep 16 19:28:42.656366 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:42.656334    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67d1d96b-13cc-4343-af92-a64272ba9182-dshm\") pod \"roll-v2-kserve-d4dbf9d67-56wxp\" (UID: \"67d1d96b-13cc-4343-af92-a64272ba9182\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-56wxp"
Sep 16 19:28:42.656541 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:42.656373    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67d1d96b-13cc-4343-af92-a64272ba9182-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-56wxp\" (UID: \"67d1d96b-13cc-4343-af92-a64272ba9182\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-56wxp"
Sep 16 19:28:42.656541 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:42.656398    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67d1d96b-13cc-4343-af92-a64272ba9182-home\") pod \"roll-v2-kserve-d4dbf9d67-56wxp\" (UID: \"67d1d96b-13cc-4343-af92-a64272ba9182\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-56wxp"
Sep 16 19:28:42.656541 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:42.656481    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67d1d96b-13cc-4343-af92-a64272ba9182-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-56wxp\" (UID: \"67d1d96b-13cc-4343-af92-a64272ba9182\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-56wxp"
Sep 16 19:28:42.656541 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:42.656510    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7gw4\" (UniqueName: \"kubernetes.io/projected/67d1d96b-13cc-4343-af92-a64272ba9182-kube-api-access-p7gw4\") pod \"roll-v2-kserve-d4dbf9d67-56wxp\" (UID: \"67d1d96b-13cc-4343-af92-a64272ba9182\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-56wxp"
Sep 16 19:28:42.656541 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:42.656537    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67d1d96b-13cc-4343-af92-a64272ba9182-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-56wxp\" (UID: \"67d1d96b-13cc-4343-af92-a64272ba9182\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-56wxp"
Sep 16 19:28:42.756990 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:42.756945    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67d1d96b-13cc-4343-af92-a64272ba9182-dshm\") pod \"roll-v2-kserve-d4dbf9d67-56wxp\" (UID: \"67d1d96b-13cc-4343-af92-a64272ba9182\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-56wxp"
Sep 16 19:28:42.756990 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:42.756998    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67d1d96b-13cc-4343-af92-a64272ba9182-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-56wxp\" (UID: \"67d1d96b-13cc-4343-af92-a64272ba9182\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-56wxp"
Sep 16 19:28:42.757265 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:42.757024    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67d1d96b-13cc-4343-af92-a64272ba9182-home\") pod \"roll-v2-kserve-d4dbf9d67-56wxp\" (UID: \"67d1d96b-13cc-4343-af92-a64272ba9182\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-56wxp"
Sep 16 19:28:42.757265 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:42.757058    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67d1d96b-13cc-4343-af92-a64272ba9182-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-56wxp\" (UID: \"67d1d96b-13cc-4343-af92-a64272ba9182\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-56wxp"
Sep 16 19:28:42.757265 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:42.757075    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7gw4\" (UniqueName: \"kubernetes.io/projected/67d1d96b-13cc-4343-af92-a64272ba9182-kube-api-access-p7gw4\") pod \"roll-v2-kserve-d4dbf9d67-56wxp\" (UID: \"67d1d96b-13cc-4343-af92-a64272ba9182\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-56wxp"
Sep 16 19:28:42.757265 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:42.757094    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67d1d96b-13cc-4343-af92-a64272ba9182-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-56wxp\" (UID: \"67d1d96b-13cc-4343-af92-a64272ba9182\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-56wxp"
Sep 16 19:28:42.757474 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:42.757449    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67d1d96b-13cc-4343-af92-a64272ba9182-home\") pod \"roll-v2-kserve-d4dbf9d67-56wxp\" (UID: \"67d1d96b-13cc-4343-af92-a64272ba9182\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-56wxp"
Sep 16 19:28:42.757515 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:42.757498    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67d1d96b-13cc-4343-af92-a64272ba9182-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-56wxp\" (UID: \"67d1d96b-13cc-4343-af92-a64272ba9182\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-56wxp"
Sep 16 19:28:42.757578 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:42.757529    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67d1d96b-13cc-4343-af92-a64272ba9182-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-56wxp\" (UID: \"67d1d96b-13cc-4343-af92-a64272ba9182\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-56wxp"
Sep 16 19:28:42.759328 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:42.759309    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67d1d96b-13cc-4343-af92-a64272ba9182-dshm\") pod \"roll-v2-kserve-d4dbf9d67-56wxp\" (UID: \"67d1d96b-13cc-4343-af92-a64272ba9182\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-56wxp"
Sep 16 19:28:42.759619 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:42.759601    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67d1d96b-13cc-4343-af92-a64272ba9182-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-56wxp\" (UID: \"67d1d96b-13cc-4343-af92-a64272ba9182\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-56wxp"
Sep 16 19:28:42.765633 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:42.765608    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7gw4\" (UniqueName: \"kubernetes.io/projected/67d1d96b-13cc-4343-af92-a64272ba9182-kube-api-access-p7gw4\") pod \"roll-v2-kserve-d4dbf9d67-56wxp\" (UID: \"67d1d96b-13cc-4343-af92-a64272ba9182\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-56wxp"
Sep 16 19:28:42.915446 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:42.915357    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-56wxp"
Sep 16 19:28:43.186967 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:43.186938    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-56wxp"]
Sep 16 19:28:43.189519 ip-10-0-129-194 kubenswrapper[2582]: W0916 19:28:43.189494    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67d1d96b_13cc_4343_af92_a64272ba9182.slice/crio-2821beb31e0dacda518b0e748ffc4b8ecc3464c2c129a435e8eccd592b7c9b5a WatchSource:0}: Error finding container 2821beb31e0dacda518b0e748ffc4b8ecc3464c2c129a435e8eccd592b7c9b5a: Status 404 returned error can't find the container with id 2821beb31e0dacda518b0e748ffc4b8ecc3464c2c129a435e8eccd592b7c9b5a
Sep 16 19:28:43.654767 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:43.654705    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-56wxp" event={"ID":"67d1d96b-13cc-4343-af92-a64272ba9182","Type":"ContainerStarted","Data":"41aae99ff80c9bf1c95b29388fd4ab08f3dd2cee0c5d6bb85370683ba1f39166"}
Sep 16 19:28:43.655249 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:43.654776    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-56wxp" event={"ID":"67d1d96b-13cc-4343-af92-a64272ba9182","Type":"ContainerStarted","Data":"2821beb31e0dacda518b0e748ffc4b8ecc3464c2c129a435e8eccd592b7c9b5a"}
Sep 16 19:28:43.676054 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:43.675998    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-56wxp" podStartSLOduration=1.675979569 podStartE2EDuration="1.675979569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:28:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:28:43.675228687 +0000 UTC m=+2727.316944352" watchObservedRunningTime="2026-09-16 19:28:43.675979569 +0000 UTC m=+2727.317695270"
Sep 16 19:28:43.915826 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:43.915734    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-56wxp"
Sep 16 19:28:52.915959 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:52.915919    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-56wxp"
Sep 16 19:28:52.928551 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:52.928515    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-56wxp"
Sep 16 19:28:53.709472 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:28:53.709443    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-56wxp"
Sep 16 19:29:45.199514 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:29:45.199436    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5lp6l"]
Sep 16 19:29:45.202633 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:29:45.202615    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5lp6l"
Sep 16 19:29:45.206342 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:29:45.206307    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 16 19:29:45.207238 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:29:45.207213    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Sep 16 19:29:45.207366 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:29:45.207213    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 16 19:29:45.207366 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:29:45.207260    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-nj5jj\""
Sep 16 19:29:45.218873 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:29:45.218849    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5lp6l"]
Sep 16 19:29:45.284787 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:29:45.284696    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8487bf88-614c-4685-8157-260db24d45ee-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5lp6l\" (UID: \"8487bf88-614c-4685-8157-260db24d45ee\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5lp6l"
Sep 16 19:29:45.284963 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:29:45.284817    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8487bf88-614c-4685-8157-260db24d45ee-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5lp6l\" (UID: \"8487bf88-614c-4685-8157-260db24d45ee\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5lp6l"
Sep 16 19:29:45.284963 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:29:45.284841    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8487bf88-614c-4685-8157-260db24d45ee-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5lp6l\" (UID: \"8487bf88-614c-4685-8157-260db24d45ee\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5lp6l"
Sep 16 19:29:45.284963 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:29:45.284869    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bdcl\" (UniqueName: \"kubernetes.io/projected/8487bf88-614c-4685-8157-260db24d45ee-kube-api-access-5bdcl\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5lp6l\" (UID: \"8487bf88-614c-4685-8157-260db24d45ee\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5lp6l"
Sep 16 19:29:45.285078 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:29:45.284968    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8487bf88-614c-4685-8157-260db24d45ee-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5lp6l\" (UID: \"8487bf88-614c-4685-8157-260db24d45ee\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5lp6l"
Sep 16 19:29:45.285078 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:29:45.285020    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8487bf88-614c-4685-8157-260db24d45ee-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5lp6l\" (UID: \"8487bf88-614c-4685-8157-260db24d45ee\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5lp6l"
Sep 16 19:29:45.386234 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:29:45.386186    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8487bf88-614c-4685-8157-260db24d45ee-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5lp6l\" (UID: \"8487bf88-614c-4685-8157-260db24d45ee\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5lp6l"
Sep 16 19:29:45.386431 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:29:45.386253    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8487bf88-614c-4685-8157-260db24d45ee-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5lp6l\" (UID: \"8487bf88-614c-4685-8157-260db24d45ee\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5lp6l"
Sep 16 19:29:45.386431 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:29:45.386286    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8487bf88-614c-4685-8157-260db24d45ee-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5lp6l\" (UID: \"8487bf88-614c-4685-8157-260db24d45ee\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5lp6l"
Sep 16 19:29:45.386431 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:29:45.386304    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8487bf88-614c-4685-8157-260db24d45ee-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5lp6l\" (UID: \"8487bf88-614c-4685-8157-260db24d45ee\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5lp6l"
Sep 16 19:29:45.386431 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:29:45.386327    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bdcl\" (UniqueName: \"kubernetes.io/projected/8487bf88-614c-4685-8157-260db24d45ee-kube-api-access-5bdcl\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5lp6l\" (UID: \"8487bf88-614c-4685-8157-260db24d45ee\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5lp6l"
Sep 16 19:29:45.386431 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:29:45.386370    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8487bf88-614c-4685-8157-260db24d45ee-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5lp6l\" (UID: \"8487bf88-614c-4685-8157-260db24d45ee\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5lp6l"
Sep 16 19:29:45.386702 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:29:45.386647    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8487bf88-614c-4685-8157-260db24d45ee-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5lp6l\" (UID: \"8487bf88-614c-4685-8157-260db24d45ee\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5lp6l"
Sep 16 19:29:45.386839 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:29:45.386815    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8487bf88-614c-4685-8157-260db24d45ee-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5lp6l\" (UID: \"8487bf88-614c-4685-8157-260db24d45ee\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5lp6l"
Sep 16 19:29:45.386957 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:29:45.386849    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8487bf88-614c-4685-8157-260db24d45ee-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5lp6l\" (UID: \"8487bf88-614c-4685-8157-260db24d45ee\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5lp6l"
Sep 16 19:29:45.388670 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:29:45.388631    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8487bf88-614c-4685-8157-260db24d45ee-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5lp6l\" (UID: \"8487bf88-614c-4685-8157-260db24d45ee\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5lp6l"
Sep 16 19:29:45.388846 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:29:45.388830    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8487bf88-614c-4685-8157-260db24d45ee-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5lp6l\" (UID: \"8487bf88-614c-4685-8157-260db24d45ee\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5lp6l"
Sep 16 19:29:45.396490 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:29:45.396471    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bdcl\" (UniqueName: \"kubernetes.io/projected/8487bf88-614c-4685-8157-260db24d45ee-kube-api-access-5bdcl\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5lp6l\" (UID: \"8487bf88-614c-4685-8157-260db24d45ee\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5lp6l"
Sep 16 19:29:45.513235 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:29:45.513199    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5lp6l"
Sep 16 19:29:45.787671 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:29:45.787639    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5lp6l"]
Sep 16 19:29:45.790477 ip-10-0-129-194 kubenswrapper[2582]: W0916 19:29:45.790455    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8487bf88_614c_4685_8157_260db24d45ee.slice/crio-9cccfba7ccfe8c492b0f0b3c451b8f0dcd22f136b270576609aa8880d27919da WatchSource:0}: Error finding container 9cccfba7ccfe8c492b0f0b3c451b8f0dcd22f136b270576609aa8880d27919da: Status 404 returned error can't find the container with id 9cccfba7ccfe8c492b0f0b3c451b8f0dcd22f136b270576609aa8880d27919da
Sep 16 19:29:45.925853 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:29:45.925810    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5lp6l" event={"ID":"8487bf88-614c-4685-8157-260db24d45ee","Type":"ContainerStarted","Data":"48651017246edc522d3b1bc98ccaad3ca51badc3e4160deb6e20ff07c718c0b4"}
Sep 16 19:29:45.926034 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:29:45.925871    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5lp6l" event={"ID":"8487bf88-614c-4685-8157-260db24d45ee","Type":"ContainerStarted","Data":"9cccfba7ccfe8c492b0f0b3c451b8f0dcd22f136b270576609aa8880d27919da"}
Sep 16 19:29:45.943896 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:29:45.943840    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5lp6l" podStartSLOduration=0.943823888 podStartE2EDuration="943.823888ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:29:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:29:45.942189839 +0000 UTC m=+2789.583905503" watchObservedRunningTime="2026-09-16 19:29:45.943823888 +0000 UTC m=+2789.585539589"
Sep 16 19:29:46.514132 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:29:46.514095    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5lp6l"
Sep 16 19:29:55.513799 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:29:55.513758    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5lp6l"
Sep 16 19:29:55.526209 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:29:55.526181    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5lp6l"
Sep 16 19:29:55.990585 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:29:55.990490    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5lp6l"
Sep 16 19:30:19.759615 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:30:19.759576    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5lp6l"]
Sep 16 19:30:19.768811 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:30:19.768770    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:30:19.768811 ip-10-0-129-194 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:30:19.768811 ip-10-0-129-194 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 19:30:19.768811 ip-10-0-129-194 kubenswrapper[2582]:  > containerID="48651017246edc522d3b1bc98ccaad3ca51badc3e4160deb6e20ff07c718c0b4" cmd=["/bin/sleep","15"]
Sep 16 19:30:19.769119 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:30:19.768838    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5lp6l" message="" err=<
Sep 16 19:30:19.769119 ip-10-0-129-194 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:30:19.769119 ip-10-0-129-194 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 19:30:19.769119 ip-10-0-129-194 kubenswrapper[2582]:  >
Sep 16 19:30:19.769119 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:30:19.768856    2582 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:30:19.769119 ip-10-0-129-194 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:30:19.769119 ip-10-0-129-194 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 19:30:19.769119 ip-10-0-129-194 kubenswrapper[2582]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5lp6l" podUID="8487bf88-614c-4685-8157-260db24d45ee" containerName="main" containerID="cri-o://48651017246edc522d3b1bc98ccaad3ca51badc3e4160deb6e20ff07c718c0b4"
Sep 16 19:30:19.769119 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:30:19.768912    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5lp6l" podUID="8487bf88-614c-4685-8157-260db24d45ee" containerName="main" containerID="cri-o://48651017246edc522d3b1bc98ccaad3ca51badc3e4160deb6e20ff07c718c0b4" gracePeriod=60
Sep 16 19:30:19.980331 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:30:19.980289    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5lp6l" podUID="8487bf88-614c-4685-8157-260db24d45ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Sep 16 19:30:20.085709 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:30:20.085677    2582 generic.go:356] "Generic (PLEG): container finished" podID="8487bf88-614c-4685-8157-260db24d45ee" containerID="48651017246edc522d3b1bc98ccaad3ca51badc3e4160deb6e20ff07c718c0b4" exitCode=0
Sep 16 19:30:20.085903 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:30:20.085754    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5lp6l" event={"ID":"8487bf88-614c-4685-8157-260db24d45ee","Type":"ContainerDied","Data":"48651017246edc522d3b1bc98ccaad3ca51badc3e4160deb6e20ff07c718c0b4"}
Sep 16 19:30:20.114655 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:30:20.114632    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5lp6l"
Sep 16 19:30:20.194960 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:30:20.194927    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8487bf88-614c-4685-8157-260db24d45ee-home\") pod \"8487bf88-614c-4685-8157-260db24d45ee\" (UID: \"8487bf88-614c-4685-8157-260db24d45ee\") "
Sep 16 19:30:20.195135 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:30:20.194980    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8487bf88-614c-4685-8157-260db24d45ee-tmp-dir\") pod \"8487bf88-614c-4685-8157-260db24d45ee\" (UID: \"8487bf88-614c-4685-8157-260db24d45ee\") "
Sep 16 19:30:20.195135 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:30:20.195006    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8487bf88-614c-4685-8157-260db24d45ee-model-cache\") pod \"8487bf88-614c-4685-8157-260db24d45ee\" (UID: \"8487bf88-614c-4685-8157-260db24d45ee\") "
Sep 16 19:30:20.195135 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:30:20.195025    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8487bf88-614c-4685-8157-260db24d45ee-tls-certs\") pod \"8487bf88-614c-4685-8157-260db24d45ee\" (UID: \"8487bf88-614c-4685-8157-260db24d45ee\") "
Sep 16 19:30:20.195135 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:30:20.195067    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5bdcl\" (UniqueName: \"kubernetes.io/projected/8487bf88-614c-4685-8157-260db24d45ee-kube-api-access-5bdcl\") pod \"8487bf88-614c-4685-8157-260db24d45ee\" (UID: \"8487bf88-614c-4685-8157-260db24d45ee\") "
Sep 16 19:30:20.195135 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:30:20.195105    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8487bf88-614c-4685-8157-260db24d45ee-dshm\") pod \"8487bf88-614c-4685-8157-260db24d45ee\" (UID: \"8487bf88-614c-4685-8157-260db24d45ee\") "
Sep 16 19:30:20.195479 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:30:20.195237    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8487bf88-614c-4685-8157-260db24d45ee-home" (OuterVolumeSpecName: "home") pod "8487bf88-614c-4685-8157-260db24d45ee" (UID: "8487bf88-614c-4685-8157-260db24d45ee"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:30:20.195479 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:30:20.195251    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8487bf88-614c-4685-8157-260db24d45ee-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8487bf88-614c-4685-8157-260db24d45ee" (UID: "8487bf88-614c-4685-8157-260db24d45ee"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:30:20.195479 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:30:20.195337    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8487bf88-614c-4685-8157-260db24d45ee-model-cache" (OuterVolumeSpecName: "model-cache") pod "8487bf88-614c-4685-8157-260db24d45ee" (UID: "8487bf88-614c-4685-8157-260db24d45ee"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:30:20.195479 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:30:20.195406    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8487bf88-614c-4685-8157-260db24d45ee-tmp-dir\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:30:20.195479 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:30:20.195427    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8487bf88-614c-4685-8157-260db24d45ee-home\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:30:20.197426 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:30:20.197399    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8487bf88-614c-4685-8157-260db24d45ee-dshm" (OuterVolumeSpecName: "dshm") pod "8487bf88-614c-4685-8157-260db24d45ee" (UID: "8487bf88-614c-4685-8157-260db24d45ee"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:30:20.197531 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:30:20.197466    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8487bf88-614c-4685-8157-260db24d45ee-kube-api-access-5bdcl" (OuterVolumeSpecName: "kube-api-access-5bdcl") pod "8487bf88-614c-4685-8157-260db24d45ee" (UID: "8487bf88-614c-4685-8157-260db24d45ee"). InnerVolumeSpecName "kube-api-access-5bdcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:30:20.197531 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:30:20.197486    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8487bf88-614c-4685-8157-260db24d45ee-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8487bf88-614c-4685-8157-260db24d45ee" (UID: "8487bf88-614c-4685-8157-260db24d45ee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:30:20.296576 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:30:20.296456    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8487bf88-614c-4685-8157-260db24d45ee-model-cache\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:30:20.296576 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:30:20.296495    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8487bf88-614c-4685-8157-260db24d45ee-tls-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:30:20.296576 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:30:20.296508    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bdcl\" (UniqueName: \"kubernetes.io/projected/8487bf88-614c-4685-8157-260db24d45ee-kube-api-access-5bdcl\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:30:20.296576 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:30:20.296521    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8487bf88-614c-4685-8157-260db24d45ee-dshm\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:30:21.091341 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:30:21.091310    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5lp6l"
Sep 16 19:30:21.091780 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:30:21.091309    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5lp6l" event={"ID":"8487bf88-614c-4685-8157-260db24d45ee","Type":"ContainerDied","Data":"9cccfba7ccfe8c492b0f0b3c451b8f0dcd22f136b270576609aa8880d27919da"}
Sep 16 19:30:21.091780 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:30:21.091465    2582 scope.go:117] "RemoveContainer" containerID="48651017246edc522d3b1bc98ccaad3ca51badc3e4160deb6e20ff07c718c0b4"
Sep 16 19:30:21.112240 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:30:21.112213    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5lp6l"]
Sep 16 19:30:21.116761 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:30:21.116731    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5lp6l"]
Sep 16 19:30:22.872147 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:30:22.872106    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8487bf88-614c-4685-8157-260db24d45ee" path="/var/lib/kubelet/pods/8487bf88-614c-4685-8157-260db24d45ee/volumes"
Sep 16 19:31:05.871057 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:05.871017    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-56wxp"]
Sep 16 19:31:06.522790 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:06.522751    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27hpt"]
Sep 16 19:31:14.085982 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:14.085890    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-kd8fq"]
Sep 16 19:31:14.086373 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:14.086268    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8487bf88-614c-4685-8157-260db24d45ee" containerName="main"
Sep 16 19:31:14.086373 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:14.086282    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8487bf88-614c-4685-8157-260db24d45ee" containerName="main"
Sep 16 19:31:14.086373 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:14.086342    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8487bf88-614c-4685-8157-260db24d45ee" containerName="main"
Sep 16 19:31:14.089273 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:14.089256    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-kd8fq"
Sep 16 19:31:14.093836 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:14.093810    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"default-dockercfg-v7lvz\""
Sep 16 19:31:14.093836 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:14.093825    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"openshift-service-ca.crt\""
Sep 16 19:31:14.094018 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:14.093825    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"owner-v1-kserve-self-signed-certs\""
Sep 16 19:31:14.094018 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:14.093827    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"kube-root-ca.crt\""
Sep 16 19:31:14.104791 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:14.104761    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-kd8fq"]
Sep 16 19:31:14.253698 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:14.253653    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13eb1c3d-d900-42d4-bb08-947c06c918a5-home\") pod \"owner-v1-kserve-75579ff5bc-kd8fq\" (UID: \"13eb1c3d-d900-42d4-bb08-947c06c918a5\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-kd8fq"
Sep 16 19:31:14.253698 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:14.253700    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13eb1c3d-d900-42d4-bb08-947c06c918a5-dshm\") pod \"owner-v1-kserve-75579ff5bc-kd8fq\" (UID: \"13eb1c3d-d900-42d4-bb08-947c06c918a5\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-kd8fq"
Sep 16 19:31:14.253956 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:14.253777    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13eb1c3d-d900-42d4-bb08-947c06c918a5-model-cache\") pod \"owner-v1-kserve-75579ff5bc-kd8fq\" (UID: \"13eb1c3d-d900-42d4-bb08-947c06c918a5\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-kd8fq"
Sep 16 19:31:14.253956 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:14.253837    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmxtn\" (UniqueName: \"kubernetes.io/projected/13eb1c3d-d900-42d4-bb08-947c06c918a5-kube-api-access-gmxtn\") pod \"owner-v1-kserve-75579ff5bc-kd8fq\" (UID: \"13eb1c3d-d900-42d4-bb08-947c06c918a5\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-kd8fq"
Sep 16 19:31:14.253956 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:14.253883    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13eb1c3d-d900-42d4-bb08-947c06c918a5-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-kd8fq\" (UID: \"13eb1c3d-d900-42d4-bb08-947c06c918a5\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-kd8fq"
Sep 16 19:31:14.253956 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:14.253902    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13eb1c3d-d900-42d4-bb08-947c06c918a5-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-kd8fq\" (UID: \"13eb1c3d-d900-42d4-bb08-947c06c918a5\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-kd8fq"
Sep 16 19:31:14.354589 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:14.354496    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13eb1c3d-d900-42d4-bb08-947c06c918a5-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-kd8fq\" (UID: \"13eb1c3d-d900-42d4-bb08-947c06c918a5\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-kd8fq"
Sep 16 19:31:14.354589 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:14.354536    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13eb1c3d-d900-42d4-bb08-947c06c918a5-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-kd8fq\" (UID: \"13eb1c3d-d900-42d4-bb08-947c06c918a5\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-kd8fq"
Sep 16 19:31:14.354589 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:14.354576    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13eb1c3d-d900-42d4-bb08-947c06c918a5-home\") pod \"owner-v1-kserve-75579ff5bc-kd8fq\" (UID: \"13eb1c3d-d900-42d4-bb08-947c06c918a5\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-kd8fq"
Sep 16 19:31:14.354897 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:14.354595    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13eb1c3d-d900-42d4-bb08-947c06c918a5-dshm\") pod \"owner-v1-kserve-75579ff5bc-kd8fq\" (UID: \"13eb1c3d-d900-42d4-bb08-947c06c918a5\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-kd8fq"
Sep 16 19:31:14.354897 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:14.354614    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13eb1c3d-d900-42d4-bb08-947c06c918a5-model-cache\") pod \"owner-v1-kserve-75579ff5bc-kd8fq\" (UID: \"13eb1c3d-d900-42d4-bb08-947c06c918a5\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-kd8fq"
Sep 16 19:31:14.354897 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:14.354649    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmxtn\" (UniqueName: \"kubernetes.io/projected/13eb1c3d-d900-42d4-bb08-947c06c918a5-kube-api-access-gmxtn\") pod \"owner-v1-kserve-75579ff5bc-kd8fq\" (UID: \"13eb1c3d-d900-42d4-bb08-947c06c918a5\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-kd8fq"
Sep 16 19:31:14.355110 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:14.355087    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13eb1c3d-d900-42d4-bb08-947c06c918a5-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-kd8fq\" (UID: \"13eb1c3d-d900-42d4-bb08-947c06c918a5\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-kd8fq"
Sep 16 19:31:14.355184 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:14.355142    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13eb1c3d-d900-42d4-bb08-947c06c918a5-home\") pod \"owner-v1-kserve-75579ff5bc-kd8fq\" (UID: \"13eb1c3d-d900-42d4-bb08-947c06c918a5\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-kd8fq"
Sep 16 19:31:14.355254 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:14.355239    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13eb1c3d-d900-42d4-bb08-947c06c918a5-model-cache\") pod \"owner-v1-kserve-75579ff5bc-kd8fq\" (UID: \"13eb1c3d-d900-42d4-bb08-947c06c918a5\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-kd8fq"
Sep 16 19:31:14.356948 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:14.356925    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13eb1c3d-d900-42d4-bb08-947c06c918a5-dshm\") pod \"owner-v1-kserve-75579ff5bc-kd8fq\" (UID: \"13eb1c3d-d900-42d4-bb08-947c06c918a5\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-kd8fq"
Sep 16 19:31:14.357130 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:14.357113    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13eb1c3d-d900-42d4-bb08-947c06c918a5-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-kd8fq\" (UID: \"13eb1c3d-d900-42d4-bb08-947c06c918a5\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-kd8fq"
Sep 16 19:31:14.366126 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:14.366096    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmxtn\" (UniqueName: \"kubernetes.io/projected/13eb1c3d-d900-42d4-bb08-947c06c918a5-kube-api-access-gmxtn\") pod \"owner-v1-kserve-75579ff5bc-kd8fq\" (UID: \"13eb1c3d-d900-42d4-bb08-947c06c918a5\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-kd8fq"
Sep 16 19:31:14.399141 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:14.399108    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-kd8fq"
Sep 16 19:31:14.674451 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:14.674364    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-kd8fq"]
Sep 16 19:31:14.677651 ip-10-0-129-194 kubenswrapper[2582]: W0916 19:31:14.677629    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13eb1c3d_d900_42d4_bb08_947c06c918a5.slice/crio-ec4da5f31878ea2ac029fd534eaf9dc015e816540b83c4dc246864ca8bbdd69f WatchSource:0}: Error finding container ec4da5f31878ea2ac029fd534eaf9dc015e816540b83c4dc246864ca8bbdd69f: Status 404 returned error can't find the container with id ec4da5f31878ea2ac029fd534eaf9dc015e816540b83c4dc246864ca8bbdd69f
Sep 16 19:31:15.348131 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:15.348096    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-kd8fq" event={"ID":"13eb1c3d-d900-42d4-bb08-947c06c918a5","Type":"ContainerStarted","Data":"f44f8805ddc41d512aadb660b1913f451dfcc25cc130827afae0e0a680435d0b"}
Sep 16 19:31:15.348539 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:15.348157    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-kd8fq" event={"ID":"13eb1c3d-d900-42d4-bb08-947c06c918a5","Type":"ContainerStarted","Data":"ec4da5f31878ea2ac029fd534eaf9dc015e816540b83c4dc246864ca8bbdd69f"}
Sep 16 19:31:15.399850 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:15.399806    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-kd8fq"
Sep 16 19:31:20.889876 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:20.889810    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-56wxp" podUID="67d1d96b-13cc-4343-af92-a64272ba9182" containerName="main" containerID="cri-o://41aae99ff80c9bf1c95b29388fd4ab08f3dd2cee0c5d6bb85370683ba1f39166" gracePeriod=45
Sep 16 19:31:21.145752 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:21.145663    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-56wxp"
Sep 16 19:31:21.169220 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:21.169170    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-kd8fq" podStartSLOduration=7.169152456 podStartE2EDuration="7.169152456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:31:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:31:15.390628745 +0000 UTC m=+2879.032344421" watchObservedRunningTime="2026-09-16 19:31:21.169152456 +0000 UTC m=+2884.810868119"
Sep 16 19:31:21.312852 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:21.312815    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p7gw4\" (UniqueName: \"kubernetes.io/projected/67d1d96b-13cc-4343-af92-a64272ba9182-kube-api-access-p7gw4\") pod \"67d1d96b-13cc-4343-af92-a64272ba9182\" (UID: \"67d1d96b-13cc-4343-af92-a64272ba9182\") "
Sep 16 19:31:21.313049 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:21.312889    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67d1d96b-13cc-4343-af92-a64272ba9182-tls-certs\") pod \"67d1d96b-13cc-4343-af92-a64272ba9182\" (UID: \"67d1d96b-13cc-4343-af92-a64272ba9182\") "
Sep 16 19:31:21.313049 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:21.312933    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67d1d96b-13cc-4343-af92-a64272ba9182-dshm\") pod \"67d1d96b-13cc-4343-af92-a64272ba9182\" (UID: \"67d1d96b-13cc-4343-af92-a64272ba9182\") "
Sep 16 19:31:21.313049 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:21.312962    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67d1d96b-13cc-4343-af92-a64272ba9182-home\") pod \"67d1d96b-13cc-4343-af92-a64272ba9182\" (UID: \"67d1d96b-13cc-4343-af92-a64272ba9182\") "
Sep 16 19:31:21.313049 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:21.312989    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67d1d96b-13cc-4343-af92-a64272ba9182-tmp-dir\") pod \"67d1d96b-13cc-4343-af92-a64272ba9182\" (UID: \"67d1d96b-13cc-4343-af92-a64272ba9182\") "
Sep 16 19:31:21.313262 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:21.313065    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67d1d96b-13cc-4343-af92-a64272ba9182-model-cache\") pod \"67d1d96b-13cc-4343-af92-a64272ba9182\" (UID: \"67d1d96b-13cc-4343-af92-a64272ba9182\") "
Sep 16 19:31:21.313262 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:21.313237    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67d1d96b-13cc-4343-af92-a64272ba9182-home" (OuterVolumeSpecName: "home") pod "67d1d96b-13cc-4343-af92-a64272ba9182" (UID: "67d1d96b-13cc-4343-af92-a64272ba9182"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:31:21.313363 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:21.313336    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67d1d96b-13cc-4343-af92-a64272ba9182-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "67d1d96b-13cc-4343-af92-a64272ba9182" (UID: "67d1d96b-13cc-4343-af92-a64272ba9182"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:31:21.313408 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:21.313368    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67d1d96b-13cc-4343-af92-a64272ba9182-home\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:31:21.313408 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:21.313363    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67d1d96b-13cc-4343-af92-a64272ba9182-model-cache" (OuterVolumeSpecName: "model-cache") pod "67d1d96b-13cc-4343-af92-a64272ba9182" (UID: "67d1d96b-13cc-4343-af92-a64272ba9182"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:31:21.315238 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:21.315203    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67d1d96b-13cc-4343-af92-a64272ba9182-dshm" (OuterVolumeSpecName: "dshm") pod "67d1d96b-13cc-4343-af92-a64272ba9182" (UID: "67d1d96b-13cc-4343-af92-a64272ba9182"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:31:21.315352 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:21.315289    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67d1d96b-13cc-4343-af92-a64272ba9182-kube-api-access-p7gw4" (OuterVolumeSpecName: "kube-api-access-p7gw4") pod "67d1d96b-13cc-4343-af92-a64272ba9182" (UID: "67d1d96b-13cc-4343-af92-a64272ba9182"). InnerVolumeSpecName "kube-api-access-p7gw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:31:21.315352 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:21.315297    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67d1d96b-13cc-4343-af92-a64272ba9182-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "67d1d96b-13cc-4343-af92-a64272ba9182" (UID: "67d1d96b-13cc-4343-af92-a64272ba9182"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:31:21.376309 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:21.376273    2582 generic.go:356] "Generic (PLEG): container finished" podID="67d1d96b-13cc-4343-af92-a64272ba9182" containerID="41aae99ff80c9bf1c95b29388fd4ab08f3dd2cee0c5d6bb85370683ba1f39166" exitCode=0
Sep 16 19:31:21.376489 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:21.376327    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-56wxp"
Sep 16 19:31:21.376489 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:21.376362    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-56wxp" event={"ID":"67d1d96b-13cc-4343-af92-a64272ba9182","Type":"ContainerDied","Data":"41aae99ff80c9bf1c95b29388fd4ab08f3dd2cee0c5d6bb85370683ba1f39166"}
Sep 16 19:31:21.376489 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:21.376420    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-56wxp" event={"ID":"67d1d96b-13cc-4343-af92-a64272ba9182","Type":"ContainerDied","Data":"2821beb31e0dacda518b0e748ffc4b8ecc3464c2c129a435e8eccd592b7c9b5a"}
Sep 16 19:31:21.376489 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:21.376443    2582 scope.go:117] "RemoveContainer" containerID="41aae99ff80c9bf1c95b29388fd4ab08f3dd2cee0c5d6bb85370683ba1f39166"
Sep 16 19:31:21.386742 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:21.386648    2582 scope.go:117] "RemoveContainer" containerID="41aae99ff80c9bf1c95b29388fd4ab08f3dd2cee0c5d6bb85370683ba1f39166"
Sep 16 19:31:21.387193 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:31:21.387122    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41aae99ff80c9bf1c95b29388fd4ab08f3dd2cee0c5d6bb85370683ba1f39166\": container with ID starting with 41aae99ff80c9bf1c95b29388fd4ab08f3dd2cee0c5d6bb85370683ba1f39166 not found: ID does not exist" containerID="41aae99ff80c9bf1c95b29388fd4ab08f3dd2cee0c5d6bb85370683ba1f39166"
Sep 16 19:31:21.387193 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:21.387157    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41aae99ff80c9bf1c95b29388fd4ab08f3dd2cee0c5d6bb85370683ba1f39166"} err="failed to get container status \"41aae99ff80c9bf1c95b29388fd4ab08f3dd2cee0c5d6bb85370683ba1f39166\": rpc error: code = NotFound desc = could not find container \"41aae99ff80c9bf1c95b29388fd4ab08f3dd2cee0c5d6bb85370683ba1f39166\": container with ID starting with 41aae99ff80c9bf1c95b29388fd4ab08f3dd2cee0c5d6bb85370683ba1f39166 not found: ID does not exist"
Sep 16 19:31:21.400107 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:21.400050    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-56wxp"]
Sep 16 19:31:21.408988 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:21.408961    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-56wxp"]
Sep 16 19:31:21.413806 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:21.413785    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7gw4\" (UniqueName: \"kubernetes.io/projected/67d1d96b-13cc-4343-af92-a64272ba9182-kube-api-access-p7gw4\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:31:21.413868 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:21.413810    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67d1d96b-13cc-4343-af92-a64272ba9182-tls-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:31:21.413868 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:21.413821    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67d1d96b-13cc-4343-af92-a64272ba9182-dshm\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:31:21.413868 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:21.413829    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67d1d96b-13cc-4343-af92-a64272ba9182-tmp-dir\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:31:21.413868 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:21.413839    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67d1d96b-13cc-4343-af92-a64272ba9182-model-cache\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:31:21.541215 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:21.541172    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27hpt" podUID="85f990bd-5573-47e8-bb26-58ede0d2e38d" containerName="main" containerID="cri-o://74aa5c01bf39491054fba27ce59c326c0ee55bdb1e27b4c0606946cf2f4f416c" gracePeriod=45
Sep 16 19:31:21.623475 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:21.623428    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27hpt" podUID="85f990bd-5573-47e8-bb26-58ede0d2e38d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Sep 16 19:31:21.769863 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:21.769839    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27hpt"
Sep 16 19:31:21.918861 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:21.918784    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85f990bd-5573-47e8-bb26-58ede0d2e38d-model-cache\") pod \"85f990bd-5573-47e8-bb26-58ede0d2e38d\" (UID: \"85f990bd-5573-47e8-bb26-58ede0d2e38d\") "
Sep 16 19:31:21.919262 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:21.918863    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85f990bd-5573-47e8-bb26-58ede0d2e38d-tmp-dir\") pod \"85f990bd-5573-47e8-bb26-58ede0d2e38d\" (UID: \"85f990bd-5573-47e8-bb26-58ede0d2e38d\") "
Sep 16 19:31:21.919262 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:21.918886    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85f990bd-5573-47e8-bb26-58ede0d2e38d-home\") pod \"85f990bd-5573-47e8-bb26-58ede0d2e38d\" (UID: \"85f990bd-5573-47e8-bb26-58ede0d2e38d\") "
Sep 16 19:31:21.919262 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:21.918928    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jz4s5\" (UniqueName: \"kubernetes.io/projected/85f990bd-5573-47e8-bb26-58ede0d2e38d-kube-api-access-jz4s5\") pod \"85f990bd-5573-47e8-bb26-58ede0d2e38d\" (UID: \"85f990bd-5573-47e8-bb26-58ede0d2e38d\") "
Sep 16 19:31:21.919262 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:21.918973    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85f990bd-5573-47e8-bb26-58ede0d2e38d-tls-certs\") pod \"85f990bd-5573-47e8-bb26-58ede0d2e38d\" (UID: \"85f990bd-5573-47e8-bb26-58ede0d2e38d\") "
Sep 16 19:31:21.919262 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:21.919001    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85f990bd-5573-47e8-bb26-58ede0d2e38d-dshm\") pod \"85f990bd-5573-47e8-bb26-58ede0d2e38d\" (UID: \"85f990bd-5573-47e8-bb26-58ede0d2e38d\") "
Sep 16 19:31:21.919262 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:21.919079    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85f990bd-5573-47e8-bb26-58ede0d2e38d-model-cache" (OuterVolumeSpecName: "model-cache") pod "85f990bd-5573-47e8-bb26-58ede0d2e38d" (UID: "85f990bd-5573-47e8-bb26-58ede0d2e38d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:31:21.919262 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:21.919161    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85f990bd-5573-47e8-bb26-58ede0d2e38d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "85f990bd-5573-47e8-bb26-58ede0d2e38d" (UID: "85f990bd-5573-47e8-bb26-58ede0d2e38d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:31:21.919262 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:21.919179    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85f990bd-5573-47e8-bb26-58ede0d2e38d-home" (OuterVolumeSpecName: "home") pod "85f990bd-5573-47e8-bb26-58ede0d2e38d" (UID: "85f990bd-5573-47e8-bb26-58ede0d2e38d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:31:21.919614 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:21.919274    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85f990bd-5573-47e8-bb26-58ede0d2e38d-model-cache\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:31:21.919614 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:21.919293    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85f990bd-5573-47e8-bb26-58ede0d2e38d-tmp-dir\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:31:21.919614 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:21.919306    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85f990bd-5573-47e8-bb26-58ede0d2e38d-home\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:31:21.921153 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:21.921129    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85f990bd-5573-47e8-bb26-58ede0d2e38d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "85f990bd-5573-47e8-bb26-58ede0d2e38d" (UID: "85f990bd-5573-47e8-bb26-58ede0d2e38d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:31:21.921307 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:21.921292    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85f990bd-5573-47e8-bb26-58ede0d2e38d-kube-api-access-jz4s5" (OuterVolumeSpecName: "kube-api-access-jz4s5") pod "85f990bd-5573-47e8-bb26-58ede0d2e38d" (UID: "85f990bd-5573-47e8-bb26-58ede0d2e38d"). InnerVolumeSpecName "kube-api-access-jz4s5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:31:21.921351 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:21.921324    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85f990bd-5573-47e8-bb26-58ede0d2e38d-dshm" (OuterVolumeSpecName: "dshm") pod "85f990bd-5573-47e8-bb26-58ede0d2e38d" (UID: "85f990bd-5573-47e8-bb26-58ede0d2e38d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:31:22.020118 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:22.020066    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85f990bd-5573-47e8-bb26-58ede0d2e38d-tls-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:31:22.020118 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:22.020110    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85f990bd-5573-47e8-bb26-58ede0d2e38d-dshm\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:31:22.020118 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:22.020122    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz4s5\" (UniqueName: \"kubernetes.io/projected/85f990bd-5573-47e8-bb26-58ede0d2e38d-kube-api-access-jz4s5\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:31:22.382573 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:22.382534    2582 generic.go:356] "Generic (PLEG): container finished" podID="85f990bd-5573-47e8-bb26-58ede0d2e38d" containerID="74aa5c01bf39491054fba27ce59c326c0ee55bdb1e27b4c0606946cf2f4f416c" exitCode=0
Sep 16 19:31:22.382764 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:22.382579    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27hpt" event={"ID":"85f990bd-5573-47e8-bb26-58ede0d2e38d","Type":"ContainerDied","Data":"74aa5c01bf39491054fba27ce59c326c0ee55bdb1e27b4c0606946cf2f4f416c"}
Sep 16 19:31:22.382764 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:22.382587    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27hpt"
Sep 16 19:31:22.382764 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:22.382629    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27hpt" event={"ID":"85f990bd-5573-47e8-bb26-58ede0d2e38d","Type":"ContainerDied","Data":"c9370671fc305b7c8b44322e478157387af3789f5c2d01c8eac26cca8b14c900"}
Sep 16 19:31:22.382764 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:22.382660    2582 scope.go:117] "RemoveContainer" containerID="74aa5c01bf39491054fba27ce59c326c0ee55bdb1e27b4c0606946cf2f4f416c"
Sep 16 19:31:22.391797 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:22.391776    2582 scope.go:117] "RemoveContainer" containerID="74aa5c01bf39491054fba27ce59c326c0ee55bdb1e27b4c0606946cf2f4f416c"
Sep 16 19:31:22.392064 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:31:22.392042    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74aa5c01bf39491054fba27ce59c326c0ee55bdb1e27b4c0606946cf2f4f416c\": container with ID starting with 74aa5c01bf39491054fba27ce59c326c0ee55bdb1e27b4c0606946cf2f4f416c not found: ID does not exist" containerID="74aa5c01bf39491054fba27ce59c326c0ee55bdb1e27b4c0606946cf2f4f416c"
Sep 16 19:31:22.392110 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:22.392074    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74aa5c01bf39491054fba27ce59c326c0ee55bdb1e27b4c0606946cf2f4f416c"} err="failed to get container status \"74aa5c01bf39491054fba27ce59c326c0ee55bdb1e27b4c0606946cf2f4f416c\": rpc error: code = NotFound desc = could not find container \"74aa5c01bf39491054fba27ce59c326c0ee55bdb1e27b4c0606946cf2f4f416c\": container with ID starting with 74aa5c01bf39491054fba27ce59c326c0ee55bdb1e27b4c0606946cf2f4f416c not found: ID does not exist"
Sep 16 19:31:22.404839 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:22.404812    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27hpt"]
Sep 16 19:31:22.410243 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:22.410217    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27hpt"]
Sep 16 19:31:22.872529 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:22.872491    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67d1d96b-13cc-4343-af92-a64272ba9182" path="/var/lib/kubelet/pods/67d1d96b-13cc-4343-af92-a64272ba9182/volumes"
Sep 16 19:31:22.872938 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:22.872922    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85f990bd-5573-47e8-bb26-58ede0d2e38d" path="/var/lib/kubelet/pods/85f990bd-5573-47e8-bb26-58ede0d2e38d/volumes"
Sep 16 19:31:24.399795 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:24.399763    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-kd8fq"
Sep 16 19:31:24.411976 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:24.411947    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-kd8fq"
Sep 16 19:31:25.408643 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:25.408612    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-kd8fq"
Sep 16 19:31:37.212875 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:37.212842    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-7zngx"]
Sep 16 19:31:37.213383 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:37.213151    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67d1d96b-13cc-4343-af92-a64272ba9182" containerName="main"
Sep 16 19:31:37.213383 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:37.213162    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="67d1d96b-13cc-4343-af92-a64272ba9182" containerName="main"
Sep 16 19:31:37.213383 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:37.213172    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85f990bd-5573-47e8-bb26-58ede0d2e38d" containerName="main"
Sep 16 19:31:37.213383 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:37.213177    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="85f990bd-5573-47e8-bb26-58ede0d2e38d" containerName="main"
Sep 16 19:31:37.213383 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:37.213230    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="85f990bd-5573-47e8-bb26-58ede0d2e38d" containerName="main"
Sep 16 19:31:37.213383 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:37.213238    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="67d1d96b-13cc-4343-af92-a64272ba9182" containerName="main"
Sep 16 19:31:37.216307 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:37.216290    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-7zngx"
Sep 16 19:31:37.219539 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:37.219517    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"owner-v2-kserve-self-signed-certs\""
Sep 16 19:31:37.227063 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:37.227037    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-7zngx"]
Sep 16 19:31:37.239570 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:37.239540    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a10f04c9-0500-4382-a2e9-975e6f217d9b-home\") pod \"owner-v2-kserve-5cf8959f95-7zngx\" (UID: \"a10f04c9-0500-4382-a2e9-975e6f217d9b\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-7zngx"
Sep 16 19:31:37.239672 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:37.239644    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a10f04c9-0500-4382-a2e9-975e6f217d9b-tls-certs\") pod \"owner-v2-kserve-5cf8959f95-7zngx\" (UID: \"a10f04c9-0500-4382-a2e9-975e6f217d9b\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-7zngx"
Sep 16 19:31:37.239714 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:37.239686    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a10f04c9-0500-4382-a2e9-975e6f217d9b-dshm\") pod \"owner-v2-kserve-5cf8959f95-7zngx\" (UID: \"a10f04c9-0500-4382-a2e9-975e6f217d9b\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-7zngx"
Sep 16 19:31:37.239714 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:37.239703    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bp9w\" (UniqueName: \"kubernetes.io/projected/a10f04c9-0500-4382-a2e9-975e6f217d9b-kube-api-access-5bp9w\") pod \"owner-v2-kserve-5cf8959f95-7zngx\" (UID: \"a10f04c9-0500-4382-a2e9-975e6f217d9b\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-7zngx"
Sep 16 19:31:37.239811 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:37.239738    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a10f04c9-0500-4382-a2e9-975e6f217d9b-tmp-dir\") pod \"owner-v2-kserve-5cf8959f95-7zngx\" (UID: \"a10f04c9-0500-4382-a2e9-975e6f217d9b\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-7zngx"
Sep 16 19:31:37.239843 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:37.239807    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a10f04c9-0500-4382-a2e9-975e6f217d9b-model-cache\") pod \"owner-v2-kserve-5cf8959f95-7zngx\" (UID: \"a10f04c9-0500-4382-a2e9-975e6f217d9b\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-7zngx"
Sep 16 19:31:37.340394 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:37.340353    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a10f04c9-0500-4382-a2e9-975e6f217d9b-tls-certs\") pod \"owner-v2-kserve-5cf8959f95-7zngx\" (UID: \"a10f04c9-0500-4382-a2e9-975e6f217d9b\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-7zngx"
Sep 16 19:31:37.340394 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:37.340401    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a10f04c9-0500-4382-a2e9-975e6f217d9b-dshm\") pod \"owner-v2-kserve-5cf8959f95-7zngx\" (UID: \"a10f04c9-0500-4382-a2e9-975e6f217d9b\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-7zngx"
Sep 16 19:31:37.340666 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:37.340418    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bp9w\" (UniqueName: \"kubernetes.io/projected/a10f04c9-0500-4382-a2e9-975e6f217d9b-kube-api-access-5bp9w\") pod \"owner-v2-kserve-5cf8959f95-7zngx\" (UID: \"a10f04c9-0500-4382-a2e9-975e6f217d9b\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-7zngx"
Sep 16 19:31:37.340666 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:37.340435    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a10f04c9-0500-4382-a2e9-975e6f217d9b-tmp-dir\") pod \"owner-v2-kserve-5cf8959f95-7zngx\" (UID: \"a10f04c9-0500-4382-a2e9-975e6f217d9b\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-7zngx"
Sep 16 19:31:37.340666 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:37.340457    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a10f04c9-0500-4382-a2e9-975e6f217d9b-model-cache\") pod \"owner-v2-kserve-5cf8959f95-7zngx\" (UID: \"a10f04c9-0500-4382-a2e9-975e6f217d9b\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-7zngx"
Sep 16 19:31:37.340666 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:37.340496    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a10f04c9-0500-4382-a2e9-975e6f217d9b-home\") pod \"owner-v2-kserve-5cf8959f95-7zngx\" (UID: \"a10f04c9-0500-4382-a2e9-975e6f217d9b\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-7zngx"
Sep 16 19:31:37.340945 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:37.340922    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a10f04c9-0500-4382-a2e9-975e6f217d9b-home\") pod \"owner-v2-kserve-5cf8959f95-7zngx\" (UID: \"a10f04c9-0500-4382-a2e9-975e6f217d9b\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-7zngx"
Sep 16 19:31:37.340990 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:37.340945    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a10f04c9-0500-4382-a2e9-975e6f217d9b-tmp-dir\") pod \"owner-v2-kserve-5cf8959f95-7zngx\" (UID: \"a10f04c9-0500-4382-a2e9-975e6f217d9b\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-7zngx"
Sep 16 19:31:37.341025 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:37.341005    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a10f04c9-0500-4382-a2e9-975e6f217d9b-model-cache\") pod \"owner-v2-kserve-5cf8959f95-7zngx\" (UID: \"a10f04c9-0500-4382-a2e9-975e6f217d9b\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-7zngx"
Sep 16 19:31:37.342826 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:37.342806    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a10f04c9-0500-4382-a2e9-975e6f217d9b-dshm\") pod \"owner-v2-kserve-5cf8959f95-7zngx\" (UID: \"a10f04c9-0500-4382-a2e9-975e6f217d9b\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-7zngx"
Sep 16 19:31:37.343262 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:37.343245    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a10f04c9-0500-4382-a2e9-975e6f217d9b-tls-certs\") pod \"owner-v2-kserve-5cf8959f95-7zngx\" (UID: \"a10f04c9-0500-4382-a2e9-975e6f217d9b\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-7zngx"
Sep 16 19:31:37.348247 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:37.348221    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bp9w\" (UniqueName: \"kubernetes.io/projected/a10f04c9-0500-4382-a2e9-975e6f217d9b-kube-api-access-5bp9w\") pod \"owner-v2-kserve-5cf8959f95-7zngx\" (UID: \"a10f04c9-0500-4382-a2e9-975e6f217d9b\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-7zngx"
Sep 16 19:31:37.526552 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:37.526522    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-7zngx"
Sep 16 19:31:37.800290 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:37.800262    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-7zngx"]
Sep 16 19:31:37.802751 ip-10-0-129-194 kubenswrapper[2582]: W0916 19:31:37.802698    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda10f04c9_0500_4382_a2e9_975e6f217d9b.slice/crio-46a54e8f0e299dd7f949be163a70aa7464dbc622e163f6f68338f2d53803f218 WatchSource:0}: Error finding container 46a54e8f0e299dd7f949be163a70aa7464dbc622e163f6f68338f2d53803f218: Status 404 returned error can't find the container with id 46a54e8f0e299dd7f949be163a70aa7464dbc622e163f6f68338f2d53803f218
Sep 16 19:31:38.456473 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:38.456437    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-7zngx" event={"ID":"a10f04c9-0500-4382-a2e9-975e6f217d9b","Type":"ContainerStarted","Data":"1d416d5239daa336b876544ead50a1d931ab44fbfaff5ea5bb8c82d3a2a59553"}
Sep 16 19:31:38.456907 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:38.456489    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-7zngx" event={"ID":"a10f04c9-0500-4382-a2e9-975e6f217d9b","Type":"ContainerStarted","Data":"46a54e8f0e299dd7f949be163a70aa7464dbc622e163f6f68338f2d53803f218"}
Sep 16 19:31:38.473868 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:38.473818    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-7zngx" podStartSLOduration=1.473802213 podStartE2EDuration="1.473802213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:31:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:31:38.472578942 +0000 UTC m=+2902.114294619" watchObservedRunningTime="2026-09-16 19:31:38.473802213 +0000 UTC m=+2902.115517876"
Sep 16 19:31:38.526946 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:38.526906    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-7zngx"
Sep 16 19:31:47.527222 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:47.527178    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-7zngx"
Sep 16 19:31:47.539671 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:47.539642    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-7zngx"
Sep 16 19:31:48.509830 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:31:48.509800    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-7zngx"
Sep 16 19:33:17.034518 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:33:17.034488    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7qwl2_b6695f86-08c0-4836-87b1-f2a9d76084bd/console-operator/1.log"
Sep 16 19:33:17.040760 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:33:17.040734    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7qwl2_b6695f86-08c0-4836-87b1-f2a9d76084bd/console-operator/1.log"
Sep 16 19:33:27.520595 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:33:27.520562    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-kd8fq"]
Sep 16 19:33:42.539711 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:33:42.539654    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-kd8fq" podUID="13eb1c3d-d900-42d4-bb08-947c06c918a5" containerName="main" containerID="cri-o://f44f8805ddc41d512aadb660b1913f451dfcc25cc130827afae0e0a680435d0b" gracePeriod=45
Sep 16 19:33:42.780821 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:33:42.780795    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-kd8fq"
Sep 16 19:33:42.893829 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:33:42.893713    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmxtn\" (UniqueName: \"kubernetes.io/projected/13eb1c3d-d900-42d4-bb08-947c06c918a5-kube-api-access-gmxtn\") pod \"13eb1c3d-d900-42d4-bb08-947c06c918a5\" (UID: \"13eb1c3d-d900-42d4-bb08-947c06c918a5\") "
Sep 16 19:33:42.893829 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:33:42.893776    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13eb1c3d-d900-42d4-bb08-947c06c918a5-tls-certs\") pod \"13eb1c3d-d900-42d4-bb08-947c06c918a5\" (UID: \"13eb1c3d-d900-42d4-bb08-947c06c918a5\") "
Sep 16 19:33:42.893829 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:33:42.893811    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13eb1c3d-d900-42d4-bb08-947c06c918a5-model-cache\") pod \"13eb1c3d-d900-42d4-bb08-947c06c918a5\" (UID: \"13eb1c3d-d900-42d4-bb08-947c06c918a5\") "
Sep 16 19:33:42.894108 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:33:42.893873    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13eb1c3d-d900-42d4-bb08-947c06c918a5-tmp-dir\") pod \"13eb1c3d-d900-42d4-bb08-947c06c918a5\" (UID: \"13eb1c3d-d900-42d4-bb08-947c06c918a5\") "
Sep 16 19:33:42.894108 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:33:42.893911    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13eb1c3d-d900-42d4-bb08-947c06c918a5-dshm\") pod \"13eb1c3d-d900-42d4-bb08-947c06c918a5\" (UID: \"13eb1c3d-d900-42d4-bb08-947c06c918a5\") "
Sep 16 19:33:42.894108 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:33:42.893962    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13eb1c3d-d900-42d4-bb08-947c06c918a5-home\") pod \"13eb1c3d-d900-42d4-bb08-947c06c918a5\" (UID: \"13eb1c3d-d900-42d4-bb08-947c06c918a5\") "
Sep 16 19:33:42.894264 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:33:42.894177    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13eb1c3d-d900-42d4-bb08-947c06c918a5-model-cache" (OuterVolumeSpecName: "model-cache") pod "13eb1c3d-d900-42d4-bb08-947c06c918a5" (UID: "13eb1c3d-d900-42d4-bb08-947c06c918a5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:33:42.894264 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:33:42.894186    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13eb1c3d-d900-42d4-bb08-947c06c918a5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "13eb1c3d-d900-42d4-bb08-947c06c918a5" (UID: "13eb1c3d-d900-42d4-bb08-947c06c918a5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:33:42.894422 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:33:42.894308    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13eb1c3d-d900-42d4-bb08-947c06c918a5-home" (OuterVolumeSpecName: "home") pod "13eb1c3d-d900-42d4-bb08-947c06c918a5" (UID: "13eb1c3d-d900-42d4-bb08-947c06c918a5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:33:42.896035 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:33:42.896007    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13eb1c3d-d900-42d4-bb08-947c06c918a5-dshm" (OuterVolumeSpecName: "dshm") pod "13eb1c3d-d900-42d4-bb08-947c06c918a5" (UID: "13eb1c3d-d900-42d4-bb08-947c06c918a5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:33:42.896162 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:33:42.896056    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13eb1c3d-d900-42d4-bb08-947c06c918a5-kube-api-access-gmxtn" (OuterVolumeSpecName: "kube-api-access-gmxtn") pod "13eb1c3d-d900-42d4-bb08-947c06c918a5" (UID: "13eb1c3d-d900-42d4-bb08-947c06c918a5"). InnerVolumeSpecName "kube-api-access-gmxtn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:33:42.896206 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:33:42.896164    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13eb1c3d-d900-42d4-bb08-947c06c918a5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "13eb1c3d-d900-42d4-bb08-947c06c918a5" (UID: "13eb1c3d-d900-42d4-bb08-947c06c918a5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:33:42.994835 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:33:42.994797    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13eb1c3d-d900-42d4-bb08-947c06c918a5-home\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:33:42.994835 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:33:42.994829    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmxtn\" (UniqueName: \"kubernetes.io/projected/13eb1c3d-d900-42d4-bb08-947c06c918a5-kube-api-access-gmxtn\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:33:42.994835 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:33:42.994840    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13eb1c3d-d900-42d4-bb08-947c06c918a5-tls-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:33:42.995072 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:33:42.994851    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13eb1c3d-d900-42d4-bb08-947c06c918a5-model-cache\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:33:42.995072 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:33:42.994858    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13eb1c3d-d900-42d4-bb08-947c06c918a5-tmp-dir\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:33:42.995072 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:33:42.994865    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13eb1c3d-d900-42d4-bb08-947c06c918a5-dshm\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:33:43.008579 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:33:43.008545    2582 generic.go:356] "Generic (PLEG): container finished" podID="13eb1c3d-d900-42d4-bb08-947c06c918a5" containerID="f44f8805ddc41d512aadb660b1913f451dfcc25cc130827afae0e0a680435d0b" exitCode=0
Sep 16 19:33:43.008738 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:33:43.008593    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-kd8fq"
Sep 16 19:33:43.008738 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:33:43.008624    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-kd8fq" event={"ID":"13eb1c3d-d900-42d4-bb08-947c06c918a5","Type":"ContainerDied","Data":"f44f8805ddc41d512aadb660b1913f451dfcc25cc130827afae0e0a680435d0b"}
Sep 16 19:33:43.008738 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:33:43.008681    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-kd8fq" event={"ID":"13eb1c3d-d900-42d4-bb08-947c06c918a5","Type":"ContainerDied","Data":"ec4da5f31878ea2ac029fd534eaf9dc015e816540b83c4dc246864ca8bbdd69f"}
Sep 16 19:33:43.008738 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:33:43.008706    2582 scope.go:117] "RemoveContainer" containerID="f44f8805ddc41d512aadb660b1913f451dfcc25cc130827afae0e0a680435d0b"
Sep 16 19:33:43.017911 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:33:43.017888    2582 scope.go:117] "RemoveContainer" containerID="f44f8805ddc41d512aadb660b1913f451dfcc25cc130827afae0e0a680435d0b"
Sep 16 19:33:43.018211 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:33:43.018190    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f44f8805ddc41d512aadb660b1913f451dfcc25cc130827afae0e0a680435d0b\": container with ID starting with f44f8805ddc41d512aadb660b1913f451dfcc25cc130827afae0e0a680435d0b not found: ID does not exist" containerID="f44f8805ddc41d512aadb660b1913f451dfcc25cc130827afae0e0a680435d0b"
Sep 16 19:33:43.018291 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:33:43.018224    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f44f8805ddc41d512aadb660b1913f451dfcc25cc130827afae0e0a680435d0b"} err="failed to get container status \"f44f8805ddc41d512aadb660b1913f451dfcc25cc130827afae0e0a680435d0b\": rpc error: code = NotFound desc = could not find container \"f44f8805ddc41d512aadb660b1913f451dfcc25cc130827afae0e0a680435d0b\": container with ID starting with f44f8805ddc41d512aadb660b1913f451dfcc25cc130827afae0e0a680435d0b not found: ID does not exist"
Sep 16 19:33:43.035878 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:33:43.033439    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-kd8fq"]
Sep 16 19:33:43.038755 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:33:43.038712    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-kd8fq"]
Sep 16 19:33:44.872301 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:33:44.872265    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13eb1c3d-d900-42d4-bb08-947c06c918a5" path="/var/lib/kubelet/pods/13eb1c3d-d900-42d4-bb08-947c06c918a5/volumes"
Sep 16 19:33:55.222665 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:33:55.222621    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-7zngx"]
Sep 16 19:34:10.241325 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:10.241216    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-7zngx" podUID="a10f04c9-0500-4382-a2e9-975e6f217d9b" containerName="main" containerID="cri-o://1d416d5239daa336b876544ead50a1d931ab44fbfaff5ea5bb8c82d3a2a59553" gracePeriod=45
Sep 16 19:34:10.488297 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:10.488270    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-7zngx"
Sep 16 19:34:10.519779 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:10.519747    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a10f04c9-0500-4382-a2e9-975e6f217d9b-dshm\") pod \"a10f04c9-0500-4382-a2e9-975e6f217d9b\" (UID: \"a10f04c9-0500-4382-a2e9-975e6f217d9b\") "
Sep 16 19:34:10.519973 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:10.519813    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a10f04c9-0500-4382-a2e9-975e6f217d9b-tls-certs\") pod \"a10f04c9-0500-4382-a2e9-975e6f217d9b\" (UID: \"a10f04c9-0500-4382-a2e9-975e6f217d9b\") "
Sep 16 19:34:10.519973 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:10.519863    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5bp9w\" (UniqueName: \"kubernetes.io/projected/a10f04c9-0500-4382-a2e9-975e6f217d9b-kube-api-access-5bp9w\") pod \"a10f04c9-0500-4382-a2e9-975e6f217d9b\" (UID: \"a10f04c9-0500-4382-a2e9-975e6f217d9b\") "
Sep 16 19:34:10.519973 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:10.519895    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a10f04c9-0500-4382-a2e9-975e6f217d9b-home\") pod \"a10f04c9-0500-4382-a2e9-975e6f217d9b\" (UID: \"a10f04c9-0500-4382-a2e9-975e6f217d9b\") "
Sep 16 19:34:10.520166 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:10.520090    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a10f04c9-0500-4382-a2e9-975e6f217d9b-model-cache\") pod \"a10f04c9-0500-4382-a2e9-975e6f217d9b\" (UID: \"a10f04c9-0500-4382-a2e9-975e6f217d9b\") "
Sep 16 19:34:10.520231 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:10.520176    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a10f04c9-0500-4382-a2e9-975e6f217d9b-home" (OuterVolumeSpecName: "home") pod "a10f04c9-0500-4382-a2e9-975e6f217d9b" (UID: "a10f04c9-0500-4382-a2e9-975e6f217d9b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:34:10.520448 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:10.520408    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a10f04c9-0500-4382-a2e9-975e6f217d9b-tmp-dir\") pod \"a10f04c9-0500-4382-a2e9-975e6f217d9b\" (UID: \"a10f04c9-0500-4382-a2e9-975e6f217d9b\") "
Sep 16 19:34:10.520826 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:10.520739    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a10f04c9-0500-4382-a2e9-975e6f217d9b-home\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:34:10.520942 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:10.520819    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a10f04c9-0500-4382-a2e9-975e6f217d9b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a10f04c9-0500-4382-a2e9-975e6f217d9b" (UID: "a10f04c9-0500-4382-a2e9-975e6f217d9b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:34:10.521248 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:10.521217    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a10f04c9-0500-4382-a2e9-975e6f217d9b-model-cache" (OuterVolumeSpecName: "model-cache") pod "a10f04c9-0500-4382-a2e9-975e6f217d9b" (UID: "a10f04c9-0500-4382-a2e9-975e6f217d9b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:34:10.522584 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:10.522562    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a10f04c9-0500-4382-a2e9-975e6f217d9b-dshm" (OuterVolumeSpecName: "dshm") pod "a10f04c9-0500-4382-a2e9-975e6f217d9b" (UID: "a10f04c9-0500-4382-a2e9-975e6f217d9b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:34:10.522704 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:10.522593    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a10f04c9-0500-4382-a2e9-975e6f217d9b-kube-api-access-5bp9w" (OuterVolumeSpecName: "kube-api-access-5bp9w") pod "a10f04c9-0500-4382-a2e9-975e6f217d9b" (UID: "a10f04c9-0500-4382-a2e9-975e6f217d9b"). InnerVolumeSpecName "kube-api-access-5bp9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:34:10.522704 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:10.522678    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a10f04c9-0500-4382-a2e9-975e6f217d9b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a10f04c9-0500-4382-a2e9-975e6f217d9b" (UID: "a10f04c9-0500-4382-a2e9-975e6f217d9b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:34:10.621275 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:10.621242    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a10f04c9-0500-4382-a2e9-975e6f217d9b-dshm\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:34:10.621275 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:10.621272    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a10f04c9-0500-4382-a2e9-975e6f217d9b-tls-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:34:10.621275 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:10.621283    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bp9w\" (UniqueName: \"kubernetes.io/projected/a10f04c9-0500-4382-a2e9-975e6f217d9b-kube-api-access-5bp9w\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:34:10.621516 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:10.621292    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a10f04c9-0500-4382-a2e9-975e6f217d9b-model-cache\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:34:10.621516 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:10.621301    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a10f04c9-0500-4382-a2e9-975e6f217d9b-tmp-dir\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:34:11.140087 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:11.139996    2582 generic.go:356] "Generic (PLEG): container finished" podID="a10f04c9-0500-4382-a2e9-975e6f217d9b" containerID="1d416d5239daa336b876544ead50a1d931ab44fbfaff5ea5bb8c82d3a2a59553" exitCode=0
Sep 16 19:34:11.140087 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:11.140050    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-7zngx"
Sep 16 19:34:11.140087 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:11.140063    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-7zngx" event={"ID":"a10f04c9-0500-4382-a2e9-975e6f217d9b","Type":"ContainerDied","Data":"1d416d5239daa336b876544ead50a1d931ab44fbfaff5ea5bb8c82d3a2a59553"}
Sep 16 19:34:11.140342 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:11.140127    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-7zngx" event={"ID":"a10f04c9-0500-4382-a2e9-975e6f217d9b","Type":"ContainerDied","Data":"46a54e8f0e299dd7f949be163a70aa7464dbc622e163f6f68338f2d53803f218"}
Sep 16 19:34:11.140342 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:11.140149    2582 scope.go:117] "RemoveContainer" containerID="1d416d5239daa336b876544ead50a1d931ab44fbfaff5ea5bb8c82d3a2a59553"
Sep 16 19:34:11.148528 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:11.148512    2582 scope.go:117] "RemoveContainer" containerID="1d416d5239daa336b876544ead50a1d931ab44fbfaff5ea5bb8c82d3a2a59553"
Sep 16 19:34:11.148862 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:34:11.148842    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d416d5239daa336b876544ead50a1d931ab44fbfaff5ea5bb8c82d3a2a59553\": container with ID starting with 1d416d5239daa336b876544ead50a1d931ab44fbfaff5ea5bb8c82d3a2a59553 not found: ID does not exist" containerID="1d416d5239daa336b876544ead50a1d931ab44fbfaff5ea5bb8c82d3a2a59553"
Sep 16 19:34:11.148926 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:11.148869    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d416d5239daa336b876544ead50a1d931ab44fbfaff5ea5bb8c82d3a2a59553"} err="failed to get container status \"1d416d5239daa336b876544ead50a1d931ab44fbfaff5ea5bb8c82d3a2a59553\": rpc error: code = NotFound desc = could not find container \"1d416d5239daa336b876544ead50a1d931ab44fbfaff5ea5bb8c82d3a2a59553\": container with ID starting with 1d416d5239daa336b876544ead50a1d931ab44fbfaff5ea5bb8c82d3a2a59553 not found: ID does not exist"
Sep 16 19:34:11.159917 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:11.159890    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-7zngx"]
Sep 16 19:34:11.164010 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:11.163987    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-7zngx"]
Sep 16 19:34:12.872750 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:12.872697    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a10f04c9-0500-4382-a2e9-975e6f217d9b" path="/var/lib/kubelet/pods/a10f04c9-0500-4382-a2e9-975e6f217d9b/volumes"
Sep 16 19:34:24.132950 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:24.132917    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997xp8nm"]
Sep 16 19:34:24.133367 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:24.133226    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13eb1c3d-d900-42d4-bb08-947c06c918a5" containerName="main"
Sep 16 19:34:24.133367 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:24.133237    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="13eb1c3d-d900-42d4-bb08-947c06c918a5" containerName="main"
Sep 16 19:34:24.133367 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:24.133247    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a10f04c9-0500-4382-a2e9-975e6f217d9b" containerName="main"
Sep 16 19:34:24.133367 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:24.133253    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a10f04c9-0500-4382-a2e9-975e6f217d9b" containerName="main"
Sep 16 19:34:24.133367 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:24.133307    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a10f04c9-0500-4382-a2e9-975e6f217d9b" containerName="main"
Sep 16 19:34:24.133367 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:24.133317    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="13eb1c3d-d900-42d4-bb08-947c06c918a5" containerName="main"
Sep 16 19:34:24.137651 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:24.137623    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997xp8nm"
Sep 16 19:34:24.143085 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:24.143055    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Sep 16 19:34:24.143230 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:24.143178    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"default-dockercfg-rxttn\""
Sep 16 19:34:24.143230 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:24.143183    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Sep 16 19:34:24.143365 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:24.143274    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs\""
Sep 16 19:34:24.156524 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:24.156497    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997xp8nm"]
Sep 16 19:34:24.237564 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:24.237523    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad13b154-5bbb-400f-8a0a-70fd88e1fa89-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997xp8nm\" (UID: \"ad13b154-5bbb-400f-8a0a-70fd88e1fa89\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997xp8nm"
Sep 16 19:34:24.237564 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:24.237568    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpg4l\" (UniqueName: \"kubernetes.io/projected/ad13b154-5bbb-400f-8a0a-70fd88e1fa89-kube-api-access-rpg4l\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997xp8nm\" (UID: \"ad13b154-5bbb-400f-8a0a-70fd88e1fa89\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997xp8nm"
Sep 16 19:34:24.237882 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:24.237592    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad13b154-5bbb-400f-8a0a-70fd88e1fa89-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997xp8nm\" (UID: \"ad13b154-5bbb-400f-8a0a-70fd88e1fa89\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997xp8nm"
Sep 16 19:34:24.237882 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:24.237713    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad13b154-5bbb-400f-8a0a-70fd88e1fa89-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997xp8nm\" (UID: \"ad13b154-5bbb-400f-8a0a-70fd88e1fa89\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997xp8nm"
Sep 16 19:34:24.237882 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:24.237803    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad13b154-5bbb-400f-8a0a-70fd88e1fa89-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997xp8nm\" (UID: \"ad13b154-5bbb-400f-8a0a-70fd88e1fa89\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997xp8nm"
Sep 16 19:34:24.237882 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:24.237831    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad13b154-5bbb-400f-8a0a-70fd88e1fa89-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997xp8nm\" (UID: \"ad13b154-5bbb-400f-8a0a-70fd88e1fa89\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997xp8nm"
Sep 16 19:34:24.339267 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:24.339226    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad13b154-5bbb-400f-8a0a-70fd88e1fa89-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997xp8nm\" (UID: \"ad13b154-5bbb-400f-8a0a-70fd88e1fa89\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997xp8nm"
Sep 16 19:34:24.339441 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:24.339276    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad13b154-5bbb-400f-8a0a-70fd88e1fa89-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997xp8nm\" (UID: \"ad13b154-5bbb-400f-8a0a-70fd88e1fa89\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997xp8nm"
Sep 16 19:34:24.339441 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:24.339364    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad13b154-5bbb-400f-8a0a-70fd88e1fa89-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997xp8nm\" (UID: \"ad13b154-5bbb-400f-8a0a-70fd88e1fa89\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997xp8nm"
Sep 16 19:34:24.339441 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:24.339403    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad13b154-5bbb-400f-8a0a-70fd88e1fa89-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997xp8nm\" (UID: \"ad13b154-5bbb-400f-8a0a-70fd88e1fa89\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997xp8nm"
Sep 16 19:34:24.339441 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:24.339422    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpg4l\" (UniqueName: \"kubernetes.io/projected/ad13b154-5bbb-400f-8a0a-70fd88e1fa89-kube-api-access-rpg4l\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997xp8nm\" (UID: \"ad13b154-5bbb-400f-8a0a-70fd88e1fa89\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997xp8nm"
Sep 16 19:34:24.339667 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:24.339445    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad13b154-5bbb-400f-8a0a-70fd88e1fa89-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997xp8nm\" (UID: \"ad13b154-5bbb-400f-8a0a-70fd88e1fa89\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997xp8nm"
Sep 16 19:34:24.339812 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:24.339777    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad13b154-5bbb-400f-8a0a-70fd88e1fa89-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997xp8nm\" (UID: \"ad13b154-5bbb-400f-8a0a-70fd88e1fa89\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997xp8nm"
Sep 16 19:34:24.339915 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:24.339890    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad13b154-5bbb-400f-8a0a-70fd88e1fa89-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997xp8nm\" (UID: \"ad13b154-5bbb-400f-8a0a-70fd88e1fa89\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997xp8nm"
Sep 16 19:34:24.340111 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:24.340084    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad13b154-5bbb-400f-8a0a-70fd88e1fa89-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997xp8nm\" (UID: \"ad13b154-5bbb-400f-8a0a-70fd88e1fa89\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997xp8nm"
Sep 16 19:34:24.341660 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:24.341638    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad13b154-5bbb-400f-8a0a-70fd88e1fa89-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997xp8nm\" (UID: \"ad13b154-5bbb-400f-8a0a-70fd88e1fa89\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997xp8nm"
Sep 16 19:34:24.342111 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:24.342092    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad13b154-5bbb-400f-8a0a-70fd88e1fa89-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997xp8nm\" (UID: \"ad13b154-5bbb-400f-8a0a-70fd88e1fa89\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997xp8nm"
Sep 16 19:34:24.353352 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:24.353319    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpg4l\" (UniqueName: \"kubernetes.io/projected/ad13b154-5bbb-400f-8a0a-70fd88e1fa89-kube-api-access-rpg4l\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997xp8nm\" (UID: \"ad13b154-5bbb-400f-8a0a-70fd88e1fa89\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997xp8nm"
Sep 16 19:34:24.446901 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:24.446805    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997xp8nm"
Sep 16 19:34:24.725878 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:24.725853    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997xp8nm"]
Sep 16 19:34:24.727909 ip-10-0-129-194 kubenswrapper[2582]: W0916 19:34:24.727879    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad13b154_5bbb_400f_8a0a_70fd88e1fa89.slice/crio-2f40eaa0e3cc24be36e018f233322ae2eb478ee088f4b9f4952486c5b769e9fe WatchSource:0}: Error finding container 2f40eaa0e3cc24be36e018f233322ae2eb478ee088f4b9f4952486c5b769e9fe: Status 404 returned error can't find the container with id 2f40eaa0e3cc24be36e018f233322ae2eb478ee088f4b9f4952486c5b769e9fe
Sep 16 19:34:25.200494 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:25.200457    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997xp8nm" event={"ID":"ad13b154-5bbb-400f-8a0a-70fd88e1fa89","Type":"ContainerStarted","Data":"bd99db0db89b4ae1e7c077b74c89dab6762acc9ad5408976399d81f108d6836b"}
Sep 16 19:34:25.200922 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:25.200516    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997xp8nm" event={"ID":"ad13b154-5bbb-400f-8a0a-70fd88e1fa89","Type":"ContainerStarted","Data":"2f40eaa0e3cc24be36e018f233322ae2eb478ee088f4b9f4952486c5b769e9fe"}
Sep 16 19:34:25.447590 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:25.447552    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997xp8nm"
Sep 16 19:34:34.447808 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:34.447767    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997xp8nm"
Sep 16 19:34:34.460840 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:34.460809    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997xp8nm"
Sep 16 19:34:34.501385 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:34.501333    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997xp8nm" podStartSLOduration=10.501316967 podStartE2EDuration="10.501316967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:34:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:34:25.231526675 +0000 UTC m=+3068.873242339" watchObservedRunningTime="2026-09-16 19:34:34.501316967 +0000 UTC m=+3078.143032631"
Sep 16 19:34:35.252083 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:35.252047    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997xp8nm"
Sep 16 19:34:58.955818 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:58.955777    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997xp8nm"]
Sep 16 19:34:58.964871 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:34:58.964833    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:34:58.964871 ip-10-0-129-194 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:34:58.964871 ip-10-0-129-194 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 19:34:58.964871 ip-10-0-129-194 kubenswrapper[2582]:  > containerID="bd99db0db89b4ae1e7c077b74c89dab6762acc9ad5408976399d81f108d6836b" cmd=["/bin/sleep","15"]
Sep 16 19:34:58.965184 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:58.964887    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997xp8nm" message="" err=<
Sep 16 19:34:58.965184 ip-10-0-129-194 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:34:58.965184 ip-10-0-129-194 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 19:34:58.965184 ip-10-0-129-194 kubenswrapper[2582]:  >
Sep 16 19:34:58.965184 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:34:58.964901    2582 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:34:58.965184 ip-10-0-129-194 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:34:58.965184 ip-10-0-129-194 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 19:34:58.965184 ip-10-0-129-194 kubenswrapper[2582]:  > pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997xp8nm" podUID="ad13b154-5bbb-400f-8a0a-70fd88e1fa89" containerName="main" containerID="cri-o://bd99db0db89b4ae1e7c077b74c89dab6762acc9ad5408976399d81f108d6836b"
Sep 16 19:34:58.965184 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:58.964948    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997xp8nm" podUID="ad13b154-5bbb-400f-8a0a-70fd88e1fa89" containerName="main" containerID="cri-o://bd99db0db89b4ae1e7c077b74c89dab6762acc9ad5408976399d81f108d6836b" gracePeriod=60
Sep 16 19:34:59.310173 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:59.310142    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997xp8nm"
Sep 16 19:34:59.320188 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:59.320158    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad13b154-5bbb-400f-8a0a-70fd88e1fa89-home\") pod \"ad13b154-5bbb-400f-8a0a-70fd88e1fa89\" (UID: \"ad13b154-5bbb-400f-8a0a-70fd88e1fa89\") "
Sep 16 19:34:59.320349 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:59.320216    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rpg4l\" (UniqueName: \"kubernetes.io/projected/ad13b154-5bbb-400f-8a0a-70fd88e1fa89-kube-api-access-rpg4l\") pod \"ad13b154-5bbb-400f-8a0a-70fd88e1fa89\" (UID: \"ad13b154-5bbb-400f-8a0a-70fd88e1fa89\") "
Sep 16 19:34:59.320349 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:59.320233    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad13b154-5bbb-400f-8a0a-70fd88e1fa89-model-cache\") pod \"ad13b154-5bbb-400f-8a0a-70fd88e1fa89\" (UID: \"ad13b154-5bbb-400f-8a0a-70fd88e1fa89\") "
Sep 16 19:34:59.320349 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:59.320270    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad13b154-5bbb-400f-8a0a-70fd88e1fa89-tls-certs\") pod \"ad13b154-5bbb-400f-8a0a-70fd88e1fa89\" (UID: \"ad13b154-5bbb-400f-8a0a-70fd88e1fa89\") "
Sep 16 19:34:59.320349 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:59.320291    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad13b154-5bbb-400f-8a0a-70fd88e1fa89-dshm\") pod \"ad13b154-5bbb-400f-8a0a-70fd88e1fa89\" (UID: \"ad13b154-5bbb-400f-8a0a-70fd88e1fa89\") "
Sep 16 19:34:59.320349 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:59.320308    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad13b154-5bbb-400f-8a0a-70fd88e1fa89-tmp-dir\") pod \"ad13b154-5bbb-400f-8a0a-70fd88e1fa89\" (UID: \"ad13b154-5bbb-400f-8a0a-70fd88e1fa89\") "
Sep 16 19:34:59.320620 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:59.320457    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad13b154-5bbb-400f-8a0a-70fd88e1fa89-home" (OuterVolumeSpecName: "home") pod "ad13b154-5bbb-400f-8a0a-70fd88e1fa89" (UID: "ad13b154-5bbb-400f-8a0a-70fd88e1fa89"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:34:59.320620 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:59.320531    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad13b154-5bbb-400f-8a0a-70fd88e1fa89-model-cache" (OuterVolumeSpecName: "model-cache") pod "ad13b154-5bbb-400f-8a0a-70fd88e1fa89" (UID: "ad13b154-5bbb-400f-8a0a-70fd88e1fa89"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:34:59.320737 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:59.320659    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad13b154-5bbb-400f-8a0a-70fd88e1fa89-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ad13b154-5bbb-400f-8a0a-70fd88e1fa89" (UID: "ad13b154-5bbb-400f-8a0a-70fd88e1fa89"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:34:59.322573 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:59.322548    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad13b154-5bbb-400f-8a0a-70fd88e1fa89-dshm" (OuterVolumeSpecName: "dshm") pod "ad13b154-5bbb-400f-8a0a-70fd88e1fa89" (UID: "ad13b154-5bbb-400f-8a0a-70fd88e1fa89"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:34:59.322683 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:59.322584    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad13b154-5bbb-400f-8a0a-70fd88e1fa89-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ad13b154-5bbb-400f-8a0a-70fd88e1fa89" (UID: "ad13b154-5bbb-400f-8a0a-70fd88e1fa89"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:34:59.322683 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:59.322605    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad13b154-5bbb-400f-8a0a-70fd88e1fa89-kube-api-access-rpg4l" (OuterVolumeSpecName: "kube-api-access-rpg4l") pod "ad13b154-5bbb-400f-8a0a-70fd88e1fa89" (UID: "ad13b154-5bbb-400f-8a0a-70fd88e1fa89"). InnerVolumeSpecName "kube-api-access-rpg4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:34:59.340485 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:59.340449    2582 generic.go:356] "Generic (PLEG): container finished" podID="ad13b154-5bbb-400f-8a0a-70fd88e1fa89" containerID="bd99db0db89b4ae1e7c077b74c89dab6762acc9ad5408976399d81f108d6836b" exitCode=0
Sep 16 19:34:59.340653 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:59.340493    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997xp8nm"
Sep 16 19:34:59.340653 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:59.340538    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997xp8nm" event={"ID":"ad13b154-5bbb-400f-8a0a-70fd88e1fa89","Type":"ContainerDied","Data":"bd99db0db89b4ae1e7c077b74c89dab6762acc9ad5408976399d81f108d6836b"}
Sep 16 19:34:59.340653 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:59.340606    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997xp8nm" event={"ID":"ad13b154-5bbb-400f-8a0a-70fd88e1fa89","Type":"ContainerDied","Data":"2f40eaa0e3cc24be36e018f233322ae2eb478ee088f4b9f4952486c5b769e9fe"}
Sep 16 19:34:59.340653 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:59.340640    2582 scope.go:117] "RemoveContainer" containerID="bd99db0db89b4ae1e7c077b74c89dab6762acc9ad5408976399d81f108d6836b"
Sep 16 19:34:59.350068 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:59.350047    2582 scope.go:117] "RemoveContainer" containerID="bd99db0db89b4ae1e7c077b74c89dab6762acc9ad5408976399d81f108d6836b"
Sep 16 19:34:59.350466 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:34:59.350447    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd99db0db89b4ae1e7c077b74c89dab6762acc9ad5408976399d81f108d6836b\": container with ID starting with bd99db0db89b4ae1e7c077b74c89dab6762acc9ad5408976399d81f108d6836b not found: ID does not exist" containerID="bd99db0db89b4ae1e7c077b74c89dab6762acc9ad5408976399d81f108d6836b"
Sep 16 19:34:59.350560 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:59.350474    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd99db0db89b4ae1e7c077b74c89dab6762acc9ad5408976399d81f108d6836b"} err="failed to get container status \"bd99db0db89b4ae1e7c077b74c89dab6762acc9ad5408976399d81f108d6836b\": rpc error: code = NotFound desc = could not find container \"bd99db0db89b4ae1e7c077b74c89dab6762acc9ad5408976399d81f108d6836b\": container with ID starting with bd99db0db89b4ae1e7c077b74c89dab6762acc9ad5408976399d81f108d6836b not found: ID does not exist"
Sep 16 19:34:59.368361 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:59.368321    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997xp8nm"]
Sep 16 19:34:59.374222 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:59.374194    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997xp8nm"]
Sep 16 19:34:59.421864 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:59.421831    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad13b154-5bbb-400f-8a0a-70fd88e1fa89-home\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:34:59.421864 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:59.421863    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpg4l\" (UniqueName: \"kubernetes.io/projected/ad13b154-5bbb-400f-8a0a-70fd88e1fa89-kube-api-access-rpg4l\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:34:59.422066 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:59.421880    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad13b154-5bbb-400f-8a0a-70fd88e1fa89-model-cache\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:34:59.422066 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:59.421892    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad13b154-5bbb-400f-8a0a-70fd88e1fa89-tls-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:34:59.422066 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:59.421902    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad13b154-5bbb-400f-8a0a-70fd88e1fa89-dshm\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:34:59.422066 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:34:59.421912    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad13b154-5bbb-400f-8a0a-70fd88e1fa89-tmp-dir\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:35:00.241806 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:35:00.241746    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997xp8nm" podUID="ad13b154-5bbb-400f-8a0a-70fd88e1fa89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": context deadline exceeded"
Sep 16 19:35:00.873187 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:35:00.873149    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad13b154-5bbb-400f-8a0a-70fd88e1fa89" path="/var/lib/kubelet/pods/ad13b154-5bbb-400f-8a0a-70fd88e1fa89/volumes"
Sep 16 19:35:12.002193 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:35:12.002081    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr"]
Sep 16 19:35:12.002661 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:35:12.002533    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad13b154-5bbb-400f-8a0a-70fd88e1fa89" containerName="main"
Sep 16 19:35:12.002661 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:35:12.002552    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad13b154-5bbb-400f-8a0a-70fd88e1fa89" containerName="main"
Sep 16 19:35:12.002661 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:35:12.002639    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad13b154-5bbb-400f-8a0a-70fd88e1fa89" containerName="main"
Sep 16 19:35:12.006988 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:35:12.006963    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr"
Sep 16 19:35:12.010064 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:35:12.010033    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Sep 16 19:35:12.010217 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:35:12.010086    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"seaweedfs-s3-creds\""
Sep 16 19:35:12.010217 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:35:12.010050    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:35:12.010217 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:35:12.010047    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-264n7\""
Sep 16 19:35:12.010217 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:35:12.010167    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Sep 16 19:35:12.010217 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:35:12.010054    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Sep 16 19:35:12.017289 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:35:12.017258    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr"]
Sep 16 19:35:12.126513 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:35:12.126474    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c05b007d-6d0c-46a1-8e4d-e238a53ac163-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr\" (UID: \"c05b007d-6d0c-46a1-8e4d-e238a53ac163\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr"
Sep 16 19:35:12.126513 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:35:12.126516    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c05b007d-6d0c-46a1-8e4d-e238a53ac163-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr\" (UID: \"c05b007d-6d0c-46a1-8e4d-e238a53ac163\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr"
Sep 16 19:35:12.126753 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:35:12.126541    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c05b007d-6d0c-46a1-8e4d-e238a53ac163-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr\" (UID: \"c05b007d-6d0c-46a1-8e4d-e238a53ac163\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr"
Sep 16 19:35:12.126753 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:35:12.126622    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c05b007d-6d0c-46a1-8e4d-e238a53ac163-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr\" (UID: \"c05b007d-6d0c-46a1-8e4d-e238a53ac163\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr"
Sep 16 19:35:12.126753 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:35:12.126659    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c05b007d-6d0c-46a1-8e4d-e238a53ac163-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr\" (UID: \"c05b007d-6d0c-46a1-8e4d-e238a53ac163\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr"
Sep 16 19:35:12.126891 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:35:12.126752    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c05b007d-6d0c-46a1-8e4d-e238a53ac163-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr\" (UID: \"c05b007d-6d0c-46a1-8e4d-e238a53ac163\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr"
Sep 16 19:35:12.126891 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:35:12.126782    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c05b007d-6d0c-46a1-8e4d-e238a53ac163-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr\" (UID: \"c05b007d-6d0c-46a1-8e4d-e238a53ac163\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr"
Sep 16 19:35:12.126891 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:35:12.126806    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmhn7\" (UniqueName: \"kubernetes.io/projected/c05b007d-6d0c-46a1-8e4d-e238a53ac163-kube-api-access-xmhn7\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr\" (UID: \"c05b007d-6d0c-46a1-8e4d-e238a53ac163\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr"
Sep 16 19:35:12.227982 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:35:12.227943    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c05b007d-6d0c-46a1-8e4d-e238a53ac163-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr\" (UID: \"c05b007d-6d0c-46a1-8e4d-e238a53ac163\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr"
Sep 16 19:35:12.227982 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:35:12.227989    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c05b007d-6d0c-46a1-8e4d-e238a53ac163-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr\" (UID: \"c05b007d-6d0c-46a1-8e4d-e238a53ac163\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr"
Sep 16 19:35:12.228236 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:35:12.228011    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmhn7\" (UniqueName: \"kubernetes.io/projected/c05b007d-6d0c-46a1-8e4d-e238a53ac163-kube-api-access-xmhn7\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr\" (UID: \"c05b007d-6d0c-46a1-8e4d-e238a53ac163\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr"
Sep 16 19:35:12.228236 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:35:12.228044    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c05b007d-6d0c-46a1-8e4d-e238a53ac163-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr\" (UID: \"c05b007d-6d0c-46a1-8e4d-e238a53ac163\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr"
Sep 16 19:35:12.228236 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:35:12.228076    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c05b007d-6d0c-46a1-8e4d-e238a53ac163-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr\" (UID: \"c05b007d-6d0c-46a1-8e4d-e238a53ac163\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr"
Sep 16 19:35:12.228236 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:35:12.228101    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c05b007d-6d0c-46a1-8e4d-e238a53ac163-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr\" (UID: \"c05b007d-6d0c-46a1-8e4d-e238a53ac163\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr"
Sep 16 19:35:12.228236 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:35:12.228136    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c05b007d-6d0c-46a1-8e4d-e238a53ac163-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr\" (UID: \"c05b007d-6d0c-46a1-8e4d-e238a53ac163\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr"
Sep 16 19:35:12.228236 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:35:12.228168    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c05b007d-6d0c-46a1-8e4d-e238a53ac163-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr\" (UID: \"c05b007d-6d0c-46a1-8e4d-e238a53ac163\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr"
Sep 16 19:35:12.228551 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:35:12.228467    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c05b007d-6d0c-46a1-8e4d-e238a53ac163-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr\" (UID: \"c05b007d-6d0c-46a1-8e4d-e238a53ac163\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr"
Sep 16 19:35:12.228551 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:35:12.228524    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c05b007d-6d0c-46a1-8e4d-e238a53ac163-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr\" (UID: \"c05b007d-6d0c-46a1-8e4d-e238a53ac163\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr"
Sep 16 19:35:12.228635 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:35:12.228582    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c05b007d-6d0c-46a1-8e4d-e238a53ac163-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr\" (UID: \"c05b007d-6d0c-46a1-8e4d-e238a53ac163\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr"
Sep 16 19:35:12.228673 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:35:12.228653    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c05b007d-6d0c-46a1-8e4d-e238a53ac163-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr\" (UID: \"c05b007d-6d0c-46a1-8e4d-e238a53ac163\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr"
Sep 16 19:35:12.228879 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:35:12.228857    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c05b007d-6d0c-46a1-8e4d-e238a53ac163-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr\" (UID: \"c05b007d-6d0c-46a1-8e4d-e238a53ac163\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr"
Sep 16 19:35:12.230576 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:35:12.230553    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c05b007d-6d0c-46a1-8e4d-e238a53ac163-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr\" (UID: \"c05b007d-6d0c-46a1-8e4d-e238a53ac163\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr"
Sep 16 19:35:12.230711 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:35:12.230694    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c05b007d-6d0c-46a1-8e4d-e238a53ac163-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr\" (UID: \"c05b007d-6d0c-46a1-8e4d-e238a53ac163\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr"
Sep 16 19:35:12.236809 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:35:12.236786    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmhn7\" (UniqueName: \"kubernetes.io/projected/c05b007d-6d0c-46a1-8e4d-e238a53ac163-kube-api-access-xmhn7\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr\" (UID: \"c05b007d-6d0c-46a1-8e4d-e238a53ac163\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr"
Sep 16 19:35:12.317893 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:35:12.317851    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr"
Sep 16 19:35:12.608567 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:35:12.608514    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr"]
Sep 16 19:35:12.610910 ip-10-0-129-194 kubenswrapper[2582]: W0916 19:35:12.610878    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc05b007d_6d0c_46a1_8e4d_e238a53ac163.slice/crio-93435832291b3ed4f480f6245ba6c3f3966eebc66adebb79122bc6f613289a93 WatchSource:0}: Error finding container 93435832291b3ed4f480f6245ba6c3f3966eebc66adebb79122bc6f613289a93: Status 404 returned error can't find the container with id 93435832291b3ed4f480f6245ba6c3f3966eebc66adebb79122bc6f613289a93
Sep 16 19:35:13.406856 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:35:13.406818    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr" event={"ID":"c05b007d-6d0c-46a1-8e4d-e238a53ac163","Type":"ContainerStarted","Data":"9a4009103356f3d8cfa95bc13d6c41e14799f63496dcaa1dfc8fef8f67edf11f"}
Sep 16 19:35:13.407265 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:35:13.406875    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr" event={"ID":"c05b007d-6d0c-46a1-8e4d-e238a53ac163","Type":"ContainerStarted","Data":"93435832291b3ed4f480f6245ba6c3f3966eebc66adebb79122bc6f613289a93"}
Sep 16 19:35:15.416942 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:35:15.416907    2582 generic.go:356] "Generic (PLEG): container finished" podID="c05b007d-6d0c-46a1-8e4d-e238a53ac163" containerID="9a4009103356f3d8cfa95bc13d6c41e14799f63496dcaa1dfc8fef8f67edf11f" exitCode=0
Sep 16 19:35:15.417340 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:35:15.416947    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr" event={"ID":"c05b007d-6d0c-46a1-8e4d-e238a53ac163","Type":"ContainerDied","Data":"9a4009103356f3d8cfa95bc13d6c41e14799f63496dcaa1dfc8fef8f67edf11f"}
Sep 16 19:35:16.424655 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:35:16.424621    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr" event={"ID":"c05b007d-6d0c-46a1-8e4d-e238a53ac163","Type":"ContainerStarted","Data":"3fe2a53b1f8217207e86cfa2baab7a595b900bfcf108bc14c9921f0fac7e5a82"}
Sep 16 19:35:16.449040 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:35:16.448982    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr" podStartSLOduration=3.276054507 podStartE2EDuration="5.448966247s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.17291174s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:35:16.446284702 +0000 UTC m=+3120.088000366" watchObservedRunningTime="2026-09-16 19:35:16.448966247 +0000 UTC m=+3120.090681911"
Sep 16 19:35:17.317967 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:35:17.317922    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr"
Sep 16 19:35:22.318325 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:35:22.318279    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr"
Sep 16 19:35:22.319884 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:35:22.319853    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr" podUID="c05b007d-6d0c-46a1-8e4d-e238a53ac163" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Sep 16 19:35:32.318579 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:35:32.318533    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr" podUID="c05b007d-6d0c-46a1-8e4d-e238a53ac163" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Sep 16 19:35:42.318787 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:35:42.318674    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr" podUID="c05b007d-6d0c-46a1-8e4d-e238a53ac163" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Sep 16 19:35:52.319209 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:35:52.319153    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr" podUID="c05b007d-6d0c-46a1-8e4d-e238a53ac163" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Sep 16 19:36:02.319164 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:36:02.319117    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr" podUID="c05b007d-6d0c-46a1-8e4d-e238a53ac163" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Sep 16 19:36:12.318888 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:36:12.318844    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr" podUID="c05b007d-6d0c-46a1-8e4d-e238a53ac163" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Sep 16 19:36:22.318531 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:36:22.318479    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr" podUID="c05b007d-6d0c-46a1-8e4d-e238a53ac163" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Sep 16 19:36:32.318498 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:36:32.318449    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr" podUID="c05b007d-6d0c-46a1-8e4d-e238a53ac163" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Sep 16 19:36:42.318634 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:36:42.318591    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr" podUID="c05b007d-6d0c-46a1-8e4d-e238a53ac163" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Sep 16 19:36:52.318524 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:36:52.318479    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr" podUID="c05b007d-6d0c-46a1-8e4d-e238a53ac163" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Sep 16 19:37:02.319137 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:37:02.319092    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr" podUID="c05b007d-6d0c-46a1-8e4d-e238a53ac163" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Sep 16 19:37:12.329230 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:37:12.329149    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr"
Sep 16 19:37:12.337059 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:37:12.337037    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr"
Sep 16 19:37:17.686949 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:37:17.686912    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr"]
Sep 16 19:37:32.708834 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:37:32.708769    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr" podUID="c05b007d-6d0c-46a1-8e4d-e238a53ac163" containerName="main" containerID="cri-o://3fe2a53b1f8217207e86cfa2baab7a595b900bfcf108bc14c9921f0fac7e5a82" gracePeriod=45
Sep 16 19:37:33.329884 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:37:33.329838    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr" podUID="c05b007d-6d0c-46a1-8e4d-e238a53ac163" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Sep 16 19:37:33.453835 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:37:33.453801    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr"
Sep 16 19:37:33.548732 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:37:33.548682    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c05b007d-6d0c-46a1-8e4d-e238a53ac163-cabundle-cert\") pod \"c05b007d-6d0c-46a1-8e4d-e238a53ac163\" (UID: \"c05b007d-6d0c-46a1-8e4d-e238a53ac163\") "
Sep 16 19:37:33.548912 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:37:33.548742    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmhn7\" (UniqueName: \"kubernetes.io/projected/c05b007d-6d0c-46a1-8e4d-e238a53ac163-kube-api-access-xmhn7\") pod \"c05b007d-6d0c-46a1-8e4d-e238a53ac163\" (UID: \"c05b007d-6d0c-46a1-8e4d-e238a53ac163\") "
Sep 16 19:37:33.548912 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:37:33.548811    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c05b007d-6d0c-46a1-8e4d-e238a53ac163-dshm\") pod \"c05b007d-6d0c-46a1-8e4d-e238a53ac163\" (UID: \"c05b007d-6d0c-46a1-8e4d-e238a53ac163\") "
Sep 16 19:37:33.548912 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:37:33.548837    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c05b007d-6d0c-46a1-8e4d-e238a53ac163-home\") pod \"c05b007d-6d0c-46a1-8e4d-e238a53ac163\" (UID: \"c05b007d-6d0c-46a1-8e4d-e238a53ac163\") "
Sep 16 19:37:33.548912 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:37:33.548891    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c05b007d-6d0c-46a1-8e4d-e238a53ac163-tmp-dir\") pod \"c05b007d-6d0c-46a1-8e4d-e238a53ac163\" (UID: \"c05b007d-6d0c-46a1-8e4d-e238a53ac163\") "
Sep 16 19:37:33.549144 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:37:33.548917    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c05b007d-6d0c-46a1-8e4d-e238a53ac163-tls-certs\") pod \"c05b007d-6d0c-46a1-8e4d-e238a53ac163\" (UID: \"c05b007d-6d0c-46a1-8e4d-e238a53ac163\") "
Sep 16 19:37:33.549144 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:37:33.548942    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c05b007d-6d0c-46a1-8e4d-e238a53ac163-model-cache\") pod \"c05b007d-6d0c-46a1-8e4d-e238a53ac163\" (UID: \"c05b007d-6d0c-46a1-8e4d-e238a53ac163\") "
Sep 16 19:37:33.549144 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:37:33.548995    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c05b007d-6d0c-46a1-8e4d-e238a53ac163-kserve-provision-location\") pod \"c05b007d-6d0c-46a1-8e4d-e238a53ac163\" (UID: \"c05b007d-6d0c-46a1-8e4d-e238a53ac163\") "
Sep 16 19:37:33.549144 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:37:33.549087    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c05b007d-6d0c-46a1-8e4d-e238a53ac163-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c05b007d-6d0c-46a1-8e4d-e238a53ac163" (UID: "c05b007d-6d0c-46a1-8e4d-e238a53ac163"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:37:33.549357 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:37:33.549240    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c05b007d-6d0c-46a1-8e4d-e238a53ac163-model-cache" (OuterVolumeSpecName: "model-cache") pod "c05b007d-6d0c-46a1-8e4d-e238a53ac163" (UID: "c05b007d-6d0c-46a1-8e4d-e238a53ac163"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:37:33.549357 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:37:33.549251    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c05b007d-6d0c-46a1-8e4d-e238a53ac163-cabundle-cert\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:37:33.549824 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:37:33.549790    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c05b007d-6d0c-46a1-8e4d-e238a53ac163-home" (OuterVolumeSpecName: "home") pod "c05b007d-6d0c-46a1-8e4d-e238a53ac163" (UID: "c05b007d-6d0c-46a1-8e4d-e238a53ac163"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:37:33.551196 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:37:33.551173    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c05b007d-6d0c-46a1-8e4d-e238a53ac163-dshm" (OuterVolumeSpecName: "dshm") pod "c05b007d-6d0c-46a1-8e4d-e238a53ac163" (UID: "c05b007d-6d0c-46a1-8e4d-e238a53ac163"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:37:33.551196 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:37:33.551188    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c05b007d-6d0c-46a1-8e4d-e238a53ac163-kube-api-access-xmhn7" (OuterVolumeSpecName: "kube-api-access-xmhn7") pod "c05b007d-6d0c-46a1-8e4d-e238a53ac163" (UID: "c05b007d-6d0c-46a1-8e4d-e238a53ac163"). InnerVolumeSpecName "kube-api-access-xmhn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:37:33.551486 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:37:33.551463    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c05b007d-6d0c-46a1-8e4d-e238a53ac163-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c05b007d-6d0c-46a1-8e4d-e238a53ac163" (UID: "c05b007d-6d0c-46a1-8e4d-e238a53ac163"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:37:33.567375 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:37:33.567340    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c05b007d-6d0c-46a1-8e4d-e238a53ac163-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c05b007d-6d0c-46a1-8e4d-e238a53ac163" (UID: "c05b007d-6d0c-46a1-8e4d-e238a53ac163"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:37:33.571651 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:37:33.571619    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c05b007d-6d0c-46a1-8e4d-e238a53ac163-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c05b007d-6d0c-46a1-8e4d-e238a53ac163" (UID: "c05b007d-6d0c-46a1-8e4d-e238a53ac163"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:37:33.650682 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:37:33.650637    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmhn7\" (UniqueName: \"kubernetes.io/projected/c05b007d-6d0c-46a1-8e4d-e238a53ac163-kube-api-access-xmhn7\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:37:33.650682 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:37:33.650679    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c05b007d-6d0c-46a1-8e4d-e238a53ac163-dshm\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:37:33.650682 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:37:33.650692    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c05b007d-6d0c-46a1-8e4d-e238a53ac163-home\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:37:33.650964 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:37:33.650704    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c05b007d-6d0c-46a1-8e4d-e238a53ac163-tmp-dir\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:37:33.650964 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:37:33.650742    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c05b007d-6d0c-46a1-8e4d-e238a53ac163-tls-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:37:33.650964 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:37:33.650754    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c05b007d-6d0c-46a1-8e4d-e238a53ac163-model-cache\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:37:33.650964 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:37:33.650765    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c05b007d-6d0c-46a1-8e4d-e238a53ac163-kserve-provision-location\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:37:34.022683 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:37:34.022649    2582 generic.go:356] "Generic (PLEG): container finished" podID="c05b007d-6d0c-46a1-8e4d-e238a53ac163" containerID="3fe2a53b1f8217207e86cfa2baab7a595b900bfcf108bc14c9921f0fac7e5a82" exitCode=0
Sep 16 19:37:34.023112 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:37:34.022703    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr"
Sep 16 19:37:34.023112 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:37:34.022761    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr" event={"ID":"c05b007d-6d0c-46a1-8e4d-e238a53ac163","Type":"ContainerDied","Data":"3fe2a53b1f8217207e86cfa2baab7a595b900bfcf108bc14c9921f0fac7e5a82"}
Sep 16 19:37:34.023112 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:37:34.022817    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr" event={"ID":"c05b007d-6d0c-46a1-8e4d-e238a53ac163","Type":"ContainerDied","Data":"93435832291b3ed4f480f6245ba6c3f3966eebc66adebb79122bc6f613289a93"}
Sep 16 19:37:34.023112 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:37:34.022840    2582 scope.go:117] "RemoveContainer" containerID="3fe2a53b1f8217207e86cfa2baab7a595b900bfcf108bc14c9921f0fac7e5a82"
Sep 16 19:37:34.031753 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:37:34.031710    2582 scope.go:117] "RemoveContainer" containerID="9a4009103356f3d8cfa95bc13d6c41e14799f63496dcaa1dfc8fef8f67edf11f"
Sep 16 19:37:34.040045 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:37:34.040025    2582 scope.go:117] "RemoveContainer" containerID="3fe2a53b1f8217207e86cfa2baab7a595b900bfcf108bc14c9921f0fac7e5a82"
Sep 16 19:37:34.040334 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:37:34.040314    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fe2a53b1f8217207e86cfa2baab7a595b900bfcf108bc14c9921f0fac7e5a82\": container with ID starting with 3fe2a53b1f8217207e86cfa2baab7a595b900bfcf108bc14c9921f0fac7e5a82 not found: ID does not exist" containerID="3fe2a53b1f8217207e86cfa2baab7a595b900bfcf108bc14c9921f0fac7e5a82"
Sep 16 19:37:34.040413 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:37:34.040347    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fe2a53b1f8217207e86cfa2baab7a595b900bfcf108bc14c9921f0fac7e5a82"} err="failed to get container status \"3fe2a53b1f8217207e86cfa2baab7a595b900bfcf108bc14c9921f0fac7e5a82\": rpc error: code = NotFound desc = could not find container \"3fe2a53b1f8217207e86cfa2baab7a595b900bfcf108bc14c9921f0fac7e5a82\": container with ID starting with 3fe2a53b1f8217207e86cfa2baab7a595b900bfcf108bc14c9921f0fac7e5a82 not found: ID does not exist"
Sep 16 19:37:34.040413 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:37:34.040370    2582 scope.go:117] "RemoveContainer" containerID="9a4009103356f3d8cfa95bc13d6c41e14799f63496dcaa1dfc8fef8f67edf11f"
Sep 16 19:37:34.040645 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:37:34.040626    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a4009103356f3d8cfa95bc13d6c41e14799f63496dcaa1dfc8fef8f67edf11f\": container with ID starting with 9a4009103356f3d8cfa95bc13d6c41e14799f63496dcaa1dfc8fef8f67edf11f not found: ID does not exist" containerID="9a4009103356f3d8cfa95bc13d6c41e14799f63496dcaa1dfc8fef8f67edf11f"
Sep 16 19:37:34.040689 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:37:34.040650    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a4009103356f3d8cfa95bc13d6c41e14799f63496dcaa1dfc8fef8f67edf11f"} err="failed to get container status \"9a4009103356f3d8cfa95bc13d6c41e14799f63496dcaa1dfc8fef8f67edf11f\": rpc error: code = NotFound desc = could not find container \"9a4009103356f3d8cfa95bc13d6c41e14799f63496dcaa1dfc8fef8f67edf11f\": container with ID starting with 9a4009103356f3d8cfa95bc13d6c41e14799f63496dcaa1dfc8fef8f67edf11f not found: ID does not exist"
Sep 16 19:37:34.074033 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:37:34.074000    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr"]
Sep 16 19:37:34.086304 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:37:34.086271    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-585b5894bcb2hcr"]
Sep 16 19:37:34.872932 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:37:34.872899    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c05b007d-6d0c-46a1-8e4d-e238a53ac163" path="/var/lib/kubelet/pods/c05b007d-6d0c-46a1-8e4d-e238a53ac163/volumes"
Sep 16 19:38:17.061790 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:38:17.061757    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7qwl2_b6695f86-08c0-4836-87b1-f2a9d76084bd/console-operator/1.log"
Sep 16 19:38:17.068587 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:38:17.068565    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7qwl2_b6695f86-08c0-4836-87b1-f2a9d76084bd/console-operator/1.log"
Sep 16 19:39:04.752142 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:39:04.752059    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-dfph6"]
Sep 16 19:39:04.752601 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:39:04.752408    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c05b007d-6d0c-46a1-8e4d-e238a53ac163" containerName="storage-initializer"
Sep 16 19:39:04.752601 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:39:04.752423    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c05b007d-6d0c-46a1-8e4d-e238a53ac163" containerName="storage-initializer"
Sep 16 19:39:04.752601 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:39:04.752445    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c05b007d-6d0c-46a1-8e4d-e238a53ac163" containerName="main"
Sep 16 19:39:04.752601 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:39:04.752452    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c05b007d-6d0c-46a1-8e4d-e238a53ac163" containerName="main"
Sep 16 19:39:04.752601 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:39:04.752508    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c05b007d-6d0c-46a1-8e4d-e238a53ac163" containerName="main"
Sep 16 19:39:04.755438 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:39:04.755418    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-dfph6"
Sep 16 19:39:04.758655 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:39:04.758630    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Sep 16 19:39:04.758805 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:39:04.758736    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-9766v\""
Sep 16 19:39:04.759114 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:39:04.759094    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 16 19:39:04.759114 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:39:04.759103    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 16 19:39:04.759439 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:39:04.759425    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 16 19:39:04.759963 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:39:04.759946    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:39:04.782827 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:39:04.782799    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-dfph6"]
Sep 16 19:39:04.858793 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:39:04.858749    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8938281-3070-4c3d-8a47-181c7f6fa6a9-tmp-dir\") pod \"stop-feature-test-kserve-7b545f4984-dfph6\" (UID: \"e8938281-3070-4c3d-8a47-181c7f6fa6a9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-dfph6"
Sep 16 19:39:04.858793 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:39:04.858794    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8938281-3070-4c3d-8a47-181c7f6fa6a9-model-cache\") pod \"stop-feature-test-kserve-7b545f4984-dfph6\" (UID: \"e8938281-3070-4c3d-8a47-181c7f6fa6a9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-dfph6"
Sep 16 19:39:04.859006 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:39:04.858864    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcv8j\" (UniqueName: \"kubernetes.io/projected/e8938281-3070-4c3d-8a47-181c7f6fa6a9-kube-api-access-fcv8j\") pod \"stop-feature-test-kserve-7b545f4984-dfph6\" (UID: \"e8938281-3070-4c3d-8a47-181c7f6fa6a9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-dfph6"
Sep 16 19:39:04.859006 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:39:04.858899    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8938281-3070-4c3d-8a47-181c7f6fa6a9-home\") pod \"stop-feature-test-kserve-7b545f4984-dfph6\" (UID: \"e8938281-3070-4c3d-8a47-181c7f6fa6a9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-dfph6"
Sep 16 19:39:04.859006 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:39:04.858921    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8938281-3070-4c3d-8a47-181c7f6fa6a9-kserve-provision-location\") pod \"stop-feature-test-kserve-7b545f4984-dfph6\" (UID: \"e8938281-3070-4c3d-8a47-181c7f6fa6a9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-dfph6"
Sep 16 19:39:04.859006 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:39:04.858949    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8938281-3070-4c3d-8a47-181c7f6fa6a9-dshm\") pod \"stop-feature-test-kserve-7b545f4984-dfph6\" (UID: \"e8938281-3070-4c3d-8a47-181c7f6fa6a9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-dfph6"
Sep 16 19:39:04.859006 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:39:04.858972    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8938281-3070-4c3d-8a47-181c7f6fa6a9-cabundle-cert\") pod \"stop-feature-test-kserve-7b545f4984-dfph6\" (UID: \"e8938281-3070-4c3d-8a47-181c7f6fa6a9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-dfph6"
Sep 16 19:39:04.859006 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:39:04.858991    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8938281-3070-4c3d-8a47-181c7f6fa6a9-tls-certs\") pod \"stop-feature-test-kserve-7b545f4984-dfph6\" (UID: \"e8938281-3070-4c3d-8a47-181c7f6fa6a9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-dfph6"
Sep 16 19:39:04.959354 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:39:04.959312    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8938281-3070-4c3d-8a47-181c7f6fa6a9-tmp-dir\") pod \"stop-feature-test-kserve-7b545f4984-dfph6\" (UID: \"e8938281-3070-4c3d-8a47-181c7f6fa6a9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-dfph6"
Sep 16 19:39:04.959354 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:39:04.959353    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8938281-3070-4c3d-8a47-181c7f6fa6a9-model-cache\") pod \"stop-feature-test-kserve-7b545f4984-dfph6\" (UID: \"e8938281-3070-4c3d-8a47-181c7f6fa6a9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-dfph6"
Sep 16 19:39:04.959618 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:39:04.959402    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcv8j\" (UniqueName: \"kubernetes.io/projected/e8938281-3070-4c3d-8a47-181c7f6fa6a9-kube-api-access-fcv8j\") pod \"stop-feature-test-kserve-7b545f4984-dfph6\" (UID: \"e8938281-3070-4c3d-8a47-181c7f6fa6a9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-dfph6"
Sep 16 19:39:04.959618 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:39:04.959429    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8938281-3070-4c3d-8a47-181c7f6fa6a9-home\") pod \"stop-feature-test-kserve-7b545f4984-dfph6\" (UID: \"e8938281-3070-4c3d-8a47-181c7f6fa6a9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-dfph6"
Sep 16 19:39:04.959618 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:39:04.959458    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8938281-3070-4c3d-8a47-181c7f6fa6a9-kserve-provision-location\") pod \"stop-feature-test-kserve-7b545f4984-dfph6\" (UID: \"e8938281-3070-4c3d-8a47-181c7f6fa6a9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-dfph6"
Sep 16 19:39:04.959618 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:39:04.959484    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8938281-3070-4c3d-8a47-181c7f6fa6a9-dshm\") pod \"stop-feature-test-kserve-7b545f4984-dfph6\" (UID: \"e8938281-3070-4c3d-8a47-181c7f6fa6a9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-dfph6"
Sep 16 19:39:04.959618 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:39:04.959510    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8938281-3070-4c3d-8a47-181c7f6fa6a9-cabundle-cert\") pod \"stop-feature-test-kserve-7b545f4984-dfph6\" (UID: \"e8938281-3070-4c3d-8a47-181c7f6fa6a9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-dfph6"
Sep 16 19:39:04.959618 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:39:04.959582    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8938281-3070-4c3d-8a47-181c7f6fa6a9-tls-certs\") pod \"stop-feature-test-kserve-7b545f4984-dfph6\" (UID: \"e8938281-3070-4c3d-8a47-181c7f6fa6a9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-dfph6"
Sep 16 19:39:04.959910 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:39:04.959796    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8938281-3070-4c3d-8a47-181c7f6fa6a9-tmp-dir\") pod \"stop-feature-test-kserve-7b545f4984-dfph6\" (UID: \"e8938281-3070-4c3d-8a47-181c7f6fa6a9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-dfph6"
Sep 16 19:39:04.959910 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:39:04.959849    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8938281-3070-4c3d-8a47-181c7f6fa6a9-model-cache\") pod \"stop-feature-test-kserve-7b545f4984-dfph6\" (UID: \"e8938281-3070-4c3d-8a47-181c7f6fa6a9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-dfph6"
Sep 16 19:39:04.959979 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:39:04.959926    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8938281-3070-4c3d-8a47-181c7f6fa6a9-home\") pod \"stop-feature-test-kserve-7b545f4984-dfph6\" (UID: \"e8938281-3070-4c3d-8a47-181c7f6fa6a9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-dfph6"
Sep 16 19:39:04.960040 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:39:04.960026    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8938281-3070-4c3d-8a47-181c7f6fa6a9-kserve-provision-location\") pod \"stop-feature-test-kserve-7b545f4984-dfph6\" (UID: \"e8938281-3070-4c3d-8a47-181c7f6fa6a9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-dfph6"
Sep 16 19:39:04.960235 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:39:04.960214    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8938281-3070-4c3d-8a47-181c7f6fa6a9-cabundle-cert\") pod \"stop-feature-test-kserve-7b545f4984-dfph6\" (UID: \"e8938281-3070-4c3d-8a47-181c7f6fa6a9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-dfph6"
Sep 16 19:39:04.961908 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:39:04.961889    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8938281-3070-4c3d-8a47-181c7f6fa6a9-dshm\") pod \"stop-feature-test-kserve-7b545f4984-dfph6\" (UID: \"e8938281-3070-4c3d-8a47-181c7f6fa6a9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-dfph6"
Sep 16 19:39:04.962096 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:39:04.962071    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8938281-3070-4c3d-8a47-181c7f6fa6a9-tls-certs\") pod \"stop-feature-test-kserve-7b545f4984-dfph6\" (UID: \"e8938281-3070-4c3d-8a47-181c7f6fa6a9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-dfph6"
Sep 16 19:39:04.969397 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:39:04.969375    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcv8j\" (UniqueName: \"kubernetes.io/projected/e8938281-3070-4c3d-8a47-181c7f6fa6a9-kube-api-access-fcv8j\") pod \"stop-feature-test-kserve-7b545f4984-dfph6\" (UID: \"e8938281-3070-4c3d-8a47-181c7f6fa6a9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-dfph6"
Sep 16 19:39:05.065837 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:39:05.065806    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-dfph6"
Sep 16 19:39:05.352983 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:39:05.352892    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-dfph6"]
Sep 16 19:39:05.359260 ip-10-0-129-194 kubenswrapper[2582]: W0916 19:39:05.359220    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8938281_3070_4c3d_8a47_181c7f6fa6a9.slice/crio-65f540264f2daacc709bf6871990d622cbbad29f7db1a43c94ace0175db35b10 WatchSource:0}: Error finding container 65f540264f2daacc709bf6871990d622cbbad29f7db1a43c94ace0175db35b10: Status 404 returned error can't find the container with id 65f540264f2daacc709bf6871990d622cbbad29f7db1a43c94ace0175db35b10
Sep 16 19:39:05.399581 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:39:05.399544    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-dfph6" event={"ID":"e8938281-3070-4c3d-8a47-181c7f6fa6a9","Type":"ContainerStarted","Data":"65f540264f2daacc709bf6871990d622cbbad29f7db1a43c94ace0175db35b10"}
Sep 16 19:39:06.405120 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:39:06.405087    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-dfph6" event={"ID":"e8938281-3070-4c3d-8a47-181c7f6fa6a9","Type":"ContainerStarted","Data":"638548d1df3879926dd14b353adb255d94b081e4e5ca27ec00c273acb772019b"}
Sep 16 19:39:08.415265 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:39:08.415232    2582 generic.go:356] "Generic (PLEG): container finished" podID="e8938281-3070-4c3d-8a47-181c7f6fa6a9" containerID="638548d1df3879926dd14b353adb255d94b081e4e5ca27ec00c273acb772019b" exitCode=0
Sep 16 19:39:08.415657 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:39:08.415294    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-dfph6" event={"ID":"e8938281-3070-4c3d-8a47-181c7f6fa6a9","Type":"ContainerDied","Data":"638548d1df3879926dd14b353adb255d94b081e4e5ca27ec00c273acb772019b"}
Sep 16 19:39:09.422643 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:39:09.422602    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-dfph6" event={"ID":"e8938281-3070-4c3d-8a47-181c7f6fa6a9","Type":"ContainerStarted","Data":"59458a1828875b17813c5428a5955b98ed02e9a4c3ee957399bb0ea678082b6a"}
Sep 16 19:39:09.474277 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:39:09.474220    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-dfph6" podStartSLOduration=3.299703689 podStartE2EDuration="5.474204666s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.174500977s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:39:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:39:09.473740977 +0000 UTC m=+3353.115456635" watchObservedRunningTime="2026-09-16 19:39:09.474204666 +0000 UTC m=+3353.115920330"
Sep 16 19:39:10.066840 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:39:10.066799    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-dfph6"
Sep 16 19:39:15.066162 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:39:15.066123    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-dfph6"
Sep 16 19:39:15.067557 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:39:15.067528    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-dfph6" podUID="e8938281-3070-4c3d-8a47-181c7f6fa6a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 16 19:39:25.066965 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:39:25.066916    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-dfph6" podUID="e8938281-3070-4c3d-8a47-181c7f6fa6a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 16 19:39:35.067140 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:39:35.067097    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-dfph6" podUID="e8938281-3070-4c3d-8a47-181c7f6fa6a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 16 19:39:45.066642 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:39:45.066592    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-dfph6" podUID="e8938281-3070-4c3d-8a47-181c7f6fa6a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 16 19:39:55.067127 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:39:55.067075    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-dfph6" podUID="e8938281-3070-4c3d-8a47-181c7f6fa6a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 16 19:40:05.066490 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:40:05.066439    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-dfph6" podUID="e8938281-3070-4c3d-8a47-181c7f6fa6a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 16 19:40:15.066655 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:40:15.066604    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-dfph6" podUID="e8938281-3070-4c3d-8a47-181c7f6fa6a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 16 19:40:25.066795 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:40:25.066742    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-dfph6" podUID="e8938281-3070-4c3d-8a47-181c7f6fa6a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 16 19:40:35.066857 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:40:35.066794    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-dfph6" podUID="e8938281-3070-4c3d-8a47-181c7f6fa6a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 16 19:40:45.066566 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:40:45.066517    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-dfph6" podUID="e8938281-3070-4c3d-8a47-181c7f6fa6a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 16 19:40:55.067155 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:40:55.067102    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-dfph6" podUID="e8938281-3070-4c3d-8a47-181c7f6fa6a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 16 19:41:05.077099 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:41:05.077056    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-dfph6"
Sep 16 19:41:05.085056 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:41:05.085022    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-dfph6"
Sep 16 19:41:07.479943 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:41:07.479908    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-dfph6"]
Sep 16 19:41:22.501914 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:41:22.501802    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-dfph6" podUID="e8938281-3070-4c3d-8a47-181c7f6fa6a9" containerName="main" containerID="cri-o://59458a1828875b17813c5428a5955b98ed02e9a4c3ee957399bb0ea678082b6a" gracePeriod=45
Sep 16 19:41:23.078454 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:41:23.078415    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-dfph6" podUID="e8938281-3070-4c3d-8a47-181c7f6fa6a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 16 19:41:23.247472 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:41:23.247445    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-dfph6"
Sep 16 19:41:23.261658 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:41:23.261632    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fcv8j\" (UniqueName: \"kubernetes.io/projected/e8938281-3070-4c3d-8a47-181c7f6fa6a9-kube-api-access-fcv8j\") pod \"e8938281-3070-4c3d-8a47-181c7f6fa6a9\" (UID: \"e8938281-3070-4c3d-8a47-181c7f6fa6a9\") "
Sep 16 19:41:23.261784 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:41:23.261664    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8938281-3070-4c3d-8a47-181c7f6fa6a9-home\") pod \"e8938281-3070-4c3d-8a47-181c7f6fa6a9\" (UID: \"e8938281-3070-4c3d-8a47-181c7f6fa6a9\") "
Sep 16 19:41:23.261784 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:41:23.261689    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8938281-3070-4c3d-8a47-181c7f6fa6a9-cabundle-cert\") pod \"e8938281-3070-4c3d-8a47-181c7f6fa6a9\" (UID: \"e8938281-3070-4c3d-8a47-181c7f6fa6a9\") "
Sep 16 19:41:23.261784 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:41:23.261729    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8938281-3070-4c3d-8a47-181c7f6fa6a9-kserve-provision-location\") pod \"e8938281-3070-4c3d-8a47-181c7f6fa6a9\" (UID: \"e8938281-3070-4c3d-8a47-181c7f6fa6a9\") "
Sep 16 19:41:23.261784 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:41:23.261759    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8938281-3070-4c3d-8a47-181c7f6fa6a9-dshm\") pod \"e8938281-3070-4c3d-8a47-181c7f6fa6a9\" (UID: \"e8938281-3070-4c3d-8a47-181c7f6fa6a9\") "
Sep 16 19:41:23.261952 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:41:23.261828    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8938281-3070-4c3d-8a47-181c7f6fa6a9-tls-certs\") pod \"e8938281-3070-4c3d-8a47-181c7f6fa6a9\" (UID: \"e8938281-3070-4c3d-8a47-181c7f6fa6a9\") "
Sep 16 19:41:23.262126 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:41:23.262094    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8938281-3070-4c3d-8a47-181c7f6fa6a9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e8938281-3070-4c3d-8a47-181c7f6fa6a9" (UID: "e8938281-3070-4c3d-8a47-181c7f6fa6a9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:41:23.262369 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:41:23.262349    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8938281-3070-4c3d-8a47-181c7f6fa6a9-home" (OuterVolumeSpecName: "home") pod "e8938281-3070-4c3d-8a47-181c7f6fa6a9" (UID: "e8938281-3070-4c3d-8a47-181c7f6fa6a9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:41:23.264160 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:41:23.264122    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8938281-3070-4c3d-8a47-181c7f6fa6a9-kube-api-access-fcv8j" (OuterVolumeSpecName: "kube-api-access-fcv8j") pod "e8938281-3070-4c3d-8a47-181c7f6fa6a9" (UID: "e8938281-3070-4c3d-8a47-181c7f6fa6a9"). InnerVolumeSpecName "kube-api-access-fcv8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:41:23.264257 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:41:23.264196    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8938281-3070-4c3d-8a47-181c7f6fa6a9-dshm" (OuterVolumeSpecName: "dshm") pod "e8938281-3070-4c3d-8a47-181c7f6fa6a9" (UID: "e8938281-3070-4c3d-8a47-181c7f6fa6a9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:41:23.264467 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:41:23.264450    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8938281-3070-4c3d-8a47-181c7f6fa6a9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e8938281-3070-4c3d-8a47-181c7f6fa6a9" (UID: "e8938281-3070-4c3d-8a47-181c7f6fa6a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:41:23.285027 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:41:23.284945    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8938281-3070-4c3d-8a47-181c7f6fa6a9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e8938281-3070-4c3d-8a47-181c7f6fa6a9" (UID: "e8938281-3070-4c3d-8a47-181c7f6fa6a9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:41:23.363149 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:41:23.363112    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8938281-3070-4c3d-8a47-181c7f6fa6a9-tmp-dir\") pod \"e8938281-3070-4c3d-8a47-181c7f6fa6a9\" (UID: \"e8938281-3070-4c3d-8a47-181c7f6fa6a9\") "
Sep 16 19:41:23.363149 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:41:23.363157    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8938281-3070-4c3d-8a47-181c7f6fa6a9-model-cache\") pod \"e8938281-3070-4c3d-8a47-181c7f6fa6a9\" (UID: \"e8938281-3070-4c3d-8a47-181c7f6fa6a9\") "
Sep 16 19:41:23.363397 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:41:23.363312    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcv8j\" (UniqueName: \"kubernetes.io/projected/e8938281-3070-4c3d-8a47-181c7f6fa6a9-kube-api-access-fcv8j\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:41:23.363397 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:41:23.363323    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8938281-3070-4c3d-8a47-181c7f6fa6a9-home\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:41:23.363397 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:41:23.363332    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8938281-3070-4c3d-8a47-181c7f6fa6a9-cabundle-cert\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:41:23.363397 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:41:23.363341    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8938281-3070-4c3d-8a47-181c7f6fa6a9-kserve-provision-location\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:41:23.363397 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:41:23.363349    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8938281-3070-4c3d-8a47-181c7f6fa6a9-dshm\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:41:23.363397 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:41:23.363357    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8938281-3070-4c3d-8a47-181c7f6fa6a9-tls-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:41:23.363636 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:41:23.363472    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8938281-3070-4c3d-8a47-181c7f6fa6a9-model-cache" (OuterVolumeSpecName: "model-cache") pod "e8938281-3070-4c3d-8a47-181c7f6fa6a9" (UID: "e8938281-3070-4c3d-8a47-181c7f6fa6a9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:41:23.373299 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:41:23.373265    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8938281-3070-4c3d-8a47-181c7f6fa6a9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e8938281-3070-4c3d-8a47-181c7f6fa6a9" (UID: "e8938281-3070-4c3d-8a47-181c7f6fa6a9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:41:23.464760 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:41:23.464697    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8938281-3070-4c3d-8a47-181c7f6fa6a9-tmp-dir\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:41:23.464760 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:41:23.464753    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8938281-3070-4c3d-8a47-181c7f6fa6a9-model-cache\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:41:24.002682 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:41:24.002645    2582 generic.go:356] "Generic (PLEG): container finished" podID="e8938281-3070-4c3d-8a47-181c7f6fa6a9" containerID="59458a1828875b17813c5428a5955b98ed02e9a4c3ee957399bb0ea678082b6a" exitCode=0
Sep 16 19:41:24.003105 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:41:24.002695    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-dfph6"
Sep 16 19:41:24.003105 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:41:24.002707    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-dfph6" event={"ID":"e8938281-3070-4c3d-8a47-181c7f6fa6a9","Type":"ContainerDied","Data":"59458a1828875b17813c5428a5955b98ed02e9a4c3ee957399bb0ea678082b6a"}
Sep 16 19:41:24.003105 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:41:24.002776    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-dfph6" event={"ID":"e8938281-3070-4c3d-8a47-181c7f6fa6a9","Type":"ContainerDied","Data":"65f540264f2daacc709bf6871990d622cbbad29f7db1a43c94ace0175db35b10"}
Sep 16 19:41:24.003105 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:41:24.002800    2582 scope.go:117] "RemoveContainer" containerID="59458a1828875b17813c5428a5955b98ed02e9a4c3ee957399bb0ea678082b6a"
Sep 16 19:41:24.011731 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:41:24.011701    2582 scope.go:117] "RemoveContainer" containerID="638548d1df3879926dd14b353adb255d94b081e4e5ca27ec00c273acb772019b"
Sep 16 19:41:24.019643 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:41:24.019623    2582 scope.go:117] "RemoveContainer" containerID="59458a1828875b17813c5428a5955b98ed02e9a4c3ee957399bb0ea678082b6a"
Sep 16 19:41:24.019944 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:41:24.019926    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59458a1828875b17813c5428a5955b98ed02e9a4c3ee957399bb0ea678082b6a\": container with ID starting with 59458a1828875b17813c5428a5955b98ed02e9a4c3ee957399bb0ea678082b6a not found: ID does not exist" containerID="59458a1828875b17813c5428a5955b98ed02e9a4c3ee957399bb0ea678082b6a"
Sep 16 19:41:24.019993 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:41:24.019956    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59458a1828875b17813c5428a5955b98ed02e9a4c3ee957399bb0ea678082b6a"} err="failed to get container status \"59458a1828875b17813c5428a5955b98ed02e9a4c3ee957399bb0ea678082b6a\": rpc error: code = NotFound desc = could not find container \"59458a1828875b17813c5428a5955b98ed02e9a4c3ee957399bb0ea678082b6a\": container with ID starting with 59458a1828875b17813c5428a5955b98ed02e9a4c3ee957399bb0ea678082b6a not found: ID does not exist"
Sep 16 19:41:24.019993 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:41:24.019972    2582 scope.go:117] "RemoveContainer" containerID="638548d1df3879926dd14b353adb255d94b081e4e5ca27ec00c273acb772019b"
Sep 16 19:41:24.020178 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:41:24.020162    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"638548d1df3879926dd14b353adb255d94b081e4e5ca27ec00c273acb772019b\": container with ID starting with 638548d1df3879926dd14b353adb255d94b081e4e5ca27ec00c273acb772019b not found: ID does not exist" containerID="638548d1df3879926dd14b353adb255d94b081e4e5ca27ec00c273acb772019b"
Sep 16 19:41:24.020222 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:41:24.020183    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"638548d1df3879926dd14b353adb255d94b081e4e5ca27ec00c273acb772019b"} err="failed to get container status \"638548d1df3879926dd14b353adb255d94b081e4e5ca27ec00c273acb772019b\": rpc error: code = NotFound desc = could not find container \"638548d1df3879926dd14b353adb255d94b081e4e5ca27ec00c273acb772019b\": container with ID starting with 638548d1df3879926dd14b353adb255d94b081e4e5ca27ec00c273acb772019b not found: ID does not exist"
Sep 16 19:41:24.026624 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:41:24.026592    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-dfph6"]
Sep 16 19:41:24.033291 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:41:24.033266    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-dfph6"]
Sep 16 19:41:24.873138 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:41:24.873105    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8938281-3070-4c3d-8a47-181c7f6fa6a9" path="/var/lib/kubelet/pods/e8938281-3070-4c3d-8a47-181c7f6fa6a9/volumes"
Sep 16 19:42:46.826407 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:42:46.826369    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6f7f557d78-rg8d9"]
Sep 16 19:42:46.826850 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:42:46.826711    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8938281-3070-4c3d-8a47-181c7f6fa6a9" containerName="main"
Sep 16 19:42:46.826850 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:42:46.826740    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8938281-3070-4c3d-8a47-181c7f6fa6a9" containerName="main"
Sep 16 19:42:46.826850 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:42:46.826759    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8938281-3070-4c3d-8a47-181c7f6fa6a9" containerName="storage-initializer"
Sep 16 19:42:46.826850 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:42:46.826765    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8938281-3070-4c3d-8a47-181c7f6fa6a9" containerName="storage-initializer"
Sep 16 19:42:46.826850 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:42:46.826816    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8938281-3070-4c3d-8a47-181c7f6fa6a9" containerName="main"
Sep 16 19:42:46.829750 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:42:46.829711    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6f7f557d78-rg8d9"
Sep 16 19:42:46.832628 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:42:46.832602    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:42:46.832814 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:42:46.832666    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"seaweedfs-s3-creds\""
Sep 16 19:42:46.832814 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:42:46.832711    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Sep 16 19:42:46.832814 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:42:46.832758    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Sep 16 19:42:46.833009 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:42:46.832982    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Sep 16 19:42:46.833988 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:42:46.833970    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-djbgh\""
Sep 16 19:42:46.844488 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:42:46.844464    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6f7f557d78-rg8d9"]
Sep 16 19:42:46.955805 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:42:46.955763    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb544b28-2283-4de1-ad12-98af966e9fc5-dshm\") pod \"custom-route-timeout-test-kserve-6f7f557d78-rg8d9\" (UID: \"fb544b28-2283-4de1-ad12-98af966e9fc5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6f7f557d78-rg8d9"
Sep 16 19:42:46.955805 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:42:46.955810    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb544b28-2283-4de1-ad12-98af966e9fc5-tmp-dir\") pod \"custom-route-timeout-test-kserve-6f7f557d78-rg8d9\" (UID: \"fb544b28-2283-4de1-ad12-98af966e9fc5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6f7f557d78-rg8d9"
Sep 16 19:42:46.956027 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:42:46.955828    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb544b28-2283-4de1-ad12-98af966e9fc5-tls-certs\") pod \"custom-route-timeout-test-kserve-6f7f557d78-rg8d9\" (UID: \"fb544b28-2283-4de1-ad12-98af966e9fc5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6f7f557d78-rg8d9"
Sep 16 19:42:46.956027 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:42:46.955906    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb544b28-2283-4de1-ad12-98af966e9fc5-cabundle-cert\") pod \"custom-route-timeout-test-kserve-6f7f557d78-rg8d9\" (UID: \"fb544b28-2283-4de1-ad12-98af966e9fc5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6f7f557d78-rg8d9"
Sep 16 19:42:46.956027 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:42:46.955955    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb544b28-2283-4de1-ad12-98af966e9fc5-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-6f7f557d78-rg8d9\" (UID: \"fb544b28-2283-4de1-ad12-98af966e9fc5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6f7f557d78-rg8d9"
Sep 16 19:42:46.956027 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:42:46.955976    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb544b28-2283-4de1-ad12-98af966e9fc5-model-cache\") pod \"custom-route-timeout-test-kserve-6f7f557d78-rg8d9\" (UID: \"fb544b28-2283-4de1-ad12-98af966e9fc5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6f7f557d78-rg8d9"
Sep 16 19:42:46.956160 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:42:46.956041    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb544b28-2283-4de1-ad12-98af966e9fc5-home\") pod \"custom-route-timeout-test-kserve-6f7f557d78-rg8d9\" (UID: \"fb544b28-2283-4de1-ad12-98af966e9fc5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6f7f557d78-rg8d9"
Sep 16 19:42:46.956160 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:42:46.956085    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qhjr\" (UniqueName: \"kubernetes.io/projected/fb544b28-2283-4de1-ad12-98af966e9fc5-kube-api-access-4qhjr\") pod \"custom-route-timeout-test-kserve-6f7f557d78-rg8d9\" (UID: \"fb544b28-2283-4de1-ad12-98af966e9fc5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6f7f557d78-rg8d9"
Sep 16 19:42:47.056553 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:42:47.056506    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb544b28-2283-4de1-ad12-98af966e9fc5-home\") pod \"custom-route-timeout-test-kserve-6f7f557d78-rg8d9\" (UID: \"fb544b28-2283-4de1-ad12-98af966e9fc5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6f7f557d78-rg8d9"
Sep 16 19:42:47.056785 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:42:47.056559    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qhjr\" (UniqueName: \"kubernetes.io/projected/fb544b28-2283-4de1-ad12-98af966e9fc5-kube-api-access-4qhjr\") pod \"custom-route-timeout-test-kserve-6f7f557d78-rg8d9\" (UID: \"fb544b28-2283-4de1-ad12-98af966e9fc5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6f7f557d78-rg8d9"
Sep 16 19:42:47.056785 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:42:47.056613    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb544b28-2283-4de1-ad12-98af966e9fc5-dshm\") pod \"custom-route-timeout-test-kserve-6f7f557d78-rg8d9\" (UID: \"fb544b28-2283-4de1-ad12-98af966e9fc5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6f7f557d78-rg8d9"
Sep 16 19:42:47.056785 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:42:47.056640    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb544b28-2283-4de1-ad12-98af966e9fc5-tmp-dir\") pod \"custom-route-timeout-test-kserve-6f7f557d78-rg8d9\" (UID: \"fb544b28-2283-4de1-ad12-98af966e9fc5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6f7f557d78-rg8d9"
Sep 16 19:42:47.056785 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:42:47.056664    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb544b28-2283-4de1-ad12-98af966e9fc5-tls-certs\") pod \"custom-route-timeout-test-kserve-6f7f557d78-rg8d9\" (UID: \"fb544b28-2283-4de1-ad12-98af966e9fc5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6f7f557d78-rg8d9"
Sep 16 19:42:47.056785 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:42:47.056690    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb544b28-2283-4de1-ad12-98af966e9fc5-cabundle-cert\") pod \"custom-route-timeout-test-kserve-6f7f557d78-rg8d9\" (UID: \"fb544b28-2283-4de1-ad12-98af966e9fc5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6f7f557d78-rg8d9"
Sep 16 19:42:47.056785 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:42:47.056753    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb544b28-2283-4de1-ad12-98af966e9fc5-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-6f7f557d78-rg8d9\" (UID: \"fb544b28-2283-4de1-ad12-98af966e9fc5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6f7f557d78-rg8d9"
Sep 16 19:42:47.056785 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:42:47.056782    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb544b28-2283-4de1-ad12-98af966e9fc5-model-cache\") pod \"custom-route-timeout-test-kserve-6f7f557d78-rg8d9\" (UID: \"fb544b28-2283-4de1-ad12-98af966e9fc5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6f7f557d78-rg8d9"
Sep 16 19:42:47.057259 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:42:47.057034    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb544b28-2283-4de1-ad12-98af966e9fc5-home\") pod \"custom-route-timeout-test-kserve-6f7f557d78-rg8d9\" (UID: \"fb544b28-2283-4de1-ad12-98af966e9fc5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6f7f557d78-rg8d9"
Sep 16 19:42:47.057259 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:42:47.057063    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb544b28-2283-4de1-ad12-98af966e9fc5-tmp-dir\") pod \"custom-route-timeout-test-kserve-6f7f557d78-rg8d9\" (UID: \"fb544b28-2283-4de1-ad12-98af966e9fc5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6f7f557d78-rg8d9"
Sep 16 19:42:47.057259 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:42:47.057098    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb544b28-2283-4de1-ad12-98af966e9fc5-model-cache\") pod \"custom-route-timeout-test-kserve-6f7f557d78-rg8d9\" (UID: \"fb544b28-2283-4de1-ad12-98af966e9fc5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6f7f557d78-rg8d9"
Sep 16 19:42:47.057437 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:42:47.057292    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb544b28-2283-4de1-ad12-98af966e9fc5-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-6f7f557d78-rg8d9\" (UID: \"fb544b28-2283-4de1-ad12-98af966e9fc5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6f7f557d78-rg8d9"
Sep 16 19:42:47.057551 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:42:47.057528    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb544b28-2283-4de1-ad12-98af966e9fc5-cabundle-cert\") pod \"custom-route-timeout-test-kserve-6f7f557d78-rg8d9\" (UID: \"fb544b28-2283-4de1-ad12-98af966e9fc5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6f7f557d78-rg8d9"
Sep 16 19:42:47.058959 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:42:47.058942    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb544b28-2283-4de1-ad12-98af966e9fc5-dshm\") pod \"custom-route-timeout-test-kserve-6f7f557d78-rg8d9\" (UID: \"fb544b28-2283-4de1-ad12-98af966e9fc5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6f7f557d78-rg8d9"
Sep 16 19:42:47.059271 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:42:47.059253    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb544b28-2283-4de1-ad12-98af966e9fc5-tls-certs\") pod \"custom-route-timeout-test-kserve-6f7f557d78-rg8d9\" (UID: \"fb544b28-2283-4de1-ad12-98af966e9fc5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6f7f557d78-rg8d9"
Sep 16 19:42:47.071450 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:42:47.071415    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qhjr\" (UniqueName: \"kubernetes.io/projected/fb544b28-2283-4de1-ad12-98af966e9fc5-kube-api-access-4qhjr\") pod \"custom-route-timeout-test-kserve-6f7f557d78-rg8d9\" (UID: \"fb544b28-2283-4de1-ad12-98af966e9fc5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6f7f557d78-rg8d9"
Sep 16 19:42:47.140511 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:42:47.140419    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6f7f557d78-rg8d9"
Sep 16 19:42:47.424487 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:42:47.424457    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6f7f557d78-rg8d9"]
Sep 16 19:42:47.427011 ip-10-0-129-194 kubenswrapper[2582]: W0916 19:42:47.426985    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb544b28_2283_4de1_ad12_98af966e9fc5.slice/crio-59029e4e3f27c086a4ec77207a8e1786cfd6759cf97bb3c75096bbb28314c2c0 WatchSource:0}: Error finding container 59029e4e3f27c086a4ec77207a8e1786cfd6759cf97bb3c75096bbb28314c2c0: Status 404 returned error can't find the container with id 59029e4e3f27c086a4ec77207a8e1786cfd6759cf97bb3c75096bbb28314c2c0
Sep 16 19:42:48.351891 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:42:48.351851    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6f7f557d78-rg8d9" event={"ID":"fb544b28-2283-4de1-ad12-98af966e9fc5","Type":"ContainerStarted","Data":"a31ce3f92a2dbec243bf380f1aca8adbacd697b23d8fc033d2235fdd29ee692b"}
Sep 16 19:42:48.352422 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:42:48.351916    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6f7f557d78-rg8d9" event={"ID":"fb544b28-2283-4de1-ad12-98af966e9fc5","Type":"ContainerStarted","Data":"59029e4e3f27c086a4ec77207a8e1786cfd6759cf97bb3c75096bbb28314c2c0"}
Sep 16 19:42:50.360926 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:42:50.360889    2582 generic.go:356] "Generic (PLEG): container finished" podID="fb544b28-2283-4de1-ad12-98af966e9fc5" containerID="a31ce3f92a2dbec243bf380f1aca8adbacd697b23d8fc033d2235fdd29ee692b" exitCode=0
Sep 16 19:42:50.361337 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:42:50.360958    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6f7f557d78-rg8d9" event={"ID":"fb544b28-2283-4de1-ad12-98af966e9fc5","Type":"ContainerDied","Data":"a31ce3f92a2dbec243bf380f1aca8adbacd697b23d8fc033d2235fdd29ee692b"}
Sep 16 19:42:51.368151 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:42:51.368116    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6f7f557d78-rg8d9" event={"ID":"fb544b28-2283-4de1-ad12-98af966e9fc5","Type":"ContainerStarted","Data":"16a28b06d607f380ca5cc82c578ee0edc852940bdadfb6f693e98c0ed788334a"}
Sep 16 19:42:51.396181 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:42:51.396132    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6f7f557d78-rg8d9" podStartSLOduration=3.211382483 podStartE2EDuration="5.396115523s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.18473304s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:42:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:42:51.394349307 +0000 UTC m=+3575.036064970" watchObservedRunningTime="2026-09-16 19:42:51.396115523 +0000 UTC m=+3575.037831226"
Sep 16 19:42:52.140733 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:42:52.140677    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6f7f557d78-rg8d9"
Sep 16 19:42:57.141165 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:42:57.141127    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6f7f557d78-rg8d9"
Sep 16 19:42:57.142601 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:42:57.142572    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6f7f557d78-rg8d9" podUID="fb544b28-2283-4de1-ad12-98af966e9fc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Sep 16 19:43:07.141851 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:43:07.141806    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6f7f557d78-rg8d9" podUID="fb544b28-2283-4de1-ad12-98af966e9fc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Sep 16 19:43:17.090395 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:43:17.090359    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7qwl2_b6695f86-08c0-4836-87b1-f2a9d76084bd/console-operator/1.log"
Sep 16 19:43:17.101864 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:43:17.101833    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7qwl2_b6695f86-08c0-4836-87b1-f2a9d76084bd/console-operator/1.log"
Sep 16 19:43:17.141741 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:43:17.141677    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6f7f557d78-rg8d9" podUID="fb544b28-2283-4de1-ad12-98af966e9fc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Sep 16 19:43:27.140850 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:43:27.140806    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6f7f557d78-rg8d9" podUID="fb544b28-2283-4de1-ad12-98af966e9fc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Sep 16 19:43:37.141089 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:43:37.141035    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6f7f557d78-rg8d9" podUID="fb544b28-2283-4de1-ad12-98af966e9fc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Sep 16 19:43:47.141300 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:43:47.141249    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6f7f557d78-rg8d9" podUID="fb544b28-2283-4de1-ad12-98af966e9fc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Sep 16 19:43:57.141563 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:43:57.141515    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6f7f557d78-rg8d9" podUID="fb544b28-2283-4de1-ad12-98af966e9fc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Sep 16 19:44:07.141004 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:44:07.140958    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6f7f557d78-rg8d9" podUID="fb544b28-2283-4de1-ad12-98af966e9fc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Sep 16 19:44:17.141065 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:44:17.141002    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6f7f557d78-rg8d9" podUID="fb544b28-2283-4de1-ad12-98af966e9fc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Sep 16 19:44:27.141021 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:44:27.140972    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6f7f557d78-rg8d9" podUID="fb544b28-2283-4de1-ad12-98af966e9fc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Sep 16 19:44:37.140860 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:44:37.140807    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6f7f557d78-rg8d9" podUID="fb544b28-2283-4de1-ad12-98af966e9fc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Sep 16 19:44:47.151204 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:44:47.151119    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6f7f557d78-rg8d9"
Sep 16 19:44:47.159305 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:44:47.159271    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6f7f557d78-rg8d9"
Sep 16 19:44:52.110470 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:44:52.110432    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6f7f557d78-rg8d9"]
Sep 16 19:45:07.136226 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:07.136179    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6f7f557d78-rg8d9" podUID="fb544b28-2283-4de1-ad12-98af966e9fc5" containerName="main" containerID="cri-o://16a28b06d607f380ca5cc82c578ee0edc852940bdadfb6f693e98c0ed788334a" gracePeriod=45
Sep 16 19:45:07.152050 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:07.152001    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6f7f557d78-rg8d9" podUID="fb544b28-2283-4de1-ad12-98af966e9fc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Sep 16 19:45:07.873138 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:07.873106    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6f7f557d78-rg8d9"
Sep 16 19:45:07.972785 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:07.972663    2582 generic.go:356] "Generic (PLEG): container finished" podID="fb544b28-2283-4de1-ad12-98af966e9fc5" containerID="16a28b06d607f380ca5cc82c578ee0edc852940bdadfb6f693e98c0ed788334a" exitCode=0
Sep 16 19:45:07.972785 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:07.972733    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6f7f557d78-rg8d9"
Sep 16 19:45:07.972785 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:07.972750    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6f7f557d78-rg8d9" event={"ID":"fb544b28-2283-4de1-ad12-98af966e9fc5","Type":"ContainerDied","Data":"16a28b06d607f380ca5cc82c578ee0edc852940bdadfb6f693e98c0ed788334a"}
Sep 16 19:45:07.973158 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:07.972839    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6f7f557d78-rg8d9" event={"ID":"fb544b28-2283-4de1-ad12-98af966e9fc5","Type":"ContainerDied","Data":"59029e4e3f27c086a4ec77207a8e1786cfd6759cf97bb3c75096bbb28314c2c0"}
Sep 16 19:45:07.973158 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:07.972865    2582 scope.go:117] "RemoveContainer" containerID="16a28b06d607f380ca5cc82c578ee0edc852940bdadfb6f693e98c0ed788334a"
Sep 16 19:45:07.981782 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:07.981757    2582 scope.go:117] "RemoveContainer" containerID="a31ce3f92a2dbec243bf380f1aca8adbacd697b23d8fc033d2235fdd29ee692b"
Sep 16 19:45:07.990321 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:07.990297    2582 scope.go:117] "RemoveContainer" containerID="16a28b06d607f380ca5cc82c578ee0edc852940bdadfb6f693e98c0ed788334a"
Sep 16 19:45:07.990634 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:45:07.990611    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16a28b06d607f380ca5cc82c578ee0edc852940bdadfb6f693e98c0ed788334a\": container with ID starting with 16a28b06d607f380ca5cc82c578ee0edc852940bdadfb6f693e98c0ed788334a not found: ID does not exist" containerID="16a28b06d607f380ca5cc82c578ee0edc852940bdadfb6f693e98c0ed788334a"
Sep 16 19:45:07.990731 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:07.990648    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16a28b06d607f380ca5cc82c578ee0edc852940bdadfb6f693e98c0ed788334a"} err="failed to get container status \"16a28b06d607f380ca5cc82c578ee0edc852940bdadfb6f693e98c0ed788334a\": rpc error: code = NotFound desc = could not find container \"16a28b06d607f380ca5cc82c578ee0edc852940bdadfb6f693e98c0ed788334a\": container with ID starting with 16a28b06d607f380ca5cc82c578ee0edc852940bdadfb6f693e98c0ed788334a not found: ID does not exist"
Sep 16 19:45:07.990731 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:07.990673    2582 scope.go:117] "RemoveContainer" containerID="a31ce3f92a2dbec243bf380f1aca8adbacd697b23d8fc033d2235fdd29ee692b"
Sep 16 19:45:07.990988 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:45:07.990969    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a31ce3f92a2dbec243bf380f1aca8adbacd697b23d8fc033d2235fdd29ee692b\": container with ID starting with a31ce3f92a2dbec243bf380f1aca8adbacd697b23d8fc033d2235fdd29ee692b not found: ID does not exist" containerID="a31ce3f92a2dbec243bf380f1aca8adbacd697b23d8fc033d2235fdd29ee692b"
Sep 16 19:45:07.991028 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:07.990994    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a31ce3f92a2dbec243bf380f1aca8adbacd697b23d8fc033d2235fdd29ee692b"} err="failed to get container status \"a31ce3f92a2dbec243bf380f1aca8adbacd697b23d8fc033d2235fdd29ee692b\": rpc error: code = NotFound desc = could not find container \"a31ce3f92a2dbec243bf380f1aca8adbacd697b23d8fc033d2235fdd29ee692b\": container with ID starting with a31ce3f92a2dbec243bf380f1aca8adbacd697b23d8fc033d2235fdd29ee692b not found: ID does not exist"
Sep 16 19:45:07.994198 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:07.994178    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb544b28-2283-4de1-ad12-98af966e9fc5-cabundle-cert\") pod \"fb544b28-2283-4de1-ad12-98af966e9fc5\" (UID: \"fb544b28-2283-4de1-ad12-98af966e9fc5\") "
Sep 16 19:45:07.994260 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:07.994253    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb544b28-2283-4de1-ad12-98af966e9fc5-tmp-dir\") pod \"fb544b28-2283-4de1-ad12-98af966e9fc5\" (UID: \"fb544b28-2283-4de1-ad12-98af966e9fc5\") "
Sep 16 19:45:07.994315 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:07.994270    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb544b28-2283-4de1-ad12-98af966e9fc5-kserve-provision-location\") pod \"fb544b28-2283-4de1-ad12-98af966e9fc5\" (UID: \"fb544b28-2283-4de1-ad12-98af966e9fc5\") "
Sep 16 19:45:07.994315 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:07.994301    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb544b28-2283-4de1-ad12-98af966e9fc5-dshm\") pod \"fb544b28-2283-4de1-ad12-98af966e9fc5\" (UID: \"fb544b28-2283-4de1-ad12-98af966e9fc5\") "
Sep 16 19:45:07.994418 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:07.994330    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb544b28-2283-4de1-ad12-98af966e9fc5-model-cache\") pod \"fb544b28-2283-4de1-ad12-98af966e9fc5\" (UID: \"fb544b28-2283-4de1-ad12-98af966e9fc5\") "
Sep 16 19:45:07.994418 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:07.994359    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb544b28-2283-4de1-ad12-98af966e9fc5-tls-certs\") pod \"fb544b28-2283-4de1-ad12-98af966e9fc5\" (UID: \"fb544b28-2283-4de1-ad12-98af966e9fc5\") "
Sep 16 19:45:07.994418 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:07.994374    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4qhjr\" (UniqueName: \"kubernetes.io/projected/fb544b28-2283-4de1-ad12-98af966e9fc5-kube-api-access-4qhjr\") pod \"fb544b28-2283-4de1-ad12-98af966e9fc5\" (UID: \"fb544b28-2283-4de1-ad12-98af966e9fc5\") "
Sep 16 19:45:07.994418 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:07.994397    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb544b28-2283-4de1-ad12-98af966e9fc5-home\") pod \"fb544b28-2283-4de1-ad12-98af966e9fc5\" (UID: \"fb544b28-2283-4de1-ad12-98af966e9fc5\") "
Sep 16 19:45:07.994632 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:07.994606    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb544b28-2283-4de1-ad12-98af966e9fc5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fb544b28-2283-4de1-ad12-98af966e9fc5" (UID: "fb544b28-2283-4de1-ad12-98af966e9fc5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:45:07.994696 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:07.994659    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb544b28-2283-4de1-ad12-98af966e9fc5-model-cache" (OuterVolumeSpecName: "model-cache") pod "fb544b28-2283-4de1-ad12-98af966e9fc5" (UID: "fb544b28-2283-4de1-ad12-98af966e9fc5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:45:07.995316 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:07.995288    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb544b28-2283-4de1-ad12-98af966e9fc5-home" (OuterVolumeSpecName: "home") pod "fb544b28-2283-4de1-ad12-98af966e9fc5" (UID: "fb544b28-2283-4de1-ad12-98af966e9fc5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:45:07.996803 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:07.996766    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb544b28-2283-4de1-ad12-98af966e9fc5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fb544b28-2283-4de1-ad12-98af966e9fc5" (UID: "fb544b28-2283-4de1-ad12-98af966e9fc5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:45:07.996889 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:07.996831    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb544b28-2283-4de1-ad12-98af966e9fc5-dshm" (OuterVolumeSpecName: "dshm") pod "fb544b28-2283-4de1-ad12-98af966e9fc5" (UID: "fb544b28-2283-4de1-ad12-98af966e9fc5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:45:07.996889 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:07.996858    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb544b28-2283-4de1-ad12-98af966e9fc5-kube-api-access-4qhjr" (OuterVolumeSpecName: "kube-api-access-4qhjr") pod "fb544b28-2283-4de1-ad12-98af966e9fc5" (UID: "fb544b28-2283-4de1-ad12-98af966e9fc5"). InnerVolumeSpecName "kube-api-access-4qhjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:45:08.012209 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:08.012166    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb544b28-2283-4de1-ad12-98af966e9fc5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fb544b28-2283-4de1-ad12-98af966e9fc5" (UID: "fb544b28-2283-4de1-ad12-98af966e9fc5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:45:08.017656 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:08.017621    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb544b28-2283-4de1-ad12-98af966e9fc5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fb544b28-2283-4de1-ad12-98af966e9fc5" (UID: "fb544b28-2283-4de1-ad12-98af966e9fc5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:45:08.095876 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:08.095837    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb544b28-2283-4de1-ad12-98af966e9fc5-tmp-dir\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:45:08.095876 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:08.095874    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb544b28-2283-4de1-ad12-98af966e9fc5-kserve-provision-location\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:45:08.095876 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:08.095883    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb544b28-2283-4de1-ad12-98af966e9fc5-dshm\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:45:08.096094 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:08.095892    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb544b28-2283-4de1-ad12-98af966e9fc5-model-cache\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:45:08.096094 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:08.095900    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb544b28-2283-4de1-ad12-98af966e9fc5-tls-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:45:08.096094 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:08.095908    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qhjr\" (UniqueName: \"kubernetes.io/projected/fb544b28-2283-4de1-ad12-98af966e9fc5-kube-api-access-4qhjr\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:45:08.096094 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:08.095916    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb544b28-2283-4de1-ad12-98af966e9fc5-home\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:45:08.096094 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:08.095925    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb544b28-2283-4de1-ad12-98af966e9fc5-cabundle-cert\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:45:08.298643 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:08.298602    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6f7f557d78-rg8d9"]
Sep 16 19:45:08.302730 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:08.302690    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6f7f557d78-rg8d9"]
Sep 16 19:45:08.872706 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:08.872664    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb544b28-2283-4de1-ad12-98af966e9fc5" path="/var/lib/kubelet/pods/fb544b28-2283-4de1-ad12-98af966e9fc5/volumes"
Sep 16 19:45:58.127747 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:58.127696    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c9f8cb8d6-hq2k7"]
Sep 16 19:45:58.128206 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:58.128013    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb544b28-2283-4de1-ad12-98af966e9fc5" containerName="storage-initializer"
Sep 16 19:45:58.128206 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:58.128024    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb544b28-2283-4de1-ad12-98af966e9fc5" containerName="storage-initializer"
Sep 16 19:45:58.128206 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:58.128037    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb544b28-2283-4de1-ad12-98af966e9fc5" containerName="main"
Sep 16 19:45:58.128206 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:58.128043    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb544b28-2283-4de1-ad12-98af966e9fc5" containerName="main"
Sep 16 19:45:58.128206 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:58.128102    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb544b28-2283-4de1-ad12-98af966e9fc5" containerName="main"
Sep 16 19:45:58.131085 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:58.131067    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c9f8cb8d6-hq2k7"
Sep 16 19:45:58.135573 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:58.135548    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"seaweedfs-s3-creds\""
Sep 16 19:45:58.135731 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:58.135578    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:45:58.135804 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:58.135761    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 16 19:45:58.135804 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:58.135794    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 16 19:45:58.135903 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:58.135883    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Sep 16 19:45:58.135953 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:58.135944    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-fr6f2\""
Sep 16 19:45:58.153510 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:58.153475    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c9f8cb8d6-hq2k7"]
Sep 16 19:45:58.202291 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:58.202256    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8a1a69d-2daf-41c3-a898-c8a09b4fdce9-cabundle-cert\") pod \"router-with-refs-test-kserve-c9f8cb8d6-hq2k7\" (UID: \"b8a1a69d-2daf-41c3-a898-c8a09b4fdce9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c9f8cb8d6-hq2k7"
Sep 16 19:45:58.202291 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:58.202291    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8a1a69d-2daf-41c3-a898-c8a09b4fdce9-home\") pod \"router-with-refs-test-kserve-c9f8cb8d6-hq2k7\" (UID: \"b8a1a69d-2daf-41c3-a898-c8a09b4fdce9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c9f8cb8d6-hq2k7"
Sep 16 19:45:58.202516 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:58.202317    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8a1a69d-2daf-41c3-a898-c8a09b4fdce9-model-cache\") pod \"router-with-refs-test-kserve-c9f8cb8d6-hq2k7\" (UID: \"b8a1a69d-2daf-41c3-a898-c8a09b4fdce9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c9f8cb8d6-hq2k7"
Sep 16 19:45:58.202516 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:58.202337    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8a1a69d-2daf-41c3-a898-c8a09b4fdce9-tls-certs\") pod \"router-with-refs-test-kserve-c9f8cb8d6-hq2k7\" (UID: \"b8a1a69d-2daf-41c3-a898-c8a09b4fdce9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c9f8cb8d6-hq2k7"
Sep 16 19:45:58.202516 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:58.202375    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8a1a69d-2daf-41c3-a898-c8a09b4fdce9-tmp-dir\") pod \"router-with-refs-test-kserve-c9f8cb8d6-hq2k7\" (UID: \"b8a1a69d-2daf-41c3-a898-c8a09b4fdce9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c9f8cb8d6-hq2k7"
Sep 16 19:45:58.202516 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:58.202392    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8a1a69d-2daf-41c3-a898-c8a09b4fdce9-kserve-provision-location\") pod \"router-with-refs-test-kserve-c9f8cb8d6-hq2k7\" (UID: \"b8a1a69d-2daf-41c3-a898-c8a09b4fdce9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c9f8cb8d6-hq2k7"
Sep 16 19:45:58.202516 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:58.202409    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls4ps\" (UniqueName: \"kubernetes.io/projected/b8a1a69d-2daf-41c3-a898-c8a09b4fdce9-kube-api-access-ls4ps\") pod \"router-with-refs-test-kserve-c9f8cb8d6-hq2k7\" (UID: \"b8a1a69d-2daf-41c3-a898-c8a09b4fdce9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c9f8cb8d6-hq2k7"
Sep 16 19:45:58.202516 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:58.202434    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8a1a69d-2daf-41c3-a898-c8a09b4fdce9-dshm\") pod \"router-with-refs-test-kserve-c9f8cb8d6-hq2k7\" (UID: \"b8a1a69d-2daf-41c3-a898-c8a09b4fdce9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c9f8cb8d6-hq2k7"
Sep 16 19:45:58.303306 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:58.303269    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8a1a69d-2daf-41c3-a898-c8a09b4fdce9-cabundle-cert\") pod \"router-with-refs-test-kserve-c9f8cb8d6-hq2k7\" (UID: \"b8a1a69d-2daf-41c3-a898-c8a09b4fdce9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c9f8cb8d6-hq2k7"
Sep 16 19:45:58.303306 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:58.303307    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8a1a69d-2daf-41c3-a898-c8a09b4fdce9-home\") pod \"router-with-refs-test-kserve-c9f8cb8d6-hq2k7\" (UID: \"b8a1a69d-2daf-41c3-a898-c8a09b4fdce9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c9f8cb8d6-hq2k7"
Sep 16 19:45:58.303559 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:58.303334    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8a1a69d-2daf-41c3-a898-c8a09b4fdce9-model-cache\") pod \"router-with-refs-test-kserve-c9f8cb8d6-hq2k7\" (UID: \"b8a1a69d-2daf-41c3-a898-c8a09b4fdce9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c9f8cb8d6-hq2k7"
Sep 16 19:45:58.303559 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:58.303361    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8a1a69d-2daf-41c3-a898-c8a09b4fdce9-tls-certs\") pod \"router-with-refs-test-kserve-c9f8cb8d6-hq2k7\" (UID: \"b8a1a69d-2daf-41c3-a898-c8a09b4fdce9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c9f8cb8d6-hq2k7"
Sep 16 19:45:58.303559 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:58.303382    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8a1a69d-2daf-41c3-a898-c8a09b4fdce9-tmp-dir\") pod \"router-with-refs-test-kserve-c9f8cb8d6-hq2k7\" (UID: \"b8a1a69d-2daf-41c3-a898-c8a09b4fdce9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c9f8cb8d6-hq2k7"
Sep 16 19:45:58.303559 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:58.303400    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8a1a69d-2daf-41c3-a898-c8a09b4fdce9-kserve-provision-location\") pod \"router-with-refs-test-kserve-c9f8cb8d6-hq2k7\" (UID: \"b8a1a69d-2daf-41c3-a898-c8a09b4fdce9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c9f8cb8d6-hq2k7"
Sep 16 19:45:58.303559 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:58.303423    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls4ps\" (UniqueName: \"kubernetes.io/projected/b8a1a69d-2daf-41c3-a898-c8a09b4fdce9-kube-api-access-ls4ps\") pod \"router-with-refs-test-kserve-c9f8cb8d6-hq2k7\" (UID: \"b8a1a69d-2daf-41c3-a898-c8a09b4fdce9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c9f8cb8d6-hq2k7"
Sep 16 19:45:58.303559 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:58.303457    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8a1a69d-2daf-41c3-a898-c8a09b4fdce9-dshm\") pod \"router-with-refs-test-kserve-c9f8cb8d6-hq2k7\" (UID: \"b8a1a69d-2daf-41c3-a898-c8a09b4fdce9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c9f8cb8d6-hq2k7"
Sep 16 19:45:58.303929 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:58.303785    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8a1a69d-2daf-41c3-a898-c8a09b4fdce9-home\") pod \"router-with-refs-test-kserve-c9f8cb8d6-hq2k7\" (UID: \"b8a1a69d-2daf-41c3-a898-c8a09b4fdce9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c9f8cb8d6-hq2k7"
Sep 16 19:45:58.303929 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:58.303801    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8a1a69d-2daf-41c3-a898-c8a09b4fdce9-kserve-provision-location\") pod \"router-with-refs-test-kserve-c9f8cb8d6-hq2k7\" (UID: \"b8a1a69d-2daf-41c3-a898-c8a09b4fdce9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c9f8cb8d6-hq2k7"
Sep 16 19:45:58.303929 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:58.303886    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8a1a69d-2daf-41c3-a898-c8a09b4fdce9-tmp-dir\") pod \"router-with-refs-test-kserve-c9f8cb8d6-hq2k7\" (UID: \"b8a1a69d-2daf-41c3-a898-c8a09b4fdce9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c9f8cb8d6-hq2k7"
Sep 16 19:45:58.304097 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:58.303958    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8a1a69d-2daf-41c3-a898-c8a09b4fdce9-model-cache\") pod \"router-with-refs-test-kserve-c9f8cb8d6-hq2k7\" (UID: \"b8a1a69d-2daf-41c3-a898-c8a09b4fdce9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c9f8cb8d6-hq2k7"
Sep 16 19:45:58.304212 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:58.304191    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8a1a69d-2daf-41c3-a898-c8a09b4fdce9-cabundle-cert\") pod \"router-with-refs-test-kserve-c9f8cb8d6-hq2k7\" (UID: \"b8a1a69d-2daf-41c3-a898-c8a09b4fdce9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c9f8cb8d6-hq2k7"
Sep 16 19:45:58.305885 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:58.305856    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8a1a69d-2daf-41c3-a898-c8a09b4fdce9-dshm\") pod \"router-with-refs-test-kserve-c9f8cb8d6-hq2k7\" (UID: \"b8a1a69d-2daf-41c3-a898-c8a09b4fdce9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c9f8cb8d6-hq2k7"
Sep 16 19:45:58.306107 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:58.306084    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8a1a69d-2daf-41c3-a898-c8a09b4fdce9-tls-certs\") pod \"router-with-refs-test-kserve-c9f8cb8d6-hq2k7\" (UID: \"b8a1a69d-2daf-41c3-a898-c8a09b4fdce9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c9f8cb8d6-hq2k7"
Sep 16 19:45:58.315494 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:58.315469    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls4ps\" (UniqueName: \"kubernetes.io/projected/b8a1a69d-2daf-41c3-a898-c8a09b4fdce9-kube-api-access-ls4ps\") pod \"router-with-refs-test-kserve-c9f8cb8d6-hq2k7\" (UID: \"b8a1a69d-2daf-41c3-a898-c8a09b4fdce9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c9f8cb8d6-hq2k7"
Sep 16 19:45:58.441151 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:58.441053    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c9f8cb8d6-hq2k7"
Sep 16 19:45:58.753492 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:58.753411    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c9f8cb8d6-hq2k7"]
Sep 16 19:45:58.757298 ip-10-0-129-194 kubenswrapper[2582]: W0916 19:45:58.757268    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8a1a69d_2daf_41c3_a898_c8a09b4fdce9.slice/crio-bb416b8e5ddc45b08e9b823848cc7d372e326364e683d32eec216e8bb4c982c3 WatchSource:0}: Error finding container bb416b8e5ddc45b08e9b823848cc7d372e326364e683d32eec216e8bb4c982c3: Status 404 returned error can't find the container with id bb416b8e5ddc45b08e9b823848cc7d372e326364e683d32eec216e8bb4c982c3
Sep 16 19:45:59.183229 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:59.183185    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c9f8cb8d6-hq2k7" event={"ID":"b8a1a69d-2daf-41c3-a898-c8a09b4fdce9","Type":"ContainerStarted","Data":"35eaa6b6e7239e346e5efe451fa51d170b9bfe3b73e04eb626f3df08c979ff0c"}
Sep 16 19:45:59.183673 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:45:59.183247    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c9f8cb8d6-hq2k7" event={"ID":"b8a1a69d-2daf-41c3-a898-c8a09b4fdce9","Type":"ContainerStarted","Data":"bb416b8e5ddc45b08e9b823848cc7d372e326364e683d32eec216e8bb4c982c3"}
Sep 16 19:46:01.194128 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:46:01.194081    2582 generic.go:356] "Generic (PLEG): container finished" podID="b8a1a69d-2daf-41c3-a898-c8a09b4fdce9" containerID="35eaa6b6e7239e346e5efe451fa51d170b9bfe3b73e04eb626f3df08c979ff0c" exitCode=0
Sep 16 19:46:01.194613 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:46:01.194154    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c9f8cb8d6-hq2k7" event={"ID":"b8a1a69d-2daf-41c3-a898-c8a09b4fdce9","Type":"ContainerDied","Data":"35eaa6b6e7239e346e5efe451fa51d170b9bfe3b73e04eb626f3df08c979ff0c"}
Sep 16 19:46:02.199979 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:46:02.199945    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c9f8cb8d6-hq2k7" event={"ID":"b8a1a69d-2daf-41c3-a898-c8a09b4fdce9","Type":"ContainerStarted","Data":"8f8b0e892397017ee087de2380e022fe9a812c9bac2695fc6d35d1d40018b20b"}
Sep 16 19:46:02.224665 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:46:02.224610    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c9f8cb8d6-hq2k7" podStartSLOduration=2.046114904 podStartE2EDuration="4.224593077s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.178478173s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:45:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:46:02.222164347 +0000 UTC m=+3765.863880008" watchObservedRunningTime="2026-09-16 19:46:02.224593077 +0000 UTC m=+3765.866308741"
Sep 16 19:46:02.442205 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:46:02.442161    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c9f8cb8d6-hq2k7"
Sep 16 19:46:08.441401 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:46:08.441359    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c9f8cb8d6-hq2k7"
Sep 16 19:46:08.442889 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:46:08.442855    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c9f8cb8d6-hq2k7" podUID="b8a1a69d-2daf-41c3-a898-c8a09b4fdce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Sep 16 19:46:18.442350 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:46:18.442301    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c9f8cb8d6-hq2k7" podUID="b8a1a69d-2daf-41c3-a898-c8a09b4fdce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Sep 16 19:46:28.441790 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:46:28.441741    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c9f8cb8d6-hq2k7" podUID="b8a1a69d-2daf-41c3-a898-c8a09b4fdce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Sep 16 19:46:38.442442 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:46:38.442394    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c9f8cb8d6-hq2k7" podUID="b8a1a69d-2daf-41c3-a898-c8a09b4fdce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Sep 16 19:46:48.441958 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:46:48.441911    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c9f8cb8d6-hq2k7" podUID="b8a1a69d-2daf-41c3-a898-c8a09b4fdce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Sep 16 19:46:58.442523 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:46:58.442466    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c9f8cb8d6-hq2k7" podUID="b8a1a69d-2daf-41c3-a898-c8a09b4fdce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Sep 16 19:47:08.441916 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:47:08.441876    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c9f8cb8d6-hq2k7" podUID="b8a1a69d-2daf-41c3-a898-c8a09b4fdce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Sep 16 19:47:18.441886 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:47:18.441844    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c9f8cb8d6-hq2k7" podUID="b8a1a69d-2daf-41c3-a898-c8a09b4fdce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Sep 16 19:47:28.441775 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:47:28.441667    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c9f8cb8d6-hq2k7" podUID="b8a1a69d-2daf-41c3-a898-c8a09b4fdce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Sep 16 19:47:38.442199 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:47:38.442153    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c9f8cb8d6-hq2k7" podUID="b8a1a69d-2daf-41c3-a898-c8a09b4fdce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Sep 16 19:47:48.442654 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:47:48.442540    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c9f8cb8d6-hq2k7" podUID="b8a1a69d-2daf-41c3-a898-c8a09b4fdce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Sep 16 19:47:58.452004 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:47:58.451966    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c9f8cb8d6-hq2k7"
Sep 16 19:47:58.459951 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:47:58.459923    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c9f8cb8d6-hq2k7"
Sep 16 19:48:03.272542 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:48:03.272495    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c9f8cb8d6-hq2k7"]
Sep 16 19:48:17.121330 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:48:17.121296    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7qwl2_b6695f86-08c0-4836-87b1-f2a9d76084bd/console-operator/1.log"
Sep 16 19:48:17.130479 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:48:17.130454    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7qwl2_b6695f86-08c0-4836-87b1-f2a9d76084bd/console-operator/1.log"
Sep 16 19:48:18.296375 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:48:18.296316    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c9f8cb8d6-hq2k7" podUID="b8a1a69d-2daf-41c3-a898-c8a09b4fdce9" containerName="main" containerID="cri-o://8f8b0e892397017ee087de2380e022fe9a812c9bac2695fc6d35d1d40018b20b" gracePeriod=45
Sep 16 19:48:18.453623 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:48:18.453573    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c9f8cb8d6-hq2k7" podUID="b8a1a69d-2daf-41c3-a898-c8a09b4fdce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Sep 16 19:48:19.048205 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:48:19.048178    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c9f8cb8d6-hq2k7"
Sep 16 19:48:19.111093 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:48:19.111000    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8a1a69d-2daf-41c3-a898-c8a09b4fdce9-home\") pod \"b8a1a69d-2daf-41c3-a898-c8a09b4fdce9\" (UID: \"b8a1a69d-2daf-41c3-a898-c8a09b4fdce9\") "
Sep 16 19:48:19.111093 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:48:19.111040    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8a1a69d-2daf-41c3-a898-c8a09b4fdce9-model-cache\") pod \"b8a1a69d-2daf-41c3-a898-c8a09b4fdce9\" (UID: \"b8a1a69d-2daf-41c3-a898-c8a09b4fdce9\") "
Sep 16 19:48:19.111093 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:48:19.111074    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8a1a69d-2daf-41c3-a898-c8a09b4fdce9-tmp-dir\") pod \"b8a1a69d-2daf-41c3-a898-c8a09b4fdce9\" (UID: \"b8a1a69d-2daf-41c3-a898-c8a09b4fdce9\") "
Sep 16 19:48:19.111380 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:48:19.111110    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8a1a69d-2daf-41c3-a898-c8a09b4fdce9-cabundle-cert\") pod \"b8a1a69d-2daf-41c3-a898-c8a09b4fdce9\" (UID: \"b8a1a69d-2daf-41c3-a898-c8a09b4fdce9\") "
Sep 16 19:48:19.111380 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:48:19.111130    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8a1a69d-2daf-41c3-a898-c8a09b4fdce9-kserve-provision-location\") pod \"b8a1a69d-2daf-41c3-a898-c8a09b4fdce9\" (UID: \"b8a1a69d-2daf-41c3-a898-c8a09b4fdce9\") "
Sep 16 19:48:19.111380 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:48:19.111156    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8a1a69d-2daf-41c3-a898-c8a09b4fdce9-dshm\") pod \"b8a1a69d-2daf-41c3-a898-c8a09b4fdce9\" (UID: \"b8a1a69d-2daf-41c3-a898-c8a09b4fdce9\") "
Sep 16 19:48:19.111380 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:48:19.111219    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8a1a69d-2daf-41c3-a898-c8a09b4fdce9-tls-certs\") pod \"b8a1a69d-2daf-41c3-a898-c8a09b4fdce9\" (UID: \"b8a1a69d-2daf-41c3-a898-c8a09b4fdce9\") "
Sep 16 19:48:19.111380 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:48:19.111248    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ls4ps\" (UniqueName: \"kubernetes.io/projected/b8a1a69d-2daf-41c3-a898-c8a09b4fdce9-kube-api-access-ls4ps\") pod \"b8a1a69d-2daf-41c3-a898-c8a09b4fdce9\" (UID: \"b8a1a69d-2daf-41c3-a898-c8a09b4fdce9\") "
Sep 16 19:48:19.111380 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:48:19.111351    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a1a69d-2daf-41c3-a898-c8a09b4fdce9-model-cache" (OuterVolumeSpecName: "model-cache") pod "b8a1a69d-2daf-41c3-a898-c8a09b4fdce9" (UID: "b8a1a69d-2daf-41c3-a898-c8a09b4fdce9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:48:19.111706 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:48:19.111526    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8a1a69d-2daf-41c3-a898-c8a09b4fdce9-model-cache\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:48:19.111706 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:48:19.111540    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8a1a69d-2daf-41c3-a898-c8a09b4fdce9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b8a1a69d-2daf-41c3-a898-c8a09b4fdce9" (UID: "b8a1a69d-2daf-41c3-a898-c8a09b4fdce9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:48:19.111910 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:48:19.111827    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a1a69d-2daf-41c3-a898-c8a09b4fdce9-home" (OuterVolumeSpecName: "home") pod "b8a1a69d-2daf-41c3-a898-c8a09b4fdce9" (UID: "b8a1a69d-2daf-41c3-a898-c8a09b4fdce9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:48:19.113608 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:48:19.113567    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8a1a69d-2daf-41c3-a898-c8a09b4fdce9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b8a1a69d-2daf-41c3-a898-c8a09b4fdce9" (UID: "b8a1a69d-2daf-41c3-a898-c8a09b4fdce9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:48:19.113796 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:48:19.113671    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a1a69d-2daf-41c3-a898-c8a09b4fdce9-dshm" (OuterVolumeSpecName: "dshm") pod "b8a1a69d-2daf-41c3-a898-c8a09b4fdce9" (UID: "b8a1a69d-2daf-41c3-a898-c8a09b4fdce9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:48:19.114049 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:48:19.114027    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8a1a69d-2daf-41c3-a898-c8a09b4fdce9-kube-api-access-ls4ps" (OuterVolumeSpecName: "kube-api-access-ls4ps") pod "b8a1a69d-2daf-41c3-a898-c8a09b4fdce9" (UID: "b8a1a69d-2daf-41c3-a898-c8a09b4fdce9"). InnerVolumeSpecName "kube-api-access-ls4ps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:48:19.124180 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:48:19.124152    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a1a69d-2daf-41c3-a898-c8a09b4fdce9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b8a1a69d-2daf-41c3-a898-c8a09b4fdce9" (UID: "b8a1a69d-2daf-41c3-a898-c8a09b4fdce9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:48:19.134012 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:48:19.133985    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a1a69d-2daf-41c3-a898-c8a09b4fdce9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b8a1a69d-2daf-41c3-a898-c8a09b4fdce9" (UID: "b8a1a69d-2daf-41c3-a898-c8a09b4fdce9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:48:19.212141 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:48:19.212106    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8a1a69d-2daf-41c3-a898-c8a09b4fdce9-home\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:48:19.212303 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:48:19.212154    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8a1a69d-2daf-41c3-a898-c8a09b4fdce9-tmp-dir\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:48:19.212303 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:48:19.212169    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8a1a69d-2daf-41c3-a898-c8a09b4fdce9-cabundle-cert\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:48:19.212303 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:48:19.212181    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8a1a69d-2daf-41c3-a898-c8a09b4fdce9-kserve-provision-location\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:48:19.212303 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:48:19.212190    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8a1a69d-2daf-41c3-a898-c8a09b4fdce9-dshm\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:48:19.212303 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:48:19.212201    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8a1a69d-2daf-41c3-a898-c8a09b4fdce9-tls-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:48:19.212303 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:48:19.212209    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ls4ps\" (UniqueName: \"kubernetes.io/projected/b8a1a69d-2daf-41c3-a898-c8a09b4fdce9-kube-api-access-ls4ps\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:48:19.806896 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:48:19.806861    2582 generic.go:356] "Generic (PLEG): container finished" podID="b8a1a69d-2daf-41c3-a898-c8a09b4fdce9" containerID="8f8b0e892397017ee087de2380e022fe9a812c9bac2695fc6d35d1d40018b20b" exitCode=0
Sep 16 19:48:19.807319 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:48:19.806928    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c9f8cb8d6-hq2k7"
Sep 16 19:48:19.807319 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:48:19.806953    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c9f8cb8d6-hq2k7" event={"ID":"b8a1a69d-2daf-41c3-a898-c8a09b4fdce9","Type":"ContainerDied","Data":"8f8b0e892397017ee087de2380e022fe9a812c9bac2695fc6d35d1d40018b20b"}
Sep 16 19:48:19.807319 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:48:19.807036    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c9f8cb8d6-hq2k7" event={"ID":"b8a1a69d-2daf-41c3-a898-c8a09b4fdce9","Type":"ContainerDied","Data":"bb416b8e5ddc45b08e9b823848cc7d372e326364e683d32eec216e8bb4c982c3"}
Sep 16 19:48:19.807319 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:48:19.807060    2582 scope.go:117] "RemoveContainer" containerID="8f8b0e892397017ee087de2380e022fe9a812c9bac2695fc6d35d1d40018b20b"
Sep 16 19:48:19.816201 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:48:19.816182    2582 scope.go:117] "RemoveContainer" containerID="35eaa6b6e7239e346e5efe451fa51d170b9bfe3b73e04eb626f3df08c979ff0c"
Sep 16 19:48:19.824208 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:48:19.824183    2582 scope.go:117] "RemoveContainer" containerID="8f8b0e892397017ee087de2380e022fe9a812c9bac2695fc6d35d1d40018b20b"
Sep 16 19:48:19.824520 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:48:19.824499    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f8b0e892397017ee087de2380e022fe9a812c9bac2695fc6d35d1d40018b20b\": container with ID starting with 8f8b0e892397017ee087de2380e022fe9a812c9bac2695fc6d35d1d40018b20b not found: ID does not exist" containerID="8f8b0e892397017ee087de2380e022fe9a812c9bac2695fc6d35d1d40018b20b"
Sep 16 19:48:19.824577 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:48:19.824531    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f8b0e892397017ee087de2380e022fe9a812c9bac2695fc6d35d1d40018b20b"} err="failed to get container status \"8f8b0e892397017ee087de2380e022fe9a812c9bac2695fc6d35d1d40018b20b\": rpc error: code = NotFound desc = could not find container \"8f8b0e892397017ee087de2380e022fe9a812c9bac2695fc6d35d1d40018b20b\": container with ID starting with 8f8b0e892397017ee087de2380e022fe9a812c9bac2695fc6d35d1d40018b20b not found: ID does not exist"
Sep 16 19:48:19.824577 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:48:19.824549    2582 scope.go:117] "RemoveContainer" containerID="35eaa6b6e7239e346e5efe451fa51d170b9bfe3b73e04eb626f3df08c979ff0c"
Sep 16 19:48:19.824850 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:48:19.824830    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35eaa6b6e7239e346e5efe451fa51d170b9bfe3b73e04eb626f3df08c979ff0c\": container with ID starting with 35eaa6b6e7239e346e5efe451fa51d170b9bfe3b73e04eb626f3df08c979ff0c not found: ID does not exist" containerID="35eaa6b6e7239e346e5efe451fa51d170b9bfe3b73e04eb626f3df08c979ff0c"
Sep 16 19:48:19.824909 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:48:19.824855    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35eaa6b6e7239e346e5efe451fa51d170b9bfe3b73e04eb626f3df08c979ff0c"} err="failed to get container status \"35eaa6b6e7239e346e5efe451fa51d170b9bfe3b73e04eb626f3df08c979ff0c\": rpc error: code = NotFound desc = could not find container \"35eaa6b6e7239e346e5efe451fa51d170b9bfe3b73e04eb626f3df08c979ff0c\": container with ID starting with 35eaa6b6e7239e346e5efe451fa51d170b9bfe3b73e04eb626f3df08c979ff0c not found: ID does not exist"
Sep 16 19:48:19.830136 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:48:19.830113    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c9f8cb8d6-hq2k7"]
Sep 16 19:48:19.836622 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:48:19.836595    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c9f8cb8d6-hq2k7"]
Sep 16 19:48:20.872710 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:48:20.872673    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8a1a69d-2daf-41c3-a898-c8a09b4fdce9" path="/var/lib/kubelet/pods/b8a1a69d-2daf-41c3-a898-c8a09b4fdce9/volumes"
Sep 16 19:49:10.760117 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:10.760030    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt"]
Sep 16 19:49:10.760528 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:10.760348    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8a1a69d-2daf-41c3-a898-c8a09b4fdce9" containerName="main"
Sep 16 19:49:10.760528 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:10.760362    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8a1a69d-2daf-41c3-a898-c8a09b4fdce9" containerName="main"
Sep 16 19:49:10.760528 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:10.760384    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8a1a69d-2daf-41c3-a898-c8a09b4fdce9" containerName="storage-initializer"
Sep 16 19:49:10.760528 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:10.760390    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8a1a69d-2daf-41c3-a898-c8a09b4fdce9" containerName="storage-initializer"
Sep 16 19:49:10.760528 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:10.760454    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8a1a69d-2daf-41c3-a898-c8a09b4fdce9" containerName="main"
Sep 16 19:49:10.763542 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:10.763523    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt"
Sep 16 19:49:10.766519 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:10.766459    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 16 19:49:10.766519 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:10.766487    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 16 19:49:10.766741 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:10.766577    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-hsx46\""
Sep 16 19:49:10.766741 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:10.766584    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Sep 16 19:49:10.766741 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:10.766714    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 16 19:49:10.766923 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:10.766828    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:49:10.767699 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:10.767682    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dockercfg-n8mf8\""
Sep 16 19:49:10.782176 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:10.782148    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt"]
Sep 16 19:49:10.837806 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:10.837768    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfc0c17b-bfda-42e4-ad94-f18d3562aa5c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt\" (UID: \"dfc0c17b-bfda-42e4-ad94-f18d3562aa5c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt"
Sep 16 19:49:10.837977 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:10.837815    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfc0c17b-bfda-42e4-ad94-f18d3562aa5c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt\" (UID: \"dfc0c17b-bfda-42e4-ad94-f18d3562aa5c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt"
Sep 16 19:49:10.837977 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:10.837895    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfc0c17b-bfda-42e4-ad94-f18d3562aa5c-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt\" (UID: \"dfc0c17b-bfda-42e4-ad94-f18d3562aa5c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt"
Sep 16 19:49:10.837977 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:10.837936    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfc0c17b-bfda-42e4-ad94-f18d3562aa5c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt\" (UID: \"dfc0c17b-bfda-42e4-ad94-f18d3562aa5c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt"
Sep 16 19:49:10.837977 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:10.837964    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dfc0c17b-bfda-42e4-ad94-f18d3562aa5c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt\" (UID: \"dfc0c17b-bfda-42e4-ad94-f18d3562aa5c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt"
Sep 16 19:49:10.838125 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:10.838045    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dfc0c17b-bfda-42e4-ad94-f18d3562aa5c-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt\" (UID: \"dfc0c17b-bfda-42e4-ad94-f18d3562aa5c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt"
Sep 16 19:49:10.838125 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:10.838069    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8qbg\" (UniqueName: \"kubernetes.io/projected/dfc0c17b-bfda-42e4-ad94-f18d3562aa5c-kube-api-access-d8qbg\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt\" (UID: \"dfc0c17b-bfda-42e4-ad94-f18d3562aa5c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt"
Sep 16 19:49:10.838125 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:10.838105    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfc0c17b-bfda-42e4-ad94-f18d3562aa5c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt\" (UID: \"dfc0c17b-bfda-42e4-ad94-f18d3562aa5c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt"
Sep 16 19:49:10.939338 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:10.939302    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dfc0c17b-bfda-42e4-ad94-f18d3562aa5c-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt\" (UID: \"dfc0c17b-bfda-42e4-ad94-f18d3562aa5c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt"
Sep 16 19:49:10.939338 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:10.939341    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8qbg\" (UniqueName: \"kubernetes.io/projected/dfc0c17b-bfda-42e4-ad94-f18d3562aa5c-kube-api-access-d8qbg\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt\" (UID: \"dfc0c17b-bfda-42e4-ad94-f18d3562aa5c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt"
Sep 16 19:49:10.939581 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:10.939369    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfc0c17b-bfda-42e4-ad94-f18d3562aa5c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt\" (UID: \"dfc0c17b-bfda-42e4-ad94-f18d3562aa5c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt"
Sep 16 19:49:10.939581 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:10.939398    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfc0c17b-bfda-42e4-ad94-f18d3562aa5c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt\" (UID: \"dfc0c17b-bfda-42e4-ad94-f18d3562aa5c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt"
Sep 16 19:49:10.939581 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:10.939419    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfc0c17b-bfda-42e4-ad94-f18d3562aa5c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt\" (UID: \"dfc0c17b-bfda-42e4-ad94-f18d3562aa5c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt"
Sep 16 19:49:10.939581 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:10.939440    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfc0c17b-bfda-42e4-ad94-f18d3562aa5c-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt\" (UID: \"dfc0c17b-bfda-42e4-ad94-f18d3562aa5c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt"
Sep 16 19:49:10.939581 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:10.939461    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfc0c17b-bfda-42e4-ad94-f18d3562aa5c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt\" (UID: \"dfc0c17b-bfda-42e4-ad94-f18d3562aa5c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt"
Sep 16 19:49:10.939581 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:10.939493    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dfc0c17b-bfda-42e4-ad94-f18d3562aa5c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt\" (UID: \"dfc0c17b-bfda-42e4-ad94-f18d3562aa5c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt"
Sep 16 19:49:10.940137 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:10.939941    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfc0c17b-bfda-42e4-ad94-f18d3562aa5c-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt\" (UID: \"dfc0c17b-bfda-42e4-ad94-f18d3562aa5c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt"
Sep 16 19:49:10.940137 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:10.940013    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfc0c17b-bfda-42e4-ad94-f18d3562aa5c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt\" (UID: \"dfc0c17b-bfda-42e4-ad94-f18d3562aa5c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt"
Sep 16 19:49:10.940137 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:10.940094    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dfc0c17b-bfda-42e4-ad94-f18d3562aa5c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt\" (UID: \"dfc0c17b-bfda-42e4-ad94-f18d3562aa5c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt"
Sep 16 19:49:10.940349 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:10.940141    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfc0c17b-bfda-42e4-ad94-f18d3562aa5c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt\" (UID: \"dfc0c17b-bfda-42e4-ad94-f18d3562aa5c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt"
Sep 16 19:49:10.940349 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:10.940166    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dfc0c17b-bfda-42e4-ad94-f18d3562aa5c-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt\" (UID: \"dfc0c17b-bfda-42e4-ad94-f18d3562aa5c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt"
Sep 16 19:49:10.941927 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:10.941902    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfc0c17b-bfda-42e4-ad94-f18d3562aa5c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt\" (UID: \"dfc0c17b-bfda-42e4-ad94-f18d3562aa5c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt"
Sep 16 19:49:10.942058 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:10.942040    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfc0c17b-bfda-42e4-ad94-f18d3562aa5c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt\" (UID: \"dfc0c17b-bfda-42e4-ad94-f18d3562aa5c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt"
Sep 16 19:49:10.948108 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:10.948080    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8qbg\" (UniqueName: \"kubernetes.io/projected/dfc0c17b-bfda-42e4-ad94-f18d3562aa5c-kube-api-access-d8qbg\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt\" (UID: \"dfc0c17b-bfda-42e4-ad94-f18d3562aa5c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt"
Sep 16 19:49:11.073072 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:11.072982    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt"
Sep 16 19:49:11.352533 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:11.352499    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt"]
Sep 16 19:49:11.355198 ip-10-0-129-194 kubenswrapper[2582]: W0916 19:49:11.355163    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfc0c17b_bfda_42e4_ad94_f18d3562aa5c.slice/crio-2cc3e7c018e21acc16505015a39207be4855caf1be070362af7b2314e7dd322d WatchSource:0}: Error finding container 2cc3e7c018e21acc16505015a39207be4855caf1be070362af7b2314e7dd322d: Status 404 returned error can't find the container with id 2cc3e7c018e21acc16505015a39207be4855caf1be070362af7b2314e7dd322d
Sep 16 19:49:12.018896 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:12.018853    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt" event={"ID":"dfc0c17b-bfda-42e4-ad94-f18d3562aa5c","Type":"ContainerStarted","Data":"22562f1d1685a4fedff1fb3e90c5d147f0f28ee5b6a12feb4bc6c22088db6251"}
Sep 16 19:49:12.019301 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:12.018922    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt" event={"ID":"dfc0c17b-bfda-42e4-ad94-f18d3562aa5c","Type":"ContainerStarted","Data":"2cc3e7c018e21acc16505015a39207be4855caf1be070362af7b2314e7dd322d"}
Sep 16 19:49:12.019301 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:12.018991    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt"
Sep 16 19:49:13.025068 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:13.025016    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt" event={"ID":"dfc0c17b-bfda-42e4-ad94-f18d3562aa5c","Type":"ContainerStarted","Data":"8339c7e6067923d7b71ceb01ec775e4b7334b614020d3d3ec8bef6ed21bf9b5c"}
Sep 16 19:49:15.035522 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:15.035485    2582 generic.go:356] "Generic (PLEG): container finished" podID="dfc0c17b-bfda-42e4-ad94-f18d3562aa5c" containerID="8339c7e6067923d7b71ceb01ec775e4b7334b614020d3d3ec8bef6ed21bf9b5c" exitCode=0
Sep 16 19:49:15.035881 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:15.035556    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt" event={"ID":"dfc0c17b-bfda-42e4-ad94-f18d3562aa5c","Type":"ContainerDied","Data":"8339c7e6067923d7b71ceb01ec775e4b7334b614020d3d3ec8bef6ed21bf9b5c"}
Sep 16 19:49:16.043017 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:16.042984    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt" event={"ID":"dfc0c17b-bfda-42e4-ad94-f18d3562aa5c","Type":"ContainerStarted","Data":"7d3553331f6f0c19d203006b5eabde92b5318b9447a98229c9bfa3f0c1881961"}
Sep 16 19:49:16.067419 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:16.067348    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt" podStartSLOduration=3.917270524 podStartE2EDuration="6.067325322s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.150054798s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:49:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:49:16.066529755 +0000 UTC m=+3959.708245420" watchObservedRunningTime="2026-09-16 19:49:16.067325322 +0000 UTC m=+3959.709040989"
Sep 16 19:49:21.073371 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:21.073327    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt"
Sep 16 19:49:21.073957 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:21.073395    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt"
Sep 16 19:49:21.074806 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:21.074772    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt" podUID="dfc0c17b-bfda-42e4-ad94-f18d3562aa5c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8001/health\": dial tcp 10.132.0.57:8001: connect: connection refused"
Sep 16 19:49:21.087963 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:21.087937    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt"
Sep 16 19:49:31.074299 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:31.074233    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt" podUID="dfc0c17b-bfda-42e4-ad94-f18d3562aa5c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8001/health\": dial tcp 10.132.0.57:8001: connect: connection refused"
Sep 16 19:49:34.934912 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:34.934877    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8"]
Sep 16 19:49:34.938052 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:34.938032    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8"
Sep 16 19:49:34.941910 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:34.941876    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"lora-pvc-single-adapter-test-kserve-self-signed-certs\""
Sep 16 19:49:34.942031 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:34.941933    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"kube-root-ca.crt\""
Sep 16 19:49:34.942031 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:34.941993    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"default-dockercfg-wpcqv\""
Sep 16 19:49:34.942479 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:34.942455    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:49:34.942581 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:34.942459    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"seaweedfs-s3-creds\""
Sep 16 19:49:34.942581 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:34.942459    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"openshift-service-ca.crt\""
Sep 16 19:49:34.963781 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:34.963741    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8"]
Sep 16 19:49:35.056446 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:35.056407    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8\" (UID: \"31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8"
Sep 16 19:49:35.056643 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:35.056466    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcxfv\" (UniqueName: \"kubernetes.io/projected/31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023-kube-api-access-gcxfv\") pod \"lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8\" (UID: \"31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8"
Sep 16 19:49:35.056643 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:35.056509    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8\" (UID: \"31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8"
Sep 16 19:49:35.056643 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:35.056537    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023-dshm\") pod \"lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8\" (UID: \"31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8"
Sep 16 19:49:35.056643 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:35.056562    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8\" (UID: \"31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8"
Sep 16 19:49:35.056643 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:35.056597    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023-home\") pod \"lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8\" (UID: \"31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8"
Sep 16 19:49:35.056643 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:35.056623    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8\" (UID: \"31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8"
Sep 16 19:49:35.056897 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:35.056674    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8\" (UID: \"31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8"
Sep 16 19:49:35.157143 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:35.157102    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8\" (UID: \"31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8"
Sep 16 19:49:35.157353 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:35.157260    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8\" (UID: \"31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8"
Sep 16 19:49:35.157353 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:35.157294    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcxfv\" (UniqueName: \"kubernetes.io/projected/31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023-kube-api-access-gcxfv\") pod \"lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8\" (UID: \"31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8"
Sep 16 19:49:35.157353 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:35.157330    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8\" (UID: \"31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8"
Sep 16 19:49:35.157353 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:35.157352    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023-dshm\") pod \"lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8\" (UID: \"31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8"
Sep 16 19:49:35.157591 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:35.157374    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8\" (UID: \"31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8"
Sep 16 19:49:35.157591 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:35.157411    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023-home\") pod \"lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8\" (UID: \"31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8"
Sep 16 19:49:35.157591 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:35.157436    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8\" (UID: \"31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8"
Sep 16 19:49:35.157791 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:35.157689    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8\" (UID: \"31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8"
Sep 16 19:49:35.157892 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:35.157870    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023-home\") pod \"lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8\" (UID: \"31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8"
Sep 16 19:49:35.158038 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:35.158015    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8\" (UID: \"31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8"
Sep 16 19:49:35.158111 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:35.158070    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8\" (UID: \"31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8"
Sep 16 19:49:35.158211 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:35.158188    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8\" (UID: \"31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8"
Sep 16 19:49:35.159799 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:35.159779    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023-dshm\") pod \"lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8\" (UID: \"31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8"
Sep 16 19:49:35.160022 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:35.160005    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8\" (UID: \"31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8"
Sep 16 19:49:35.165661 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:35.165640    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcxfv\" (UniqueName: \"kubernetes.io/projected/31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023-kube-api-access-gcxfv\") pod \"lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8\" (UID: \"31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8"
Sep 16 19:49:40.811487 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:40.811441    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-9245be29-5793-431d-9414-32dff13f3f9b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bd3b235cd00981bb\") pod \"lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8\" (UID: \"31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8"
Sep 16 19:49:40.817878 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:40.817848    2582 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-9245be29-5793-431d-9414-32dff13f3f9b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bd3b235cd00981bb\") pod \"lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8\" (UID: \"31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8"
Sep 16 19:49:40.912035 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:40.911989    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-9245be29-5793-431d-9414-32dff13f3f9b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bd3b235cd00981bb\") pod \"lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8\" (UID: \"31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8"
Sep 16 19:49:40.912242 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:40.912092    2582 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-9245be29-5793-431d-9414-32dff13f3f9b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bd3b235cd00981bb\") pod \"lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8\" (UID: \"31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8"
Sep 16 19:49:40.915750 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:40.915691    2582 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-9245be29-5793-431d-9414-32dff13f3f9b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bd3b235cd00981bb\") pod \"lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8\" (UID: \"31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023\") DevicePath \"csi-ae685b6817d1bf78ac23b4d1f71f834f99f2d2fd2eddaa7c60fa2c493145dd65\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8"
Sep 16 19:49:41.011604 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:41.011569    2582 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-9245be29-5793-431d-9414-32dff13f3f9b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bd3b235cd00981bb\") pod \"lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8\" (UID: \"31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/d9afeddae0d033bfca5c3cae2353cb957b5c56e40c32d6e74ff955208c9fa6f1/globalmount\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8"
Sep 16 19:49:41.035971 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:41.035937    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-9245be29-5793-431d-9414-32dff13f3f9b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bd3b235cd00981bb\") pod \"lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8\" (UID: \"31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8"
Sep 16 19:49:41.074566 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:41.074448    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt" podUID="dfc0c17b-bfda-42e4-ad94-f18d3562aa5c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8001/health\": dial tcp 10.132.0.57:8001: connect: connection refused"
Sep 16 19:49:41.249604 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:41.249567    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8"
Sep 16 19:49:41.533947 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:41.533919    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8"]
Sep 16 19:49:41.536069 ip-10-0-129-194 kubenswrapper[2582]: W0916 19:49:41.536039    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31b10ee3_bfbe_4ecf_a2e0_bd1a1abb3023.slice/crio-6bfcab00e54a9899c654b2ffa8656640b2cf16e0d09904c12a711fab803c9679 WatchSource:0}: Error finding container 6bfcab00e54a9899c654b2ffa8656640b2cf16e0d09904c12a711fab803c9679: Status 404 returned error can't find the container with id 6bfcab00e54a9899c654b2ffa8656640b2cf16e0d09904c12a711fab803c9679
Sep 16 19:49:42.163858 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:42.163810    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8" event={"ID":"31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023","Type":"ContainerStarted","Data":"6dd3971698dc1f2aaf1d711e0ce9bdb2b7f0fe42c1a6408f25fce30386423175"}
Sep 16 19:49:42.164264 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:42.163878    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8" event={"ID":"31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023","Type":"ContainerStarted","Data":"6bfcab00e54a9899c654b2ffa8656640b2cf16e0d09904c12a711fab803c9679"}
Sep 16 19:49:44.174497 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:44.174406    2582 generic.go:356] "Generic (PLEG): container finished" podID="31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023" containerID="6dd3971698dc1f2aaf1d711e0ce9bdb2b7f0fe42c1a6408f25fce30386423175" exitCode=0
Sep 16 19:49:44.174497 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:44.174473    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8" event={"ID":"31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023","Type":"ContainerDied","Data":"6dd3971698dc1f2aaf1d711e0ce9bdb2b7f0fe42c1a6408f25fce30386423175"}
Sep 16 19:49:45.181958 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:45.181919    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8" event={"ID":"31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023","Type":"ContainerStarted","Data":"be548665f927a15272fca2ec012378896ab2816a67b1aece19069b2df4240b8f"}
Sep 16 19:49:45.209896 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:45.209827    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8" podStartSLOduration=8.926317082 podStartE2EDuration="11.209804365s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.283487283s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:49:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:49:45.209263527 +0000 UTC m=+3988.850979204" watchObservedRunningTime="2026-09-16 19:49:45.209804365 +0000 UTC m=+3988.851520029"
Sep 16 19:49:45.250699 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:45.250658    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8"
Sep 16 19:49:51.074369 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:51.074314    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt" podUID="dfc0c17b-bfda-42e4-ad94-f18d3562aa5c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8001/health\": dial tcp 10.132.0.57:8001: connect: connection refused"
Sep 16 19:49:51.250137 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:51.250094    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8"
Sep 16 19:49:51.251825 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:49:51.251787    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8" podUID="31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 16 19:50:01.073543 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:50:01.073476    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt" podUID="dfc0c17b-bfda-42e4-ad94-f18d3562aa5c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8001/health\": dial tcp 10.132.0.57:8001: connect: connection refused"
Sep 16 19:50:01.250270 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:50:01.250222    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8" podUID="31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 16 19:50:11.074201 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:50:11.074134    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt" podUID="dfc0c17b-bfda-42e4-ad94-f18d3562aa5c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8001/health\": dial tcp 10.132.0.57:8001: connect: connection refused"
Sep 16 19:50:11.250539 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:50:11.250479    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8" podUID="31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 16 19:50:21.074452 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:50:21.074393    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt" podUID="dfc0c17b-bfda-42e4-ad94-f18d3562aa5c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8001/health\": dial tcp 10.132.0.57:8001: connect: connection refused"
Sep 16 19:50:21.250347 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:50:21.250296    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8" podUID="31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 16 19:50:31.074410 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:50:31.074303    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt" podUID="dfc0c17b-bfda-42e4-ad94-f18d3562aa5c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8001/health\": dial tcp 10.132.0.57:8001: connect: connection refused"
Sep 16 19:50:31.250793 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:50:31.250741    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8" podUID="31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 16 19:50:41.074389 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:50:41.074163    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt" podUID="dfc0c17b-bfda-42e4-ad94-f18d3562aa5c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8001/health\": dial tcp 10.132.0.57:8001: connect: connection refused"
Sep 16 19:50:41.250889 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:50:41.250839    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8" podUID="31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 16 19:50:51.074430 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:50:51.074378    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt" podUID="dfc0c17b-bfda-42e4-ad94-f18d3562aa5c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8001/health\": dial tcp 10.132.0.57:8001: connect: connection refused"
Sep 16 19:50:51.250030 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:50:51.249988    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8" podUID="31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 16 19:51:01.074202 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:01.074144    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt" podUID="dfc0c17b-bfda-42e4-ad94-f18d3562aa5c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8001/health\": dial tcp 10.132.0.57:8001: connect: connection refused"
Sep 16 19:51:01.260883 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:01.260848    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8"
Sep 16 19:51:01.269010 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:01.268978    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8"
Sep 16 19:51:11.074030 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:11.073981    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt" podUID="dfc0c17b-bfda-42e4-ad94-f18d3562aa5c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8001/health\": dial tcp 10.132.0.57:8001: connect: connection refused"
Sep 16 19:51:21.090347 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:21.090295    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt"
Sep 16 19:51:21.102938 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:21.102904    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt"
Sep 16 19:51:35.205471 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:35.205416    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt"]
Sep 16 19:51:39.359607 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:39.359569    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-4412bdb9_lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8_31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023/storage-initializer/0.log"
Sep 16 19:51:39.369447 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:39.369413    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-4412bdb9_lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8_31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023/main/0.log"
Sep 16 19:51:40.385431 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:40.385388    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8"]
Sep 16 19:51:50.230879 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:50.230809    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt" podUID="dfc0c17b-bfda-42e4-ad94-f18d3562aa5c" containerName="main" containerID="cri-o://7d3553331f6f0c19d203006b5eabde92b5318b9447a98229c9bfa3f0c1881961" gracePeriod=45
Sep 16 19:51:50.847903 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:50.847844    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt" podUID="dfc0c17b-bfda-42e4-ad94-f18d3562aa5c" containerName="llm-d-routing-sidecar" containerID="cri-o://22562f1d1685a4fedff1fb3e90c5d147f0f28ee5b6a12feb4bc6c22088db6251" gracePeriod=45
Sep 16 19:51:51.098831 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:51.098797    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt"
Sep 16 19:51:51.222313 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:51.222277    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfc0c17b-bfda-42e4-ad94-f18d3562aa5c-model-cache\") pod \"dfc0c17b-bfda-42e4-ad94-f18d3562aa5c\" (UID: \"dfc0c17b-bfda-42e4-ad94-f18d3562aa5c\") "
Sep 16 19:51:51.222313 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:51.222322    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dfc0c17b-bfda-42e4-ad94-f18d3562aa5c-cabundle-cert\") pod \"dfc0c17b-bfda-42e4-ad94-f18d3562aa5c\" (UID: \"dfc0c17b-bfda-42e4-ad94-f18d3562aa5c\") "
Sep 16 19:51:51.222585 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:51.222353    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfc0c17b-bfda-42e4-ad94-f18d3562aa5c-tls-certs\") pod \"dfc0c17b-bfda-42e4-ad94-f18d3562aa5c\" (UID: \"dfc0c17b-bfda-42e4-ad94-f18d3562aa5c\") "
Sep 16 19:51:51.222585 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:51.222478    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfc0c17b-bfda-42e4-ad94-f18d3562aa5c-dshm\") pod \"dfc0c17b-bfda-42e4-ad94-f18d3562aa5c\" (UID: \"dfc0c17b-bfda-42e4-ad94-f18d3562aa5c\") "
Sep 16 19:51:51.222585 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:51.222518    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfc0c17b-bfda-42e4-ad94-f18d3562aa5c-tmp-dir\") pod \"dfc0c17b-bfda-42e4-ad94-f18d3562aa5c\" (UID: \"dfc0c17b-bfda-42e4-ad94-f18d3562aa5c\") "
Sep 16 19:51:51.222585 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:51.222549    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dfc0c17b-bfda-42e4-ad94-f18d3562aa5c-kserve-provision-location\") pod \"dfc0c17b-bfda-42e4-ad94-f18d3562aa5c\" (UID: \"dfc0c17b-bfda-42e4-ad94-f18d3562aa5c\") "
Sep 16 19:51:51.222585 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:51.222582    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8qbg\" (UniqueName: \"kubernetes.io/projected/dfc0c17b-bfda-42e4-ad94-f18d3562aa5c-kube-api-access-d8qbg\") pod \"dfc0c17b-bfda-42e4-ad94-f18d3562aa5c\" (UID: \"dfc0c17b-bfda-42e4-ad94-f18d3562aa5c\") "
Sep 16 19:51:51.222894 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:51.222622    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfc0c17b-bfda-42e4-ad94-f18d3562aa5c-home\") pod \"dfc0c17b-bfda-42e4-ad94-f18d3562aa5c\" (UID: \"dfc0c17b-bfda-42e4-ad94-f18d3562aa5c\") "
Sep 16 19:51:51.222894 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:51.222637    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfc0c17b-bfda-42e4-ad94-f18d3562aa5c-model-cache" (OuterVolumeSpecName: "model-cache") pod "dfc0c17b-bfda-42e4-ad94-f18d3562aa5c" (UID: "dfc0c17b-bfda-42e4-ad94-f18d3562aa5c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:51:51.223010 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:51.222980    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfc0c17b-bfda-42e4-ad94-f18d3562aa5c-model-cache\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:51:51.223293 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:51.223264    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfc0c17b-bfda-42e4-ad94-f18d3562aa5c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dfc0c17b-bfda-42e4-ad94-f18d3562aa5c" (UID: "dfc0c17b-bfda-42e4-ad94-f18d3562aa5c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:51:51.223430 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:51.223390    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfc0c17b-bfda-42e4-ad94-f18d3562aa5c-home" (OuterVolumeSpecName: "home") pod "dfc0c17b-bfda-42e4-ad94-f18d3562aa5c" (UID: "dfc0c17b-bfda-42e4-ad94-f18d3562aa5c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:51:51.224963 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:51.224913    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfc0c17b-bfda-42e4-ad94-f18d3562aa5c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dfc0c17b-bfda-42e4-ad94-f18d3562aa5c" (UID: "dfc0c17b-bfda-42e4-ad94-f18d3562aa5c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:51:51.225136 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:51.225080    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfc0c17b-bfda-42e4-ad94-f18d3562aa5c-kube-api-access-d8qbg" (OuterVolumeSpecName: "kube-api-access-d8qbg") pod "dfc0c17b-bfda-42e4-ad94-f18d3562aa5c" (UID: "dfc0c17b-bfda-42e4-ad94-f18d3562aa5c"). InnerVolumeSpecName "kube-api-access-d8qbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:51:51.225136 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:51.225093    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfc0c17b-bfda-42e4-ad94-f18d3562aa5c-dshm" (OuterVolumeSpecName: "dshm") pod "dfc0c17b-bfda-42e4-ad94-f18d3562aa5c" (UID: "dfc0c17b-bfda-42e4-ad94-f18d3562aa5c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:51:51.239607 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:51.239557    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfc0c17b-bfda-42e4-ad94-f18d3562aa5c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dfc0c17b-bfda-42e4-ad94-f18d3562aa5c" (UID: "dfc0c17b-bfda-42e4-ad94-f18d3562aa5c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:51:51.248461 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:51.248416    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfc0c17b-bfda-42e4-ad94-f18d3562aa5c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dfc0c17b-bfda-42e4-ad94-f18d3562aa5c" (UID: "dfc0c17b-bfda-42e4-ad94-f18d3562aa5c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:51:51.324024 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:51.323978    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfc0c17b-bfda-42e4-ad94-f18d3562aa5c-tmp-dir\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:51:51.324024 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:51.324016    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dfc0c17b-bfda-42e4-ad94-f18d3562aa5c-kserve-provision-location\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:51:51.324024 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:51.324027    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8qbg\" (UniqueName: \"kubernetes.io/projected/dfc0c17b-bfda-42e4-ad94-f18d3562aa5c-kube-api-access-d8qbg\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:51:51.324024 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:51.324036    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfc0c17b-bfda-42e4-ad94-f18d3562aa5c-home\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:51:51.324312 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:51.324045    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dfc0c17b-bfda-42e4-ad94-f18d3562aa5c-cabundle-cert\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:51:51.324312 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:51.324053    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfc0c17b-bfda-42e4-ad94-f18d3562aa5c-tls-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:51:51.324312 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:51.324061    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfc0c17b-bfda-42e4-ad94-f18d3562aa5c-dshm\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:51:51.812519 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:51.812477    2582 generic.go:356] "Generic (PLEG): container finished" podID="dfc0c17b-bfda-42e4-ad94-f18d3562aa5c" containerID="7d3553331f6f0c19d203006b5eabde92b5318b9447a98229c9bfa3f0c1881961" exitCode=0
Sep 16 19:51:51.812519 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:51.812513    2582 generic.go:356] "Generic (PLEG): container finished" podID="dfc0c17b-bfda-42e4-ad94-f18d3562aa5c" containerID="22562f1d1685a4fedff1fb3e90c5d147f0f28ee5b6a12feb4bc6c22088db6251" exitCode=0
Sep 16 19:51:51.812775 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:51.812540    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt" event={"ID":"dfc0c17b-bfda-42e4-ad94-f18d3562aa5c","Type":"ContainerDied","Data":"7d3553331f6f0c19d203006b5eabde92b5318b9447a98229c9bfa3f0c1881961"}
Sep 16 19:51:51.812775 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:51.812546    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt"
Sep 16 19:51:51.812775 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:51.812615    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt" event={"ID":"dfc0c17b-bfda-42e4-ad94-f18d3562aa5c","Type":"ContainerDied","Data":"22562f1d1685a4fedff1fb3e90c5d147f0f28ee5b6a12feb4bc6c22088db6251"}
Sep 16 19:51:51.812775 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:51.812632    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt" event={"ID":"dfc0c17b-bfda-42e4-ad94-f18d3562aa5c","Type":"ContainerDied","Data":"2cc3e7c018e21acc16505015a39207be4855caf1be070362af7b2314e7dd322d"}
Sep 16 19:51:51.812775 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:51.812653    2582 scope.go:117] "RemoveContainer" containerID="7d3553331f6f0c19d203006b5eabde92b5318b9447a98229c9bfa3f0c1881961"
Sep 16 19:51:51.822648 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:51.822626    2582 scope.go:117] "RemoveContainer" containerID="8339c7e6067923d7b71ceb01ec775e4b7334b614020d3d3ec8bef6ed21bf9b5c"
Sep 16 19:51:51.831004 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:51.830983    2582 scope.go:117] "RemoveContainer" containerID="22562f1d1685a4fedff1fb3e90c5d147f0f28ee5b6a12feb4bc6c22088db6251"
Sep 16 19:51:51.839086 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:51.839053    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt"]
Sep 16 19:51:51.842844 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:51.842818    2582 scope.go:117] "RemoveContainer" containerID="7d3553331f6f0c19d203006b5eabde92b5318b9447a98229c9bfa3f0c1881961"
Sep 16 19:51:51.843241 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:51:51.843216    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d3553331f6f0c19d203006b5eabde92b5318b9447a98229c9bfa3f0c1881961\": container with ID starting with 7d3553331f6f0c19d203006b5eabde92b5318b9447a98229c9bfa3f0c1881961 not found: ID does not exist" containerID="7d3553331f6f0c19d203006b5eabde92b5318b9447a98229c9bfa3f0c1881961"
Sep 16 19:51:51.843340 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:51.843250    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d3553331f6f0c19d203006b5eabde92b5318b9447a98229c9bfa3f0c1881961"} err="failed to get container status \"7d3553331f6f0c19d203006b5eabde92b5318b9447a98229c9bfa3f0c1881961\": rpc error: code = NotFound desc = could not find container \"7d3553331f6f0c19d203006b5eabde92b5318b9447a98229c9bfa3f0c1881961\": container with ID starting with 7d3553331f6f0c19d203006b5eabde92b5318b9447a98229c9bfa3f0c1881961 not found: ID does not exist"
Sep 16 19:51:51.843340 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:51.843270    2582 scope.go:117] "RemoveContainer" containerID="8339c7e6067923d7b71ceb01ec775e4b7334b614020d3d3ec8bef6ed21bf9b5c"
Sep 16 19:51:51.843567 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:51:51.843549    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8339c7e6067923d7b71ceb01ec775e4b7334b614020d3d3ec8bef6ed21bf9b5c\": container with ID starting with 8339c7e6067923d7b71ceb01ec775e4b7334b614020d3d3ec8bef6ed21bf9b5c not found: ID does not exist" containerID="8339c7e6067923d7b71ceb01ec775e4b7334b614020d3d3ec8bef6ed21bf9b5c"
Sep 16 19:51:51.843615 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:51.843572    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8339c7e6067923d7b71ceb01ec775e4b7334b614020d3d3ec8bef6ed21bf9b5c"} err="failed to get container status \"8339c7e6067923d7b71ceb01ec775e4b7334b614020d3d3ec8bef6ed21bf9b5c\": rpc error: code = NotFound desc = could not find container \"8339c7e6067923d7b71ceb01ec775e4b7334b614020d3d3ec8bef6ed21bf9b5c\": container with ID starting with 8339c7e6067923d7b71ceb01ec775e4b7334b614020d3d3ec8bef6ed21bf9b5c not found: ID does not exist"
Sep 16 19:51:51.843615 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:51.843588    2582 scope.go:117] "RemoveContainer" containerID="22562f1d1685a4fedff1fb3e90c5d147f0f28ee5b6a12feb4bc6c22088db6251"
Sep 16 19:51:51.843824 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:51:51.843806    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22562f1d1685a4fedff1fb3e90c5d147f0f28ee5b6a12feb4bc6c22088db6251\": container with ID starting with 22562f1d1685a4fedff1fb3e90c5d147f0f28ee5b6a12feb4bc6c22088db6251 not found: ID does not exist" containerID="22562f1d1685a4fedff1fb3e90c5d147f0f28ee5b6a12feb4bc6c22088db6251"
Sep 16 19:51:51.843880 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:51.843827    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22562f1d1685a4fedff1fb3e90c5d147f0f28ee5b6a12feb4bc6c22088db6251"} err="failed to get container status \"22562f1d1685a4fedff1fb3e90c5d147f0f28ee5b6a12feb4bc6c22088db6251\": rpc error: code = NotFound desc = could not find container \"22562f1d1685a4fedff1fb3e90c5d147f0f28ee5b6a12feb4bc6c22088db6251\": container with ID starting with 22562f1d1685a4fedff1fb3e90c5d147f0f28ee5b6a12feb4bc6c22088db6251 not found: ID does not exist"
Sep 16 19:51:51.843880 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:51.843839    2582 scope.go:117] "RemoveContainer" containerID="7d3553331f6f0c19d203006b5eabde92b5318b9447a98229c9bfa3f0c1881961"
Sep 16 19:51:51.844087 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:51.844072    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d3553331f6f0c19d203006b5eabde92b5318b9447a98229c9bfa3f0c1881961"} err="failed to get container status \"7d3553331f6f0c19d203006b5eabde92b5318b9447a98229c9bfa3f0c1881961\": rpc error: code = NotFound desc = could not find container \"7d3553331f6f0c19d203006b5eabde92b5318b9447a98229c9bfa3f0c1881961\": container with ID starting with 7d3553331f6f0c19d203006b5eabde92b5318b9447a98229c9bfa3f0c1881961 not found: ID does not exist"
Sep 16 19:51:51.844140 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:51.844086    2582 scope.go:117] "RemoveContainer" containerID="8339c7e6067923d7b71ceb01ec775e4b7334b614020d3d3ec8bef6ed21bf9b5c"
Sep 16 19:51:51.844318 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:51.844293    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8339c7e6067923d7b71ceb01ec775e4b7334b614020d3d3ec8bef6ed21bf9b5c"} err="failed to get container status \"8339c7e6067923d7b71ceb01ec775e4b7334b614020d3d3ec8bef6ed21bf9b5c\": rpc error: code = NotFound desc = could not find container \"8339c7e6067923d7b71ceb01ec775e4b7334b614020d3d3ec8bef6ed21bf9b5c\": container with ID starting with 8339c7e6067923d7b71ceb01ec775e4b7334b614020d3d3ec8bef6ed21bf9b5c not found: ID does not exist"
Sep 16 19:51:51.844362 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:51.844318    2582 scope.go:117] "RemoveContainer" containerID="22562f1d1685a4fedff1fb3e90c5d147f0f28ee5b6a12feb4bc6c22088db6251"
Sep 16 19:51:51.844571 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:51.844554    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22562f1d1685a4fedff1fb3e90c5d147f0f28ee5b6a12feb4bc6c22088db6251"} err="failed to get container status \"22562f1d1685a4fedff1fb3e90c5d147f0f28ee5b6a12feb4bc6c22088db6251\": rpc error: code = NotFound desc = could not find container \"22562f1d1685a4fedff1fb3e90c5d147f0f28ee5b6a12feb4bc6c22088db6251\": container with ID starting with 22562f1d1685a4fedff1fb3e90c5d147f0f28ee5b6a12feb4bc6c22088db6251 not found: ID does not exist"
Sep 16 19:51:51.847104 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:51.847082    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt"]
Sep 16 19:51:52.873288 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:52.873254    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfc0c17b-bfda-42e4-ad94-f18d3562aa5c" path="/var/lib/kubelet/pods/dfc0c17b-bfda-42e4-ad94-f18d3562aa5c/volumes"
Sep 16 19:51:55.408342 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:55.408277    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8" podUID="31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023" containerName="main" containerID="cri-o://be548665f927a15272fca2ec012378896ab2816a67b1aece19069b2df4240b8f" gracePeriod=45
Sep 16 19:51:56.070752 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:56.070708    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8"
Sep 16 19:51:56.074628 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:56.074572    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt" podUID="dfc0c17b-bfda-42e4-ad94-f18d3562aa5c" containerName="llm-d-routing-sidecar" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": context deadline exceeded"
Sep 16 19:51:56.092036 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:56.091960    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt" podUID="dfc0c17b-bfda-42e4-ad94-f18d3562aa5c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": context deadline exceeded"
Sep 16 19:51:56.166670 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:56.166581    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023-model-cache\") pod \"31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023\" (UID: \"31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023\") "
Sep 16 19:51:56.166670 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:56.166618    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023-home\") pod \"31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023\" (UID: \"31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023\") "
Sep 16 19:51:56.166670 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:56.166666    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023-dshm\") pod \"31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023\" (UID: \"31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023\") "
Sep 16 19:51:56.166983 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:56.166692    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gcxfv\" (UniqueName: \"kubernetes.io/projected/31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023-kube-api-access-gcxfv\") pod \"31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023\" (UID: \"31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023\") "
Sep 16 19:51:56.166983 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:56.166738    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023-kserve-provision-location\") pod \"31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023\" (UID: \"31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023\") "
Sep 16 19:51:56.166983 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:56.166872    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"lora-pvc-lora-adapter-1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bd3b235cd00981bb\") pod \"31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023\" (UID: \"31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023\") "
Sep 16 19:51:56.166983 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:56.166922    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023-tls-certs\") pod \"31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023\" (UID: \"31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023\") "
Sep 16 19:51:56.166983 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:56.166954    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023-cabundle-cert\") pod \"31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023\" (UID: \"31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023\") "
Sep 16 19:51:56.166983 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:56.166952    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023-model-cache" (OuterVolumeSpecName: "model-cache") pod "31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023" (UID: "31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:51:56.167298 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:56.166991    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023-tmp-dir\") pod \"31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023\" (UID: \"31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023\") "
Sep 16 19:51:56.167353 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:56.167297    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023-model-cache\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:51:56.167547 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:56.167523    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023" (UID: "31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:51:56.169280 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:56.169241    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023" (UID: "31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:51:56.169280 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:56.169247    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023-kube-api-access-gcxfv" (OuterVolumeSpecName: "kube-api-access-gcxfv") pod "31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023" (UID: "31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023"). InnerVolumeSpecName "kube-api-access-gcxfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:51:56.169458 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:56.169311    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023-home" (OuterVolumeSpecName: "home") pod "31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023" (UID: "31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:51:56.169688 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:56.169665    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023-dshm" (OuterVolumeSpecName: "dshm") pod "31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023" (UID: "31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:51:56.173352 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:56.173332    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0bd3b235cd00981bb" (OuterVolumeSpecName: "lora-pvc-lora-adapter-1") pod "31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023" (UID: "31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023"). InnerVolumeSpecName "pvc-9245be29-5793-431d-9414-32dff13f3f9b". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 16 19:51:56.184856 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:56.184825    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023" (UID: "31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:51:56.194951 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:56.194922    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023" (UID: "31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:51:56.267846 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:56.267810    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023-dshm\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:51:56.267846 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:56.267839    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcxfv\" (UniqueName: \"kubernetes.io/projected/31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023-kube-api-access-gcxfv\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:51:56.267846 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:56.267850    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023-kserve-provision-location\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:51:56.268079 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:56.267882    2582 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-9245be29-5793-431d-9414-32dff13f3f9b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bd3b235cd00981bb\") on node \"ip-10-0-129-194.ec2.internal\" "
Sep 16 19:51:56.268079 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:56.267897    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023-tls-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:51:56.268079 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:56.267909    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023-cabundle-cert\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:51:56.268079 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:56.267917    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023-tmp-dir\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:51:56.268079 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:56.267924    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023-home\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:51:56.278403 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:56.278382    2582 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-9245be29-5793-431d-9414-32dff13f3f9b" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0bd3b235cd00981bb") on node "ip-10-0-129-194.ec2.internal"
Sep 16 19:51:56.368849 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:56.368810    2582 reconciler_common.go:299] "Volume detached for volume \"pvc-9245be29-5793-431d-9414-32dff13f3f9b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bd3b235cd00981bb\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"csi-ae685b6817d1bf78ac23b4d1f71f834f99f2d2fd2eddaa7c60fa2c493145dd65\""
Sep 16 19:51:56.838186 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:56.838143    2582 generic.go:356] "Generic (PLEG): container finished" podID="31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023" containerID="be548665f927a15272fca2ec012378896ab2816a67b1aece19069b2df4240b8f" exitCode=0
Sep 16 19:51:56.838635 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:56.838203    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8"
Sep 16 19:51:56.838635 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:56.838216    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8" event={"ID":"31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023","Type":"ContainerDied","Data":"be548665f927a15272fca2ec012378896ab2816a67b1aece19069b2df4240b8f"}
Sep 16 19:51:56.838635 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:56.838269    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8" event={"ID":"31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023","Type":"ContainerDied","Data":"6bfcab00e54a9899c654b2ffa8656640b2cf16e0d09904c12a711fab803c9679"}
Sep 16 19:51:56.838635 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:56.838292    2582 scope.go:117] "RemoveContainer" containerID="be548665f927a15272fca2ec012378896ab2816a67b1aece19069b2df4240b8f"
Sep 16 19:51:56.854424 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:56.854382    2582 scope.go:117] "RemoveContainer" containerID="6dd3971698dc1f2aaf1d711e0ce9bdb2b7f0fe42c1a6408f25fce30386423175"
Sep 16 19:51:56.862295 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:56.862275    2582 scope.go:117] "RemoveContainer" containerID="be548665f927a15272fca2ec012378896ab2816a67b1aece19069b2df4240b8f"
Sep 16 19:51:56.862607 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:51:56.862585    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be548665f927a15272fca2ec012378896ab2816a67b1aece19069b2df4240b8f\": container with ID starting with be548665f927a15272fca2ec012378896ab2816a67b1aece19069b2df4240b8f not found: ID does not exist" containerID="be548665f927a15272fca2ec012378896ab2816a67b1aece19069b2df4240b8f"
Sep 16 19:51:56.862651 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:56.862622    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be548665f927a15272fca2ec012378896ab2816a67b1aece19069b2df4240b8f"} err="failed to get container status \"be548665f927a15272fca2ec012378896ab2816a67b1aece19069b2df4240b8f\": rpc error: code = NotFound desc = could not find container \"be548665f927a15272fca2ec012378896ab2816a67b1aece19069b2df4240b8f\": container with ID starting with be548665f927a15272fca2ec012378896ab2816a67b1aece19069b2df4240b8f not found: ID does not exist"
Sep 16 19:51:56.862651 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:56.862643    2582 scope.go:117] "RemoveContainer" containerID="6dd3971698dc1f2aaf1d711e0ce9bdb2b7f0fe42c1a6408f25fce30386423175"
Sep 16 19:51:56.862952 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:51:56.862934    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dd3971698dc1f2aaf1d711e0ce9bdb2b7f0fe42c1a6408f25fce30386423175\": container with ID starting with 6dd3971698dc1f2aaf1d711e0ce9bdb2b7f0fe42c1a6408f25fce30386423175 not found: ID does not exist" containerID="6dd3971698dc1f2aaf1d711e0ce9bdb2b7f0fe42c1a6408f25fce30386423175"
Sep 16 19:51:56.863005 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:56.862956    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dd3971698dc1f2aaf1d711e0ce9bdb2b7f0fe42c1a6408f25fce30386423175"} err="failed to get container status \"6dd3971698dc1f2aaf1d711e0ce9bdb2b7f0fe42c1a6408f25fce30386423175\": rpc error: code = NotFound desc = could not find container \"6dd3971698dc1f2aaf1d711e0ce9bdb2b7f0fe42c1a6408f25fce30386423175\": container with ID starting with 6dd3971698dc1f2aaf1d711e0ce9bdb2b7f0fe42c1a6408f25fce30386423175 not found: ID does not exist"
Sep 16 19:51:56.873776 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:56.873748    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8"]
Sep 16 19:51:56.873776 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:56.873783    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-577599dbc6-2nlw8"]
Sep 16 19:51:58.872358 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:51:58.872323    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023" path="/var/lib/kubelet/pods/31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023/volumes"
Sep 16 19:52:01.074297 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:01.074249    2582 prober.go:120] "Probe failed" probeType="Liveness" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b477b8d9dpwjrt" podUID="dfc0c17b-bfda-42e4-ad94-f18d3562aa5c" containerName="llm-d-routing-sidecar" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 16 19:52:29.259257 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:29.259220    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8"]
Sep 16 19:52:29.259693 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:29.259541    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfc0c17b-bfda-42e4-ad94-f18d3562aa5c" containerName="main"
Sep 16 19:52:29.259693 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:29.259553    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfc0c17b-bfda-42e4-ad94-f18d3562aa5c" containerName="main"
Sep 16 19:52:29.259693 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:29.259567    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023" containerName="main"
Sep 16 19:52:29.259693 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:29.259576    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023" containerName="main"
Sep 16 19:52:29.259693 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:29.259590    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfc0c17b-bfda-42e4-ad94-f18d3562aa5c" containerName="llm-d-routing-sidecar"
Sep 16 19:52:29.259693 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:29.259596    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfc0c17b-bfda-42e4-ad94-f18d3562aa5c" containerName="llm-d-routing-sidecar"
Sep 16 19:52:29.259693 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:29.259603    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023" containerName="storage-initializer"
Sep 16 19:52:29.259693 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:29.259609    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023" containerName="storage-initializer"
Sep 16 19:52:29.259693 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:29.259617    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfc0c17b-bfda-42e4-ad94-f18d3562aa5c" containerName="storage-initializer"
Sep 16 19:52:29.259693 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:29.259623    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfc0c17b-bfda-42e4-ad94-f18d3562aa5c" containerName="storage-initializer"
Sep 16 19:52:29.259693 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:29.259673    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfc0c17b-bfda-42e4-ad94-f18d3562aa5c" containerName="llm-d-routing-sidecar"
Sep 16 19:52:29.259693 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:29.259683    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="31b10ee3-bfbe-4ecf-a2e0-bd1a1abb3023" containerName="main"
Sep 16 19:52:29.259693 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:29.259692    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfc0c17b-bfda-42e4-ad94-f18d3562aa5c" containerName="main"
Sep 16 19:52:29.262572 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:29.262550    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8"
Sep 16 19:52:29.265176 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:29.265146    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"kube-root-ca.crt\""
Sep 16 19:52:29.265932 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:29.265906    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"openshift-service-ca.crt\""
Sep 16 19:52:29.266079 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:29.265905    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"default-dockercfg-czkfv\""
Sep 16 19:52:29.266079 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:29.266034    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"lora-pvc-adapters-test-kserve-self-signed-certs\""
Sep 16 19:52:29.282313 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:29.282278    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8"]
Sep 16 19:52:29.344798 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:29.344754    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79f47a45-eb8f-406d-92bc-bbf9ba83c1de-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8\" (UID: \"79f47a45-eb8f-406d-92bc-bbf9ba83c1de\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8"
Sep 16 19:52:29.344798 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:29.344796    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79f47a45-eb8f-406d-92bc-bbf9ba83c1de-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8\" (UID: \"79f47a45-eb8f-406d-92bc-bbf9ba83c1de\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8"
Sep 16 19:52:29.345072 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:29.344823    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79f47a45-eb8f-406d-92bc-bbf9ba83c1de-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8\" (UID: \"79f47a45-eb8f-406d-92bc-bbf9ba83c1de\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8"
Sep 16 19:52:29.345072 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:29.344888    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79f47a45-eb8f-406d-92bc-bbf9ba83c1de-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8\" (UID: \"79f47a45-eb8f-406d-92bc-bbf9ba83c1de\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8"
Sep 16 19:52:29.345072 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:29.344948    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79f47a45-eb8f-406d-92bc-bbf9ba83c1de-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8\" (UID: \"79f47a45-eb8f-406d-92bc-bbf9ba83c1de\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8"
Sep 16 19:52:29.446457 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:29.446409    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79f47a45-eb8f-406d-92bc-bbf9ba83c1de-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8\" (UID: \"79f47a45-eb8f-406d-92bc-bbf9ba83c1de\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8"
Sep 16 19:52:29.446457 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:29.446459    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79f47a45-eb8f-406d-92bc-bbf9ba83c1de-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8\" (UID: \"79f47a45-eb8f-406d-92bc-bbf9ba83c1de\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8"
Sep 16 19:52:29.446755 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:29.446482    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79f47a45-eb8f-406d-92bc-bbf9ba83c1de-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8\" (UID: \"79f47a45-eb8f-406d-92bc-bbf9ba83c1de\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8"
Sep 16 19:52:29.446755 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:29.446498    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79f47a45-eb8f-406d-92bc-bbf9ba83c1de-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8\" (UID: \"79f47a45-eb8f-406d-92bc-bbf9ba83c1de\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8"
Sep 16 19:52:29.446755 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:29.446556    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79f47a45-eb8f-406d-92bc-bbf9ba83c1de-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8\" (UID: \"79f47a45-eb8f-406d-92bc-bbf9ba83c1de\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8"
Sep 16 19:52:29.446755 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:29.446596    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnczz\" (UniqueName: \"kubernetes.io/projected/79f47a45-eb8f-406d-92bc-bbf9ba83c1de-kube-api-access-fnczz\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8\" (UID: \"79f47a45-eb8f-406d-92bc-bbf9ba83c1de\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8"
Sep 16 19:52:29.446979 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:29.446922    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79f47a45-eb8f-406d-92bc-bbf9ba83c1de-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8\" (UID: \"79f47a45-eb8f-406d-92bc-bbf9ba83c1de\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8"
Sep 16 19:52:29.446979 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:29.446929    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79f47a45-eb8f-406d-92bc-bbf9ba83c1de-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8\" (UID: \"79f47a45-eb8f-406d-92bc-bbf9ba83c1de\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8"
Sep 16 19:52:29.449706 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:29.446966    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79f47a45-eb8f-406d-92bc-bbf9ba83c1de-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8\" (UID: \"79f47a45-eb8f-406d-92bc-bbf9ba83c1de\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8"
Sep 16 19:52:29.449969 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:29.449942    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79f47a45-eb8f-406d-92bc-bbf9ba83c1de-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8\" (UID: \"79f47a45-eb8f-406d-92bc-bbf9ba83c1de\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8"
Sep 16 19:52:29.454607 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:29.454582    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79f47a45-eb8f-406d-92bc-bbf9ba83c1de-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8\" (UID: \"79f47a45-eb8f-406d-92bc-bbf9ba83c1de\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8"
Sep 16 19:52:29.547411 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:29.547304    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnczz\" (UniqueName: \"kubernetes.io/projected/79f47a45-eb8f-406d-92bc-bbf9ba83c1de-kube-api-access-fnczz\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8\" (UID: \"79f47a45-eb8f-406d-92bc-bbf9ba83c1de\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8"
Sep 16 19:52:29.555683 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:29.555646    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnczz\" (UniqueName: \"kubernetes.io/projected/79f47a45-eb8f-406d-92bc-bbf9ba83c1de-kube-api-access-fnczz\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8\" (UID: \"79f47a45-eb8f-406d-92bc-bbf9ba83c1de\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8"
Sep 16 19:52:33.381643 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:33.381599    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-0aedfc2c-223d-4c6c-b480-93cbde3a0e1a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0eb6e5be7fc1f4c94\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8\" (UID: \"79f47a45-eb8f-406d-92bc-bbf9ba83c1de\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8"
Sep 16 19:52:33.387814 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:33.387784    2582 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-0aedfc2c-223d-4c6c-b480-93cbde3a0e1a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0eb6e5be7fc1f4c94\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8\" (UID: \"79f47a45-eb8f-406d-92bc-bbf9ba83c1de\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8"
Sep 16 19:52:33.482378 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:33.482333    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-0aedfc2c-223d-4c6c-b480-93cbde3a0e1a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0eb6e5be7fc1f4c94\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8\" (UID: \"79f47a45-eb8f-406d-92bc-bbf9ba83c1de\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8"
Sep 16 19:52:33.482569 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:33.482451    2582 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-0aedfc2c-223d-4c6c-b480-93cbde3a0e1a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0eb6e5be7fc1f4c94\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8\" (UID: \"79f47a45-eb8f-406d-92bc-bbf9ba83c1de\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8"
Sep 16 19:52:33.486242 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:33.486208    2582 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-0aedfc2c-223d-4c6c-b480-93cbde3a0e1a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0eb6e5be7fc1f4c94\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8\" (UID: \"79f47a45-eb8f-406d-92bc-bbf9ba83c1de\") DevicePath \"csi-6e487c693fba6bbfd1e8fab9c676a3a68b7da96812722c27e22a98f4af3add8a\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8"
Sep 16 19:52:33.574364 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:33.574332    2582 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-0aedfc2c-223d-4c6c-b480-93cbde3a0e1a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0eb6e5be7fc1f4c94\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8\" (UID: \"79f47a45-eb8f-406d-92bc-bbf9ba83c1de\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/dd931231270b48cb4cee8c64e7f689b8f1564e3b0455ef756768af2b40f3848e/globalmount\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8"
Sep 16 19:52:33.596043 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:33.596008    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-0aedfc2c-223d-4c6c-b480-93cbde3a0e1a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0eb6e5be7fc1f4c94\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8\" (UID: \"79f47a45-eb8f-406d-92bc-bbf9ba83c1de\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8"
Sep 16 19:52:33.773260 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:33.773227    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8"
Sep 16 19:52:34.069659 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:34.069571    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8"]
Sep 16 19:52:34.072935 ip-10-0-129-194 kubenswrapper[2582]: W0916 19:52:34.072904    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79f47a45_eb8f_406d_92bc_bbf9ba83c1de.slice/crio-a76ffc07a1529a58f362d08ea70f541afdb2a4cde87da0f3b2cd63075a9d78dc WatchSource:0}: Error finding container a76ffc07a1529a58f362d08ea70f541afdb2a4cde87da0f3b2cd63075a9d78dc: Status 404 returned error can't find the container with id a76ffc07a1529a58f362d08ea70f541afdb2a4cde87da0f3b2cd63075a9d78dc
Sep 16 19:52:35.008233 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:35.008195    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8" event={"ID":"79f47a45-eb8f-406d-92bc-bbf9ba83c1de","Type":"ContainerStarted","Data":"36e6243e48ff93a8d0eb9b538b09f827768f068b10d5ce8c16c299fa0a19755b"}
Sep 16 19:52:35.008782 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:35.008256    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8" event={"ID":"79f47a45-eb8f-406d-92bc-bbf9ba83c1de","Type":"ContainerStarted","Data":"a76ffc07a1529a58f362d08ea70f541afdb2a4cde87da0f3b2cd63075a9d78dc"}
Sep 16 19:52:35.032827 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:35.032751    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8" podStartSLOduration=6.032730076 podStartE2EDuration="6.032730076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:52:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:52:35.030557522 +0000 UTC m=+4158.672273186" watchObservedRunningTime="2026-09-16 19:52:35.032730076 +0000 UTC m=+4158.674445732"
Sep 16 19:52:35.773562 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:35.773523    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8"
Sep 16 19:52:43.774332 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:43.774287    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8"
Sep 16 19:52:43.775797 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:43.775767    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8" podUID="79f47a45-eb8f-406d-92bc-bbf9ba83c1de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Sep 16 19:52:44.015888 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:44.015850    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv"]
Sep 16 19:52:44.019193 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:44.019173    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv"
Sep 16 19:52:44.022583 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:44.022554    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 16 19:52:44.022749 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:44.022637    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-cm5jg\""
Sep 16 19:52:44.022749 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:44.022678    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 16 19:52:44.022749 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:44.022684    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:52:44.023153 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:44.023134    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 16 19:52:44.023510 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:44.023494    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 16 19:52:44.043062 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:44.043027    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv"]
Sep 16 19:52:44.168501 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:44.168462    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f93c6a8-ab3d-4d59-a395-c050eb07c511-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv\" (UID: \"5f93c6a8-ab3d-4d59-a395-c050eb07c511\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv"
Sep 16 19:52:44.168681 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:44.168512    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f93c6a8-ab3d-4d59-a395-c050eb07c511-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv\" (UID: \"5f93c6a8-ab3d-4d59-a395-c050eb07c511\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv"
Sep 16 19:52:44.168681 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:44.168560    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f93c6a8-ab3d-4d59-a395-c050eb07c511-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv\" (UID: \"5f93c6a8-ab3d-4d59-a395-c050eb07c511\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv"
Sep 16 19:52:44.168681 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:44.168663    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f93c6a8-ab3d-4d59-a395-c050eb07c511-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv\" (UID: \"5f93c6a8-ab3d-4d59-a395-c050eb07c511\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv"
Sep 16 19:52:44.168810 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:44.168698    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrcwv\" (UniqueName: \"kubernetes.io/projected/5f93c6a8-ab3d-4d59-a395-c050eb07c511-kube-api-access-hrcwv\") pod \"custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv\" (UID: \"5f93c6a8-ab3d-4d59-a395-c050eb07c511\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv"
Sep 16 19:52:44.168810 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:44.168738    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f93c6a8-ab3d-4d59-a395-c050eb07c511-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv\" (UID: \"5f93c6a8-ab3d-4d59-a395-c050eb07c511\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv"
Sep 16 19:52:44.168877 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:44.168815    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f93c6a8-ab3d-4d59-a395-c050eb07c511-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv\" (UID: \"5f93c6a8-ab3d-4d59-a395-c050eb07c511\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv"
Sep 16 19:52:44.168877 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:44.168864    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f93c6a8-ab3d-4d59-a395-c050eb07c511-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv\" (UID: \"5f93c6a8-ab3d-4d59-a395-c050eb07c511\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv"
Sep 16 19:52:44.269571 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:44.269533    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f93c6a8-ab3d-4d59-a395-c050eb07c511-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv\" (UID: \"5f93c6a8-ab3d-4d59-a395-c050eb07c511\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv"
Sep 16 19:52:44.269571 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:44.269572    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrcwv\" (UniqueName: \"kubernetes.io/projected/5f93c6a8-ab3d-4d59-a395-c050eb07c511-kube-api-access-hrcwv\") pod \"custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv\" (UID: \"5f93c6a8-ab3d-4d59-a395-c050eb07c511\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv"
Sep 16 19:52:44.269863 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:44.269593    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f93c6a8-ab3d-4d59-a395-c050eb07c511-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv\" (UID: \"5f93c6a8-ab3d-4d59-a395-c050eb07c511\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv"
Sep 16 19:52:44.269863 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:44.269620    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f93c6a8-ab3d-4d59-a395-c050eb07c511-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv\" (UID: \"5f93c6a8-ab3d-4d59-a395-c050eb07c511\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv"
Sep 16 19:52:44.269863 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:44.269665    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f93c6a8-ab3d-4d59-a395-c050eb07c511-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv\" (UID: \"5f93c6a8-ab3d-4d59-a395-c050eb07c511\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv"
Sep 16 19:52:44.269863 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:44.269742    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f93c6a8-ab3d-4d59-a395-c050eb07c511-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv\" (UID: \"5f93c6a8-ab3d-4d59-a395-c050eb07c511\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv"
Sep 16 19:52:44.269863 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:44.269769    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f93c6a8-ab3d-4d59-a395-c050eb07c511-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv\" (UID: \"5f93c6a8-ab3d-4d59-a395-c050eb07c511\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv"
Sep 16 19:52:44.269863 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:44.269792    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f93c6a8-ab3d-4d59-a395-c050eb07c511-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv\" (UID: \"5f93c6a8-ab3d-4d59-a395-c050eb07c511\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv"
Sep 16 19:52:44.270207 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:44.270038    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f93c6a8-ab3d-4d59-a395-c050eb07c511-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv\" (UID: \"5f93c6a8-ab3d-4d59-a395-c050eb07c511\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv"
Sep 16 19:52:44.270207 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:44.270130    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f93c6a8-ab3d-4d59-a395-c050eb07c511-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv\" (UID: \"5f93c6a8-ab3d-4d59-a395-c050eb07c511\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv"
Sep 16 19:52:44.270336 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:44.270206    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f93c6a8-ab3d-4d59-a395-c050eb07c511-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv\" (UID: \"5f93c6a8-ab3d-4d59-a395-c050eb07c511\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv"
Sep 16 19:52:44.270415 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:44.270392    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f93c6a8-ab3d-4d59-a395-c050eb07c511-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv\" (UID: \"5f93c6a8-ab3d-4d59-a395-c050eb07c511\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv"
Sep 16 19:52:44.270494 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:44.270475    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f93c6a8-ab3d-4d59-a395-c050eb07c511-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv\" (UID: \"5f93c6a8-ab3d-4d59-a395-c050eb07c511\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv"
Sep 16 19:52:44.272201 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:44.272177    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f93c6a8-ab3d-4d59-a395-c050eb07c511-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv\" (UID: \"5f93c6a8-ab3d-4d59-a395-c050eb07c511\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv"
Sep 16 19:52:44.272358 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:44.272339    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f93c6a8-ab3d-4d59-a395-c050eb07c511-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv\" (UID: \"5f93c6a8-ab3d-4d59-a395-c050eb07c511\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv"
Sep 16 19:52:44.280272 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:44.280218    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrcwv\" (UniqueName: \"kubernetes.io/projected/5f93c6a8-ab3d-4d59-a395-c050eb07c511-kube-api-access-hrcwv\") pod \"custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv\" (UID: \"5f93c6a8-ab3d-4d59-a395-c050eb07c511\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv"
Sep 16 19:52:44.330824 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:44.330784    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv"
Sep 16 19:52:44.615622 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:44.615589    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv"]
Sep 16 19:52:44.617643 ip-10-0-129-194 kubenswrapper[2582]: W0916 19:52:44.617614    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f93c6a8_ab3d_4d59_a395_c050eb07c511.slice/crio-bbd74f49c308ef1184c88c5551cb52e3a80c089ff0eeff31c13344b0556fdaa0 WatchSource:0}: Error finding container bbd74f49c308ef1184c88c5551cb52e3a80c089ff0eeff31c13344b0556fdaa0: Status 404 returned error can't find the container with id bbd74f49c308ef1184c88c5551cb52e3a80c089ff0eeff31c13344b0556fdaa0
Sep 16 19:52:45.061868 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:45.061830    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv" event={"ID":"5f93c6a8-ab3d-4d59-a395-c050eb07c511","Type":"ContainerStarted","Data":"7bbf3f32f7582999f22481b3c9799c6d8d3986f68b86208855fa5bab16c93d02"}
Sep 16 19:52:45.062293 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:45.061891    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv" event={"ID":"5f93c6a8-ab3d-4d59-a395-c050eb07c511","Type":"ContainerStarted","Data":"bbd74f49c308ef1184c88c5551cb52e3a80c089ff0eeff31c13344b0556fdaa0"}
Sep 16 19:52:47.073393 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:47.073360    2582 generic.go:356] "Generic (PLEG): container finished" podID="5f93c6a8-ab3d-4d59-a395-c050eb07c511" containerID="7bbf3f32f7582999f22481b3c9799c6d8d3986f68b86208855fa5bab16c93d02" exitCode=0
Sep 16 19:52:47.073816 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:47.073435    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv" event={"ID":"5f93c6a8-ab3d-4d59-a395-c050eb07c511","Type":"ContainerDied","Data":"7bbf3f32f7582999f22481b3c9799c6d8d3986f68b86208855fa5bab16c93d02"}
Sep 16 19:52:48.080654 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:48.080611    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv" event={"ID":"5f93c6a8-ab3d-4d59-a395-c050eb07c511","Type":"ContainerStarted","Data":"8f15382b2b94232845d068f3ca7db0e6a3705a156a7f099e9ce8298d5c13f5f4"}
Sep 16 19:52:48.110900 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:48.110839    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv" podStartSLOduration=2.775257861 podStartE2EDuration="5.110818607s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.335560746s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:52:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:52:48.108404809 +0000 UTC m=+4171.750120496" watchObservedRunningTime="2026-09-16 19:52:48.110818607 +0000 UTC m=+4171.752534273"
Sep 16 19:52:53.774148 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:53.774092    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8" podUID="79f47a45-eb8f-406d-92bc-bbf9ba83c1de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Sep 16 19:52:54.331588 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:54.331534    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv"
Sep 16 19:52:54.331822 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:54.331622    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv"
Sep 16 19:52:54.333318 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:52:54.333279    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv" podUID="5f93c6a8-ab3d-4d59-a395-c050eb07c511" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Sep 16 19:53:03.774366 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:53:03.774305    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8" podUID="79f47a45-eb8f-406d-92bc-bbf9ba83c1de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Sep 16 19:53:04.332199 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:53:04.332148    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv" podUID="5f93c6a8-ab3d-4d59-a395-c050eb07c511" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Sep 16 19:53:13.774758 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:53:13.774689    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8" podUID="79f47a45-eb8f-406d-92bc-bbf9ba83c1de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Sep 16 19:53:14.331955 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:53:14.331914    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv" podUID="5f93c6a8-ab3d-4d59-a395-c050eb07c511" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Sep 16 19:53:17.154435 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:53:17.154401    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7qwl2_b6695f86-08c0-4836-87b1-f2a9d76084bd/console-operator/1.log"
Sep 16 19:53:17.162586 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:53:17.162557    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7qwl2_b6695f86-08c0-4836-87b1-f2a9d76084bd/console-operator/1.log"
Sep 16 19:53:23.773779 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:53:23.773700    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8" podUID="79f47a45-eb8f-406d-92bc-bbf9ba83c1de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Sep 16 19:53:24.331778 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:53:24.331710    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv" podUID="5f93c6a8-ab3d-4d59-a395-c050eb07c511" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Sep 16 19:53:33.774777 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:53:33.774689    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8" podUID="79f47a45-eb8f-406d-92bc-bbf9ba83c1de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Sep 16 19:53:34.331932 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:53:34.331885    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv" podUID="5f93c6a8-ab3d-4d59-a395-c050eb07c511" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Sep 16 19:53:43.774441 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:53:43.774329    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8" podUID="79f47a45-eb8f-406d-92bc-bbf9ba83c1de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Sep 16 19:53:44.331391 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:53:44.331342    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv" podUID="5f93c6a8-ab3d-4d59-a395-c050eb07c511" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Sep 16 19:53:53.784535 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:53:53.784491    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8"
Sep 16 19:53:53.793273 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:53:53.793240    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8"
Sep 16 19:53:54.331800 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:53:54.331712    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv" podUID="5f93c6a8-ab3d-4d59-a395-c050eb07c511" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Sep 16 19:54:04.331833 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:54:04.331788    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv" podUID="5f93c6a8-ab3d-4d59-a395-c050eb07c511" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Sep 16 19:54:14.331927 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:54:14.331879    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv" podUID="5f93c6a8-ab3d-4d59-a395-c050eb07c511" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Sep 16 19:54:24.331216 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:54:24.331166    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv" podUID="5f93c6a8-ab3d-4d59-a395-c050eb07c511" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Sep 16 19:54:34.331696 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:54:34.331641    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv" podUID="5f93c6a8-ab3d-4d59-a395-c050eb07c511" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Sep 16 19:54:42.649604 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:54:42.649556    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-a290d4b0_lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8_79f47a45-eb8f-406d-92bc-bbf9ba83c1de/main/0.log"
Sep 16 19:54:43.525332 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:54:43.525294    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8"]
Sep 16 19:54:44.332240 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:54:44.332189    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv" podUID="5f93c6a8-ab3d-4d59-a395-c050eb07c511" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Sep 16 19:54:54.331619 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:54:54.331565    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv" podUID="5f93c6a8-ab3d-4d59-a395-c050eb07c511" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Sep 16 19:54:58.548557 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:54:58.548492    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8" podUID="79f47a45-eb8f-406d-92bc-bbf9ba83c1de" containerName="main" containerID="cri-o://36e6243e48ff93a8d0eb9b538b09f827768f068b10d5ce8c16c299fa0a19755b" gracePeriod=45
Sep 16 19:54:58.785680 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:54:58.785632    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8" podUID="79f47a45-eb8f-406d-92bc-bbf9ba83c1de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Sep 16 19:54:59.517807 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:54:59.517778    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8"
Sep 16 19:54:59.591598 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:54:59.591494    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79f47a45-eb8f-406d-92bc-bbf9ba83c1de-model-cache\") pod \"79f47a45-eb8f-406d-92bc-bbf9ba83c1de\" (UID: \"79f47a45-eb8f-406d-92bc-bbf9ba83c1de\") "
Sep 16 19:54:59.591598 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:54:59.591555    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fnczz\" (UniqueName: \"kubernetes.io/projected/79f47a45-eb8f-406d-92bc-bbf9ba83c1de-kube-api-access-fnczz\") pod \"79f47a45-eb8f-406d-92bc-bbf9ba83c1de\" (UID: \"79f47a45-eb8f-406d-92bc-bbf9ba83c1de\") "
Sep 16 19:54:59.591598 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:54:59.591579    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79f47a45-eb8f-406d-92bc-bbf9ba83c1de-tmp-dir\") pod \"79f47a45-eb8f-406d-92bc-bbf9ba83c1de\" (UID: \"79f47a45-eb8f-406d-92bc-bbf9ba83c1de\") "
Sep 16 19:54:59.592199 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:54:59.591612    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79f47a45-eb8f-406d-92bc-bbf9ba83c1de-dshm\") pod \"79f47a45-eb8f-406d-92bc-bbf9ba83c1de\" (UID: \"79f47a45-eb8f-406d-92bc-bbf9ba83c1de\") "
Sep 16 19:54:59.592199 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:54:59.591634    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79f47a45-eb8f-406d-92bc-bbf9ba83c1de-home\") pod \"79f47a45-eb8f-406d-92bc-bbf9ba83c1de\" (UID: \"79f47a45-eb8f-406d-92bc-bbf9ba83c1de\") "
Sep 16 19:54:59.592199 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:54:59.591759    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0eb6e5be7fc1f4c94\") pod \"79f47a45-eb8f-406d-92bc-bbf9ba83c1de\" (UID: \"79f47a45-eb8f-406d-92bc-bbf9ba83c1de\") "
Sep 16 19:54:59.592199 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:54:59.591794    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79f47a45-eb8f-406d-92bc-bbf9ba83c1de-tls-certs\") pod \"79f47a45-eb8f-406d-92bc-bbf9ba83c1de\" (UID: \"79f47a45-eb8f-406d-92bc-bbf9ba83c1de\") "
Sep 16 19:54:59.592199 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:54:59.591876    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79f47a45-eb8f-406d-92bc-bbf9ba83c1de-model-cache" (OuterVolumeSpecName: "model-cache") pod "79f47a45-eb8f-406d-92bc-bbf9ba83c1de" (UID: "79f47a45-eb8f-406d-92bc-bbf9ba83c1de"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:54:59.592199 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:54:59.592074    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79f47a45-eb8f-406d-92bc-bbf9ba83c1de-model-cache\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:54:59.592199 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:54:59.592072    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79f47a45-eb8f-406d-92bc-bbf9ba83c1de-home" (OuterVolumeSpecName: "home") pod "79f47a45-eb8f-406d-92bc-bbf9ba83c1de" (UID: "79f47a45-eb8f-406d-92bc-bbf9ba83c1de"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:54:59.594195 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:54:59.594167    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79f47a45-eb8f-406d-92bc-bbf9ba83c1de-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "79f47a45-eb8f-406d-92bc-bbf9ba83c1de" (UID: "79f47a45-eb8f-406d-92bc-bbf9ba83c1de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:54:59.594337 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:54:59.594297    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79f47a45-eb8f-406d-92bc-bbf9ba83c1de-dshm" (OuterVolumeSpecName: "dshm") pod "79f47a45-eb8f-406d-92bc-bbf9ba83c1de" (UID: "79f47a45-eb8f-406d-92bc-bbf9ba83c1de"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:54:59.594337 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:54:59.594312    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79f47a45-eb8f-406d-92bc-bbf9ba83c1de-kube-api-access-fnczz" (OuterVolumeSpecName: "kube-api-access-fnczz") pod "79f47a45-eb8f-406d-92bc-bbf9ba83c1de" (UID: "79f47a45-eb8f-406d-92bc-bbf9ba83c1de"). InnerVolumeSpecName "kube-api-access-fnczz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:54:59.599912 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:54:59.599881    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0eb6e5be7fc1f4c94" (OuterVolumeSpecName: "kserve-pvc-source") pod "79f47a45-eb8f-406d-92bc-bbf9ba83c1de" (UID: "79f47a45-eb8f-406d-92bc-bbf9ba83c1de"). InnerVolumeSpecName "pvc-0aedfc2c-223d-4c6c-b480-93cbde3a0e1a". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 16 19:54:59.610262 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:54:59.610191    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79f47a45-eb8f-406d-92bc-bbf9ba83c1de-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "79f47a45-eb8f-406d-92bc-bbf9ba83c1de" (UID: "79f47a45-eb8f-406d-92bc-bbf9ba83c1de"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:54:59.693541 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:54:59.693491    2582 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-0aedfc2c-223d-4c6c-b480-93cbde3a0e1a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0eb6e5be7fc1f4c94\") on node \"ip-10-0-129-194.ec2.internal\" "
Sep 16 19:54:59.693705 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:54:59.693544    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79f47a45-eb8f-406d-92bc-bbf9ba83c1de-tls-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:54:59.693705 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:54:59.693565    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnczz\" (UniqueName: \"kubernetes.io/projected/79f47a45-eb8f-406d-92bc-bbf9ba83c1de-kube-api-access-fnczz\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:54:59.693705 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:54:59.693578    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79f47a45-eb8f-406d-92bc-bbf9ba83c1de-tmp-dir\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:54:59.693705 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:54:59.693595    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79f47a45-eb8f-406d-92bc-bbf9ba83c1de-dshm\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:54:59.693705 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:54:59.693608    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79f47a45-eb8f-406d-92bc-bbf9ba83c1de-home\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:54:59.720950 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:54:59.720925    2582 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-0aedfc2c-223d-4c6c-b480-93cbde3a0e1a" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0eb6e5be7fc1f4c94") on node "ip-10-0-129-194.ec2.internal"
Sep 16 19:54:59.744343 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:54:59.744313    2582 generic.go:356] "Generic (PLEG): container finished" podID="79f47a45-eb8f-406d-92bc-bbf9ba83c1de" containerID="36e6243e48ff93a8d0eb9b538b09f827768f068b10d5ce8c16c299fa0a19755b" exitCode=0
Sep 16 19:54:59.744517 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:54:59.744369    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8" event={"ID":"79f47a45-eb8f-406d-92bc-bbf9ba83c1de","Type":"ContainerDied","Data":"36e6243e48ff93a8d0eb9b538b09f827768f068b10d5ce8c16c299fa0a19755b"}
Sep 16 19:54:59.744517 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:54:59.744388    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8"
Sep 16 19:54:59.744517 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:54:59.744411    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8" event={"ID":"79f47a45-eb8f-406d-92bc-bbf9ba83c1de","Type":"ContainerDied","Data":"a76ffc07a1529a58f362d08ea70f541afdb2a4cde87da0f3b2cd63075a9d78dc"}
Sep 16 19:54:59.744517 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:54:59.744432    2582 scope.go:117] "RemoveContainer" containerID="36e6243e48ff93a8d0eb9b538b09f827768f068b10d5ce8c16c299fa0a19755b"
Sep 16 19:54:59.755459 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:54:59.755429    2582 scope.go:117] "RemoveContainer" containerID="36e6243e48ff93a8d0eb9b538b09f827768f068b10d5ce8c16c299fa0a19755b"
Sep 16 19:54:59.756042 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:54:59.756015    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36e6243e48ff93a8d0eb9b538b09f827768f068b10d5ce8c16c299fa0a19755b\": container with ID starting with 36e6243e48ff93a8d0eb9b538b09f827768f068b10d5ce8c16c299fa0a19755b not found: ID does not exist" containerID="36e6243e48ff93a8d0eb9b538b09f827768f068b10d5ce8c16c299fa0a19755b"
Sep 16 19:54:59.756130 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:54:59.756050    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36e6243e48ff93a8d0eb9b538b09f827768f068b10d5ce8c16c299fa0a19755b"} err="failed to get container status \"36e6243e48ff93a8d0eb9b538b09f827768f068b10d5ce8c16c299fa0a19755b\": rpc error: code = NotFound desc = could not find container \"36e6243e48ff93a8d0eb9b538b09f827768f068b10d5ce8c16c299fa0a19755b\": container with ID starting with 36e6243e48ff93a8d0eb9b538b09f827768f068b10d5ce8c16c299fa0a19755b not found: ID does not exist"
Sep 16 19:54:59.773520 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:54:59.773486    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8"]
Sep 16 19:54:59.779165 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:54:59.779133    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-q2wm8"]
Sep 16 19:54:59.794698 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:54:59.794665    2582 reconciler_common.go:299] "Volume detached for volume \"pvc-0aedfc2c-223d-4c6c-b480-93cbde3a0e1a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0eb6e5be7fc1f4c94\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"csi-6e487c693fba6bbfd1e8fab9c676a3a68b7da96812722c27e22a98f4af3add8a\""
Sep 16 19:55:00.871919 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:00.871883    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79f47a45-eb8f-406d-92bc-bbf9ba83c1de" path="/var/lib/kubelet/pods/79f47a45-eb8f-406d-92bc-bbf9ba83c1de/volumes"
Sep 16 19:55:04.331847 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:04.331783    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv" podUID="5f93c6a8-ab3d-4d59-a395-c050eb07c511" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Sep 16 19:55:13.310701 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:13.310621    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5df597f6b8-2bscj"]
Sep 16 19:55:13.311081 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:13.310985    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79f47a45-eb8f-406d-92bc-bbf9ba83c1de" containerName="main"
Sep 16 19:55:13.311081 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:13.310999    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="79f47a45-eb8f-406d-92bc-bbf9ba83c1de" containerName="main"
Sep 16 19:55:13.311081 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:13.311062    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="79f47a45-eb8f-406d-92bc-bbf9ba83c1de" containerName="main"
Sep 16 19:55:13.314197 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:13.314179    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5df597f6b8-2bscj"
Sep 16 19:55:13.316829 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:13.316798    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Sep 16 19:55:13.317016 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:13.316994    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Sep 16 19:55:13.317107 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:13.316993    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"seaweedfs-s3-creds\""
Sep 16 19:55:13.317169 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:13.317122    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Sep 16 19:55:13.318035 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:13.318014    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:55:13.318636 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:13.318612    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-qtj6f\""
Sep 16 19:55:13.327575 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:13.327552    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5df597f6b8-2bscj"]
Sep 16 19:55:13.421454 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:13.421411    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8675a010-fb9e-4c2a-ae3f-11ad391ddb6c-tmp-dir\") pod \"tls-verification-test-kserve-5df597f6b8-2bscj\" (UID: \"8675a010-fb9e-4c2a-ae3f-11ad391ddb6c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5df597f6b8-2bscj"
Sep 16 19:55:13.421637 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:13.421462    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8675a010-fb9e-4c2a-ae3f-11ad391ddb6c-model-cache\") pod \"tls-verification-test-kserve-5df597f6b8-2bscj\" (UID: \"8675a010-fb9e-4c2a-ae3f-11ad391ddb6c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5df597f6b8-2bscj"
Sep 16 19:55:13.421637 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:13.421482    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8675a010-fb9e-4c2a-ae3f-11ad391ddb6c-tls-certs\") pod \"tls-verification-test-kserve-5df597f6b8-2bscj\" (UID: \"8675a010-fb9e-4c2a-ae3f-11ad391ddb6c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5df597f6b8-2bscj"
Sep 16 19:55:13.421637 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:13.421500    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxwkk\" (UniqueName: \"kubernetes.io/projected/8675a010-fb9e-4c2a-ae3f-11ad391ddb6c-kube-api-access-wxwkk\") pod \"tls-verification-test-kserve-5df597f6b8-2bscj\" (UID: \"8675a010-fb9e-4c2a-ae3f-11ad391ddb6c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5df597f6b8-2bscj"
Sep 16 19:55:13.421637 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:13.421531    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8675a010-fb9e-4c2a-ae3f-11ad391ddb6c-cabundle-cert\") pod \"tls-verification-test-kserve-5df597f6b8-2bscj\" (UID: \"8675a010-fb9e-4c2a-ae3f-11ad391ddb6c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5df597f6b8-2bscj"
Sep 16 19:55:13.421637 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:13.421625    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8675a010-fb9e-4c2a-ae3f-11ad391ddb6c-dshm\") pod \"tls-verification-test-kserve-5df597f6b8-2bscj\" (UID: \"8675a010-fb9e-4c2a-ae3f-11ad391ddb6c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5df597f6b8-2bscj"
Sep 16 19:55:13.421888 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:13.421666    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8675a010-fb9e-4c2a-ae3f-11ad391ddb6c-home\") pod \"tls-verification-test-kserve-5df597f6b8-2bscj\" (UID: \"8675a010-fb9e-4c2a-ae3f-11ad391ddb6c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5df597f6b8-2bscj"
Sep 16 19:55:13.421888 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:13.421685    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8675a010-fb9e-4c2a-ae3f-11ad391ddb6c-kserve-provision-location\") pod \"tls-verification-test-kserve-5df597f6b8-2bscj\" (UID: \"8675a010-fb9e-4c2a-ae3f-11ad391ddb6c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5df597f6b8-2bscj"
Sep 16 19:55:13.522785 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:13.522715    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8675a010-fb9e-4c2a-ae3f-11ad391ddb6c-dshm\") pod \"tls-verification-test-kserve-5df597f6b8-2bscj\" (UID: \"8675a010-fb9e-4c2a-ae3f-11ad391ddb6c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5df597f6b8-2bscj"
Sep 16 19:55:13.522968 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:13.522811    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8675a010-fb9e-4c2a-ae3f-11ad391ddb6c-home\") pod \"tls-verification-test-kserve-5df597f6b8-2bscj\" (UID: \"8675a010-fb9e-4c2a-ae3f-11ad391ddb6c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5df597f6b8-2bscj"
Sep 16 19:55:13.522968 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:13.522830    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8675a010-fb9e-4c2a-ae3f-11ad391ddb6c-kserve-provision-location\") pod \"tls-verification-test-kserve-5df597f6b8-2bscj\" (UID: \"8675a010-fb9e-4c2a-ae3f-11ad391ddb6c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5df597f6b8-2bscj"
Sep 16 19:55:13.522968 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:13.522869    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8675a010-fb9e-4c2a-ae3f-11ad391ddb6c-tmp-dir\") pod \"tls-verification-test-kserve-5df597f6b8-2bscj\" (UID: \"8675a010-fb9e-4c2a-ae3f-11ad391ddb6c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5df597f6b8-2bscj"
Sep 16 19:55:13.523200 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:13.523009    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8675a010-fb9e-4c2a-ae3f-11ad391ddb6c-model-cache\") pod \"tls-verification-test-kserve-5df597f6b8-2bscj\" (UID: \"8675a010-fb9e-4c2a-ae3f-11ad391ddb6c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5df597f6b8-2bscj"
Sep 16 19:55:13.523200 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:13.523057    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8675a010-fb9e-4c2a-ae3f-11ad391ddb6c-tls-certs\") pod \"tls-verification-test-kserve-5df597f6b8-2bscj\" (UID: \"8675a010-fb9e-4c2a-ae3f-11ad391ddb6c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5df597f6b8-2bscj"
Sep 16 19:55:13.523200 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:13.523088    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxwkk\" (UniqueName: \"kubernetes.io/projected/8675a010-fb9e-4c2a-ae3f-11ad391ddb6c-kube-api-access-wxwkk\") pod \"tls-verification-test-kserve-5df597f6b8-2bscj\" (UID: \"8675a010-fb9e-4c2a-ae3f-11ad391ddb6c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5df597f6b8-2bscj"
Sep 16 19:55:13.523200 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:13.523130    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8675a010-fb9e-4c2a-ae3f-11ad391ddb6c-cabundle-cert\") pod \"tls-verification-test-kserve-5df597f6b8-2bscj\" (UID: \"8675a010-fb9e-4c2a-ae3f-11ad391ddb6c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5df597f6b8-2bscj"
Sep 16 19:55:13.523413 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:13.523212    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8675a010-fb9e-4c2a-ae3f-11ad391ddb6c-tmp-dir\") pod \"tls-verification-test-kserve-5df597f6b8-2bscj\" (UID: \"8675a010-fb9e-4c2a-ae3f-11ad391ddb6c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5df597f6b8-2bscj"
Sep 16 19:55:13.523413 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:13.523261    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8675a010-fb9e-4c2a-ae3f-11ad391ddb6c-kserve-provision-location\") pod \"tls-verification-test-kserve-5df597f6b8-2bscj\" (UID: \"8675a010-fb9e-4c2a-ae3f-11ad391ddb6c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5df597f6b8-2bscj"
Sep 16 19:55:13.523413 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:13.523268    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8675a010-fb9e-4c2a-ae3f-11ad391ddb6c-home\") pod \"tls-verification-test-kserve-5df597f6b8-2bscj\" (UID: \"8675a010-fb9e-4c2a-ae3f-11ad391ddb6c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5df597f6b8-2bscj"
Sep 16 19:55:13.523413 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:13.523318    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8675a010-fb9e-4c2a-ae3f-11ad391ddb6c-model-cache\") pod \"tls-verification-test-kserve-5df597f6b8-2bscj\" (UID: \"8675a010-fb9e-4c2a-ae3f-11ad391ddb6c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5df597f6b8-2bscj"
Sep 16 19:55:13.523916 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:13.523895    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8675a010-fb9e-4c2a-ae3f-11ad391ddb6c-cabundle-cert\") pod \"tls-verification-test-kserve-5df597f6b8-2bscj\" (UID: \"8675a010-fb9e-4c2a-ae3f-11ad391ddb6c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5df597f6b8-2bscj"
Sep 16 19:55:13.525177 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:13.525159    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8675a010-fb9e-4c2a-ae3f-11ad391ddb6c-dshm\") pod \"tls-verification-test-kserve-5df597f6b8-2bscj\" (UID: \"8675a010-fb9e-4c2a-ae3f-11ad391ddb6c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5df597f6b8-2bscj"
Sep 16 19:55:13.525630 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:13.525612    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8675a010-fb9e-4c2a-ae3f-11ad391ddb6c-tls-certs\") pod \"tls-verification-test-kserve-5df597f6b8-2bscj\" (UID: \"8675a010-fb9e-4c2a-ae3f-11ad391ddb6c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5df597f6b8-2bscj"
Sep 16 19:55:13.532379 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:13.532359    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxwkk\" (UniqueName: \"kubernetes.io/projected/8675a010-fb9e-4c2a-ae3f-11ad391ddb6c-kube-api-access-wxwkk\") pod \"tls-verification-test-kserve-5df597f6b8-2bscj\" (UID: \"8675a010-fb9e-4c2a-ae3f-11ad391ddb6c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5df597f6b8-2bscj"
Sep 16 19:55:13.624455 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:13.624365    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5df597f6b8-2bscj"
Sep 16 19:55:13.906456 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:13.906417    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5df597f6b8-2bscj"]
Sep 16 19:55:13.909427 ip-10-0-129-194 kubenswrapper[2582]: W0916 19:55:13.909395    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8675a010_fb9e_4c2a_ae3f_11ad391ddb6c.slice/crio-2c6381b83ea5bebac5db3989b7a2550707ce47f1ef4021981b31580a0f9db2d6 WatchSource:0}: Error finding container 2c6381b83ea5bebac5db3989b7a2550707ce47f1ef4021981b31580a0f9db2d6: Status 404 returned error can't find the container with id 2c6381b83ea5bebac5db3989b7a2550707ce47f1ef4021981b31580a0f9db2d6
Sep 16 19:55:14.341729 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:14.341682    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv"
Sep 16 19:55:14.350543 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:14.350508    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv"
Sep 16 19:55:14.823012 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:14.822970    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5df597f6b8-2bscj" event={"ID":"8675a010-fb9e-4c2a-ae3f-11ad391ddb6c","Type":"ContainerStarted","Data":"21076671560b683cfb23866c854dc7c740b1508ccaa3a2309703c74c99d65789"}
Sep 16 19:55:14.823174 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:14.823065    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5df597f6b8-2bscj" event={"ID":"8675a010-fb9e-4c2a-ae3f-11ad391ddb6c","Type":"ContainerStarted","Data":"2c6381b83ea5bebac5db3989b7a2550707ce47f1ef4021981b31580a0f9db2d6"}
Sep 16 19:55:16.837444 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:16.837409    2582 generic.go:356] "Generic (PLEG): container finished" podID="8675a010-fb9e-4c2a-ae3f-11ad391ddb6c" containerID="21076671560b683cfb23866c854dc7c740b1508ccaa3a2309703c74c99d65789" exitCode=0
Sep 16 19:55:16.837869 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:16.837478    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5df597f6b8-2bscj" event={"ID":"8675a010-fb9e-4c2a-ae3f-11ad391ddb6c","Type":"ContainerDied","Data":"21076671560b683cfb23866c854dc7c740b1508ccaa3a2309703c74c99d65789"}
Sep 16 19:55:17.844284 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:17.844248    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5df597f6b8-2bscj" event={"ID":"8675a010-fb9e-4c2a-ae3f-11ad391ddb6c","Type":"ContainerStarted","Data":"0b50c038e9081a142e87e9e0da5ddff748366340f91145c382abdc90f4417b90"}
Sep 16 19:55:17.869464 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:17.869401    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5df597f6b8-2bscj" podStartSLOduration=2.552518836 podStartE2EDuration="4.869379349s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.316860513s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:55:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:55:17.867876009 +0000 UTC m=+4321.509591685" watchObservedRunningTime="2026-09-16 19:55:17.869379349 +0000 UTC m=+4321.511095014"
Sep 16 19:55:18.625531 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:18.625492    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5df597f6b8-2bscj"
Sep 16 19:55:23.625287 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:23.625243    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5df597f6b8-2bscj"
Sep 16 19:55:23.626903 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:23.626871    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5df597f6b8-2bscj" podUID="8675a010-fb9e-4c2a-ae3f-11ad391ddb6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Sep 16 19:55:32.006127 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:32.006079    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv"]
Sep 16 19:55:33.625838 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:33.625790    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5df597f6b8-2bscj" podUID="8675a010-fb9e-4c2a-ae3f-11ad391ddb6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Sep 16 19:55:43.625107 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:43.625057    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5df597f6b8-2bscj" podUID="8675a010-fb9e-4c2a-ae3f-11ad391ddb6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Sep 16 19:55:47.030642 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:47.030593    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv" podUID="5f93c6a8-ab3d-4d59-a395-c050eb07c511" containerName="main" containerID="cri-o://8f15382b2b94232845d068f3ca7db0e6a3705a156a7f099e9ce8298d5c13f5f4" gracePeriod=45
Sep 16 19:55:47.780456 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:47.780424    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv"
Sep 16 19:55:47.924881 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:47.924757    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f93c6a8-ab3d-4d59-a395-c050eb07c511-tls-certs\") pod \"5f93c6a8-ab3d-4d59-a395-c050eb07c511\" (UID: \"5f93c6a8-ab3d-4d59-a395-c050eb07c511\") "
Sep 16 19:55:47.924881 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:47.924831    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f93c6a8-ab3d-4d59-a395-c050eb07c511-tmp-dir\") pod \"5f93c6a8-ab3d-4d59-a395-c050eb07c511\" (UID: \"5f93c6a8-ab3d-4d59-a395-c050eb07c511\") "
Sep 16 19:55:47.924881 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:47.924860    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f93c6a8-ab3d-4d59-a395-c050eb07c511-kserve-provision-location\") pod \"5f93c6a8-ab3d-4d59-a395-c050eb07c511\" (UID: \"5f93c6a8-ab3d-4d59-a395-c050eb07c511\") "
Sep 16 19:55:47.924881 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:47.924881    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrcwv\" (UniqueName: \"kubernetes.io/projected/5f93c6a8-ab3d-4d59-a395-c050eb07c511-kube-api-access-hrcwv\") pod \"5f93c6a8-ab3d-4d59-a395-c050eb07c511\" (UID: \"5f93c6a8-ab3d-4d59-a395-c050eb07c511\") "
Sep 16 19:55:47.925243 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:47.924895    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f93c6a8-ab3d-4d59-a395-c050eb07c511-dshm\") pod \"5f93c6a8-ab3d-4d59-a395-c050eb07c511\" (UID: \"5f93c6a8-ab3d-4d59-a395-c050eb07c511\") "
Sep 16 19:55:47.925243 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:47.924916    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f93c6a8-ab3d-4d59-a395-c050eb07c511-model-cache\") pod \"5f93c6a8-ab3d-4d59-a395-c050eb07c511\" (UID: \"5f93c6a8-ab3d-4d59-a395-c050eb07c511\") "
Sep 16 19:55:47.925243 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:47.924950    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f93c6a8-ab3d-4d59-a395-c050eb07c511-home\") pod \"5f93c6a8-ab3d-4d59-a395-c050eb07c511\" (UID: \"5f93c6a8-ab3d-4d59-a395-c050eb07c511\") "
Sep 16 19:55:47.925243 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:47.925031    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f93c6a8-ab3d-4d59-a395-c050eb07c511-cabundle-cert\") pod \"5f93c6a8-ab3d-4d59-a395-c050eb07c511\" (UID: \"5f93c6a8-ab3d-4d59-a395-c050eb07c511\") "
Sep 16 19:55:47.925921 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:47.925880    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f93c6a8-ab3d-4d59-a395-c050eb07c511-model-cache" (OuterVolumeSpecName: "model-cache") pod "5f93c6a8-ab3d-4d59-a395-c050eb07c511" (UID: "5f93c6a8-ab3d-4d59-a395-c050eb07c511"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:55:47.925921 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:47.925903    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f93c6a8-ab3d-4d59-a395-c050eb07c511-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5f93c6a8-ab3d-4d59-a395-c050eb07c511" (UID: "5f93c6a8-ab3d-4d59-a395-c050eb07c511"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:55:47.926308 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:47.926279    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f93c6a8-ab3d-4d59-a395-c050eb07c511-home" (OuterVolumeSpecName: "home") pod "5f93c6a8-ab3d-4d59-a395-c050eb07c511" (UID: "5f93c6a8-ab3d-4d59-a395-c050eb07c511"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:55:47.927869 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:47.927745    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f93c6a8-ab3d-4d59-a395-c050eb07c511-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5f93c6a8-ab3d-4d59-a395-c050eb07c511" (UID: "5f93c6a8-ab3d-4d59-a395-c050eb07c511"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:55:47.927869 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:47.927823    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f93c6a8-ab3d-4d59-a395-c050eb07c511-dshm" (OuterVolumeSpecName: "dshm") pod "5f93c6a8-ab3d-4d59-a395-c050eb07c511" (UID: "5f93c6a8-ab3d-4d59-a395-c050eb07c511"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:55:47.928267 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:47.928240    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f93c6a8-ab3d-4d59-a395-c050eb07c511-kube-api-access-hrcwv" (OuterVolumeSpecName: "kube-api-access-hrcwv") pod "5f93c6a8-ab3d-4d59-a395-c050eb07c511" (UID: "5f93c6a8-ab3d-4d59-a395-c050eb07c511"). InnerVolumeSpecName "kube-api-access-hrcwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:55:47.938623 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:47.938581    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f93c6a8-ab3d-4d59-a395-c050eb07c511-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5f93c6a8-ab3d-4d59-a395-c050eb07c511" (UID: "5f93c6a8-ab3d-4d59-a395-c050eb07c511"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:55:47.949236 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:47.949191    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f93c6a8-ab3d-4d59-a395-c050eb07c511-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5f93c6a8-ab3d-4d59-a395-c050eb07c511" (UID: "5f93c6a8-ab3d-4d59-a395-c050eb07c511"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:55:48.013369 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:48.013328    2582 generic.go:356] "Generic (PLEG): container finished" podID="5f93c6a8-ab3d-4d59-a395-c050eb07c511" containerID="8f15382b2b94232845d068f3ca7db0e6a3705a156a7f099e9ce8298d5c13f5f4" exitCode=0
Sep 16 19:55:48.013560 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:48.013414    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv"
Sep 16 19:55:48.013560 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:48.013417    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv" event={"ID":"5f93c6a8-ab3d-4d59-a395-c050eb07c511","Type":"ContainerDied","Data":"8f15382b2b94232845d068f3ca7db0e6a3705a156a7f099e9ce8298d5c13f5f4"}
Sep 16 19:55:48.013560 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:48.013519    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv" event={"ID":"5f93c6a8-ab3d-4d59-a395-c050eb07c511","Type":"ContainerDied","Data":"bbd74f49c308ef1184c88c5551cb52e3a80c089ff0eeff31c13344b0556fdaa0"}
Sep 16 19:55:48.013560 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:48.013552    2582 scope.go:117] "RemoveContainer" containerID="8f15382b2b94232845d068f3ca7db0e6a3705a156a7f099e9ce8298d5c13f5f4"
Sep 16 19:55:48.024030 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:48.024002    2582 scope.go:117] "RemoveContainer" containerID="7bbf3f32f7582999f22481b3c9799c6d8d3986f68b86208855fa5bab16c93d02"
Sep 16 19:55:48.026696 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:48.026664    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f93c6a8-ab3d-4d59-a395-c050eb07c511-kserve-provision-location\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:55:48.026696 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:48.026696    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrcwv\" (UniqueName: \"kubernetes.io/projected/5f93c6a8-ab3d-4d59-a395-c050eb07c511-kube-api-access-hrcwv\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:55:48.026848 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:48.026708    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f93c6a8-ab3d-4d59-a395-c050eb07c511-dshm\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:55:48.026848 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:48.026757    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f93c6a8-ab3d-4d59-a395-c050eb07c511-model-cache\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:55:48.026848 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:48.026767    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f93c6a8-ab3d-4d59-a395-c050eb07c511-home\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:55:48.026848 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:48.026774    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f93c6a8-ab3d-4d59-a395-c050eb07c511-cabundle-cert\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:55:48.026848 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:48.026782    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f93c6a8-ab3d-4d59-a395-c050eb07c511-tls-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:55:48.026848 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:48.026809    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f93c6a8-ab3d-4d59-a395-c050eb07c511-tmp-dir\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:55:48.035241 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:48.035088    2582 scope.go:117] "RemoveContainer" containerID="8f15382b2b94232845d068f3ca7db0e6a3705a156a7f099e9ce8298d5c13f5f4"
Sep 16 19:55:48.035526 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:55:48.035484    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f15382b2b94232845d068f3ca7db0e6a3705a156a7f099e9ce8298d5c13f5f4\": container with ID starting with 8f15382b2b94232845d068f3ca7db0e6a3705a156a7f099e9ce8298d5c13f5f4 not found: ID does not exist" containerID="8f15382b2b94232845d068f3ca7db0e6a3705a156a7f099e9ce8298d5c13f5f4"
Sep 16 19:55:48.035580 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:48.035517    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f15382b2b94232845d068f3ca7db0e6a3705a156a7f099e9ce8298d5c13f5f4"} err="failed to get container status \"8f15382b2b94232845d068f3ca7db0e6a3705a156a7f099e9ce8298d5c13f5f4\": rpc error: code = NotFound desc = could not find container \"8f15382b2b94232845d068f3ca7db0e6a3705a156a7f099e9ce8298d5c13f5f4\": container with ID starting with 8f15382b2b94232845d068f3ca7db0e6a3705a156a7f099e9ce8298d5c13f5f4 not found: ID does not exist"
Sep 16 19:55:48.035580 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:48.035537    2582 scope.go:117] "RemoveContainer" containerID="7bbf3f32f7582999f22481b3c9799c6d8d3986f68b86208855fa5bab16c93d02"
Sep 16 19:55:48.035892 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:55:48.035869    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bbf3f32f7582999f22481b3c9799c6d8d3986f68b86208855fa5bab16c93d02\": container with ID starting with 7bbf3f32f7582999f22481b3c9799c6d8d3986f68b86208855fa5bab16c93d02 not found: ID does not exist" containerID="7bbf3f32f7582999f22481b3c9799c6d8d3986f68b86208855fa5bab16c93d02"
Sep 16 19:55:48.035976 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:48.035902    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bbf3f32f7582999f22481b3c9799c6d8d3986f68b86208855fa5bab16c93d02"} err="failed to get container status \"7bbf3f32f7582999f22481b3c9799c6d8d3986f68b86208855fa5bab16c93d02\": rpc error: code = NotFound desc = could not find container \"7bbf3f32f7582999f22481b3c9799c6d8d3986f68b86208855fa5bab16c93d02\": container with ID starting with 7bbf3f32f7582999f22481b3c9799c6d8d3986f68b86208855fa5bab16c93d02 not found: ID does not exist"
Sep 16 19:55:48.050009 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:48.049975    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv"]
Sep 16 19:55:48.061149 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:48.061112    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cc4874fd9-nhbxv"]
Sep 16 19:55:48.874104 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:48.874063    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f93c6a8-ab3d-4d59-a395-c050eb07c511" path="/var/lib/kubelet/pods/5f93c6a8-ab3d-4d59-a395-c050eb07c511/volumes"
Sep 16 19:55:53.625696 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:55:53.625645    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5df597f6b8-2bscj" podUID="8675a010-fb9e-4c2a-ae3f-11ad391ddb6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Sep 16 19:56:03.625470 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:56:03.625423    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5df597f6b8-2bscj" podUID="8675a010-fb9e-4c2a-ae3f-11ad391ddb6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Sep 16 19:56:13.625203 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:56:13.625152    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5df597f6b8-2bscj" podUID="8675a010-fb9e-4c2a-ae3f-11ad391ddb6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Sep 16 19:56:23.625299 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:56:23.625246    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5df597f6b8-2bscj" podUID="8675a010-fb9e-4c2a-ae3f-11ad391ddb6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Sep 16 19:56:33.625645 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:56:33.625592    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5df597f6b8-2bscj" podUID="8675a010-fb9e-4c2a-ae3f-11ad391ddb6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Sep 16 19:56:42.703366 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:56:42.703291    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6"]
Sep 16 19:56:42.703811 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:56:42.703603    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f93c6a8-ab3d-4d59-a395-c050eb07c511" containerName="storage-initializer"
Sep 16 19:56:42.703811 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:56:42.703614    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f93c6a8-ab3d-4d59-a395-c050eb07c511" containerName="storage-initializer"
Sep 16 19:56:42.703811 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:56:42.703624    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f93c6a8-ab3d-4d59-a395-c050eb07c511" containerName="main"
Sep 16 19:56:42.703811 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:56:42.703630    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f93c6a8-ab3d-4d59-a395-c050eb07c511" containerName="main"
Sep 16 19:56:42.703811 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:56:42.703685    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f93c6a8-ab3d-4d59-a395-c050eb07c511" containerName="main"
Sep 16 19:56:42.706749 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:56:42.706728    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6"
Sep 16 19:56:42.710178 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:56:42.710157    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 16 19:56:42.710465 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:56:42.710434    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:56:42.711000 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:56:42.710980    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-4m4fc\""
Sep 16 19:56:42.711117 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:56:42.711010    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 16 19:56:42.711117 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:56:42.711034    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 16 19:56:42.714406 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:56:42.714387    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 16 19:56:42.741131 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:56:42.741098    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6"]
Sep 16 19:56:42.778831 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:56:42.778790    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3296120-91c5-4a5e-92d5-ca891652f649-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6\" (UID: \"e3296120-91c5-4a5e-92d5-ca891652f649\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6"
Sep 16 19:56:42.779005 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:56:42.778944    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3296120-91c5-4a5e-92d5-ca891652f649-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6\" (UID: \"e3296120-91c5-4a5e-92d5-ca891652f649\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6"
Sep 16 19:56:42.779005 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:56:42.778994    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3296120-91c5-4a5e-92d5-ca891652f649-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6\" (UID: \"e3296120-91c5-4a5e-92d5-ca891652f649\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6"
Sep 16 19:56:42.779084 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:56:42.779020    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3296120-91c5-4a5e-92d5-ca891652f649-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6\" (UID: \"e3296120-91c5-4a5e-92d5-ca891652f649\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6"
Sep 16 19:56:42.779084 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:56:42.779059    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3296120-91c5-4a5e-92d5-ca891652f649-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6\" (UID: \"e3296120-91c5-4a5e-92d5-ca891652f649\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6"
Sep 16 19:56:42.779164 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:56:42.779146    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cxf7\" (UniqueName: \"kubernetes.io/projected/e3296120-91c5-4a5e-92d5-ca891652f649-kube-api-access-2cxf7\") pod \"router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6\" (UID: \"e3296120-91c5-4a5e-92d5-ca891652f649\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6"
Sep 16 19:56:42.779203 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:56:42.779192    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3296120-91c5-4a5e-92d5-ca891652f649-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6\" (UID: \"e3296120-91c5-4a5e-92d5-ca891652f649\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6"
Sep 16 19:56:42.779238 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:56:42.779221    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3296120-91c5-4a5e-92d5-ca891652f649-home\") pod \"router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6\" (UID: \"e3296120-91c5-4a5e-92d5-ca891652f649\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6"
Sep 16 19:56:42.880188 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:56:42.880146    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3296120-91c5-4a5e-92d5-ca891652f649-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6\" (UID: \"e3296120-91c5-4a5e-92d5-ca891652f649\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6"
Sep 16 19:56:42.880188 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:56:42.880188    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3296120-91c5-4a5e-92d5-ca891652f649-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6\" (UID: \"e3296120-91c5-4a5e-92d5-ca891652f649\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6"
Sep 16 19:56:42.880442 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:56:42.880207    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3296120-91c5-4a5e-92d5-ca891652f649-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6\" (UID: \"e3296120-91c5-4a5e-92d5-ca891652f649\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6"
Sep 16 19:56:42.880442 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:56:42.880229    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3296120-91c5-4a5e-92d5-ca891652f649-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6\" (UID: \"e3296120-91c5-4a5e-92d5-ca891652f649\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6"
Sep 16 19:56:42.880442 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:56:42.880287    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cxf7\" (UniqueName: \"kubernetes.io/projected/e3296120-91c5-4a5e-92d5-ca891652f649-kube-api-access-2cxf7\") pod \"router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6\" (UID: \"e3296120-91c5-4a5e-92d5-ca891652f649\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6"
Sep 16 19:56:42.880442 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:56:42.880317    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3296120-91c5-4a5e-92d5-ca891652f649-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6\" (UID: \"e3296120-91c5-4a5e-92d5-ca891652f649\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6"
Sep 16 19:56:42.880442 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:56:42.880342    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3296120-91c5-4a5e-92d5-ca891652f649-home\") pod \"router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6\" (UID: \"e3296120-91c5-4a5e-92d5-ca891652f649\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6"
Sep 16 19:56:42.880442 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:56:42.880367    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3296120-91c5-4a5e-92d5-ca891652f649-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6\" (UID: \"e3296120-91c5-4a5e-92d5-ca891652f649\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6"
Sep 16 19:56:42.880785 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:56:42.880669    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3296120-91c5-4a5e-92d5-ca891652f649-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6\" (UID: \"e3296120-91c5-4a5e-92d5-ca891652f649\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6"
Sep 16 19:56:42.880785 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:56:42.880696    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3296120-91c5-4a5e-92d5-ca891652f649-home\") pod \"router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6\" (UID: \"e3296120-91c5-4a5e-92d5-ca891652f649\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6"
Sep 16 19:56:42.880894 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:56:42.880820    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3296120-91c5-4a5e-92d5-ca891652f649-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6\" (UID: \"e3296120-91c5-4a5e-92d5-ca891652f649\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6"
Sep 16 19:56:42.880942 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:56:42.880926    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3296120-91c5-4a5e-92d5-ca891652f649-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6\" (UID: \"e3296120-91c5-4a5e-92d5-ca891652f649\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6"
Sep 16 19:56:42.881208 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:56:42.881184    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3296120-91c5-4a5e-92d5-ca891652f649-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6\" (UID: \"e3296120-91c5-4a5e-92d5-ca891652f649\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6"
Sep 16 19:56:42.883170 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:56:42.883144    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3296120-91c5-4a5e-92d5-ca891652f649-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6\" (UID: \"e3296120-91c5-4a5e-92d5-ca891652f649\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6"
Sep 16 19:56:42.883843 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:56:42.883823    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3296120-91c5-4a5e-92d5-ca891652f649-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6\" (UID: \"e3296120-91c5-4a5e-92d5-ca891652f649\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6"
Sep 16 19:56:42.907345 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:56:42.907318    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cxf7\" (UniqueName: \"kubernetes.io/projected/e3296120-91c5-4a5e-92d5-ca891652f649-kube-api-access-2cxf7\") pod \"router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6\" (UID: \"e3296120-91c5-4a5e-92d5-ca891652f649\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6"
Sep 16 19:56:43.017247 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:56:43.017204    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6"
Sep 16 19:56:43.308030 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:56:43.308006    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6"]
Sep 16 19:56:43.309830 ip-10-0-129-194 kubenswrapper[2582]: W0916 19:56:43.309799    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3296120_91c5_4a5e_92d5_ca891652f649.slice/crio-6cd82143144efce26c8b4a1c395f1336c7622a9746f7ed9cdde33d75a6e79716 WatchSource:0}: Error finding container 6cd82143144efce26c8b4a1c395f1336c7622a9746f7ed9cdde33d75a6e79716: Status 404 returned error can't find the container with id 6cd82143144efce26c8b4a1c395f1336c7622a9746f7ed9cdde33d75a6e79716
Sep 16 19:56:43.625992 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:56:43.625895    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5df597f6b8-2bscj" podUID="8675a010-fb9e-4c2a-ae3f-11ad391ddb6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Sep 16 19:56:44.274417 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:56:44.274367    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6" event={"ID":"e3296120-91c5-4a5e-92d5-ca891652f649","Type":"ContainerStarted","Data":"8660a1a948bc56b1103170023e5fa44897d18d7b370380cc30f985bb5a970a54"}
Sep 16 19:56:44.274948 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:56:44.274450    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6" event={"ID":"e3296120-91c5-4a5e-92d5-ca891652f649","Type":"ContainerStarted","Data":"6cd82143144efce26c8b4a1c395f1336c7622a9746f7ed9cdde33d75a6e79716"}
Sep 16 19:56:46.286296 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:56:46.286260    2582 generic.go:356] "Generic (PLEG): container finished" podID="e3296120-91c5-4a5e-92d5-ca891652f649" containerID="8660a1a948bc56b1103170023e5fa44897d18d7b370380cc30f985bb5a970a54" exitCode=0
Sep 16 19:56:46.286822 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:56:46.286340    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6" event={"ID":"e3296120-91c5-4a5e-92d5-ca891652f649","Type":"ContainerDied","Data":"8660a1a948bc56b1103170023e5fa44897d18d7b370380cc30f985bb5a970a54"}
Sep 16 19:56:47.297047 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:56:47.296993    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6" event={"ID":"e3296120-91c5-4a5e-92d5-ca891652f649","Type":"ContainerStarted","Data":"98cd48cf768515756314d6c818ebd8ca8705b913915c191edc520f7c364878b2"}
Sep 16 19:56:47.324705 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:56:47.324647    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6" podStartSLOduration=3.051983585 podStartE2EDuration="5.324626978s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.272643393s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:56:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:56:47.323019221 +0000 UTC m=+4410.964734886" watchObservedRunningTime="2026-09-16 19:56:47.324626978 +0000 UTC m=+4410.966342633"
Sep 16 19:56:53.017992 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:56:53.017941    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6"
Sep 16 19:56:53.018618 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:56:53.018014    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6"
Sep 16 19:56:53.019498 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:56:53.019467    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6" podUID="e3296120-91c5-4a5e-92d5-ca891652f649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Sep 16 19:56:53.625774 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:56:53.625694    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5df597f6b8-2bscj" podUID="8675a010-fb9e-4c2a-ae3f-11ad391ddb6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Sep 16 19:57:03.018232 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:57:03.018173    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6" podUID="e3296120-91c5-4a5e-92d5-ca891652f649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Sep 16 19:57:03.625498 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:57:03.625437    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5df597f6b8-2bscj" podUID="8675a010-fb9e-4c2a-ae3f-11ad391ddb6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Sep 16 19:57:13.018030 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:57:13.017982    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6" podUID="e3296120-91c5-4a5e-92d5-ca891652f649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Sep 16 19:57:13.635349 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:57:13.635318    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5df597f6b8-2bscj"
Sep 16 19:57:13.642990 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:57:13.642959    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5df597f6b8-2bscj"
Sep 16 19:57:18.945024 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:57:18.944975    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5df597f6b8-2bscj"]
Sep 16 19:57:23.017658 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:57:23.017608    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6" podUID="e3296120-91c5-4a5e-92d5-ca891652f649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Sep 16 19:57:33.018273 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:57:33.018231    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6" podUID="e3296120-91c5-4a5e-92d5-ca891652f649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Sep 16 19:57:33.967994 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:57:33.967941    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5df597f6b8-2bscj" podUID="8675a010-fb9e-4c2a-ae3f-11ad391ddb6c" containerName="main" containerID="cri-o://0b50c038e9081a142e87e9e0da5ddff748366340f91145c382abdc90f4417b90" gracePeriod=45
Sep 16 19:57:34.724440 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:57:34.724408    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5df597f6b8-2bscj"
Sep 16 19:57:34.850489 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:57:34.850403    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8675a010-fb9e-4c2a-ae3f-11ad391ddb6c-tmp-dir\") pod \"8675a010-fb9e-4c2a-ae3f-11ad391ddb6c\" (UID: \"8675a010-fb9e-4c2a-ae3f-11ad391ddb6c\") "
Sep 16 19:57:34.850489 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:57:34.850464    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8675a010-fb9e-4c2a-ae3f-11ad391ddb6c-tls-certs\") pod \"8675a010-fb9e-4c2a-ae3f-11ad391ddb6c\" (UID: \"8675a010-fb9e-4c2a-ae3f-11ad391ddb6c\") "
Sep 16 19:57:34.850698 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:57:34.850492    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8675a010-fb9e-4c2a-ae3f-11ad391ddb6c-cabundle-cert\") pod \"8675a010-fb9e-4c2a-ae3f-11ad391ddb6c\" (UID: \"8675a010-fb9e-4c2a-ae3f-11ad391ddb6c\") "
Sep 16 19:57:34.850698 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:57:34.850571    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8675a010-fb9e-4c2a-ae3f-11ad391ddb6c-model-cache\") pod \"8675a010-fb9e-4c2a-ae3f-11ad391ddb6c\" (UID: \"8675a010-fb9e-4c2a-ae3f-11ad391ddb6c\") "
Sep 16 19:57:34.850698 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:57:34.850612    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8675a010-fb9e-4c2a-ae3f-11ad391ddb6c-kserve-provision-location\") pod \"8675a010-fb9e-4c2a-ae3f-11ad391ddb6c\" (UID: \"8675a010-fb9e-4c2a-ae3f-11ad391ddb6c\") "
Sep 16 19:57:34.850698 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:57:34.850645    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8675a010-fb9e-4c2a-ae3f-11ad391ddb6c-home\") pod \"8675a010-fb9e-4c2a-ae3f-11ad391ddb6c\" (UID: \"8675a010-fb9e-4c2a-ae3f-11ad391ddb6c\") "
Sep 16 19:57:34.850698 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:57:34.850671    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8675a010-fb9e-4c2a-ae3f-11ad391ddb6c-dshm\") pod \"8675a010-fb9e-4c2a-ae3f-11ad391ddb6c\" (UID: \"8675a010-fb9e-4c2a-ae3f-11ad391ddb6c\") "
Sep 16 19:57:34.850997 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:57:34.850702    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxwkk\" (UniqueName: \"kubernetes.io/projected/8675a010-fb9e-4c2a-ae3f-11ad391ddb6c-kube-api-access-wxwkk\") pod \"8675a010-fb9e-4c2a-ae3f-11ad391ddb6c\" (UID: \"8675a010-fb9e-4c2a-ae3f-11ad391ddb6c\") "
Sep 16 19:57:34.850997 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:57:34.850915    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8675a010-fb9e-4c2a-ae3f-11ad391ddb6c-model-cache" (OuterVolumeSpecName: "model-cache") pod "8675a010-fb9e-4c2a-ae3f-11ad391ddb6c" (UID: "8675a010-fb9e-4c2a-ae3f-11ad391ddb6c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:57:34.851637 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:57:34.851608    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8675a010-fb9e-4c2a-ae3f-11ad391ddb6c-home" (OuterVolumeSpecName: "home") pod "8675a010-fb9e-4c2a-ae3f-11ad391ddb6c" (UID: "8675a010-fb9e-4c2a-ae3f-11ad391ddb6c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:57:34.851824 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:57:34.851802    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8675a010-fb9e-4c2a-ae3f-11ad391ddb6c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8675a010-fb9e-4c2a-ae3f-11ad391ddb6c" (UID: "8675a010-fb9e-4c2a-ae3f-11ad391ddb6c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:57:34.853050 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:57:34.853025    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8675a010-fb9e-4c2a-ae3f-11ad391ddb6c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8675a010-fb9e-4c2a-ae3f-11ad391ddb6c" (UID: "8675a010-fb9e-4c2a-ae3f-11ad391ddb6c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:57:34.853495 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:57:34.853476    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8675a010-fb9e-4c2a-ae3f-11ad391ddb6c-kube-api-access-wxwkk" (OuterVolumeSpecName: "kube-api-access-wxwkk") pod "8675a010-fb9e-4c2a-ae3f-11ad391ddb6c" (UID: "8675a010-fb9e-4c2a-ae3f-11ad391ddb6c"). InnerVolumeSpecName "kube-api-access-wxwkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:57:34.853897 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:57:34.853875    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8675a010-fb9e-4c2a-ae3f-11ad391ddb6c-dshm" (OuterVolumeSpecName: "dshm") pod "8675a010-fb9e-4c2a-ae3f-11ad391ddb6c" (UID: "8675a010-fb9e-4c2a-ae3f-11ad391ddb6c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:57:34.866149 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:57:34.866096    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8675a010-fb9e-4c2a-ae3f-11ad391ddb6c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8675a010-fb9e-4c2a-ae3f-11ad391ddb6c" (UID: "8675a010-fb9e-4c2a-ae3f-11ad391ddb6c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:57:34.877320 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:57:34.877286    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8675a010-fb9e-4c2a-ae3f-11ad391ddb6c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8675a010-fb9e-4c2a-ae3f-11ad391ddb6c" (UID: "8675a010-fb9e-4c2a-ae3f-11ad391ddb6c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:57:34.951858 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:57:34.951813    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8675a010-fb9e-4c2a-ae3f-11ad391ddb6c-tmp-dir\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:57:34.951858 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:57:34.951850    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8675a010-fb9e-4c2a-ae3f-11ad391ddb6c-tls-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:57:34.951858 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:57:34.951860    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8675a010-fb9e-4c2a-ae3f-11ad391ddb6c-cabundle-cert\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:57:34.951858 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:57:34.951870    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8675a010-fb9e-4c2a-ae3f-11ad391ddb6c-model-cache\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:57:34.952248 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:57:34.951880    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8675a010-fb9e-4c2a-ae3f-11ad391ddb6c-kserve-provision-location\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:57:34.952248 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:57:34.951888    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8675a010-fb9e-4c2a-ae3f-11ad391ddb6c-home\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:57:34.952248 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:57:34.951898    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8675a010-fb9e-4c2a-ae3f-11ad391ddb6c-dshm\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:57:34.952248 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:57:34.951911    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxwkk\" (UniqueName: \"kubernetes.io/projected/8675a010-fb9e-4c2a-ae3f-11ad391ddb6c-kube-api-access-wxwkk\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:57:35.544600 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:57:35.544557    2582 generic.go:356] "Generic (PLEG): container finished" podID="8675a010-fb9e-4c2a-ae3f-11ad391ddb6c" containerID="0b50c038e9081a142e87e9e0da5ddff748366340f91145c382abdc90f4417b90" exitCode=0
Sep 16 19:57:35.544796 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:57:35.544606    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5df597f6b8-2bscj" event={"ID":"8675a010-fb9e-4c2a-ae3f-11ad391ddb6c","Type":"ContainerDied","Data":"0b50c038e9081a142e87e9e0da5ddff748366340f91145c382abdc90f4417b90"}
Sep 16 19:57:35.544796 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:57:35.544672    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5df597f6b8-2bscj" event={"ID":"8675a010-fb9e-4c2a-ae3f-11ad391ddb6c","Type":"ContainerDied","Data":"2c6381b83ea5bebac5db3989b7a2550707ce47f1ef4021981b31580a0f9db2d6"}
Sep 16 19:57:35.544796 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:57:35.544617    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5df597f6b8-2bscj"
Sep 16 19:57:35.544796 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:57:35.544702    2582 scope.go:117] "RemoveContainer" containerID="0b50c038e9081a142e87e9e0da5ddff748366340f91145c382abdc90f4417b90"
Sep 16 19:57:35.554080 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:57:35.554061    2582 scope.go:117] "RemoveContainer" containerID="21076671560b683cfb23866c854dc7c740b1508ccaa3a2309703c74c99d65789"
Sep 16 19:57:35.562146 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:57:35.562128    2582 scope.go:117] "RemoveContainer" containerID="0b50c038e9081a142e87e9e0da5ddff748366340f91145c382abdc90f4417b90"
Sep 16 19:57:35.562429 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:57:35.562411    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b50c038e9081a142e87e9e0da5ddff748366340f91145c382abdc90f4417b90\": container with ID starting with 0b50c038e9081a142e87e9e0da5ddff748366340f91145c382abdc90f4417b90 not found: ID does not exist" containerID="0b50c038e9081a142e87e9e0da5ddff748366340f91145c382abdc90f4417b90"
Sep 16 19:57:35.562474 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:57:35.562439    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b50c038e9081a142e87e9e0da5ddff748366340f91145c382abdc90f4417b90"} err="failed to get container status \"0b50c038e9081a142e87e9e0da5ddff748366340f91145c382abdc90f4417b90\": rpc error: code = NotFound desc = could not find container \"0b50c038e9081a142e87e9e0da5ddff748366340f91145c382abdc90f4417b90\": container with ID starting with 0b50c038e9081a142e87e9e0da5ddff748366340f91145c382abdc90f4417b90 not found: ID does not exist"
Sep 16 19:57:35.562474 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:57:35.562454    2582 scope.go:117] "RemoveContainer" containerID="21076671560b683cfb23866c854dc7c740b1508ccaa3a2309703c74c99d65789"
Sep 16 19:57:35.562691 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:57:35.562670    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21076671560b683cfb23866c854dc7c740b1508ccaa3a2309703c74c99d65789\": container with ID starting with 21076671560b683cfb23866c854dc7c740b1508ccaa3a2309703c74c99d65789 not found: ID does not exist" containerID="21076671560b683cfb23866c854dc7c740b1508ccaa3a2309703c74c99d65789"
Sep 16 19:57:35.562778 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:57:35.562703    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21076671560b683cfb23866c854dc7c740b1508ccaa3a2309703c74c99d65789"} err="failed to get container status \"21076671560b683cfb23866c854dc7c740b1508ccaa3a2309703c74c99d65789\": rpc error: code = NotFound desc = could not find container \"21076671560b683cfb23866c854dc7c740b1508ccaa3a2309703c74c99d65789\": container with ID starting with 21076671560b683cfb23866c854dc7c740b1508ccaa3a2309703c74c99d65789 not found: ID does not exist"
Sep 16 19:57:35.568755 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:57:35.568711    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5df597f6b8-2bscj"]
Sep 16 19:57:35.573565 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:57:35.573540    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5df597f6b8-2bscj"]
Sep 16 19:57:35.636249 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:57:35.636204    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5df597f6b8-2bscj" podUID="8675a010-fb9e-4c2a-ae3f-11ad391ddb6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": context deadline exceeded"
Sep 16 19:57:36.871904 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:57:36.871869    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8675a010-fb9e-4c2a-ae3f-11ad391ddb6c" path="/var/lib/kubelet/pods/8675a010-fb9e-4c2a-ae3f-11ad391ddb6c/volumes"
Sep 16 19:57:43.018523 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:57:43.018475    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6" podUID="e3296120-91c5-4a5e-92d5-ca891652f649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Sep 16 19:57:53.018002 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:57:53.017958    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6" podUID="e3296120-91c5-4a5e-92d5-ca891652f649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Sep 16 19:58:03.018535 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:03.018487    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6" podUID="e3296120-91c5-4a5e-92d5-ca891652f649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Sep 16 19:58:13.018202 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:13.018096    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6" podUID="e3296120-91c5-4a5e-92d5-ca891652f649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Sep 16 19:58:17.188471 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:17.188434    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7qwl2_b6695f86-08c0-4836-87b1-f2a9d76084bd/console-operator/1.log"
Sep 16 19:58:17.194311 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:17.194286    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7qwl2_b6695f86-08c0-4836-87b1-f2a9d76084bd/console-operator/1.log"
Sep 16 19:58:19.629979 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:19.629942    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-89dd8"]
Sep 16 19:58:19.630566 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:19.630272    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8675a010-fb9e-4c2a-ae3f-11ad391ddb6c" containerName="storage-initializer"
Sep 16 19:58:19.630566 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:19.630285    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8675a010-fb9e-4c2a-ae3f-11ad391ddb6c" containerName="storage-initializer"
Sep 16 19:58:19.630566 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:19.630298    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8675a010-fb9e-4c2a-ae3f-11ad391ddb6c" containerName="main"
Sep 16 19:58:19.630566 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:19.630303    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8675a010-fb9e-4c2a-ae3f-11ad391ddb6c" containerName="main"
Sep 16 19:58:19.630566 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:19.630362    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8675a010-fb9e-4c2a-ae3f-11ad391ddb6c" containerName="main"
Sep 16 19:58:19.634574 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:19.634551    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-89dd8"
Sep 16 19:58:19.637254 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:19.637233    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:58:19.637383 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:19.637235    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 16 19:58:19.637383 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:19.637235    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 16 19:58:19.637383 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:19.637369    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Sep 16 19:58:19.637763 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:19.637737    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-frrmd\""
Sep 16 19:58:19.637875 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:19.637739    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 16 19:58:19.647859 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:19.647823    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-89dd8"]
Sep 16 19:58:19.733620 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:19.733570    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e056b4e3-e914-4dee-8202-8df521d4a398-kserve-provision-location\") pod \"prestop-hook-test-kserve-69b4ffd8cb-89dd8\" (UID: \"e056b4e3-e914-4dee-8202-8df521d4a398\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-89dd8"
Sep 16 19:58:19.733620 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:19.733605    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e056b4e3-e914-4dee-8202-8df521d4a398-cabundle-cert\") pod \"prestop-hook-test-kserve-69b4ffd8cb-89dd8\" (UID: \"e056b4e3-e914-4dee-8202-8df521d4a398\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-89dd8"
Sep 16 19:58:19.733936 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:19.733640    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e056b4e3-e914-4dee-8202-8df521d4a398-model-cache\") pod \"prestop-hook-test-kserve-69b4ffd8cb-89dd8\" (UID: \"e056b4e3-e914-4dee-8202-8df521d4a398\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-89dd8"
Sep 16 19:58:19.733936 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:19.733738    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e056b4e3-e914-4dee-8202-8df521d4a398-home\") pod \"prestop-hook-test-kserve-69b4ffd8cb-89dd8\" (UID: \"e056b4e3-e914-4dee-8202-8df521d4a398\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-89dd8"
Sep 16 19:58:19.733936 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:19.733771    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e056b4e3-e914-4dee-8202-8df521d4a398-tls-certs\") pod \"prestop-hook-test-kserve-69b4ffd8cb-89dd8\" (UID: \"e056b4e3-e914-4dee-8202-8df521d4a398\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-89dd8"
Sep 16 19:58:19.733936 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:19.733809    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e056b4e3-e914-4dee-8202-8df521d4a398-dshm\") pod \"prestop-hook-test-kserve-69b4ffd8cb-89dd8\" (UID: \"e056b4e3-e914-4dee-8202-8df521d4a398\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-89dd8"
Sep 16 19:58:19.733936 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:19.733825    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntp6h\" (UniqueName: \"kubernetes.io/projected/e056b4e3-e914-4dee-8202-8df521d4a398-kube-api-access-ntp6h\") pod \"prestop-hook-test-kserve-69b4ffd8cb-89dd8\" (UID: \"e056b4e3-e914-4dee-8202-8df521d4a398\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-89dd8"
Sep 16 19:58:19.733936 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:19.733848    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e056b4e3-e914-4dee-8202-8df521d4a398-tmp-dir\") pod \"prestop-hook-test-kserve-69b4ffd8cb-89dd8\" (UID: \"e056b4e3-e914-4dee-8202-8df521d4a398\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-89dd8"
Sep 16 19:58:19.835206 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:19.835168    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e056b4e3-e914-4dee-8202-8df521d4a398-dshm\") pod \"prestop-hook-test-kserve-69b4ffd8cb-89dd8\" (UID: \"e056b4e3-e914-4dee-8202-8df521d4a398\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-89dd8"
Sep 16 19:58:19.835206 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:19.835206    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntp6h\" (UniqueName: \"kubernetes.io/projected/e056b4e3-e914-4dee-8202-8df521d4a398-kube-api-access-ntp6h\") pod \"prestop-hook-test-kserve-69b4ffd8cb-89dd8\" (UID: \"e056b4e3-e914-4dee-8202-8df521d4a398\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-89dd8"
Sep 16 19:58:19.835491 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:19.835231    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e056b4e3-e914-4dee-8202-8df521d4a398-tmp-dir\") pod \"prestop-hook-test-kserve-69b4ffd8cb-89dd8\" (UID: \"e056b4e3-e914-4dee-8202-8df521d4a398\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-89dd8"
Sep 16 19:58:19.835491 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:19.835281    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e056b4e3-e914-4dee-8202-8df521d4a398-kserve-provision-location\") pod \"prestop-hook-test-kserve-69b4ffd8cb-89dd8\" (UID: \"e056b4e3-e914-4dee-8202-8df521d4a398\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-89dd8"
Sep 16 19:58:19.835491 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:19.835305    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e056b4e3-e914-4dee-8202-8df521d4a398-cabundle-cert\") pod \"prestop-hook-test-kserve-69b4ffd8cb-89dd8\" (UID: \"e056b4e3-e914-4dee-8202-8df521d4a398\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-89dd8"
Sep 16 19:58:19.835491 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:19.835351    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e056b4e3-e914-4dee-8202-8df521d4a398-model-cache\") pod \"prestop-hook-test-kserve-69b4ffd8cb-89dd8\" (UID: \"e056b4e3-e914-4dee-8202-8df521d4a398\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-89dd8"
Sep 16 19:58:19.835491 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:19.835386    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e056b4e3-e914-4dee-8202-8df521d4a398-home\") pod \"prestop-hook-test-kserve-69b4ffd8cb-89dd8\" (UID: \"e056b4e3-e914-4dee-8202-8df521d4a398\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-89dd8"
Sep 16 19:58:19.835491 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:19.835411    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e056b4e3-e914-4dee-8202-8df521d4a398-tls-certs\") pod \"prestop-hook-test-kserve-69b4ffd8cb-89dd8\" (UID: \"e056b4e3-e914-4dee-8202-8df521d4a398\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-89dd8"
Sep 16 19:58:19.835863 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:19.835747    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e056b4e3-e914-4dee-8202-8df521d4a398-kserve-provision-location\") pod \"prestop-hook-test-kserve-69b4ffd8cb-89dd8\" (UID: \"e056b4e3-e914-4dee-8202-8df521d4a398\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-89dd8"
Sep 16 19:58:19.835863 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:19.835760    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e056b4e3-e914-4dee-8202-8df521d4a398-tmp-dir\") pod \"prestop-hook-test-kserve-69b4ffd8cb-89dd8\" (UID: \"e056b4e3-e914-4dee-8202-8df521d4a398\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-89dd8"
Sep 16 19:58:19.835863 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:19.835816    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e056b4e3-e914-4dee-8202-8df521d4a398-home\") pod \"prestop-hook-test-kserve-69b4ffd8cb-89dd8\" (UID: \"e056b4e3-e914-4dee-8202-8df521d4a398\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-89dd8"
Sep 16 19:58:19.836045 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:19.835999    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e056b4e3-e914-4dee-8202-8df521d4a398-model-cache\") pod \"prestop-hook-test-kserve-69b4ffd8cb-89dd8\" (UID: \"e056b4e3-e914-4dee-8202-8df521d4a398\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-89dd8"
Sep 16 19:58:19.836172 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:19.836153    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e056b4e3-e914-4dee-8202-8df521d4a398-cabundle-cert\") pod \"prestop-hook-test-kserve-69b4ffd8cb-89dd8\" (UID: \"e056b4e3-e914-4dee-8202-8df521d4a398\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-89dd8"
Sep 16 19:58:19.838139 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:19.838108    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e056b4e3-e914-4dee-8202-8df521d4a398-dshm\") pod \"prestop-hook-test-kserve-69b4ffd8cb-89dd8\" (UID: \"e056b4e3-e914-4dee-8202-8df521d4a398\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-89dd8"
Sep 16 19:58:19.838460 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:19.838437    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e056b4e3-e914-4dee-8202-8df521d4a398-tls-certs\") pod \"prestop-hook-test-kserve-69b4ffd8cb-89dd8\" (UID: \"e056b4e3-e914-4dee-8202-8df521d4a398\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-89dd8"
Sep 16 19:58:19.844122 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:19.844099    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntp6h\" (UniqueName: \"kubernetes.io/projected/e056b4e3-e914-4dee-8202-8df521d4a398-kube-api-access-ntp6h\") pod \"prestop-hook-test-kserve-69b4ffd8cb-89dd8\" (UID: \"e056b4e3-e914-4dee-8202-8df521d4a398\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-89dd8"
Sep 16 19:58:19.944602 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:19.944505    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-89dd8"
Sep 16 19:58:20.228320 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:20.228291    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-89dd8"]
Sep 16 19:58:20.230538 ip-10-0-129-194 kubenswrapper[2582]: W0916 19:58:20.230506    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode056b4e3_e914_4dee_8202_8df521d4a398.slice/crio-82fb4f2fd33e9c2a322e03d1e9440c4018b92435e56c96b8d9266952504bac11 WatchSource:0}: Error finding container 82fb4f2fd33e9c2a322e03d1e9440c4018b92435e56c96b8d9266952504bac11: Status 404 returned error can't find the container with id 82fb4f2fd33e9c2a322e03d1e9440c4018b92435e56c96b8d9266952504bac11
Sep 16 19:58:20.760943 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:20.760894    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-89dd8" event={"ID":"e056b4e3-e914-4dee-8202-8df521d4a398","Type":"ContainerStarted","Data":"960ad7857958432a2af5b878c913d58282e830fc9c82045747bf200e7bfc7dcc"}
Sep 16 19:58:20.761452 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:20.760963    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-89dd8" event={"ID":"e056b4e3-e914-4dee-8202-8df521d4a398","Type":"ContainerStarted","Data":"82fb4f2fd33e9c2a322e03d1e9440c4018b92435e56c96b8d9266952504bac11"}
Sep 16 19:58:22.772735 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:22.772677    2582 generic.go:356] "Generic (PLEG): container finished" podID="e056b4e3-e914-4dee-8202-8df521d4a398" containerID="960ad7857958432a2af5b878c913d58282e830fc9c82045747bf200e7bfc7dcc" exitCode=0
Sep 16 19:58:22.773219 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:22.772739    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-89dd8" event={"ID":"e056b4e3-e914-4dee-8202-8df521d4a398","Type":"ContainerDied","Data":"960ad7857958432a2af5b878c913d58282e830fc9c82045747bf200e7bfc7dcc"}
Sep 16 19:58:23.017776 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:23.017730    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6" podUID="e3296120-91c5-4a5e-92d5-ca891652f649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Sep 16 19:58:23.779181 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:23.779144    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-89dd8" event={"ID":"e056b4e3-e914-4dee-8202-8df521d4a398","Type":"ContainerStarted","Data":"c4ffabbc8a8452b40764f73ea69028a6bbbcd02d382aa829276cda608c7de6a8"}
Sep 16 19:58:23.808852 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:23.808790    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-89dd8" podStartSLOduration=2.506240989 podStartE2EDuration="4.808770235s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.302529246s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:58:23.808129155 +0000 UTC m=+4507.449844831" watchObservedRunningTime="2026-09-16 19:58:23.808770235 +0000 UTC m=+4507.450485900"
Sep 16 19:58:23.945493 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:23.945445    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-89dd8"
Sep 16 19:58:29.945057 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:29.945011    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-89dd8"
Sep 16 19:58:29.947045 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:29.947009    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-89dd8" podUID="e056b4e3-e914-4dee-8202-8df521d4a398" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Sep 16 19:58:33.018442 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:33.018393    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6" podUID="e3296120-91c5-4a5e-92d5-ca891652f649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Sep 16 19:58:39.945404 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:39.945356    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-89dd8" podUID="e056b4e3-e914-4dee-8202-8df521d4a398" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Sep 16 19:58:43.028310 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:43.028272    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6"
Sep 16 19:58:43.036878 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:43.036848    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6"
Sep 16 19:58:49.945308 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:49.945255    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-89dd8" podUID="e056b4e3-e914-4dee-8202-8df521d4a398" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Sep 16 19:58:52.866358 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:52.866314    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6"]
Sep 16 19:58:59.945867 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:58:59.945825    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-89dd8" podUID="e056b4e3-e914-4dee-8202-8df521d4a398" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Sep 16 19:59:07.889093 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:59:07.889020    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6" podUID="e3296120-91c5-4a5e-92d5-ca891652f649" containerName="main" containerID="cri-o://98cd48cf768515756314d6c818ebd8ca8705b913915c191edc520f7c364878b2" gracePeriod=45
Sep 16 19:59:08.630833 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:59:08.630807    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6"
Sep 16 19:59:08.674050 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:59:08.673966    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3296120-91c5-4a5e-92d5-ca891652f649-home\") pod \"e3296120-91c5-4a5e-92d5-ca891652f649\" (UID: \"e3296120-91c5-4a5e-92d5-ca891652f649\") "
Sep 16 19:59:08.674050 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:59:08.674033    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3296120-91c5-4a5e-92d5-ca891652f649-cabundle-cert\") pod \"e3296120-91c5-4a5e-92d5-ca891652f649\" (UID: \"e3296120-91c5-4a5e-92d5-ca891652f649\") "
Sep 16 19:59:08.674272 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:59:08.674084    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3296120-91c5-4a5e-92d5-ca891652f649-dshm\") pod \"e3296120-91c5-4a5e-92d5-ca891652f649\" (UID: \"e3296120-91c5-4a5e-92d5-ca891652f649\") "
Sep 16 19:59:08.674272 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:59:08.674106    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3296120-91c5-4a5e-92d5-ca891652f649-tmp-dir\") pod \"e3296120-91c5-4a5e-92d5-ca891652f649\" (UID: \"e3296120-91c5-4a5e-92d5-ca891652f649\") "
Sep 16 19:59:08.674272 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:59:08.674136    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3296120-91c5-4a5e-92d5-ca891652f649-kserve-provision-location\") pod \"e3296120-91c5-4a5e-92d5-ca891652f649\" (UID: \"e3296120-91c5-4a5e-92d5-ca891652f649\") "
Sep 16 19:59:08.674272 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:59:08.674167    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3296120-91c5-4a5e-92d5-ca891652f649-model-cache\") pod \"e3296120-91c5-4a5e-92d5-ca891652f649\" (UID: \"e3296120-91c5-4a5e-92d5-ca891652f649\") "
Sep 16 19:59:08.674272 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:59:08.674192    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3296120-91c5-4a5e-92d5-ca891652f649-tls-certs\") pod \"e3296120-91c5-4a5e-92d5-ca891652f649\" (UID: \"e3296120-91c5-4a5e-92d5-ca891652f649\") "
Sep 16 19:59:08.674272 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:59:08.674220    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2cxf7\" (UniqueName: \"kubernetes.io/projected/e3296120-91c5-4a5e-92d5-ca891652f649-kube-api-access-2cxf7\") pod \"e3296120-91c5-4a5e-92d5-ca891652f649\" (UID: \"e3296120-91c5-4a5e-92d5-ca891652f649\") "
Sep 16 19:59:08.674593 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:59:08.674405    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3296120-91c5-4a5e-92d5-ca891652f649-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e3296120-91c5-4a5e-92d5-ca891652f649" (UID: "e3296120-91c5-4a5e-92d5-ca891652f649"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:59:08.674660 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:59:08.674594    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3296120-91c5-4a5e-92d5-ca891652f649-model-cache" (OuterVolumeSpecName: "model-cache") pod "e3296120-91c5-4a5e-92d5-ca891652f649" (UID: "e3296120-91c5-4a5e-92d5-ca891652f649"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:59:08.674906 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:59:08.674742    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3296120-91c5-4a5e-92d5-ca891652f649-home" (OuterVolumeSpecName: "home") pod "e3296120-91c5-4a5e-92d5-ca891652f649" (UID: "e3296120-91c5-4a5e-92d5-ca891652f649"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:59:08.674906 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:59:08.674842    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3296120-91c5-4a5e-92d5-ca891652f649-cabundle-cert\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:59:08.674906 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:59:08.674865    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3296120-91c5-4a5e-92d5-ca891652f649-model-cache\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:59:08.674906 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:59:08.674882    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3296120-91c5-4a5e-92d5-ca891652f649-home\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:59:08.676962 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:59:08.676936    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3296120-91c5-4a5e-92d5-ca891652f649-dshm" (OuterVolumeSpecName: "dshm") pod "e3296120-91c5-4a5e-92d5-ca891652f649" (UID: "e3296120-91c5-4a5e-92d5-ca891652f649"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:59:08.677056 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:59:08.676943    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3296120-91c5-4a5e-92d5-ca891652f649-kube-api-access-2cxf7" (OuterVolumeSpecName: "kube-api-access-2cxf7") pod "e3296120-91c5-4a5e-92d5-ca891652f649" (UID: "e3296120-91c5-4a5e-92d5-ca891652f649"). InnerVolumeSpecName "kube-api-access-2cxf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:59:08.677056 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:59:08.677037    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3296120-91c5-4a5e-92d5-ca891652f649-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e3296120-91c5-4a5e-92d5-ca891652f649" (UID: "e3296120-91c5-4a5e-92d5-ca891652f649"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:59:08.690098 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:59:08.690066    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3296120-91c5-4a5e-92d5-ca891652f649-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e3296120-91c5-4a5e-92d5-ca891652f649" (UID: "e3296120-91c5-4a5e-92d5-ca891652f649"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:59:08.707810 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:59:08.707775    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3296120-91c5-4a5e-92d5-ca891652f649-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e3296120-91c5-4a5e-92d5-ca891652f649" (UID: "e3296120-91c5-4a5e-92d5-ca891652f649"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:59:08.776089 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:59:08.776047    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3296120-91c5-4a5e-92d5-ca891652f649-dshm\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:59:08.776219 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:59:08.776118    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3296120-91c5-4a5e-92d5-ca891652f649-tmp-dir\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:59:08.776219 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:59:08.776138    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3296120-91c5-4a5e-92d5-ca891652f649-kserve-provision-location\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:59:08.776219 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:59:08.776151    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3296120-91c5-4a5e-92d5-ca891652f649-tls-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:59:08.776219 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:59:08.776166    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cxf7\" (UniqueName: \"kubernetes.io/projected/e3296120-91c5-4a5e-92d5-ca891652f649-kube-api-access-2cxf7\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 19:59:08.999911 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:59:08.999812    2582 generic.go:356] "Generic (PLEG): container finished" podID="e3296120-91c5-4a5e-92d5-ca891652f649" containerID="98cd48cf768515756314d6c818ebd8ca8705b913915c191edc520f7c364878b2" exitCode=0
Sep 16 19:59:08.999911 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:59:08.999861    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6"
Sep 16 19:59:09.000422 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:59:08.999903    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6" event={"ID":"e3296120-91c5-4a5e-92d5-ca891652f649","Type":"ContainerDied","Data":"98cd48cf768515756314d6c818ebd8ca8705b913915c191edc520f7c364878b2"}
Sep 16 19:59:09.000422 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:59:08.999961    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6" event={"ID":"e3296120-91c5-4a5e-92d5-ca891652f649","Type":"ContainerDied","Data":"6cd82143144efce26c8b4a1c395f1336c7622a9746f7ed9cdde33d75a6e79716"}
Sep 16 19:59:09.000422 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:59:08.999984    2582 scope.go:117] "RemoveContainer" containerID="98cd48cf768515756314d6c818ebd8ca8705b913915c191edc520f7c364878b2"
Sep 16 19:59:09.009800 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:59:09.009738    2582 scope.go:117] "RemoveContainer" containerID="8660a1a948bc56b1103170023e5fa44897d18d7b370380cc30f985bb5a970a54"
Sep 16 19:59:09.018686 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:59:09.018657    2582 scope.go:117] "RemoveContainer" containerID="98cd48cf768515756314d6c818ebd8ca8705b913915c191edc520f7c364878b2"
Sep 16 19:59:09.019109 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:59:09.019076    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98cd48cf768515756314d6c818ebd8ca8705b913915c191edc520f7c364878b2\": container with ID starting with 98cd48cf768515756314d6c818ebd8ca8705b913915c191edc520f7c364878b2 not found: ID does not exist" containerID="98cd48cf768515756314d6c818ebd8ca8705b913915c191edc520f7c364878b2"
Sep 16 19:59:09.019212 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:59:09.019142    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98cd48cf768515756314d6c818ebd8ca8705b913915c191edc520f7c364878b2"} err="failed to get container status \"98cd48cf768515756314d6c818ebd8ca8705b913915c191edc520f7c364878b2\": rpc error: code = NotFound desc = could not find container \"98cd48cf768515756314d6c818ebd8ca8705b913915c191edc520f7c364878b2\": container with ID starting with 98cd48cf768515756314d6c818ebd8ca8705b913915c191edc520f7c364878b2 not found: ID does not exist"
Sep 16 19:59:09.019212 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:59:09.019165    2582 scope.go:117] "RemoveContainer" containerID="8660a1a948bc56b1103170023e5fa44897d18d7b370380cc30f985bb5a970a54"
Sep 16 19:59:09.019484 ip-10-0-129-194 kubenswrapper[2582]: E0916 19:59:09.019454    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8660a1a948bc56b1103170023e5fa44897d18d7b370380cc30f985bb5a970a54\": container with ID starting with 8660a1a948bc56b1103170023e5fa44897d18d7b370380cc30f985bb5a970a54 not found: ID does not exist" containerID="8660a1a948bc56b1103170023e5fa44897d18d7b370380cc30f985bb5a970a54"
Sep 16 19:59:09.019673 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:59:09.019481    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8660a1a948bc56b1103170023e5fa44897d18d7b370380cc30f985bb5a970a54"} err="failed to get container status \"8660a1a948bc56b1103170023e5fa44897d18d7b370380cc30f985bb5a970a54\": rpc error: code = NotFound desc = could not find container \"8660a1a948bc56b1103170023e5fa44897d18d7b370380cc30f985bb5a970a54\": container with ID starting with 8660a1a948bc56b1103170023e5fa44897d18d7b370380cc30f985bb5a970a54 not found: ID does not exist"
Sep 16 19:59:09.021097 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:59:09.021075    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6"]
Sep 16 19:59:09.024753 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:59:09.024709    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7d49c4dc5f-ngmn6"]
Sep 16 19:59:09.945348 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:59:09.945304    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-89dd8" podUID="e056b4e3-e914-4dee-8202-8df521d4a398" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Sep 16 19:59:10.874395 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:59:10.874353    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3296120-91c5-4a5e-92d5-ca891652f649" path="/var/lib/kubelet/pods/e3296120-91c5-4a5e-92d5-ca891652f649/volumes"
Sep 16 19:59:19.945769 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:59:19.945692    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-89dd8" podUID="e056b4e3-e914-4dee-8202-8df521d4a398" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Sep 16 19:59:29.945208 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:59:29.945159    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-89dd8" podUID="e056b4e3-e914-4dee-8202-8df521d4a398" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Sep 16 19:59:39.945690 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:59:39.945604    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-89dd8" podUID="e056b4e3-e914-4dee-8202-8df521d4a398" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Sep 16 19:59:49.945519 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:59:49.945468    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-89dd8" podUID="e056b4e3-e914-4dee-8202-8df521d4a398" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Sep 16 19:59:59.945444 ip-10-0-129-194 kubenswrapper[2582]: I0916 19:59:59.945401    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-89dd8" podUID="e056b4e3-e914-4dee-8202-8df521d4a398" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Sep 16 20:00:09.945819 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:09.945773    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-89dd8" podUID="e056b4e3-e914-4dee-8202-8df521d4a398" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Sep 16 20:00:19.955432 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:19.955397    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-89dd8"
Sep 16 20:00:19.963163 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:19.963135    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-89dd8"
Sep 16 20:00:24.911619 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:24.911577    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-89dd8"]
Sep 16 20:00:24.938411 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:24.938378    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-lw4fp"]
Sep 16 20:00:24.938683 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:24.938670    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3296120-91c5-4a5e-92d5-ca891652f649" containerName="main"
Sep 16 20:00:24.938752 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:24.938685    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3296120-91c5-4a5e-92d5-ca891652f649" containerName="main"
Sep 16 20:00:24.938752 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:24.938695    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3296120-91c5-4a5e-92d5-ca891652f649" containerName="storage-initializer"
Sep 16 20:00:24.938752 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:24.938701    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3296120-91c5-4a5e-92d5-ca891652f649" containerName="storage-initializer"
Sep 16 20:00:24.938897 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:24.938801    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3296120-91c5-4a5e-92d5-ca891652f649" containerName="main"
Sep 16 20:00:24.942083 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:24.942059    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-lw4fp"
Sep 16 20:00:24.958139 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:24.958104    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-lw4fp"]
Sep 16 20:00:25.034451 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:25.034420    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd265538-44a0-4409-a13a-ffe71ff42ed6-dshm\") pod \"prestop-hook-test-kserve-69b4ffd8cb-lw4fp\" (UID: \"bd265538-44a0-4409-a13a-ffe71ff42ed6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-lw4fp"
Sep 16 20:00:25.034612 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:25.034458    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd265538-44a0-4409-a13a-ffe71ff42ed6-home\") pod \"prestop-hook-test-kserve-69b4ffd8cb-lw4fp\" (UID: \"bd265538-44a0-4409-a13a-ffe71ff42ed6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-lw4fp"
Sep 16 20:00:25.034612 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:25.034477    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd265538-44a0-4409-a13a-ffe71ff42ed6-tmp-dir\") pod \"prestop-hook-test-kserve-69b4ffd8cb-lw4fp\" (UID: \"bd265538-44a0-4409-a13a-ffe71ff42ed6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-lw4fp"
Sep 16 20:00:25.034612 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:25.034498    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkz4r\" (UniqueName: \"kubernetes.io/projected/bd265538-44a0-4409-a13a-ffe71ff42ed6-kube-api-access-nkz4r\") pod \"prestop-hook-test-kserve-69b4ffd8cb-lw4fp\" (UID: \"bd265538-44a0-4409-a13a-ffe71ff42ed6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-lw4fp"
Sep 16 20:00:25.034612 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:25.034585    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd265538-44a0-4409-a13a-ffe71ff42ed6-tls-certs\") pod \"prestop-hook-test-kserve-69b4ffd8cb-lw4fp\" (UID: \"bd265538-44a0-4409-a13a-ffe71ff42ed6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-lw4fp"
Sep 16 20:00:25.034774 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:25.034632    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd265538-44a0-4409-a13a-ffe71ff42ed6-kserve-provision-location\") pod \"prestop-hook-test-kserve-69b4ffd8cb-lw4fp\" (UID: \"bd265538-44a0-4409-a13a-ffe71ff42ed6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-lw4fp"
Sep 16 20:00:25.034774 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:25.034664    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd265538-44a0-4409-a13a-ffe71ff42ed6-cabundle-cert\") pod \"prestop-hook-test-kserve-69b4ffd8cb-lw4fp\" (UID: \"bd265538-44a0-4409-a13a-ffe71ff42ed6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-lw4fp"
Sep 16 20:00:25.034774 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:25.034747    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd265538-44a0-4409-a13a-ffe71ff42ed6-model-cache\") pod \"prestop-hook-test-kserve-69b4ffd8cb-lw4fp\" (UID: \"bd265538-44a0-4409-a13a-ffe71ff42ed6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-lw4fp"
Sep 16 20:00:25.136140 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:25.136100    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd265538-44a0-4409-a13a-ffe71ff42ed6-dshm\") pod \"prestop-hook-test-kserve-69b4ffd8cb-lw4fp\" (UID: \"bd265538-44a0-4409-a13a-ffe71ff42ed6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-lw4fp"
Sep 16 20:00:25.136140 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:25.136145    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd265538-44a0-4409-a13a-ffe71ff42ed6-home\") pod \"prestop-hook-test-kserve-69b4ffd8cb-lw4fp\" (UID: \"bd265538-44a0-4409-a13a-ffe71ff42ed6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-lw4fp"
Sep 16 20:00:25.136416 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:25.136162    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd265538-44a0-4409-a13a-ffe71ff42ed6-tmp-dir\") pod \"prestop-hook-test-kserve-69b4ffd8cb-lw4fp\" (UID: \"bd265538-44a0-4409-a13a-ffe71ff42ed6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-lw4fp"
Sep 16 20:00:25.136416 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:25.136180    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkz4r\" (UniqueName: \"kubernetes.io/projected/bd265538-44a0-4409-a13a-ffe71ff42ed6-kube-api-access-nkz4r\") pod \"prestop-hook-test-kserve-69b4ffd8cb-lw4fp\" (UID: \"bd265538-44a0-4409-a13a-ffe71ff42ed6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-lw4fp"
Sep 16 20:00:25.136416 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:25.136208    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd265538-44a0-4409-a13a-ffe71ff42ed6-tls-certs\") pod \"prestop-hook-test-kserve-69b4ffd8cb-lw4fp\" (UID: \"bd265538-44a0-4409-a13a-ffe71ff42ed6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-lw4fp"
Sep 16 20:00:25.136416 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:25.136340    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd265538-44a0-4409-a13a-ffe71ff42ed6-kserve-provision-location\") pod \"prestop-hook-test-kserve-69b4ffd8cb-lw4fp\" (UID: \"bd265538-44a0-4409-a13a-ffe71ff42ed6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-lw4fp"
Sep 16 20:00:25.136416 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:25.136382    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd265538-44a0-4409-a13a-ffe71ff42ed6-cabundle-cert\") pod \"prestop-hook-test-kserve-69b4ffd8cb-lw4fp\" (UID: \"bd265538-44a0-4409-a13a-ffe71ff42ed6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-lw4fp"
Sep 16 20:00:25.136745 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:25.136453    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd265538-44a0-4409-a13a-ffe71ff42ed6-model-cache\") pod \"prestop-hook-test-kserve-69b4ffd8cb-lw4fp\" (UID: \"bd265538-44a0-4409-a13a-ffe71ff42ed6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-lw4fp"
Sep 16 20:00:25.136745 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:25.136557    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd265538-44a0-4409-a13a-ffe71ff42ed6-tmp-dir\") pod \"prestop-hook-test-kserve-69b4ffd8cb-lw4fp\" (UID: \"bd265538-44a0-4409-a13a-ffe71ff42ed6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-lw4fp"
Sep 16 20:00:25.136745 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:25.136639    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd265538-44a0-4409-a13a-ffe71ff42ed6-home\") pod \"prestop-hook-test-kserve-69b4ffd8cb-lw4fp\" (UID: \"bd265538-44a0-4409-a13a-ffe71ff42ed6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-lw4fp"
Sep 16 20:00:25.136910 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:25.136805    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd265538-44a0-4409-a13a-ffe71ff42ed6-kserve-provision-location\") pod \"prestop-hook-test-kserve-69b4ffd8cb-lw4fp\" (UID: \"bd265538-44a0-4409-a13a-ffe71ff42ed6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-lw4fp"
Sep 16 20:00:25.136910 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:25.136848    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd265538-44a0-4409-a13a-ffe71ff42ed6-model-cache\") pod \"prestop-hook-test-kserve-69b4ffd8cb-lw4fp\" (UID: \"bd265538-44a0-4409-a13a-ffe71ff42ed6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-lw4fp"
Sep 16 20:00:25.137351 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:25.137323    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd265538-44a0-4409-a13a-ffe71ff42ed6-cabundle-cert\") pod \"prestop-hook-test-kserve-69b4ffd8cb-lw4fp\" (UID: \"bd265538-44a0-4409-a13a-ffe71ff42ed6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-lw4fp"
Sep 16 20:00:25.138685 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:25.138659    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd265538-44a0-4409-a13a-ffe71ff42ed6-dshm\") pod \"prestop-hook-test-kserve-69b4ffd8cb-lw4fp\" (UID: \"bd265538-44a0-4409-a13a-ffe71ff42ed6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-lw4fp"
Sep 16 20:00:25.138852 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:25.138834    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd265538-44a0-4409-a13a-ffe71ff42ed6-tls-certs\") pod \"prestop-hook-test-kserve-69b4ffd8cb-lw4fp\" (UID: \"bd265538-44a0-4409-a13a-ffe71ff42ed6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-lw4fp"
Sep 16 20:00:25.145236 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:25.145213    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkz4r\" (UniqueName: \"kubernetes.io/projected/bd265538-44a0-4409-a13a-ffe71ff42ed6-kube-api-access-nkz4r\") pod \"prestop-hook-test-kserve-69b4ffd8cb-lw4fp\" (UID: \"bd265538-44a0-4409-a13a-ffe71ff42ed6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-lw4fp"
Sep 16 20:00:25.252354 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:25.252245    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-lw4fp"
Sep 16 20:00:25.529403 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:25.529357    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-lw4fp"]
Sep 16 20:00:25.532269 ip-10-0-129-194 kubenswrapper[2582]: W0916 20:00:25.532234    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd265538_44a0_4409_a13a_ffe71ff42ed6.slice/crio-ece37d842dad5d7c53c7ab1ea861df78cd5de8841f3f9c131a147939f5557558 WatchSource:0}: Error finding container ece37d842dad5d7c53c7ab1ea861df78cd5de8841f3f9c131a147939f5557558: Status 404 returned error can't find the container with id ece37d842dad5d7c53c7ab1ea861df78cd5de8841f3f9c131a147939f5557558
Sep 16 20:00:26.342375 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:26.342327    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-lw4fp" event={"ID":"bd265538-44a0-4409-a13a-ffe71ff42ed6","Type":"ContainerStarted","Data":"74a97ffe14e37ccf3ffe8de22e7bdc158e4feca048bac4be497bc7130c565279"}
Sep 16 20:00:26.342796 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:26.342414    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-lw4fp" event={"ID":"bd265538-44a0-4409-a13a-ffe71ff42ed6","Type":"ContainerStarted","Data":"ece37d842dad5d7c53c7ab1ea861df78cd5de8841f3f9c131a147939f5557558"}
Sep 16 20:00:28.352481 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:28.352449    2582 generic.go:356] "Generic (PLEG): container finished" podID="bd265538-44a0-4409-a13a-ffe71ff42ed6" containerID="74a97ffe14e37ccf3ffe8de22e7bdc158e4feca048bac4be497bc7130c565279" exitCode=0
Sep 16 20:00:28.352876 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:28.352525    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-lw4fp" event={"ID":"bd265538-44a0-4409-a13a-ffe71ff42ed6","Type":"ContainerDied","Data":"74a97ffe14e37ccf3ffe8de22e7bdc158e4feca048bac4be497bc7130c565279"}
Sep 16 20:00:29.359181 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:29.359143    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-lw4fp" event={"ID":"bd265538-44a0-4409-a13a-ffe71ff42ed6","Type":"ContainerStarted","Data":"d7af743a9a45f45d47414a4a8080be94836fdf815ea203beeaf9ebe93df4a839"}
Sep 16 20:00:29.385305 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:29.385251    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-lw4fp" podStartSLOduration=3.294730532 podStartE2EDuration="5.38523604s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.090505508s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:00:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:00:29.384210468 +0000 UTC m=+4633.025926133" watchObservedRunningTime="2026-09-16 20:00:29.38523604 +0000 UTC m=+4633.026951703"
Sep 16 20:00:30.252874 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:30.252837    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-lw4fp"
Sep 16 20:00:35.253292 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:35.253255    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-lw4fp"
Sep 16 20:00:35.255196 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:35.255160    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-lw4fp" podUID="bd265538-44a0-4409-a13a-ffe71ff42ed6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 16 20:00:39.933645 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:39.933599    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-89dd8" podUID="e056b4e3-e914-4dee-8202-8df521d4a398" containerName="main" containerID="cri-o://c4ffabbc8a8452b40764f73ea69028a6bbbcd02d382aa829276cda608c7de6a8" gracePeriod=45
Sep 16 20:00:40.679428 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:40.679402    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-89dd8"
Sep 16 20:00:40.772230 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:40.772200    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ntp6h\" (UniqueName: \"kubernetes.io/projected/e056b4e3-e914-4dee-8202-8df521d4a398-kube-api-access-ntp6h\") pod \"e056b4e3-e914-4dee-8202-8df521d4a398\" (UID: \"e056b4e3-e914-4dee-8202-8df521d4a398\") "
Sep 16 20:00:40.772411 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:40.772259    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e056b4e3-e914-4dee-8202-8df521d4a398-cabundle-cert\") pod \"e056b4e3-e914-4dee-8202-8df521d4a398\" (UID: \"e056b4e3-e914-4dee-8202-8df521d4a398\") "
Sep 16 20:00:40.772411 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:40.772281    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e056b4e3-e914-4dee-8202-8df521d4a398-home\") pod \"e056b4e3-e914-4dee-8202-8df521d4a398\" (UID: \"e056b4e3-e914-4dee-8202-8df521d4a398\") "
Sep 16 20:00:40.772411 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:40.772301    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e056b4e3-e914-4dee-8202-8df521d4a398-tmp-dir\") pod \"e056b4e3-e914-4dee-8202-8df521d4a398\" (UID: \"e056b4e3-e914-4dee-8202-8df521d4a398\") "
Sep 16 20:00:40.772595 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:40.772434    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e056b4e3-e914-4dee-8202-8df521d4a398-kserve-provision-location\") pod \"e056b4e3-e914-4dee-8202-8df521d4a398\" (UID: \"e056b4e3-e914-4dee-8202-8df521d4a398\") "
Sep 16 20:00:40.772595 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:40.772478    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e056b4e3-e914-4dee-8202-8df521d4a398-dshm\") pod \"e056b4e3-e914-4dee-8202-8df521d4a398\" (UID: \"e056b4e3-e914-4dee-8202-8df521d4a398\") "
Sep 16 20:00:40.772595 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:40.772559    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e056b4e3-e914-4dee-8202-8df521d4a398-model-cache\") pod \"e056b4e3-e914-4dee-8202-8df521d4a398\" (UID: \"e056b4e3-e914-4dee-8202-8df521d4a398\") "
Sep 16 20:00:40.772791 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:40.772633    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e056b4e3-e914-4dee-8202-8df521d4a398-tls-certs\") pod \"e056b4e3-e914-4dee-8202-8df521d4a398\" (UID: \"e056b4e3-e914-4dee-8202-8df521d4a398\") "
Sep 16 20:00:40.772791 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:40.772658    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e056b4e3-e914-4dee-8202-8df521d4a398-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e056b4e3-e914-4dee-8202-8df521d4a398" (UID: "e056b4e3-e914-4dee-8202-8df521d4a398"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:00:40.772926 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:40.772906    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e056b4e3-e914-4dee-8202-8df521d4a398-cabundle-cert\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 20:00:40.773958 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:40.772979    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e056b4e3-e914-4dee-8202-8df521d4a398-home" (OuterVolumeSpecName: "home") pod "e056b4e3-e914-4dee-8202-8df521d4a398" (UID: "e056b4e3-e914-4dee-8202-8df521d4a398"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:00:40.773958 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:40.773360    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e056b4e3-e914-4dee-8202-8df521d4a398-model-cache" (OuterVolumeSpecName: "model-cache") pod "e056b4e3-e914-4dee-8202-8df521d4a398" (UID: "e056b4e3-e914-4dee-8202-8df521d4a398"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:00:40.775132 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:40.775105    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e056b4e3-e914-4dee-8202-8df521d4a398-kube-api-access-ntp6h" (OuterVolumeSpecName: "kube-api-access-ntp6h") pod "e056b4e3-e914-4dee-8202-8df521d4a398" (UID: "e056b4e3-e914-4dee-8202-8df521d4a398"). InnerVolumeSpecName "kube-api-access-ntp6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:00:40.775601 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:40.775567    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e056b4e3-e914-4dee-8202-8df521d4a398-dshm" (OuterVolumeSpecName: "dshm") pod "e056b4e3-e914-4dee-8202-8df521d4a398" (UID: "e056b4e3-e914-4dee-8202-8df521d4a398"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:00:40.775688 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:40.775658    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e056b4e3-e914-4dee-8202-8df521d4a398-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e056b4e3-e914-4dee-8202-8df521d4a398" (UID: "e056b4e3-e914-4dee-8202-8df521d4a398"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:00:40.787025 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:40.786999    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e056b4e3-e914-4dee-8202-8df521d4a398-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e056b4e3-e914-4dee-8202-8df521d4a398" (UID: "e056b4e3-e914-4dee-8202-8df521d4a398"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:00:40.797198 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:40.797171    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e056b4e3-e914-4dee-8202-8df521d4a398-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e056b4e3-e914-4dee-8202-8df521d4a398" (UID: "e056b4e3-e914-4dee-8202-8df521d4a398"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:00:40.873881 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:40.873847    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e056b4e3-e914-4dee-8202-8df521d4a398-tls-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 20:00:40.873881 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:40.873881    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntp6h\" (UniqueName: \"kubernetes.io/projected/e056b4e3-e914-4dee-8202-8df521d4a398-kube-api-access-ntp6h\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 20:00:40.874072 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:40.873896    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e056b4e3-e914-4dee-8202-8df521d4a398-home\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 20:00:40.874072 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:40.873911    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e056b4e3-e914-4dee-8202-8df521d4a398-tmp-dir\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 20:00:40.874072 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:40.873925    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e056b4e3-e914-4dee-8202-8df521d4a398-kserve-provision-location\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 20:00:40.874072 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:40.873937    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e056b4e3-e914-4dee-8202-8df521d4a398-dshm\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 20:00:40.874072 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:40.873949    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e056b4e3-e914-4dee-8202-8df521d4a398-model-cache\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 20:00:41.415100 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:41.415065    2582 generic.go:356] "Generic (PLEG): container finished" podID="e056b4e3-e914-4dee-8202-8df521d4a398" containerID="c4ffabbc8a8452b40764f73ea69028a6bbbcd02d382aa829276cda608c7de6a8" exitCode=0
Sep 16 20:00:41.415507 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:41.415111    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-89dd8" event={"ID":"e056b4e3-e914-4dee-8202-8df521d4a398","Type":"ContainerDied","Data":"c4ffabbc8a8452b40764f73ea69028a6bbbcd02d382aa829276cda608c7de6a8"}
Sep 16 20:00:41.415507 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:41.415120    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-89dd8"
Sep 16 20:00:41.415507 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:41.415147    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-89dd8" event={"ID":"e056b4e3-e914-4dee-8202-8df521d4a398","Type":"ContainerDied","Data":"82fb4f2fd33e9c2a322e03d1e9440c4018b92435e56c96b8d9266952504bac11"}
Sep 16 20:00:41.415507 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:41.415168    2582 scope.go:117] "RemoveContainer" containerID="c4ffabbc8a8452b40764f73ea69028a6bbbcd02d382aa829276cda608c7de6a8"
Sep 16 20:00:41.423828 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:41.423777    2582 scope.go:117] "RemoveContainer" containerID="960ad7857958432a2af5b878c913d58282e830fc9c82045747bf200e7bfc7dcc"
Sep 16 20:00:41.431610 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:41.431588    2582 scope.go:117] "RemoveContainer" containerID="c4ffabbc8a8452b40764f73ea69028a6bbbcd02d382aa829276cda608c7de6a8"
Sep 16 20:00:41.431935 ip-10-0-129-194 kubenswrapper[2582]: E0916 20:00:41.431915    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4ffabbc8a8452b40764f73ea69028a6bbbcd02d382aa829276cda608c7de6a8\": container with ID starting with c4ffabbc8a8452b40764f73ea69028a6bbbcd02d382aa829276cda608c7de6a8 not found: ID does not exist" containerID="c4ffabbc8a8452b40764f73ea69028a6bbbcd02d382aa829276cda608c7de6a8"
Sep 16 20:00:41.432015 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:41.431944    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4ffabbc8a8452b40764f73ea69028a6bbbcd02d382aa829276cda608c7de6a8"} err="failed to get container status \"c4ffabbc8a8452b40764f73ea69028a6bbbcd02d382aa829276cda608c7de6a8\": rpc error: code = NotFound desc = could not find container \"c4ffabbc8a8452b40764f73ea69028a6bbbcd02d382aa829276cda608c7de6a8\": container with ID starting with c4ffabbc8a8452b40764f73ea69028a6bbbcd02d382aa829276cda608c7de6a8 not found: ID does not exist"
Sep 16 20:00:41.432015 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:41.431963    2582 scope.go:117] "RemoveContainer" containerID="960ad7857958432a2af5b878c913d58282e830fc9c82045747bf200e7bfc7dcc"
Sep 16 20:00:41.432224 ip-10-0-129-194 kubenswrapper[2582]: E0916 20:00:41.432205    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"960ad7857958432a2af5b878c913d58282e830fc9c82045747bf200e7bfc7dcc\": container with ID starting with 960ad7857958432a2af5b878c913d58282e830fc9c82045747bf200e7bfc7dcc not found: ID does not exist" containerID="960ad7857958432a2af5b878c913d58282e830fc9c82045747bf200e7bfc7dcc"
Sep 16 20:00:41.432281 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:41.432234    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"960ad7857958432a2af5b878c913d58282e830fc9c82045747bf200e7bfc7dcc"} err="failed to get container status \"960ad7857958432a2af5b878c913d58282e830fc9c82045747bf200e7bfc7dcc\": rpc error: code = NotFound desc = could not find container \"960ad7857958432a2af5b878c913d58282e830fc9c82045747bf200e7bfc7dcc\": container with ID starting with 960ad7857958432a2af5b878c913d58282e830fc9c82045747bf200e7bfc7dcc not found: ID does not exist"
Sep 16 20:00:41.438403 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:41.438379    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-89dd8"]
Sep 16 20:00:41.443846 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:41.443825    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-89dd8"]
Sep 16 20:00:42.873971 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:42.873929    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e056b4e3-e914-4dee-8202-8df521d4a398" path="/var/lib/kubelet/pods/e056b4e3-e914-4dee-8202-8df521d4a398/volumes"
Sep 16 20:00:43.325830 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:43.325797    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-lw4fp"]
Sep 16 20:00:58.348697 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:00:58.348632    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-lw4fp" podUID="bd265538-44a0-4409-a13a-ffe71ff42ed6" containerName="main" containerID="cri-o://d7af743a9a45f45d47414a4a8080be94836fdf815ea203beeaf9ebe93df4a839" gracePeriod=45
Sep 16 20:01:43.695365 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:01:43.695290    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-69b4ffd8cb-lw4fp_bd265538-44a0-4409-a13a-ffe71ff42ed6/main/0.log"
Sep 16 20:01:43.695829 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:01:43.695641    2582 generic.go:356] "Generic (PLEG): container finished" podID="bd265538-44a0-4409-a13a-ffe71ff42ed6" containerID="d7af743a9a45f45d47414a4a8080be94836fdf815ea203beeaf9ebe93df4a839" exitCode=137
Sep 16 20:01:43.695829 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:01:43.695695    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-lw4fp" event={"ID":"bd265538-44a0-4409-a13a-ffe71ff42ed6","Type":"ContainerDied","Data":"d7af743a9a45f45d47414a4a8080be94836fdf815ea203beeaf9ebe93df4a839"}
Sep 16 20:01:43.996093 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:01:43.996070    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-69b4ffd8cb-lw4fp_bd265538-44a0-4409-a13a-ffe71ff42ed6/main/0.log"
Sep 16 20:01:43.998805 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:01:43.998789    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-lw4fp"
Sep 16 20:01:44.015322 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:01:44.015301    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nkz4r\" (UniqueName: \"kubernetes.io/projected/bd265538-44a0-4409-a13a-ffe71ff42ed6-kube-api-access-nkz4r\") pod \"bd265538-44a0-4409-a13a-ffe71ff42ed6\" (UID: \"bd265538-44a0-4409-a13a-ffe71ff42ed6\") "
Sep 16 20:01:44.015421 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:01:44.015344    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd265538-44a0-4409-a13a-ffe71ff42ed6-tmp-dir\") pod \"bd265538-44a0-4409-a13a-ffe71ff42ed6\" (UID: \"bd265538-44a0-4409-a13a-ffe71ff42ed6\") "
Sep 16 20:01:44.015421 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:01:44.015380    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd265538-44a0-4409-a13a-ffe71ff42ed6-kserve-provision-location\") pod \"bd265538-44a0-4409-a13a-ffe71ff42ed6\" (UID: \"bd265538-44a0-4409-a13a-ffe71ff42ed6\") "
Sep 16 20:01:44.015513 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:01:44.015418    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd265538-44a0-4409-a13a-ffe71ff42ed6-dshm\") pod \"bd265538-44a0-4409-a13a-ffe71ff42ed6\" (UID: \"bd265538-44a0-4409-a13a-ffe71ff42ed6\") "
Sep 16 20:01:44.015513 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:01:44.015460    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd265538-44a0-4409-a13a-ffe71ff42ed6-tls-certs\") pod \"bd265538-44a0-4409-a13a-ffe71ff42ed6\" (UID: \"bd265538-44a0-4409-a13a-ffe71ff42ed6\") "
Sep 16 20:01:44.015513 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:01:44.015491    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd265538-44a0-4409-a13a-ffe71ff42ed6-cabundle-cert\") pod \"bd265538-44a0-4409-a13a-ffe71ff42ed6\" (UID: \"bd265538-44a0-4409-a13a-ffe71ff42ed6\") "
Sep 16 20:01:44.015679 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:01:44.015516    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd265538-44a0-4409-a13a-ffe71ff42ed6-model-cache\") pod \"bd265538-44a0-4409-a13a-ffe71ff42ed6\" (UID: \"bd265538-44a0-4409-a13a-ffe71ff42ed6\") "
Sep 16 20:01:44.015679 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:01:44.015548    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd265538-44a0-4409-a13a-ffe71ff42ed6-home\") pod \"bd265538-44a0-4409-a13a-ffe71ff42ed6\" (UID: \"bd265538-44a0-4409-a13a-ffe71ff42ed6\") "
Sep 16 20:01:44.016024 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:01:44.015899    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd265538-44a0-4409-a13a-ffe71ff42ed6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bd265538-44a0-4409-a13a-ffe71ff42ed6" (UID: "bd265538-44a0-4409-a13a-ffe71ff42ed6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:01:44.016024 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:01:44.016007    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd265538-44a0-4409-a13a-ffe71ff42ed6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bd265538-44a0-4409-a13a-ffe71ff42ed6" (UID: "bd265538-44a0-4409-a13a-ffe71ff42ed6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:01:44.016239 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:01:44.016214    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd265538-44a0-4409-a13a-ffe71ff42ed6-model-cache" (OuterVolumeSpecName: "model-cache") pod "bd265538-44a0-4409-a13a-ffe71ff42ed6" (UID: "bd265538-44a0-4409-a13a-ffe71ff42ed6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:01:44.016327 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:01:44.016246    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd265538-44a0-4409-a13a-ffe71ff42ed6-home" (OuterVolumeSpecName: "home") pod "bd265538-44a0-4409-a13a-ffe71ff42ed6" (UID: "bd265538-44a0-4409-a13a-ffe71ff42ed6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:01:44.017702 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:01:44.017672    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd265538-44a0-4409-a13a-ffe71ff42ed6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bd265538-44a0-4409-a13a-ffe71ff42ed6" (UID: "bd265538-44a0-4409-a13a-ffe71ff42ed6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:01:44.017702 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:01:44.017691    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd265538-44a0-4409-a13a-ffe71ff42ed6-kube-api-access-nkz4r" (OuterVolumeSpecName: "kube-api-access-nkz4r") pod "bd265538-44a0-4409-a13a-ffe71ff42ed6" (UID: "bd265538-44a0-4409-a13a-ffe71ff42ed6"). InnerVolumeSpecName "kube-api-access-nkz4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:01:44.017985 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:01:44.017967    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd265538-44a0-4409-a13a-ffe71ff42ed6-dshm" (OuterVolumeSpecName: "dshm") pod "bd265538-44a0-4409-a13a-ffe71ff42ed6" (UID: "bd265538-44a0-4409-a13a-ffe71ff42ed6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:01:44.040381 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:01:44.040339    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd265538-44a0-4409-a13a-ffe71ff42ed6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bd265538-44a0-4409-a13a-ffe71ff42ed6" (UID: "bd265538-44a0-4409-a13a-ffe71ff42ed6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:01:44.116426 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:01:44.116395    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd265538-44a0-4409-a13a-ffe71ff42ed6-tmp-dir\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 20:01:44.116426 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:01:44.116422    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd265538-44a0-4409-a13a-ffe71ff42ed6-kserve-provision-location\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 20:01:44.116426 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:01:44.116432    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd265538-44a0-4409-a13a-ffe71ff42ed6-dshm\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 20:01:44.116640 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:01:44.116440    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd265538-44a0-4409-a13a-ffe71ff42ed6-tls-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 20:01:44.116640 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:01:44.116449    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd265538-44a0-4409-a13a-ffe71ff42ed6-cabundle-cert\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 20:01:44.116640 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:01:44.116458    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd265538-44a0-4409-a13a-ffe71ff42ed6-model-cache\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 20:01:44.116640 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:01:44.116466    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd265538-44a0-4409-a13a-ffe71ff42ed6-home\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 20:01:44.116640 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:01:44.116474    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkz4r\" (UniqueName: \"kubernetes.io/projected/bd265538-44a0-4409-a13a-ffe71ff42ed6-kube-api-access-nkz4r\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 20:01:44.701449 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:01:44.701419    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-69b4ffd8cb-lw4fp_bd265538-44a0-4409-a13a-ffe71ff42ed6/main/0.log"
Sep 16 20:01:44.701908 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:01:44.701860    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-lw4fp" event={"ID":"bd265538-44a0-4409-a13a-ffe71ff42ed6","Type":"ContainerDied","Data":"ece37d842dad5d7c53c7ab1ea861df78cd5de8841f3f9c131a147939f5557558"}
Sep 16 20:01:44.701953 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:01:44.701912    2582 scope.go:117] "RemoveContainer" containerID="d7af743a9a45f45d47414a4a8080be94836fdf815ea203beeaf9ebe93df4a839"
Sep 16 20:01:44.701953 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:01:44.701867    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-lw4fp"
Sep 16 20:01:44.710968 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:01:44.710949    2582 scope.go:117] "RemoveContainer" containerID="74a97ffe14e37ccf3ffe8de22e7bdc158e4feca048bac4be497bc7130c565279"
Sep 16 20:01:44.725480 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:01:44.725457    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-lw4fp"]
Sep 16 20:01:44.729023 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:01:44.728999    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-69b4ffd8cb-lw4fp"]
Sep 16 20:01:44.872441 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:01:44.872411    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd265538-44a0-4409-a13a-ffe71ff42ed6" path="/var/lib/kubelet/pods/bd265538-44a0-4409-a13a-ffe71ff42ed6/volumes"
Sep 16 20:02:02.405561 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:02.405527    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx29j"]
Sep 16 20:02:02.406044 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:02.405851    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e056b4e3-e914-4dee-8202-8df521d4a398" containerName="storage-initializer"
Sep 16 20:02:02.406044 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:02.405864    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e056b4e3-e914-4dee-8202-8df521d4a398" containerName="storage-initializer"
Sep 16 20:02:02.406044 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:02.405872    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd265538-44a0-4409-a13a-ffe71ff42ed6" containerName="storage-initializer"
Sep 16 20:02:02.406044 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:02.405878    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd265538-44a0-4409-a13a-ffe71ff42ed6" containerName="storage-initializer"
Sep 16 20:02:02.406044 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:02.405887    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd265538-44a0-4409-a13a-ffe71ff42ed6" containerName="main"
Sep 16 20:02:02.406044 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:02.405893    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd265538-44a0-4409-a13a-ffe71ff42ed6" containerName="main"
Sep 16 20:02:02.406044 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:02.405909    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e056b4e3-e914-4dee-8202-8df521d4a398" containerName="main"
Sep 16 20:02:02.406044 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:02.405915    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e056b4e3-e914-4dee-8202-8df521d4a398" containerName="main"
Sep 16 20:02:02.406044 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:02.405977    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd265538-44a0-4409-a13a-ffe71ff42ed6" containerName="main"
Sep 16 20:02:02.406044 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:02.405986    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e056b4e3-e914-4dee-8202-8df521d4a398" containerName="main"
Sep 16 20:02:02.410384 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:02.410365    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx29j"
Sep 16 20:02:02.413132 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:02.413106    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 16 20:02:02.414269 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:02.414243    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-c294d\""
Sep 16 20:02:02.414390 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:02.414269    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 16 20:02:02.414390 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:02.414279    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 16 20:02:02.418577 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:02.418553    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx29j"]
Sep 16 20:02:02.468797 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:02.468754    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad422317-3573-4861-97da-f7cd470dd295-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-gx29j\" (UID: \"ad422317-3573-4861-97da-f7cd470dd295\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx29j"
Sep 16 20:02:02.468980 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:02.468807    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gzq2\" (UniqueName: \"kubernetes.io/projected/ad422317-3573-4861-97da-f7cd470dd295-kube-api-access-6gzq2\") pod \"rolling-upgrade-test-kserve-74b79d7f77-gx29j\" (UID: \"ad422317-3573-4861-97da-f7cd470dd295\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx29j"
Sep 16 20:02:02.468980 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:02.468889    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad422317-3573-4861-97da-f7cd470dd295-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-gx29j\" (UID: \"ad422317-3573-4861-97da-f7cd470dd295\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx29j"
Sep 16 20:02:02.468980 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:02.468913    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad422317-3573-4861-97da-f7cd470dd295-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-gx29j\" (UID: \"ad422317-3573-4861-97da-f7cd470dd295\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx29j"
Sep 16 20:02:02.468980 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:02.468941    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad422317-3573-4861-97da-f7cd470dd295-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-gx29j\" (UID: \"ad422317-3573-4861-97da-f7cd470dd295\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx29j"
Sep 16 20:02:02.469183 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:02.469005    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad422317-3573-4861-97da-f7cd470dd295-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-gx29j\" (UID: \"ad422317-3573-4861-97da-f7cd470dd295\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx29j"
Sep 16 20:02:02.570043 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:02.569996    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad422317-3573-4861-97da-f7cd470dd295-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-gx29j\" (UID: \"ad422317-3573-4861-97da-f7cd470dd295\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx29j"
Sep 16 20:02:02.570043 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:02.570038    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad422317-3573-4861-97da-f7cd470dd295-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-gx29j\" (UID: \"ad422317-3573-4861-97da-f7cd470dd295\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx29j"
Sep 16 20:02:02.570303 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:02.570089    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad422317-3573-4861-97da-f7cd470dd295-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-gx29j\" (UID: \"ad422317-3573-4861-97da-f7cd470dd295\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx29j"
Sep 16 20:02:02.570303 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:02.570107    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gzq2\" (UniqueName: \"kubernetes.io/projected/ad422317-3573-4861-97da-f7cd470dd295-kube-api-access-6gzq2\") pod \"rolling-upgrade-test-kserve-74b79d7f77-gx29j\" (UID: \"ad422317-3573-4861-97da-f7cd470dd295\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx29j"
Sep 16 20:02:02.570303 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:02.570152    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad422317-3573-4861-97da-f7cd470dd295-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-gx29j\" (UID: \"ad422317-3573-4861-97da-f7cd470dd295\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx29j"
Sep 16 20:02:02.570303 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:02.570176    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad422317-3573-4861-97da-f7cd470dd295-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-gx29j\" (UID: \"ad422317-3573-4861-97da-f7cd470dd295\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx29j"
Sep 16 20:02:02.570515 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:02.570495    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad422317-3573-4861-97da-f7cd470dd295-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-gx29j\" (UID: \"ad422317-3573-4861-97da-f7cd470dd295\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx29j"
Sep 16 20:02:02.570572 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:02.570529    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad422317-3573-4861-97da-f7cd470dd295-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-gx29j\" (UID: \"ad422317-3573-4861-97da-f7cd470dd295\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx29j"
Sep 16 20:02:02.571011 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:02.570984    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad422317-3573-4861-97da-f7cd470dd295-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-gx29j\" (UID: \"ad422317-3573-4861-97da-f7cd470dd295\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx29j"
Sep 16 20:02:02.572850 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:02.572821    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad422317-3573-4861-97da-f7cd470dd295-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-gx29j\" (UID: \"ad422317-3573-4861-97da-f7cd470dd295\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx29j"
Sep 16 20:02:02.572850 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:02.572840    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad422317-3573-4861-97da-f7cd470dd295-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-gx29j\" (UID: \"ad422317-3573-4861-97da-f7cd470dd295\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx29j"
Sep 16 20:02:02.579169 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:02.579138    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gzq2\" (UniqueName: \"kubernetes.io/projected/ad422317-3573-4861-97da-f7cd470dd295-kube-api-access-6gzq2\") pod \"rolling-upgrade-test-kserve-74b79d7f77-gx29j\" (UID: \"ad422317-3573-4861-97da-f7cd470dd295\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx29j"
Sep 16 20:02:02.721541 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:02.721446    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx29j"
Sep 16 20:02:02.995382 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:02.995300    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx29j"]
Sep 16 20:02:02.999002 ip-10-0-129-194 kubenswrapper[2582]: W0916 20:02:02.998970    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad422317_3573_4861_97da_f7cd470dd295.slice/crio-a148f42f0b811273ef6f6042e4ba0e84a578161ca505c4e55e865f6f2011c343 WatchSource:0}: Error finding container a148f42f0b811273ef6f6042e4ba0e84a578161ca505c4e55e865f6f2011c343: Status 404 returned error can't find the container with id a148f42f0b811273ef6f6042e4ba0e84a578161ca505c4e55e865f6f2011c343
Sep 16 20:02:03.783144 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:03.783106    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx29j" event={"ID":"ad422317-3573-4861-97da-f7cd470dd295","Type":"ContainerStarted","Data":"62a03120be5e433856aa61b1d85f94b8a3802e2ba530ebff259aa02bf3c6c669"}
Sep 16 20:02:03.783541 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:03.783158    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx29j" event={"ID":"ad422317-3573-4861-97da-f7cd470dd295","Type":"ContainerStarted","Data":"a148f42f0b811273ef6f6042e4ba0e84a578161ca505c4e55e865f6f2011c343"}
Sep 16 20:02:03.806830 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:03.806772    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx29j" podStartSLOduration=1.806757129 podStartE2EDuration="1.806757129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:02:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:02:03.80525969 +0000 UTC m=+4727.446975354" watchObservedRunningTime="2026-09-16 20:02:03.806757129 +0000 UTC m=+4727.448472792"
Sep 16 20:02:04.722151 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:04.722110    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx29j"
Sep 16 20:02:12.722186 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:12.722150    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx29j"
Sep 16 20:02:12.734736 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:12.734689    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx29j"
Sep 16 20:02:12.838543 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:12.838512    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx29j"
Sep 16 20:02:47.553092 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:47.553012    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx29j"]
Sep 16 20:02:47.562138 ip-10-0-129-194 kubenswrapper[2582]: E0916 20:02:47.562104    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 20:02:47.562138 ip-10-0-129-194 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:02:47.562138 ip-10-0-129-194 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 20:02:47.562138 ip-10-0-129-194 kubenswrapper[2582]:  > containerID="62a03120be5e433856aa61b1d85f94b8a3802e2ba530ebff259aa02bf3c6c669" cmd=["/bin/sleep","15"]
Sep 16 20:02:47.562413 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:47.562161    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx29j" message="" err=<
Sep 16 20:02:47.562413 ip-10-0-129-194 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:02:47.562413 ip-10-0-129-194 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 20:02:47.562413 ip-10-0-129-194 kubenswrapper[2582]:  >
Sep 16 20:02:47.562413 ip-10-0-129-194 kubenswrapper[2582]: E0916 20:02:47.562176    2582 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 20:02:47.562413 ip-10-0-129-194 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:02:47.562413 ip-10-0-129-194 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 20:02:47.562413 ip-10-0-129-194 kubenswrapper[2582]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx29j" podUID="ad422317-3573-4861-97da-f7cd470dd295" containerName="main" containerID="cri-o://62a03120be5e433856aa61b1d85f94b8a3802e2ba530ebff259aa02bf3c6c669"
Sep 16 20:02:47.562413 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:47.562209    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx29j" podUID="ad422317-3573-4861-97da-f7cd470dd295" containerName="main" containerID="cri-o://62a03120be5e433856aa61b1d85f94b8a3802e2ba530ebff259aa02bf3c6c669" gracePeriod=60
Sep 16 20:02:47.828421 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:47.828102    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx29j" podUID="ad422317-3573-4861-97da-f7cd470dd295" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Sep 16 20:02:47.903077 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:47.903053    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx29j"
Sep 16 20:02:47.969946 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:47.969909    2582 generic.go:356] "Generic (PLEG): container finished" podID="ad422317-3573-4861-97da-f7cd470dd295" containerID="62a03120be5e433856aa61b1d85f94b8a3802e2ba530ebff259aa02bf3c6c669" exitCode=0
Sep 16 20:02:47.970118 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:47.969960    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx29j"
Sep 16 20:02:47.970118 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:47.969998    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx29j" event={"ID":"ad422317-3573-4861-97da-f7cd470dd295","Type":"ContainerDied","Data":"62a03120be5e433856aa61b1d85f94b8a3802e2ba530ebff259aa02bf3c6c669"}
Sep 16 20:02:47.970118 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:47.970060    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx29j" event={"ID":"ad422317-3573-4861-97da-f7cd470dd295","Type":"ContainerDied","Data":"a148f42f0b811273ef6f6042e4ba0e84a578161ca505c4e55e865f6f2011c343"}
Sep 16 20:02:47.970118 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:47.970084    2582 scope.go:117] "RemoveContainer" containerID="62a03120be5e433856aa61b1d85f94b8a3802e2ba530ebff259aa02bf3c6c669"
Sep 16 20:02:47.978982 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:47.978961    2582 scope.go:117] "RemoveContainer" containerID="62a03120be5e433856aa61b1d85f94b8a3802e2ba530ebff259aa02bf3c6c669"
Sep 16 20:02:47.979244 ip-10-0-129-194 kubenswrapper[2582]: E0916 20:02:47.979225    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62a03120be5e433856aa61b1d85f94b8a3802e2ba530ebff259aa02bf3c6c669\": container with ID starting with 62a03120be5e433856aa61b1d85f94b8a3802e2ba530ebff259aa02bf3c6c669 not found: ID does not exist" containerID="62a03120be5e433856aa61b1d85f94b8a3802e2ba530ebff259aa02bf3c6c669"
Sep 16 20:02:47.979298 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:47.979255    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62a03120be5e433856aa61b1d85f94b8a3802e2ba530ebff259aa02bf3c6c669"} err="failed to get container status \"62a03120be5e433856aa61b1d85f94b8a3802e2ba530ebff259aa02bf3c6c669\": rpc error: code = NotFound desc = could not find container \"62a03120be5e433856aa61b1d85f94b8a3802e2ba530ebff259aa02bf3c6c669\": container with ID starting with 62a03120be5e433856aa61b1d85f94b8a3802e2ba530ebff259aa02bf3c6c669 not found: ID does not exist"
Sep 16 20:02:48.037631 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:48.037603    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad422317-3573-4861-97da-f7cd470dd295-model-cache\") pod \"ad422317-3573-4861-97da-f7cd470dd295\" (UID: \"ad422317-3573-4861-97da-f7cd470dd295\") "
Sep 16 20:02:48.037766 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:48.037637    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6gzq2\" (UniqueName: \"kubernetes.io/projected/ad422317-3573-4861-97da-f7cd470dd295-kube-api-access-6gzq2\") pod \"ad422317-3573-4861-97da-f7cd470dd295\" (UID: \"ad422317-3573-4861-97da-f7cd470dd295\") "
Sep 16 20:02:48.037766 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:48.037667    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad422317-3573-4861-97da-f7cd470dd295-home\") pod \"ad422317-3573-4861-97da-f7cd470dd295\" (UID: \"ad422317-3573-4861-97da-f7cd470dd295\") "
Sep 16 20:02:48.037766 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:48.037700    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad422317-3573-4861-97da-f7cd470dd295-tls-certs\") pod \"ad422317-3573-4861-97da-f7cd470dd295\" (UID: \"ad422317-3573-4861-97da-f7cd470dd295\") "
Sep 16 20:02:48.037923 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:48.037898    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad422317-3573-4861-97da-f7cd470dd295-tmp-dir\") pod \"ad422317-3573-4861-97da-f7cd470dd295\" (UID: \"ad422317-3573-4861-97da-f7cd470dd295\") "
Sep 16 20:02:48.037982 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:48.037944    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad422317-3573-4861-97da-f7cd470dd295-model-cache" (OuterVolumeSpecName: "model-cache") pod "ad422317-3573-4861-97da-f7cd470dd295" (UID: "ad422317-3573-4861-97da-f7cd470dd295"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:02:48.037982 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:48.037949    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad422317-3573-4861-97da-f7cd470dd295-home" (OuterVolumeSpecName: "home") pod "ad422317-3573-4861-97da-f7cd470dd295" (UID: "ad422317-3573-4861-97da-f7cd470dd295"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:02:48.038102 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:48.037987    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad422317-3573-4861-97da-f7cd470dd295-dshm\") pod \"ad422317-3573-4861-97da-f7cd470dd295\" (UID: \"ad422317-3573-4861-97da-f7cd470dd295\") "
Sep 16 20:02:48.038183 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:48.038149    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad422317-3573-4861-97da-f7cd470dd295-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ad422317-3573-4861-97da-f7cd470dd295" (UID: "ad422317-3573-4861-97da-f7cd470dd295"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:02:48.038298 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:48.038282    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad422317-3573-4861-97da-f7cd470dd295-model-cache\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 20:02:48.038374 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:48.038303    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad422317-3573-4861-97da-f7cd470dd295-home\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 20:02:48.038374 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:48.038317    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad422317-3573-4861-97da-f7cd470dd295-tmp-dir\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 20:02:48.039986 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:48.039959    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad422317-3573-4861-97da-f7cd470dd295-dshm" (OuterVolumeSpecName: "dshm") pod "ad422317-3573-4861-97da-f7cd470dd295" (UID: "ad422317-3573-4861-97da-f7cd470dd295"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:02:48.040082 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:48.040011    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad422317-3573-4861-97da-f7cd470dd295-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ad422317-3573-4861-97da-f7cd470dd295" (UID: "ad422317-3573-4861-97da-f7cd470dd295"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:02:48.040149 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:48.040126    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad422317-3573-4861-97da-f7cd470dd295-kube-api-access-6gzq2" (OuterVolumeSpecName: "kube-api-access-6gzq2") pod "ad422317-3573-4861-97da-f7cd470dd295" (UID: "ad422317-3573-4861-97da-f7cd470dd295"). InnerVolumeSpecName "kube-api-access-6gzq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:02:48.138980 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:48.138886    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad422317-3573-4861-97da-f7cd470dd295-tls-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 20:02:48.138980 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:48.138916    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad422317-3573-4861-97da-f7cd470dd295-dshm\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 20:02:48.138980 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:48.138929    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gzq2\" (UniqueName: \"kubernetes.io/projected/ad422317-3573-4861-97da-f7cd470dd295-kube-api-access-6gzq2\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 20:02:48.292932 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:48.292901    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx29j"]
Sep 16 20:02:48.296987 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:48.296958    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx29j"]
Sep 16 20:02:48.872981 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:02:48.872944    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad422317-3573-4861-97da-f7cd470dd295" path="/var/lib/kubelet/pods/ad422317-3573-4861-97da-f7cd470dd295/volumes"
Sep 16 20:03:17.214801 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:03:17.214771    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7qwl2_b6695f86-08c0-4836-87b1-f2a9d76084bd/console-operator/1.log"
Sep 16 20:03:17.222181 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:03:17.222157    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7qwl2_b6695f86-08c0-4836-87b1-f2a9d76084bd/console-operator/1.log"
Sep 16 20:04:00.376662 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:04:00.376628    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-g54jf/must-gather-jh4ft"]
Sep 16 20:04:00.377150 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:04:00.376946    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad422317-3573-4861-97da-f7cd470dd295" containerName="main"
Sep 16 20:04:00.377150 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:04:00.376959    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad422317-3573-4861-97da-f7cd470dd295" containerName="main"
Sep 16 20:04:00.377150 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:04:00.377024    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad422317-3573-4861-97da-f7cd470dd295" containerName="main"
Sep 16 20:04:00.379928 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:04:00.379911    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g54jf/must-gather-jh4ft"
Sep 16 20:04:00.382709 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:04:00.382687    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-g54jf\"/\"kube-root-ca.crt\""
Sep 16 20:04:00.382991 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:04:00.382970    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-g54jf\"/\"openshift-service-ca.crt\""
Sep 16 20:04:00.383895 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:04:00.383876    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-g54jf\"/\"default-dockercfg-b2g54\""
Sep 16 20:04:00.391093 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:04:00.391072    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g54jf/must-gather-jh4ft"]
Sep 16 20:04:00.526930 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:04:00.526889    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8d0cb6a0-c104-403a-8c26-0cd95ff9ad60-must-gather-output\") pod \"must-gather-jh4ft\" (UID: \"8d0cb6a0-c104-403a-8c26-0cd95ff9ad60\") " pod="openshift-must-gather-g54jf/must-gather-jh4ft"
Sep 16 20:04:00.527111 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:04:00.526947    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgnbl\" (UniqueName: \"kubernetes.io/projected/8d0cb6a0-c104-403a-8c26-0cd95ff9ad60-kube-api-access-bgnbl\") pod \"must-gather-jh4ft\" (UID: \"8d0cb6a0-c104-403a-8c26-0cd95ff9ad60\") " pod="openshift-must-gather-g54jf/must-gather-jh4ft"
Sep 16 20:04:00.627763 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:04:00.627644    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgnbl\" (UniqueName: \"kubernetes.io/projected/8d0cb6a0-c104-403a-8c26-0cd95ff9ad60-kube-api-access-bgnbl\") pod \"must-gather-jh4ft\" (UID: \"8d0cb6a0-c104-403a-8c26-0cd95ff9ad60\") " pod="openshift-must-gather-g54jf/must-gather-jh4ft"
Sep 16 20:04:00.627763 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:04:00.627711    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8d0cb6a0-c104-403a-8c26-0cd95ff9ad60-must-gather-output\") pod \"must-gather-jh4ft\" (UID: \"8d0cb6a0-c104-403a-8c26-0cd95ff9ad60\") " pod="openshift-must-gather-g54jf/must-gather-jh4ft"
Sep 16 20:04:00.628113 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:04:00.628087    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8d0cb6a0-c104-403a-8c26-0cd95ff9ad60-must-gather-output\") pod \"must-gather-jh4ft\" (UID: \"8d0cb6a0-c104-403a-8c26-0cd95ff9ad60\") " pod="openshift-must-gather-g54jf/must-gather-jh4ft"
Sep 16 20:04:00.639785 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:04:00.639753    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgnbl\" (UniqueName: \"kubernetes.io/projected/8d0cb6a0-c104-403a-8c26-0cd95ff9ad60-kube-api-access-bgnbl\") pod \"must-gather-jh4ft\" (UID: \"8d0cb6a0-c104-403a-8c26-0cd95ff9ad60\") " pod="openshift-must-gather-g54jf/must-gather-jh4ft"
Sep 16 20:04:00.688391 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:04:00.688360    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g54jf/must-gather-jh4ft"
Sep 16 20:04:00.957817 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:04:00.957747    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g54jf/must-gather-jh4ft"]
Sep 16 20:04:00.960095 ip-10-0-129-194 kubenswrapper[2582]: W0916 20:04:00.960068    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d0cb6a0_c104_403a_8c26_0cd95ff9ad60.slice/crio-4d050cf6cdfacd248152f7825da2b63f9d99acabd5c25818744700341ea3e825 WatchSource:0}: Error finding container 4d050cf6cdfacd248152f7825da2b63f9d99acabd5c25818744700341ea3e825: Status 404 returned error can't find the container with id 4d050cf6cdfacd248152f7825da2b63f9d99acabd5c25818744700341ea3e825
Sep 16 20:04:00.961972 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:04:00.961953    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 20:04:01.256960 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:04:01.256928    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g54jf/must-gather-jh4ft" event={"ID":"8d0cb6a0-c104-403a-8c26-0cd95ff9ad60","Type":"ContainerStarted","Data":"4d050cf6cdfacd248152f7825da2b63f9d99acabd5c25818744700341ea3e825"}
Sep 16 20:04:07.292022 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:04:07.291959    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g54jf/must-gather-jh4ft" event={"ID":"8d0cb6a0-c104-403a-8c26-0cd95ff9ad60","Type":"ContainerStarted","Data":"4a335210347fa0da335bf7494810dd1f23a0bd188e646145cd39fd02f180d480"}
Sep 16 20:04:08.297927 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:04:08.297889    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g54jf/must-gather-jh4ft" event={"ID":"8d0cb6a0-c104-403a-8c26-0cd95ff9ad60","Type":"ContainerStarted","Data":"e9978b06b161886ae18ccbf2b9ea8454c4fe9785c2b97fc23a0f104830a0e71b"}
Sep 16 20:04:08.329097 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:04:08.329038    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-g54jf/must-gather-jh4ft" podStartSLOduration=2.197959685 podStartE2EDuration="8.329021626s" totalImagesPullingTime="6.131061941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:04:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:04:08.326452937 +0000 UTC m=+4851.968168602" watchObservedRunningTime="2026-09-16 20:04:08.329021626 +0000 UTC m=+4851.970737291"
Sep 16 20:04:23.906230 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:04:23.906192    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt_949c2e66-950f-480a-96a3-16c9e4be467b/istio-proxy/0.log"
Sep 16 20:04:23.945020 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:04:23.944985    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt_949c2e66-950f-480a-96a3-16c9e4be467b/istio-proxy/1.log"
Sep 16 20:04:23.958082 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:04:23.958047    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7465b55b9d-njqmh_9ac2b8ef-ff60-4791-9e56-5ac64263bf1e/router/0.log"
Sep 16 20:04:24.883500 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:04:24.883471    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt_949c2e66-950f-480a-96a3-16c9e4be467b/istio-proxy/0.log"
Sep 16 20:04:24.921614 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:04:24.921582    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt_949c2e66-950f-480a-96a3-16c9e4be467b/istio-proxy/1.log"
Sep 16 20:04:24.935294 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:04:24.935266    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7465b55b9d-njqmh_9ac2b8ef-ff60-4791-9e56-5ac64263bf1e/router/0.log"
Sep 16 20:04:25.960600 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:04:25.960562    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt_949c2e66-950f-480a-96a3-16c9e4be467b/istio-proxy/0.log"
Sep 16 20:04:26.002589 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:04:26.002558    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt_949c2e66-950f-480a-96a3-16c9e4be467b/istio-proxy/1.log"
Sep 16 20:04:26.016510 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:04:26.016482    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7465b55b9d-njqmh_9ac2b8ef-ff60-4791-9e56-5ac64263bf1e/router/0.log"
Sep 16 20:04:26.921090 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:04:26.921060    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt_949c2e66-950f-480a-96a3-16c9e4be467b/istio-proxy/0.log"
Sep 16 20:04:26.959730 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:04:26.959690    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt_949c2e66-950f-480a-96a3-16c9e4be467b/istio-proxy/1.log"
Sep 16 20:04:26.973045 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:04:26.973013    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7465b55b9d-njqmh_9ac2b8ef-ff60-4791-9e56-5ac64263bf1e/router/0.log"
Sep 16 20:04:28.646101 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:04:28.646065    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-w8w2f_9d2b060c-f303-4dca-945c-42b6ef7a2b09/limitador/0.log"
Sep 16 20:04:30.566208 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:04:30.566177    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-w8w2f_9d2b060c-f303-4dca-945c-42b6ef7a2b09/limitador/0.log"
Sep 16 20:04:32.506085 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:04:32.506047    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-w8w2f_9d2b060c-f303-4dca-945c-42b6ef7a2b09/limitador/0.log"
Sep 16 20:04:34.324854 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:04:34.324817    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-w8w2f_9d2b060c-f303-4dca-945c-42b6ef7a2b09/limitador/0.log"
Sep 16 20:04:35.965977 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:04:35.965948    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-w8w2f_9d2b060c-f303-4dca-945c-42b6ef7a2b09/limitador/0.log"
Sep 16 20:04:37.640300 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:04:37.640269    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-w8w2f_9d2b060c-f303-4dca-945c-42b6ef7a2b09/limitador/0.log"
Sep 16 20:04:39.303914 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:04:39.303884    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-w8w2f_9d2b060c-f303-4dca-945c-42b6ef7a2b09/limitador/0.log"
Sep 16 20:04:41.051441 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:04:41.051401    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-w8w2f_9d2b060c-f303-4dca-945c-42b6ef7a2b09/limitador/0.log"
Sep 16 20:04:42.868316 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:04:42.868277    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-w8w2f_9d2b060c-f303-4dca-945c-42b6ef7a2b09/limitador/0.log"
Sep 16 20:04:44.491372 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:04:44.491335    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-w8w2f_9d2b060c-f303-4dca-945c-42b6ef7a2b09/limitador/0.log"
Sep 16 20:04:46.099096 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:04:46.099071    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-w8w2f_9d2b060c-f303-4dca-945c-42b6ef7a2b09/limitador/0.log"
Sep 16 20:04:47.813107 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:04:47.813069    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-w8w2f_9d2b060c-f303-4dca-945c-42b6ef7a2b09/limitador/0.log"
Sep 16 20:04:49.492615 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:04:49.492589    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-w8w2f_9d2b060c-f303-4dca-945c-42b6ef7a2b09/limitador/0.log"
Sep 16 20:04:51.100486 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:04:51.100458    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-w8w2f_9d2b060c-f303-4dca-945c-42b6ef7a2b09/limitador/0.log"
Sep 16 20:04:52.733773 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:04:52.733747    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-w8w2f_9d2b060c-f303-4dca-945c-42b6ef7a2b09/limitador/0.log"
Sep 16 20:04:54.452499 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:04:54.452470    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-w8w2f_9d2b060c-f303-4dca-945c-42b6ef7a2b09/limitador/0.log"
Sep 16 20:04:56.119260 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:04:56.119228    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-w8w2f_9d2b060c-f303-4dca-945c-42b6ef7a2b09/limitador/0.log"
Sep 16 20:04:57.823824 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:04:57.823794    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-w8w2f_9d2b060c-f303-4dca-945c-42b6ef7a2b09/limitador/0.log"
Sep 16 20:04:59.525150 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:04:59.525115    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-w8w2f_9d2b060c-f303-4dca-945c-42b6ef7a2b09/limitador/0.log"
Sep 16 20:05:01.206897 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:01.206853    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-w8w2f_9d2b060c-f303-4dca-945c-42b6ef7a2b09/limitador/0.log"
Sep 16 20:05:02.857384 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:02.857354    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-w8w2f_9d2b060c-f303-4dca-945c-42b6ef7a2b09/limitador/0.log"
Sep 16 20:05:04.632237 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:04.632205    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-w8w2f_9d2b060c-f303-4dca-945c-42b6ef7a2b09/limitador/0.log"
Sep 16 20:05:06.251582 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:06.251550    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-w8w2f_9d2b060c-f303-4dca-945c-42b6ef7a2b09/limitador/0.log"
Sep 16 20:05:07.883609 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:07.883581    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-w8w2f_9d2b060c-f303-4dca-945c-42b6ef7a2b09/limitador/0.log"
Sep 16 20:05:09.506215 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:09.506189    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-w8w2f_9d2b060c-f303-4dca-945c-42b6ef7a2b09/limitador/0.log"
Sep 16 20:05:10.589605 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:10.589567    2582 generic.go:356] "Generic (PLEG): container finished" podID="8d0cb6a0-c104-403a-8c26-0cd95ff9ad60" containerID="4a335210347fa0da335bf7494810dd1f23a0bd188e646145cd39fd02f180d480" exitCode=0
Sep 16 20:05:10.590109 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:10.589642    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g54jf/must-gather-jh4ft" event={"ID":"8d0cb6a0-c104-403a-8c26-0cd95ff9ad60","Type":"ContainerDied","Data":"4a335210347fa0da335bf7494810dd1f23a0bd188e646145cd39fd02f180d480"}
Sep 16 20:05:10.590109 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:10.590000    2582 scope.go:117] "RemoveContainer" containerID="4a335210347fa0da335bf7494810dd1f23a0bd188e646145cd39fd02f180d480"
Sep 16 20:05:11.039590 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:11.039548    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-g54jf_must-gather-jh4ft_8d0cb6a0-c104-403a-8c26-0cd95ff9ad60/gather/0.log"
Sep 16 20:05:14.772144 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:14.772107    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-tplxv_e6428035-ca16-4098-ac4e-1ca4931bb7a7/global-pull-secret-syncer/0.log"
Sep 16 20:05:14.886880 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:14.886854    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-pr4kj_bfffe95c-37f4-4aab-a7aa-19c64f66e810/konnectivity-agent/0.log"
Sep 16 20:05:14.969408 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:14.969372    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-194.ec2.internal_a8fee4f0c396919249581e1ee599be3f/haproxy/0.log"
Sep 16 20:05:16.516947 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:16.516909    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-g54jf/must-gather-jh4ft"]
Sep 16 20:05:16.517442 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:16.517105    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-g54jf/must-gather-jh4ft" podUID="8d0cb6a0-c104-403a-8c26-0cd95ff9ad60" containerName="copy" containerID="cri-o://e9978b06b161886ae18ccbf2b9ea8454c4fe9785c2b97fc23a0f104830a0e71b" gracePeriod=2
Sep 16 20:05:16.523009 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:16.522984    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-g54jf/must-gather-jh4ft"]
Sep 16 20:05:16.524478 ip-10-0-129-194 kubenswrapper[2582]: E0916 20:05:16.524445    2582 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"8d0cb6a0-c104-403a-8c26-0cd95ff9ad60\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"openshift-must-gather-g54jf\"/\"must-gather-jh4ft\": pods \"must-gather-jh4ft\" not found" pod="openshift-must-gather-g54jf/must-gather-jh4ft"
Sep 16 20:05:16.757485 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:16.757461    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-g54jf_must-gather-jh4ft_8d0cb6a0-c104-403a-8c26-0cd95ff9ad60/copy/0.log"
Sep 16 20:05:16.760081 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:16.760061    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g54jf/must-gather-jh4ft"
Sep 16 20:05:16.762320 ip-10-0-129-194 kubenswrapper[2582]: E0916 20:05:16.762299    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-jh4ft\" is forbidden: User \"system:node:ip-10-0-129-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-g54jf\": no relationship found between node 'ip-10-0-129-194.ec2.internal' and this object" podUID="8d0cb6a0-c104-403a-8c26-0cd95ff9ad60" pod="openshift-must-gather-g54jf/must-gather-jh4ft"
Sep 16 20:05:16.781457 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:16.781377    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8d0cb6a0-c104-403a-8c26-0cd95ff9ad60-must-gather-output\") pod \"8d0cb6a0-c104-403a-8c26-0cd95ff9ad60\" (UID: \"8d0cb6a0-c104-403a-8c26-0cd95ff9ad60\") "
Sep 16 20:05:16.781457 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:16.781452    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgnbl\" (UniqueName: \"kubernetes.io/projected/8d0cb6a0-c104-403a-8c26-0cd95ff9ad60-kube-api-access-bgnbl\") pod \"8d0cb6a0-c104-403a-8c26-0cd95ff9ad60\" (UID: \"8d0cb6a0-c104-403a-8c26-0cd95ff9ad60\") "
Sep 16 20:05:16.785746 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:16.784600    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d0cb6a0-c104-403a-8c26-0cd95ff9ad60-kube-api-access-bgnbl" (OuterVolumeSpecName: "kube-api-access-bgnbl") pod "8d0cb6a0-c104-403a-8c26-0cd95ff9ad60" (UID: "8d0cb6a0-c104-403a-8c26-0cd95ff9ad60"). InnerVolumeSpecName "kube-api-access-bgnbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:05:16.794529 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:16.794501    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d0cb6a0-c104-403a-8c26-0cd95ff9ad60-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "8d0cb6a0-c104-403a-8c26-0cd95ff9ad60" (UID: "8d0cb6a0-c104-403a-8c26-0cd95ff9ad60"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:05:16.872262 ip-10-0-129-194 kubenswrapper[2582]: E0916 20:05:16.872224    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-jh4ft\" is forbidden: User \"system:node:ip-10-0-129-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-g54jf\": no relationship found between node 'ip-10-0-129-194.ec2.internal' and this object" podUID="8d0cb6a0-c104-403a-8c26-0cd95ff9ad60" pod="openshift-must-gather-g54jf/must-gather-jh4ft"
Sep 16 20:05:16.873475 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:16.873452    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d0cb6a0-c104-403a-8c26-0cd95ff9ad60" path="/var/lib/kubelet/pods/8d0cb6a0-c104-403a-8c26-0cd95ff9ad60/volumes"
Sep 16 20:05:16.882956 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:16.882935    2582 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8d0cb6a0-c104-403a-8c26-0cd95ff9ad60-must-gather-output\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 20:05:16.883061 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:16.882958    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgnbl\" (UniqueName: \"kubernetes.io/projected/8d0cb6a0-c104-403a-8c26-0cd95ff9ad60-kube-api-access-bgnbl\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Sep 16 20:05:17.621613 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:17.621580    2582 generic.go:356] "Generic (PLEG): container finished" podID="8d0cb6a0-c104-403a-8c26-0cd95ff9ad60" containerID="e9978b06b161886ae18ccbf2b9ea8454c4fe9785c2b97fc23a0f104830a0e71b" exitCode=143
Sep 16 20:05:17.622092 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:17.621647    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g54jf/must-gather-jh4ft"
Sep 16 20:05:17.622092 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:17.621675    2582 scope.go:117] "RemoveContainer" containerID="e9978b06b161886ae18ccbf2b9ea8454c4fe9785c2b97fc23a0f104830a0e71b"
Sep 16 20:05:17.629352 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:17.629335    2582 scope.go:117] "RemoveContainer" containerID="4a335210347fa0da335bf7494810dd1f23a0bd188e646145cd39fd02f180d480"
Sep 16 20:05:17.642332 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:17.642315    2582 scope.go:117] "RemoveContainer" containerID="e9978b06b161886ae18ccbf2b9ea8454c4fe9785c2b97fc23a0f104830a0e71b"
Sep 16 20:05:17.642578 ip-10-0-129-194 kubenswrapper[2582]: E0916 20:05:17.642560    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9978b06b161886ae18ccbf2b9ea8454c4fe9785c2b97fc23a0f104830a0e71b\": container with ID starting with e9978b06b161886ae18ccbf2b9ea8454c4fe9785c2b97fc23a0f104830a0e71b not found: ID does not exist" containerID="e9978b06b161886ae18ccbf2b9ea8454c4fe9785c2b97fc23a0f104830a0e71b"
Sep 16 20:05:17.642622 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:17.642586    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9978b06b161886ae18ccbf2b9ea8454c4fe9785c2b97fc23a0f104830a0e71b"} err="failed to get container status \"e9978b06b161886ae18ccbf2b9ea8454c4fe9785c2b97fc23a0f104830a0e71b\": rpc error: code = NotFound desc = could not find container \"e9978b06b161886ae18ccbf2b9ea8454c4fe9785c2b97fc23a0f104830a0e71b\": container with ID starting with e9978b06b161886ae18ccbf2b9ea8454c4fe9785c2b97fc23a0f104830a0e71b not found: ID does not exist"
Sep 16 20:05:17.642622 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:17.642603    2582 scope.go:117] "RemoveContainer" containerID="4a335210347fa0da335bf7494810dd1f23a0bd188e646145cd39fd02f180d480"
Sep 16 20:05:17.642862 ip-10-0-129-194 kubenswrapper[2582]: E0916 20:05:17.642843    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a335210347fa0da335bf7494810dd1f23a0bd188e646145cd39fd02f180d480\": container with ID starting with 4a335210347fa0da335bf7494810dd1f23a0bd188e646145cd39fd02f180d480 not found: ID does not exist" containerID="4a335210347fa0da335bf7494810dd1f23a0bd188e646145cd39fd02f180d480"
Sep 16 20:05:17.642908 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:17.642868    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a335210347fa0da335bf7494810dd1f23a0bd188e646145cd39fd02f180d480"} err="failed to get container status \"4a335210347fa0da335bf7494810dd1f23a0bd188e646145cd39fd02f180d480\": rpc error: code = NotFound desc = could not find container \"4a335210347fa0da335bf7494810dd1f23a0bd188e646145cd39fd02f180d480\": container with ID starting with 4a335210347fa0da335bf7494810dd1f23a0bd188e646145cd39fd02f180d480 not found: ID does not exist"
Sep 16 20:05:20.266393 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:20.266360    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-w8w2f_9d2b060c-f303-4dca-945c-42b6ef7a2b09/limitador/0.log"
Sep 16 20:05:21.865748 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:21.865704    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-x6kvz_e040ac61-5af8-4092-9a00-ec184466deaf/cluster-monitoring-operator/0.log"
Sep 16 20:05:22.101649 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:22.101614    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4b825_ed2c35c3-6cab-4978-9ab5-958f4de177b6/node-exporter/0.log"
Sep 16 20:05:22.153561 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:22.153471    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4b825_ed2c35c3-6cab-4978-9ab5-958f4de177b6/kube-rbac-proxy/0.log"
Sep 16 20:05:22.235828 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:22.235803    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4b825_ed2c35c3-6cab-4978-9ab5-958f4de177b6/init-textfile/0.log"
Sep 16 20:05:24.519386 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:24.519352    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-l56nd_ba5d0219-c94f-41a7-a6fc-86b14d2bccd2/networking-console-plugin/0.log"
Sep 16 20:05:24.545220 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:24.545183    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qnxnw/perf-node-gather-daemonset-bsxcx"]
Sep 16 20:05:24.545509 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:24.545494    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d0cb6a0-c104-403a-8c26-0cd95ff9ad60" containerName="copy"
Sep 16 20:05:24.545509 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:24.545511    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d0cb6a0-c104-403a-8c26-0cd95ff9ad60" containerName="copy"
Sep 16 20:05:24.545602 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:24.545532    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d0cb6a0-c104-403a-8c26-0cd95ff9ad60" containerName="gather"
Sep 16 20:05:24.545602 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:24.545538    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d0cb6a0-c104-403a-8c26-0cd95ff9ad60" containerName="gather"
Sep 16 20:05:24.545602 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:24.545591    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d0cb6a0-c104-403a-8c26-0cd95ff9ad60" containerName="gather"
Sep 16 20:05:24.545602 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:24.545598    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d0cb6a0-c104-403a-8c26-0cd95ff9ad60" containerName="copy"
Sep 16 20:05:24.550873 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:24.550849    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-bsxcx"
Sep 16 20:05:24.553424 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:24.553404    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qnxnw\"/\"kube-root-ca.crt\""
Sep 16 20:05:24.553536 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:24.553437    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qnxnw\"/\"openshift-service-ca.crt\""
Sep 16 20:05:24.553536 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:24.553443    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qnxnw\"/\"default-dockercfg-kqdgr\""
Sep 16 20:05:24.559256 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:24.559236    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qnxnw/perf-node-gather-daemonset-bsxcx"]
Sep 16 20:05:24.646406 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:24.646364    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/150ec2e1-29c3-4bcf-af59-bdf8ab7a3ff4-lib-modules\") pod \"perf-node-gather-daemonset-bsxcx\" (UID: \"150ec2e1-29c3-4bcf-af59-bdf8ab7a3ff4\") " pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-bsxcx"
Sep 16 20:05:24.646599 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:24.646431    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/150ec2e1-29c3-4bcf-af59-bdf8ab7a3ff4-podres\") pod \"perf-node-gather-daemonset-bsxcx\" (UID: \"150ec2e1-29c3-4bcf-af59-bdf8ab7a3ff4\") " pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-bsxcx"
Sep 16 20:05:24.646599 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:24.646495    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/150ec2e1-29c3-4bcf-af59-bdf8ab7a3ff4-sys\") pod \"perf-node-gather-daemonset-bsxcx\" (UID: \"150ec2e1-29c3-4bcf-af59-bdf8ab7a3ff4\") " pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-bsxcx"
Sep 16 20:05:24.646599 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:24.646518    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/150ec2e1-29c3-4bcf-af59-bdf8ab7a3ff4-proc\") pod \"perf-node-gather-daemonset-bsxcx\" (UID: \"150ec2e1-29c3-4bcf-af59-bdf8ab7a3ff4\") " pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-bsxcx"
Sep 16 20:05:24.646599 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:24.646559    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8v46\" (UniqueName: \"kubernetes.io/projected/150ec2e1-29c3-4bcf-af59-bdf8ab7a3ff4-kube-api-access-d8v46\") pod \"perf-node-gather-daemonset-bsxcx\" (UID: \"150ec2e1-29c3-4bcf-af59-bdf8ab7a3ff4\") " pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-bsxcx"
Sep 16 20:05:24.747900 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:24.747859    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/150ec2e1-29c3-4bcf-af59-bdf8ab7a3ff4-podres\") pod \"perf-node-gather-daemonset-bsxcx\" (UID: \"150ec2e1-29c3-4bcf-af59-bdf8ab7a3ff4\") " pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-bsxcx"
Sep 16 20:05:24.748086 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:24.747914    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/150ec2e1-29c3-4bcf-af59-bdf8ab7a3ff4-sys\") pod \"perf-node-gather-daemonset-bsxcx\" (UID: \"150ec2e1-29c3-4bcf-af59-bdf8ab7a3ff4\") " pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-bsxcx"
Sep 16 20:05:24.748086 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:24.747933    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/150ec2e1-29c3-4bcf-af59-bdf8ab7a3ff4-proc\") pod \"perf-node-gather-daemonset-bsxcx\" (UID: \"150ec2e1-29c3-4bcf-af59-bdf8ab7a3ff4\") " pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-bsxcx"
Sep 16 20:05:24.748086 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:24.747952    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8v46\" (UniqueName: \"kubernetes.io/projected/150ec2e1-29c3-4bcf-af59-bdf8ab7a3ff4-kube-api-access-d8v46\") pod \"perf-node-gather-daemonset-bsxcx\" (UID: \"150ec2e1-29c3-4bcf-af59-bdf8ab7a3ff4\") " pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-bsxcx"
Sep 16 20:05:24.748086 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:24.747996    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/150ec2e1-29c3-4bcf-af59-bdf8ab7a3ff4-sys\") pod \"perf-node-gather-daemonset-bsxcx\" (UID: \"150ec2e1-29c3-4bcf-af59-bdf8ab7a3ff4\") " pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-bsxcx"
Sep 16 20:05:24.748086 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:24.748025    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/150ec2e1-29c3-4bcf-af59-bdf8ab7a3ff4-podres\") pod \"perf-node-gather-daemonset-bsxcx\" (UID: \"150ec2e1-29c3-4bcf-af59-bdf8ab7a3ff4\") " pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-bsxcx"
Sep 16 20:05:24.748086 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:24.748003    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/150ec2e1-29c3-4bcf-af59-bdf8ab7a3ff4-lib-modules\") pod \"perf-node-gather-daemonset-bsxcx\" (UID: \"150ec2e1-29c3-4bcf-af59-bdf8ab7a3ff4\") " pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-bsxcx"
Sep 16 20:05:24.748086 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:24.748055    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/150ec2e1-29c3-4bcf-af59-bdf8ab7a3ff4-proc\") pod \"perf-node-gather-daemonset-bsxcx\" (UID: \"150ec2e1-29c3-4bcf-af59-bdf8ab7a3ff4\") " pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-bsxcx"
Sep 16 20:05:24.748086 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:24.748089    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/150ec2e1-29c3-4bcf-af59-bdf8ab7a3ff4-lib-modules\") pod \"perf-node-gather-daemonset-bsxcx\" (UID: \"150ec2e1-29c3-4bcf-af59-bdf8ab7a3ff4\") " pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-bsxcx"
Sep 16 20:05:24.756424 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:24.756402    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8v46\" (UniqueName: \"kubernetes.io/projected/150ec2e1-29c3-4bcf-af59-bdf8ab7a3ff4-kube-api-access-d8v46\") pod \"perf-node-gather-daemonset-bsxcx\" (UID: \"150ec2e1-29c3-4bcf-af59-bdf8ab7a3ff4\") " pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-bsxcx"
Sep 16 20:05:24.861365 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:24.861277    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-bsxcx"
Sep 16 20:05:25.097766 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:25.097701    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7qwl2_b6695f86-08c0-4836-87b1-f2a9d76084bd/console-operator/1.log"
Sep 16 20:05:25.104707 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:25.104671    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7qwl2_b6695f86-08c0-4836-87b1-f2a9d76084bd/console-operator/2.log"
Sep 16 20:05:25.141679 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:25.141605    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qnxnw/perf-node-gather-daemonset-bsxcx"]
Sep 16 20:05:25.144705 ip-10-0-129-194 kubenswrapper[2582]: W0916 20:05:25.144678    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod150ec2e1_29c3_4bcf_af59_bdf8ab7a3ff4.slice/crio-e89255b8a030f50d73252d8a2cf5a21c4fc14661f3c4b7f1d833e7224e058d95 WatchSource:0}: Error finding container e89255b8a030f50d73252d8a2cf5a21c4fc14661f3c4b7f1d833e7224e058d95: Status 404 returned error can't find the container with id e89255b8a030f50d73252d8a2cf5a21c4fc14661f3c4b7f1d833e7224e058d95
Sep 16 20:05:25.655475 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:25.655420    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-bsxcx" event={"ID":"150ec2e1-29c3-4bcf-af59-bdf8ab7a3ff4","Type":"ContainerStarted","Data":"418e552569091ce6c729ff08f06615af71856ce3dce405f2181849875b8fe260"}
Sep 16 20:05:25.656016 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:25.655488    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-bsxcx" event={"ID":"150ec2e1-29c3-4bcf-af59-bdf8ab7a3ff4","Type":"ContainerStarted","Data":"e89255b8a030f50d73252d8a2cf5a21c4fc14661f3c4b7f1d833e7224e058d95"}
Sep 16 20:05:25.656016 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:25.655517    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-bsxcx"
Sep 16 20:05:25.686465 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:25.686413    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-bsxcx" podStartSLOduration=1.686394975 podStartE2EDuration="1.686394975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 20:05:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:05:25.684118264 +0000 UTC m=+4929.325833928" watchObservedRunningTime="2026-09-16 20:05:25.686394975 +0000 UTC m=+4929.328110640"
Sep 16 20:05:26.346576 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:26.346537    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-cfz6r_e4d69278-94eb-4b53-8967-c787e4575efd/volume-data-source-validator/0.log"
Sep 16 20:05:27.627049 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:27.627016    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nw2m8_d76fc261-072a-4625-818d-e8faa5710cbd/dns/0.log"
Sep 16 20:05:27.653051 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:27.653026    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nw2m8_d76fc261-072a-4625-818d-e8faa5710cbd/kube-rbac-proxy/0.log"
Sep 16 20:05:27.708278 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:27.708246    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wdq46_c60bca6d-3f8c-4725-94a2-eedd9a075586/dns-node-resolver/0.log"
Sep 16 20:05:28.459366 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:28.459332    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-w8t8m_870ff296-8673-44d5-9fc3-619486d607a1/node-ca/0.log"
Sep 16 20:05:29.835319 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:29.835291    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt_949c2e66-950f-480a-96a3-16c9e4be467b/istio-proxy/0.log"
Sep 16 20:05:29.867157 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:29.867125    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt_949c2e66-950f-480a-96a3-16c9e4be467b/istio-proxy/1.log"
Sep 16 20:05:29.910384 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:29.910351    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7465b55b9d-njqmh_9ac2b8ef-ff60-4791-9e56-5ac64263bf1e/router/0.log"
Sep 16 20:05:30.531910 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:30.531878    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wxss9_d5d54719-5c41-4b38-a2a1-367dc4bd6801/serve-healthcheck-canary/0.log"
Sep 16 20:05:30.996378 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:30.996289    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-h28sc_ed1b0dcc-4b23-4906-bc1d-4efa04e932cf/insights-operator/0.log"
Sep 16 20:05:30.998381 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:30.998360    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-h28sc_ed1b0dcc-4b23-4906-bc1d-4efa04e932cf/insights-operator/1.log"
Sep 16 20:05:31.027662 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:31.027631    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-627pg_e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339/kube-rbac-proxy/0.log"
Sep 16 20:05:31.057421 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:31.057371    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-627pg_e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339/exporter/0.log"
Sep 16 20:05:31.085031 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:31.084999    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-627pg_e603fb2e-bc3c-4c6e-9fa2-1c384bcc3339/extractor/0.log"
Sep 16 20:05:31.669197 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:31.669168    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-bsxcx"
Sep 16 20:05:34.652301 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:34.652266    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-77db69dd7c-4g645_5fdaf830-2e71-4373-aa18-0bcca69ea3e4/manager/0.log"
Sep 16 20:05:35.216417 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:35.216313    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt_949c2e66-950f-480a-96a3-16c9e4be467b/istio-proxy/0.log"
Sep 16 20:05:35.249095 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:35.249058    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt_949c2e66-950f-480a-96a3-16c9e4be467b/istio-proxy/1.log"
Sep 16 20:05:35.268899 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:35.268856    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7465b55b9d-njqmh_9ac2b8ef-ff60-4791-9e56-5ac64263bf1e/router/0.log"
Sep 16 20:05:36.891853 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:36.891821    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt_949c2e66-950f-480a-96a3-16c9e4be467b/istio-proxy/0.log"
Sep 16 20:05:36.927353 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:36.927239    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-ppvdt_949c2e66-950f-480a-96a3-16c9e4be467b/istio-proxy/1.log"
Sep 16 20:05:36.947787 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:36.947757    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7465b55b9d-njqmh_9ac2b8ef-ff60-4791-9e56-5ac64263bf1e/router/0.log"
Sep 16 20:05:37.592476 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:37.592419    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-rvnrb_6054145c-eb8c-4d2e-a1a4-8f0f4c674573/s3-init/0.log"
Sep 16 20:05:37.614304 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:37.614275    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-rvnrb_6054145c-eb8c-4d2e-a1a4-8f0f4c674573/download-hf-model/0.log"
Sep 16 20:05:42.401413 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:42.401379    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-9xhbs_c1f943cd-af90-41fe-ae52-9e2a35ac33f5/migrator/0.log"
Sep 16 20:05:42.427043 ip-10-0-129-194 kubenswrapper[2582]: I0916 20:05:42.427014    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-9xhbs_c1f943cd-af90-41fe-ae52-9e2a35ac33f5/graceful-termination/0.log"