Sep 18 19:13:47.393354 ip-10-0-129-30 systemd[1]: Starting Kubernetes Kubelet...
Sep 18 19:13:51.245088 ip-10-0-129-30 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 19:13:51.245088 ip-10-0-129-30 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 18 19:13:51.245088 ip-10-0-129-30 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 19:13:51.245088 ip-10-0-129-30 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 18 19:13:51.245088 ip-10-0-129-30 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 19:13:51.245868 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.245157    2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 18 19:13:51.249225 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249207    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:13:51.249225 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249223    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:13:51.249313 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249228    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:13:51.249313 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249231    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:13:51.249313 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249234    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:13:51.249313 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249237    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:13:51.249313 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249241    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:13:51.249313 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249244    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:13:51.249313 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249247    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:13:51.249313 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249250    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:13:51.249313 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249253    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:13:51.249313 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249256    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:13:51.249313 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249258    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:13:51.249313 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249261    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:13:51.249313 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249264    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:13:51.249313 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249267    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:13:51.249313 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249277    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:13:51.249313 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249280    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:13:51.249313 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249284    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:13:51.249313 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249288    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:13:51.249313 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249291    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:13:51.249777 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249294    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:13:51.249777 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249297    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:13:51.249777 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249300    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:13:51.249777 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249303    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:13:51.249777 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249306    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:13:51.249777 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249309    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:13:51.249777 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249311    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:13:51.249777 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249314    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:13:51.249777 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249317    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:13:51.249777 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249319    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:13:51.249777 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249322    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:13:51.249777 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249325    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:13:51.249777 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249328    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:13:51.249777 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249331    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:13:51.249777 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249333    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:13:51.249777 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249336    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:13:51.249777 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249339    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:13:51.249777 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249341    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:13:51.249777 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249345    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:13:51.249777 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249348    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:13:51.250306 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249351    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:13:51.250306 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249356    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:13:51.250306 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249359    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:13:51.250306 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249362    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:13:51.250306 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249365    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:13:51.250306 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249367    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:13:51.250306 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249370    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:13:51.250306 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249372    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:13:51.250306 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249375    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:13:51.250306 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249378    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:13:51.250306 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249380    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:13:51.250306 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249383    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:13:51.250306 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249386    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:13:51.250306 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249389    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:13:51.250306 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249392    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:13:51.250306 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249395    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:13:51.250306 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249397    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:13:51.250306 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249400    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:13:51.250306 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249402    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:13:51.250756 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249405    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:13:51.250756 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249408    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:13:51.250756 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249410    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:13:51.250756 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249413    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:13:51.250756 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249416    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:13:51.250756 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249418    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:13:51.250756 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249421    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:13:51.250756 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249423    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:13:51.250756 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249426    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:13:51.250756 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249428    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:13:51.250756 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249431    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:13:51.250756 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249434    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:13:51.250756 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249436    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:13:51.250756 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249439    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:13:51.250756 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249441    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:13:51.250756 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249444    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:13:51.250756 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249446    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:13:51.250756 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249449    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:13:51.250756 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249451    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:13:51.250756 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249454    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:13:51.251329 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249456    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:13:51.251329 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249459    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:13:51.251329 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249463    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:13:51.251329 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249466    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:13:51.251329 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249468    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:13:51.251329 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249471    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:13:51.251329 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249475    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:13:51.251329 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249478    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:13:51.251329 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249481    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:13:51.251329 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249483    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:13:51.251329 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249486    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:13:51.251329 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249489    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:13:51.251329 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249491    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:13:51.251329 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249494    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:13:51.251329 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249496    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:13:51.251329 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249499    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:13:51.251329 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249502    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:13:51.251329 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249504    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:13:51.251329 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.249506    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:13:51.251329 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250211    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:13:51.251820 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250218    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:13:51.251820 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250221    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:13:51.251820 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250224    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:13:51.251820 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250227    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:13:51.251820 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250230    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:13:51.251820 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250232    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:13:51.251820 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250235    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:13:51.251820 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250238    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:13:51.251820 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250240    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:13:51.251820 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250243    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:13:51.251820 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250246    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:13:51.251820 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250248    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:13:51.251820 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250251    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:13:51.251820 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250253    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:13:51.251820 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250256    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:13:51.251820 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250259    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:13:51.251820 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250262    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:13:51.251820 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250264    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:13:51.251820 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250268    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:13:51.251820 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250270    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:13:51.252392 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250273    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:13:51.252392 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250276    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:13:51.252392 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250278    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:13:51.252392 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250281    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:13:51.252392 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250283    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:13:51.252392 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250286    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:13:51.252392 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250288    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:13:51.252392 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250291    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:13:51.252392 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250293    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:13:51.252392 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250296    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:13:51.252392 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250298    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:13:51.252392 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250301    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:13:51.252392 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250304    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:13:51.252392 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250306    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:13:51.252392 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250311    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:13:51.252392 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250315    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:13:51.252392 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250319    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:13:51.252392 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250322    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:13:51.252392 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250325    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:13:51.252872 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250329    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:13:51.252872 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250332    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:13:51.252872 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250335    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:13:51.252872 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250338    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:13:51.252872 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250340    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:13:51.252872 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250343    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:13:51.252872 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250345    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:13:51.252872 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250349    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:13:51.252872 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250352    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:13:51.252872 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250355    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:13:51.252872 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250358    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:13:51.252872 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250361    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:13:51.252872 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250365    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:13:51.252872 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250368    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:13:51.252872 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250371    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:13:51.252872 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250374    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:13:51.252872 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250376    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:13:51.252872 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250379    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:13:51.252872 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250382    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:13:51.253349 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250384    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:13:51.253349 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250387    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:13:51.253349 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250390    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:13:51.253349 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250392    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:13:51.253349 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250395    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:13:51.253349 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250398    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:13:51.253349 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250400    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:13:51.253349 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250403    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:13:51.253349 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250405    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:13:51.253349 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250408    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:13:51.253349 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250410    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:13:51.253349 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250413    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:13:51.253349 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250415    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:13:51.253349 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250419    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:13:51.253349 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250421    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:13:51.253349 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250424    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:13:51.253349 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250427    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:13:51.253349 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250430    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:13:51.253349 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250432    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:13:51.253349 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250435    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:13:51.253824 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250438    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:13:51.253824 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250441    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:13:51.253824 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250444    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:13:51.253824 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250448    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:13:51.253824 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250450    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:13:51.253824 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250453    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:13:51.253824 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250456    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:13:51.253824 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250459    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:13:51.253824 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250461    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:13:51.253824 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250464    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:13:51.253824 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250466    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:13:51.253824 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250468    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:13:51.253824 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250472    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:13:51.253824 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250476    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:13:51.253824 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250478    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:13:51.253824 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250481    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:13:51.253824 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250483    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:13:51.253824 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250487    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:13:51.253824 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250490    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:13:51.253824 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250492    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:13:51.254316 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250854    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:13:51.254316 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250858    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:13:51.254316 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250862    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:13:51.254316 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250864    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:13:51.254316 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250867    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:13:51.254316 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250870    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:13:51.254316 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250873    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:13:51.254316 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250876    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:13:51.254316 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250879    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:13:51.254316 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250882    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:13:51.254316 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250884    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:13:51.254316 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250887    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:13:51.254316 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250890    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:13:51.254316 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250892    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:13:51.254316 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250895    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:13:51.254316 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250898    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:13:51.254316 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250901    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:13:51.254316 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250903    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:13:51.254316 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250907    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:13:51.254775 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250909    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:13:51.254775 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250912    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:13:51.254775 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250915    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:13:51.254775 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250917    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:13:51.254775 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250920    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:13:51.254775 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250923    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:13:51.254775 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250925    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:13:51.254775 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250928    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:13:51.254775 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250930    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:13:51.254775 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250933    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:13:51.254775 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250936    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:13:51.254775 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250938    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:13:51.254775 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250942    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:13:51.254775 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250946    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:13:51.254775 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250948    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:13:51.254775 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250951    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:13:51.254775 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250953    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:13:51.254775 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250956    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:13:51.254775 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250959    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:13:51.254775 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250961    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:13:51.255271 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250964    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:13:51.255271 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250967    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:13:51.255271 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250970    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:13:51.255271 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250972    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:13:51.255271 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250975    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:13:51.255271 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250978    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:13:51.255271 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250980    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:13:51.255271 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.250983    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:13:51.255271 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251001    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:13:51.255271 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251006    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:13:51.255271 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251009    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:13:51.255271 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251012    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:13:51.255271 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251015    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:13:51.255271 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251018    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:13:51.255271 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251021    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:13:51.255271 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251024    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:13:51.255271 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251027    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:13:51.255271 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251029    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:13:51.255271 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251032    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:13:51.255271 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251035    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:13:51.255757 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251038    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:13:51.255757 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251040    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:13:51.255757 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251043    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:13:51.255757 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251046    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:13:51.255757 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251048    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:13:51.255757 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251051    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:13:51.255757 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251054    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:13:51.255757 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251056    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:13:51.255757 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251060    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:13:51.255757 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251063    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:13:51.255757 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251066    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:13:51.255757 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251069    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:13:51.255757 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251072    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:13:51.255757 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251075    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:13:51.255757 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251078    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:13:51.255757 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251082    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:13:51.255757 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251085    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:13:51.255757 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251088    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:13:51.255757 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251090    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:13:51.256230 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251093    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:13:51.256230 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251095    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:13:51.256230 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251098    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:13:51.256230 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251101    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:13:51.256230 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251104    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:13:51.256230 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251106    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:13:51.256230 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251109    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:13:51.256230 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251112    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:13:51.256230 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251114    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:13:51.256230 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251117    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:13:51.256230 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251119    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:13:51.256230 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251122    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:13:51.256230 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251124    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:13:51.256230 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251127    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:13:51.256230 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251129    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:13:51.256230 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251132    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:13:51.256230 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251134    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:13:51.256230 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251137    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:13:51.256230 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251140    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:13:51.256230 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251143    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:13:51.256713 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251145    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:13:51.256713 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251219    2577 flags.go:64] FLAG: --address="0.0.0.0"
Sep 18 19:13:51.256713 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251254    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 18 19:13:51.256713 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251261    2577 flags.go:64] FLAG: --anonymous-auth="true"
Sep 18 19:13:51.256713 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251266    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 18 19:13:51.256713 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251270    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 18 19:13:51.256713 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251275    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 18 19:13:51.256713 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251279    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 18 19:13:51.256713 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251284    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 18 19:13:51.256713 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251287    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 18 19:13:51.256713 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251291    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 18 19:13:51.256713 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251294    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 18 19:13:51.256713 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251298    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 18 19:13:51.256713 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251301    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 18 19:13:51.256713 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251304    2577 flags.go:64] FLAG: --cgroup-root=""
Sep 18 19:13:51.256713 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251307    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 18 19:13:51.256713 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251310    2577 flags.go:64] FLAG: --client-ca-file=""
Sep 18 19:13:51.256713 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251313    2577 flags.go:64] FLAG: --cloud-provider="external"
Sep 18 19:13:51.256713 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251316    2577 flags.go:64] FLAG: --cluster-dns="[]"
Sep 18 19:13:51.256713 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251320    2577 flags.go:64] FLAG: --cluster-domain=""
Sep 18 19:13:51.256713 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251323    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 18 19:13:51.256713 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251326    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 18 19:13:51.256713 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251329    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 18 19:13:51.256713 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251333    2577 flags.go:64] FLAG: --container-log-max-files="5"
Sep 18 19:13:51.257355 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251336    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 18 19:13:51.257355 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251339    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 18 19:13:51.257355 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251342    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 18 19:13:51.257355 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251346    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 18 19:13:51.257355 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251349    2577 flags.go:64] FLAG: --contention-profiling="false"
Sep 18 19:13:51.257355 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251353    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 18 19:13:51.257355 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251357    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 18 19:13:51.257355 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251360    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 18 19:13:51.257355 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251363    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 18 19:13:51.257355 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251371    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 18 19:13:51.257355 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251374    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 18 19:13:51.257355 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251377    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 18 19:13:51.257355 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251380    2577 flags.go:64] FLAG: --enable-load-reader="false"
Sep 18 19:13:51.257355 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251383    2577 flags.go:64] FLAG: --enable-server="true"
Sep 18 19:13:51.257355 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251388    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 18 19:13:51.257355 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251392    2577 flags.go:64] FLAG: --event-burst="100"
Sep 18 19:13:51.257355 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251396    2577 flags.go:64] FLAG: --event-qps="50"
Sep 18 19:13:51.257355 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251399    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 18 19:13:51.257355 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251402    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 18 19:13:51.257355 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251405    2577 flags.go:64] FLAG: --eviction-hard=""
Sep 18 19:13:51.257355 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251408    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 18 19:13:51.257355 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251411    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 18 19:13:51.257355 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251414    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 18 19:13:51.257355 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251418    2577 flags.go:64] FLAG: --eviction-soft=""
Sep 18 19:13:51.257355 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251420    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 18 19:13:51.257955 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251424    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 18 19:13:51.257955 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251427    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 18 19:13:51.257955 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251430    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 18 19:13:51.257955 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251433    2577 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 18 19:13:51.257955 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251436    2577 flags.go:64] FLAG: --fail-swap-on="true"
Sep 18 19:13:51.257955 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251439    2577 flags.go:64] FLAG: --feature-gates=""
Sep 18 19:13:51.257955 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251443    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 18 19:13:51.257955 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251446    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 18 19:13:51.257955 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251449    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 18 19:13:51.257955 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251453    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 18 19:13:51.257955 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251456    2577 flags.go:64] FLAG: --healthz-port="10248"
Sep 18 19:13:51.257955 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251459    2577 flags.go:64] FLAG: --help="false"
Sep 18 19:13:51.257955 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251463    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-129-30.ec2.internal"
Sep 18 19:13:51.257955 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251466    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 18 19:13:51.257955 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251469    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 18 19:13:51.257955 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251472    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 18 19:13:51.257955 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251476    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 18 19:13:51.257955 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251479    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 18 19:13:51.257955 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251482    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 18 19:13:51.257955 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251485    2577 flags.go:64] FLAG: --image-service-endpoint=""
Sep 18 19:13:51.257955 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251488    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 18 19:13:51.257955 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251493    2577 flags.go:64] FLAG: --kube-api-burst="100"
Sep 18 19:13:51.257955 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251496    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 18 19:13:51.258552 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251499    2577 flags.go:64] FLAG: --kube-api-qps="50"
Sep 18 19:13:51.258552 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251502    2577 flags.go:64] FLAG: --kube-reserved=""
Sep 18 19:13:51.258552 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251505    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 18 19:13:51.258552 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251507    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 18 19:13:51.258552 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251510    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 18 19:13:51.258552 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251513    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 18 19:13:51.258552 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251516    2577 flags.go:64] FLAG: --lock-file=""
Sep 18 19:13:51.258552 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251519    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 18 19:13:51.258552 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251522    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 18 19:13:51.258552 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251525    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 18 19:13:51.258552 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251531    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 18 19:13:51.258552 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251534    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 18 19:13:51.258552 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251536    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 18 19:13:51.258552 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251539    2577 flags.go:64] FLAG: --logging-format="text"
Sep 18 19:13:51.258552 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251542    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 18 19:13:51.258552 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251545    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 18 19:13:51.258552 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251548    2577 flags.go:64] FLAG: --manifest-url=""
Sep 18 19:13:51.258552 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251551    2577 flags.go:64] FLAG: --manifest-url-header=""
Sep 18 19:13:51.258552 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251555    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 18 19:13:51.258552 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251558    2577 flags.go:64] FLAG: --max-open-files="1000000"
Sep 18 19:13:51.258552 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251562    2577 flags.go:64] FLAG: --max-pods="110"
Sep 18 19:13:51.258552 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251566    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 18 19:13:51.258552 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251569    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 18 19:13:51.258552 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251572    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 18 19:13:51.258552 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251575    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 18 19:13:51.259512 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251578    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 18 19:13:51.259512 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251581    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 18 19:13:51.259512 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251584    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 18 19:13:51.259512 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251591    2577 flags.go:64] FLAG: --node-status-max-images="50"
Sep 18 19:13:51.259512 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251594    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 18 19:13:51.259512 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251599    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 18 19:13:51.259512 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251602    2577 flags.go:64] FLAG: --pod-cidr=""
Sep 18 19:13:51.259512 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251605    2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 18 19:13:51.259512 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251610    2577 flags.go:64] FLAG: --pod-manifest-path=""
Sep 18 19:13:51.259512 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251613    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 18 19:13:51.259512 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251616    2577 flags.go:64] FLAG: --pods-per-core="0"
Sep 18 19:13:51.259512 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251619    2577 flags.go:64] FLAG: --port="10250"
Sep 18 19:13:51.259512 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251622    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 18 19:13:51.259512 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251625    2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03bed4e4e7b8ef7c1"
Sep 18 19:13:51.259512 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251628    2577 flags.go:64] FLAG: --qos-reserved=""
Sep 18 19:13:51.259512 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251631    2577 flags.go:64] FLAG: --read-only-port="10255"
Sep 18 19:13:51.259512 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251635    2577 flags.go:64] FLAG: --register-node="true"
Sep 18 19:13:51.259512 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251637    2577 flags.go:64] FLAG: --register-with-taints=""
Sep 18 19:13:51.259512 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251642    2577 flags.go:64] FLAG: --registry-burst="10"
Sep 18 19:13:51.259512 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251645    2577 flags.go:64] FLAG: --registry-qps="5"
Sep 18 19:13:51.259512 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251648    2577 flags.go:64] FLAG: --reserved-cpus=""
Sep 18 19:13:51.259512 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251650    2577 flags.go:64] FLAG: --reserved-memory=""
Sep 18 19:13:51.259512 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251654    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 18 19:13:51.259512 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251657    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 18 19:13:51.259512 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251660    2577 flags.go:64] FLAG: --rotate-certificates="false"
Sep 18 19:13:51.260191 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251663    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 18 19:13:51.260191 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251666    2577 flags.go:64] FLAG: --runonce="false"
Sep 18 19:13:51.260191 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251669    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 18 19:13:51.260191 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251673    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 18 19:13:51.260191 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251676    2577 flags.go:64] FLAG: --seccomp-default="false"
Sep 18 19:13:51.260191 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251679    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 18 19:13:51.260191 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251683    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 18 19:13:51.260191 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251686    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 18 19:13:51.260191 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251689    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 18 19:13:51.260191 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251692    2577 flags.go:64] FLAG: --storage-driver-password="root"
Sep 18 19:13:51.260191 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251695    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 18 19:13:51.260191 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251698    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 18 19:13:51.260191 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251700    2577 flags.go:64] FLAG: --storage-driver-user="root"
Sep 18 19:13:51.260191 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251706    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 18 19:13:51.260191 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251709    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 18 19:13:51.260191 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251712    2577 flags.go:64] FLAG: --system-cgroups=""
Sep 18 19:13:51.260191 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251715    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 18 19:13:51.260191 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251721    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 18 19:13:51.260191 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251723    2577 flags.go:64] FLAG: --tls-cert-file=""
Sep 18 19:13:51.260191 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251726    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 18 19:13:51.260191 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251731    2577 flags.go:64] FLAG: --tls-min-version=""
Sep 18 19:13:51.260191 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251734    2577 flags.go:64] FLAG: --tls-private-key-file=""
Sep 18 19:13:51.260191 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251736    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 18 19:13:51.260191 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251739    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 18 19:13:51.260191 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251742    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 18 19:13:51.261183 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251745    2577 flags.go:64] FLAG: --v="2"
Sep 18 19:13:51.261183 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251750    2577 flags.go:64] FLAG: --version="false"
Sep 18 19:13:51.261183 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251754    2577 flags.go:64] FLAG: --vmodule=""
Sep 18 19:13:51.261183 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251758    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 18 19:13:51.261183 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.251761    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 18 19:13:51.261183 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251872    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:13:51.261183 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251876    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:13:51.261183 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251879    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:13:51.261183 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251882    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:13:51.261183 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251885    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:13:51.261183 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251889    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:13:51.261183 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251892    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:13:51.261183 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251894    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:13:51.261183 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251897    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:13:51.261183 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251900    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:13:51.261183 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251902    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:13:51.261183 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251905    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:13:51.261183 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251908    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:13:51.261183 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251911    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:13:51.261183 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251913    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:13:51.261183 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251918    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:13:51.261786 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251920    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:13:51.261786 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251923    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:13:51.261786 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251926    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:13:51.261786 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251928    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:13:51.261786 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251931    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:13:51.261786 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251934    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:13:51.261786 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251937    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:13:51.261786 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251939    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:13:51.261786 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251942    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:13:51.261786 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251945    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:13:51.261786 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251948    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:13:51.261786 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251950    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:13:51.261786 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251953    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:13:51.261786 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251956    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:13:51.261786 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251959    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:13:51.261786 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251963    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:13:51.261786 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251966    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:13:51.261786 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251969    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:13:51.261786 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251972    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:13:51.261786 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251975    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:13:51.262321 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.251977    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:13:51.262321 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252001    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:13:51.262321 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252004    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:13:51.262321 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252007    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:13:51.262321 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252010    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:13:51.262321 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252012    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:13:51.262321 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252015    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:13:51.262321 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252018    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:13:51.262321 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252020    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:13:51.262321 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252023    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:13:51.262321 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252026    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:13:51.262321 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252029    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:13:51.262321 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252032    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:13:51.262321 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252035    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:13:51.262321 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252037    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:13:51.262321 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252040    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:13:51.262321 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252043    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:13:51.262321 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252045    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:13:51.262321 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252048    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:13:51.262801 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252051    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:13:51.262801 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252053    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:13:51.262801 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252056    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:13:51.262801 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252060    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:13:51.262801 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252064    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:13:51.262801 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252068    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:13:51.262801 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252071    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:13:51.262801 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252075    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:13:51.262801 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252078    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:13:51.262801 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252081    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:13:51.262801 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252084    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:13:51.262801 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252086    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:13:51.262801 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252089    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:13:51.262801 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252092    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:13:51.262801 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252095    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:13:51.262801 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252098    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:13:51.262801 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252101    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:13:51.262801 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252104    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:13:51.262801 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252107    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:13:51.263299 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252109    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:13:51.263299 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252112    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:13:51.263299 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252115    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:13:51.263299 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252117    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:13:51.263299 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252120    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:13:51.263299 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252124    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:13:51.263299 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252126    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:13:51.263299 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252129    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:13:51.263299 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252132    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:13:51.263299 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252134    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:13:51.263299 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252137    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:13:51.263299 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252140    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:13:51.263299 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252142    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:13:51.263299 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252145    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:13:51.263299 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252148    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:13:51.263299 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252151    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:13:51.263299 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252154    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:13:51.263299 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252157    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:13:51.263299 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252159    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:13:51.263299 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252162    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:13:51.263787 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252164    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:13:51.263787 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252167    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:13:51.263787 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252170    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:13:51.263787 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252173    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:13:51.263787 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.252176    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:13:51.263787 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.252782    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 19:13:51.263787 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.262549    2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 18 19:13:51.263787 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.262566    2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 18 19:13:51.263787 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.262589    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 18 19:13:51.263787 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.262595    2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 18 19:13:51.263787 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262652    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:13:51.263787 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262658    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:13:51.263787 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262661    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:13:51.263787 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262665    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:13:51.264155 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262668    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:13:51.264155 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262673    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:13:51.264155 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262677    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:13:51.264155 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262680    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:13:51.264155 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262684    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:13:51.264155 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262687    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:13:51.264155 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262689    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:13:51.264155 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262692    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:13:51.264155 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262696    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:13:51.264155 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262699    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:13:51.264155 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262702    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:13:51.264155 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262704    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:13:51.264155 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262707    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:13:51.264155 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262710    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:13:51.264155 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262713    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:13:51.264155 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262716    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:13:51.264155 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262720    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:13:51.264155 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262723    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:13:51.264155 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262726    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:13:51.264605 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262728    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:13:51.264605 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262731    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:13:51.264605 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262733    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:13:51.264605 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262736    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:13:51.264605 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262739    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:13:51.264605 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262742    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:13:51.264605 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262745    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:13:51.264605 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262748    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:13:51.264605 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262750    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:13:51.264605 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262753    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:13:51.264605 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262755    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:13:51.264605 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262758    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:13:51.264605 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262761    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:13:51.264605 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262764    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:13:51.264605 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262767    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:13:51.264605 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262770    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:13:51.264605 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262773    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:13:51.264605 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262775    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:13:51.264605 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262778    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:13:51.264605 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262780    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:13:51.265106 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262783    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:13:51.265106 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262785    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:13:51.265106 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262788    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:13:51.265106 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262791    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:13:51.265106 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262794    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:13:51.265106 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262796    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:13:51.265106 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262799    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:13:51.265106 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262802    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:13:51.265106 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262805    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:13:51.265106 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262807    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:13:51.265106 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262810    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:13:51.265106 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262813    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:13:51.265106 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262815    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:13:51.265106 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262818    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:13:51.265106 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262821    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:13:51.265106 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262823    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:13:51.265106 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262826    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:13:51.265106 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262828    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:13:51.265106 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262831    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:13:51.265106 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262834    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:13:51.265585 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262836    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:13:51.265585 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262839    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:13:51.265585 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262841    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:13:51.265585 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262844    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:13:51.265585 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262846    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:13:51.265585 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262849    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:13:51.265585 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262852    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:13:51.265585 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262855    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:13:51.265585 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262857    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:13:51.265585 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262860    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:13:51.265585 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262862    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:13:51.265585 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262865    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:13:51.265585 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262868    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:13:51.265585 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262870    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:13:51.265585 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262873    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:13:51.265585 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262875    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:13:51.265585 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262879    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:13:51.265585 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262882    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:13:51.265585 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262884    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:13:51.265585 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262887    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:13:51.266202 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262889    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:13:51.266202 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262892    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:13:51.266202 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262895    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:13:51.266202 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262897    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:13:51.266202 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262900    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:13:51.266202 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262902    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:13:51.266202 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262906    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:13:51.266202 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262909    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:13:51.266202 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262913    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:13:51.266202 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262915    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:13:51.266202 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262918    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:13:51.266202 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262921    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:13:51.266202 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262924    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:13:51.266202 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262926    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:13:51.266202 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262930    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:13:51.266202 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.262932    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:13:51.266590 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.262937    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 19:13:51.266590 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263065    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:13:51.266590 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263071    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:13:51.266590 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263074    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:13:51.266590 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263077    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:13:51.266590 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263080    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:13:51.266590 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263082    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:13:51.266590 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263085    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:13:51.266590 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263088    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:13:51.266590 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263091    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:13:51.266590 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263094    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:13:51.266590 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263097    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:13:51.266590 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263099    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:13:51.266590 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263103    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:13:51.266590 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263106    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:13:51.267000 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263109    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:13:51.267000 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263112    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:13:51.267000 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263114    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:13:51.267000 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263117    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:13:51.267000 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263120    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:13:51.267000 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263123    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:13:51.267000 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263125    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:13:51.267000 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263128    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:13:51.267000 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263131    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:13:51.267000 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263134    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:13:51.267000 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263136    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:13:51.267000 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263139    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:13:51.267000 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263141    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:13:51.267000 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263144    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:13:51.267000 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263147    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:13:51.267000 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263150    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:13:51.267000 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263152    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:13:51.267000 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263155    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:13:51.267000 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263158    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:13:51.267530 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263161    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:13:51.267530 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263163    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:13:51.267530 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263166    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:13:51.267530 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263169    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:13:51.267530 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263171    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:13:51.267530 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263174    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:13:51.267530 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263176    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:13:51.267530 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263182    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:13:51.267530 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263185    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:13:51.267530 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263189    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:13:51.267530 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263192    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:13:51.267530 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263194    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:13:51.267530 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263198    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:13:51.267530 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263200    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:13:51.267530 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263203    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:13:51.267530 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263206    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:13:51.267530 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263208    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:13:51.267530 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263211    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:13:51.267530 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263214    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:13:51.267530 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263217    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:13:51.268027 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263220    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:13:51.268027 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263222    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:13:51.268027 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263225    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:13:51.268027 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263227    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:13:51.268027 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263230    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:13:51.268027 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263233    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:13:51.268027 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263235    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:13:51.268027 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263238    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:13:51.268027 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263241    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:13:51.268027 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263243    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:13:51.268027 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263246    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:13:51.268027 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263249    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:13:51.268027 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263252    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:13:51.268027 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263254    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:13:51.268027 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263257    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:13:51.268027 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263260    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:13:51.268027 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263262    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:13:51.268027 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263265    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:13:51.268027 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263268    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:13:51.268027 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263272    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:13:51.268507 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263275    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:13:51.268507 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263277    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:13:51.268507 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263280    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:13:51.268507 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263284    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:13:51.268507 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263287    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:13:51.268507 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263291    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:13:51.268507 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263295    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:13:51.268507 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263297    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:13:51.268507 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263300    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:13:51.268507 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263304    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:13:51.268507 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263307    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:13:51.268507 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263310    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:13:51.268507 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263312    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:13:51.268507 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263315    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:13:51.268507 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263318    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:13:51.268507 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263322    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:13:51.268507 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263325    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:13:51.268507 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263328    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:13:51.268507 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263331    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:13:51.268972 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263334    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:13:51.268972 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263337    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:13:51.268972 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263340    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:13:51.268972 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263343    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:13:51.268972 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263346    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:13:51.268972 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263348    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:13:51.268972 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:51.263351    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:13:51.268972 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.263356    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 19:13:51.268972 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.263465    2577 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 18 19:13:51.268972 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.266489    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 18 19:13:51.268972 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.267369    2577 server.go:1019] "Starting client certificate rotation"
Sep 18 19:13:51.268972 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.267462    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 19:13:51.268972 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.267493    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 19:13:51.270275 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.270261    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 19:13:51.271499 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.271482    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 19:13:51.275481 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.275464    2577 log.go:25] "Validated CRI v1 runtime API"
Sep 18 19:13:51.277073 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.277059    2577 log.go:25] "Validated CRI v1 image API"
Sep 18 19:13:51.280549 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.280535    2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 18 19:13:51.285536 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.285510    2577 fs.go:135] Filesystem UUIDs: map[006456aa-51f7-49c8-af97-c9ce9b447b81:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 c0111735-0168-4aa5-89dc-e61c3a78e4d0:/dev/nvme0n1p3]
Sep 18 19:13:51.285629 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.285534    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 18 19:13:51.291118 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.290893    2577 manager.go:216] Machine: {Timestamp:2026-09-18 19:13:51.290178481 +0000 UTC m=+2.142478844 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3056529 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25541140cdaf2252af5c78afb410cf SystemUUID:ec255411-40cd-af22-52af-5c78afb410cf BootID:a37dd7db-3258-4108-b2ad-283ab2729969 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d3:c3:25:fd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d3:c3:25:fd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:02:8b:5b:af:2d:56 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 18 19:13:51.291118 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.291103    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 18 19:13:51.291339 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.291278    2577 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 18 19:13:51.293299 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.293267    2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 18 19:13:51.293487 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.293306    2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-30.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 18 19:13:51.294533 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.294521    2577 topology_manager.go:138] "Creating topology manager with none policy"
Sep 18 19:13:51.294567 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.294536    2577 container_manager_linux.go:309] "Creating device plugin manager"
Sep 18 19:13:51.294567 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.294544    2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 19:13:51.294567 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.294555    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 19:13:51.294650 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.294626    2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 18 19:13:51.297528 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.297516    2577 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 19:13:51.297640 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.297631    2577 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 18 19:13:51.297668 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.297665    2577 kubelet.go:486] "Attempting to sync node with API server"
Sep 18 19:13:51.297702 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.297674    2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 18 19:13:51.297702 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.297686    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 18 19:13:51.297702 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.297695    2577 kubelet.go:398] "Adding apiserver pod source"
Sep 18 19:13:51.297702 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.297703    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 18 19:13:51.298058 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.298044    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 19:13:51.298128 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.298066    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 19:13:51.306030 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.306011    2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 18 19:13:51.308271 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.308258    2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 18 19:13:51.308325 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.308287    2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 18 19:13:51.309378 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.309367    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 18 19:13:51.309415 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.309384    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 18 19:13:51.309415 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.309390    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 18 19:13:51.309415 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.309395    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 18 19:13:51.309415 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.309403    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 18 19:13:51.309415 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.309408    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 18 19:13:51.309542 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.309421    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 18 19:13:51.309542 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.309426    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 18 19:13:51.309542 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.309434    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 18 19:13:51.309542 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.309439    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 18 19:13:51.309542 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.309448    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 18 19:13:51.309542 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.309454    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 18 19:13:51.309542 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.309480    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 18 19:13:51.309542 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.309486    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 18 19:13:51.313073 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.313056    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 19:13:51.313383 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.313359    2577 server.go:1268] "Started kubelet"
Sep 18 19:13:51.313486 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.313417    2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 18 19:13:51.313844 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.313791    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 18 19:13:51.313919 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.313864    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 18 19:13:51.314233 ip-10-0-129-30 systemd[1]: Started Kubernetes Kubelet.
Sep 18 19:13:51.314915 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.314795    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-30.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 19:13:51.314974 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:51.314939    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-30.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 19:13:51.315221 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.315068    2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 18 19:13:51.315596 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:51.315572    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 19:13:51.315829 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.315818    2577 server.go:310] "Adding debug handlers to kubelet server"
Sep 18 19:13:51.319304 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:51.319273    2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 18 19:13:51.321018 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.321003    2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 18 19:13:51.321018 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.321015    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 18 19:13:51.321141 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:51.321073    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-30.ec2.internal\" not found"
Sep 18 19:13:51.321768 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.321740    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 18 19:13:51.321768 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.321743    2577 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 18 19:13:51.321768 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.321771    2577 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 18 19:13:51.321938 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.321746    2577 factory.go:55] Registering systemd factory
Sep 18 19:13:51.321938 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.321809    2577 factory.go:223] Registration of the systemd container factory successfully
Sep 18 19:13:51.321938 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.321843    2577 reconstruct.go:97] "Volume reconstruction finished"
Sep 18 19:13:51.321938 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.321854    2577 reconciler.go:29] "Reconciler: start to sync state"
Sep 18 19:13:51.322106 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.322014    2577 factory.go:153] Registering CRI-O factory
Sep 18 19:13:51.322106 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.322026    2577 factory.go:223] Registration of the crio container factory successfully
Sep 18 19:13:51.322106 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.322072    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 18 19:13:51.322106 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.322094    2577 factory.go:103] Registering Raw factory
Sep 18 19:13:51.322106 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.322107    2577 manager.go:1196] Started watching for new ooms in manager
Sep 18 19:13:51.322644 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.322627    2577 manager.go:349] Starting recovery of all containers
Sep 18 19:13:51.324503 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:51.324477    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 19:13:51.324608 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:51.324567    2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-30.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 18 19:13:51.348860 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.348842    2577 manager.go:354] Recovery completed
Sep 18 19:13:51.352801 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.352785    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:13:51.353938 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:51.352969    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-30.ec2.internal.18d6802cae0ca3ed  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-30.ec2.internal,UID:ip-10-0-129-30.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-30.ec2.internal,},FirstTimestamp:2026-09-18 19:13:51.313331181 +0000 UTC m=+2.165631542,LastTimestamp:2026-09-18 19:13:51.313331181 +0000 UTC m=+2.165631542,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-30.ec2.internal,}"
Sep 18 19:13:51.354546 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.354524    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-30.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:13:51.354623 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.354562    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-30.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:13:51.354623 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.354575    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-30.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:13:51.355155 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.355141    2577 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 18 19:13:51.355155 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.355153    2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 18 19:13:51.355278 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.355155    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:13:51.355278 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.355167    2577 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 19:13:51.355793 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.355777    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-30.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:13:51.355853 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.355803    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-30.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:13:51.355853 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.355813    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-30.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:13:51.356537 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.356525    2577 policy_none.go:49] "None policy: Start"
Sep 18 19:13:51.356588 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.356540    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 18 19:13:51.356588 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.356552    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 18 19:13:51.357706 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.357695    2577 policy_none.go:47] "Start"
Sep 18 19:13:51.368079 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:51.367907    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-30.ec2.internal.18d6802cb08184b0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-30.ec2.internal,UID:ip-10-0-129-30.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-30.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-30.ec2.internal,},FirstTimestamp:2026-09-18 19:13:51.354545328 +0000 UTC m=+2.206845691,LastTimestamp:2026-09-18 19:13:51.354545328 +0000 UTC m=+2.206845691,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-30.ec2.internal,}"
Sep 18 19:13:51.379503 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:51.379437    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-30.ec2.internal.18d6802cb081e3ae  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-30.ec2.internal,UID:ip-10-0-129-30.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-30.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-30.ec2.internal,},FirstTimestamp:2026-09-18 19:13:51.354569646 +0000 UTC m=+2.206870007,LastTimestamp:2026-09-18 19:13:51.354569646 +0000 UTC m=+2.206870007,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-30.ec2.internal,}"
Sep 18 19:13:51.386776 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.386755    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-szvwv"
Sep 18 19:13:51.389557 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:51.389495    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-30.ec2.internal.18d6802cb08209fd  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-30.ec2.internal,UID:ip-10-0-129-30.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-129-30.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-129-30.ec2.internal,},FirstTimestamp:2026-09-18 19:13:51.354579453 +0000 UTC m=+2.206879814,LastTimestamp:2026-09-18 19:13:51.354579453 +0000 UTC m=+2.206879814,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-30.ec2.internal,}"
Sep 18 19:13:51.405463 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.398338    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-szvwv"
Sep 18 19:13:51.405463 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.399564    2577 manager.go:337] "Starting Device Plugin manager"
Sep 18 19:13:51.405463 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:51.399602    2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 18 19:13:51.405463 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.399611    2577 server.go:87] "Starting device plugin registration server"
Sep 18 19:13:51.405463 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.399876    2577 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 18 19:13:51.405463 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.399899    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 18 19:13:51.405463 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.399969    2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 18 19:13:51.405463 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.400060    2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 18 19:13:51.405463 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.400069    2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 18 19:13:51.405463 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:51.400687    2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 18 19:13:51.405463 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:51.400732    2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-30.ec2.internal\" not found"
Sep 18 19:13:51.407011 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:51.406927    2577 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-30.ec2.internal.18d6802cb08184b0\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-30.ec2.internal.18d6802cb08184b0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-30.ec2.internal,UID:ip-10-0-129-30.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-30.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-30.ec2.internal,},FirstTimestamp:2026-09-18 19:13:51.354545328 +0000 UTC m=+2.206845691,LastTimestamp:2026-09-18 19:13:51.35579225 +0000 UTC m=+2.208092612,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-30.ec2.internal,}"
Sep 18 19:13:51.431276 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.431248    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 18 19:13:51.432396 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.432382    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 18 19:13:51.432466 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.432425    2577 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 18 19:13:51.432466 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.432444    2577 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 18 19:13:51.432559 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:51.432477    2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 18 19:13:51.434876 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.434862    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:13:51.500747 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.500669    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:13:51.501577 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.501563    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-30.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:13:51.501697 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.501592    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-30.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:13:51.501697 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.501606    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-30.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:13:51.501697 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.501629    2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-30.ec2.internal"
Sep 18 19:13:51.519472 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.519454    2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-30.ec2.internal"
Sep 18 19:13:51.519533 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:51.519480    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-30.ec2.internal\": node \"ip-10-0-129-30.ec2.internal\" not found"
Sep 18 19:13:51.533101 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.533079    2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-30.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-30.ec2.internal"]
Sep 18 19:13:51.533179 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.533161    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:13:51.534521 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.534506    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-30.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:13:51.534617 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.534537    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-30.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:13:51.534617 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.534551    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-30.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:13:51.535851 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.535836    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:13:51.536020 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.536006    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-30.ec2.internal"
Sep 18 19:13:51.536072 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.536045    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:13:51.537112 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.537096    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-30.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:13:51.537198 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.537125    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-30.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:13:51.537198 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.537149    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-30.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:13:51.537198 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.537098    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-30.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:13:51.537342 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.537199    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-30.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:13:51.537342 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.537232    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-30.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:13:51.538320 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.538305    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-30.ec2.internal"
Sep 18 19:13:51.538399 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.538336    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:13:51.541414 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.541396    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-30.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:13:51.541492 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.541426    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-30.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:13:51.541492 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.541441    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-30.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:13:51.565673 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:51.565649    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-30.ec2.internal\" not found" node="ip-10-0-129-30.ec2.internal"
Sep 18 19:13:51.570471 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:51.570457    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-30.ec2.internal\" not found" node="ip-10-0-129-30.ec2.internal"
Sep 18 19:13:51.585899 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:51.585884    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-30.ec2.internal\" not found"
Sep 18 19:13:51.623191 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.623156    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/216eb917577d10dfc85b487495afcc49-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-30.ec2.internal\" (UID: \"216eb917577d10dfc85b487495afcc49\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-30.ec2.internal"
Sep 18 19:13:51.623191 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.623189    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/216eb917577d10dfc85b487495afcc49-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-30.ec2.internal\" (UID: \"216eb917577d10dfc85b487495afcc49\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-30.ec2.internal"
Sep 18 19:13:51.623370 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.623204    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5e6343628ef74873cac4c6411141685f-config\") pod \"kube-apiserver-proxy-ip-10-0-129-30.ec2.internal\" (UID: \"5e6343628ef74873cac4c6411141685f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-30.ec2.internal"
Sep 18 19:13:51.686619 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:51.686597    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-30.ec2.internal\" not found"
Sep 18 19:13:51.723666 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.723641    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/216eb917577d10dfc85b487495afcc49-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-30.ec2.internal\" (UID: \"216eb917577d10dfc85b487495afcc49\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-30.ec2.internal"
Sep 18 19:13:51.723751 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.723669    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/216eb917577d10dfc85b487495afcc49-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-30.ec2.internal\" (UID: \"216eb917577d10dfc85b487495afcc49\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-30.ec2.internal"
Sep 18 19:13:51.723751 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.723688    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5e6343628ef74873cac4c6411141685f-config\") pod \"kube-apiserver-proxy-ip-10-0-129-30.ec2.internal\" (UID: \"5e6343628ef74873cac4c6411141685f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-30.ec2.internal"
Sep 18 19:13:51.723751 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.723729    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5e6343628ef74873cac4c6411141685f-config\") pod \"kube-apiserver-proxy-ip-10-0-129-30.ec2.internal\" (UID: \"5e6343628ef74873cac4c6411141685f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-30.ec2.internal"
Sep 18 19:13:51.723881 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.723784    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/216eb917577d10dfc85b487495afcc49-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-30.ec2.internal\" (UID: \"216eb917577d10dfc85b487495afcc49\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-30.ec2.internal"
Sep 18 19:13:51.723881 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.723804    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/216eb917577d10dfc85b487495afcc49-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-30.ec2.internal\" (UID: \"216eb917577d10dfc85b487495afcc49\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-30.ec2.internal"
Sep 18 19:13:51.787529 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:51.787474    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-30.ec2.internal\" not found"
Sep 18 19:13:51.869770 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.869740    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-30.ec2.internal"
Sep 18 19:13:51.875300 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:51.874726    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-30.ec2.internal"
Sep 18 19:13:51.887942 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:51.887923    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-30.ec2.internal\" not found"
Sep 18 19:13:51.988365 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:51.988331    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-30.ec2.internal\" not found"
Sep 18 19:13:52.088911 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:52.088846    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-30.ec2.internal\" not found"
Sep 18 19:13:52.189345 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:52.189320    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-30.ec2.internal\" not found"
Sep 18 19:13:52.266809 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:52.266774    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 18 19:13:52.267276 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:52.266927    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 19:13:52.290081 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:52.290051    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-30.ec2.internal\" not found"
Sep 18 19:13:52.390193 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:52.390164    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-30.ec2.internal\" not found"
Sep 18 19:13:52.405486 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:52.405450    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 19:08:51 +0000 UTC" deadline="2028-05-23 14:49:32.327910895 +0000 UTC"
Sep 18 19:13:52.405486 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:52.405480    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14707h35m39.922434343s"
Sep 18 19:13:52.454274 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:52.454245    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod216eb917577d10dfc85b487495afcc49.slice/crio-87921ec499739bbc114de95591acf4e769be1f6261d170966df017cb96ecf545 WatchSource:0}: Error finding container 87921ec499739bbc114de95591acf4e769be1f6261d170966df017cb96ecf545: Status 404 returned error can't find the container with id 87921ec499739bbc114de95591acf4e769be1f6261d170966df017cb96ecf545
Sep 18 19:13:52.454777 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:52.454759    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e6343628ef74873cac4c6411141685f.slice/crio-c808c1be34ffbd7ec03dd67e8299e030934c232320be1f6ccfc6d77254e01610 WatchSource:0}: Error finding container c808c1be34ffbd7ec03dd67e8299e030934c232320be1f6ccfc6d77254e01610: Status 404 returned error can't find the container with id c808c1be34ffbd7ec03dd67e8299e030934c232320be1f6ccfc6d77254e01610
Sep 18 19:13:52.459640 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:52.459623    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:13:52.490734 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:52.490713    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-30.ec2.internal\" not found"
Sep 18 19:13:52.515878 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:52.515856    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:13:52.517126 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:52.517109    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-30.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:13:52.517212 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:52.517140    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-30.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:13:52.517212 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:52.517154    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-30.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:13:52.591801 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:52.591748    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-30.ec2.internal\" not found"
Sep 18 19:13:52.692330 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:52.692225    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-30.ec2.internal\" not found"
Sep 18 19:13:52.749803 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:52.749773    2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:13:52.792463 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:52.792437    2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:13:52.822161 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:52.822127    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-30.ec2.internal"
Sep 18 19:13:52.848054 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:52.848032    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 19:13:52.848882 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:52.848866    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-30.ec2.internal"
Sep 18 19:13:52.872633 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:52.872609    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 19:13:52.912113 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:52.912080    2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:13:53.298331 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.298296    2577 apiserver.go:52] "Watching apiserver"
Sep 18 19:13:53.305869 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.305845    2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 18 19:13:53.306273 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.306250    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-s6qfb","openshift-dns/node-resolver-5jzcl","openshift-multus/network-metrics-daemon-l9g57","openshift-network-operator/iptables-alerter-lrqrg","openshift-ovn-kubernetes/ovnkube-node-z76lc","kube-system/kube-apiserver-proxy-ip-10-0-129-30.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfddm","openshift-cluster-node-tuning-operator/tuned-hbbzh","openshift-image-registry/node-ca-mx49g","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-30.ec2.internal","openshift-multus/multus-additional-cni-plugins-fq6nw","openshift-multus/multus-v7jmf","openshift-network-diagnostics/network-check-target-2fpv8","kube-system/global-pull-secret-syncer-2m5d7"]
Sep 18 19:13:53.307826 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.307807    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfddm"
Sep 18 19:13:53.309366 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.309338    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fpv8"
Sep 18 19:13:53.309459 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:53.309420    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fpv8" podUID="1bd5d395-d4f3-489a-a20b-1fd771f55da6"
Sep 18 19:13:53.310610 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.310590    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9g57"
Sep 18 19:13:53.310693 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:53.310646    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9g57" podUID="b0bb7eb5-2fe7-42f3-92a6-00d65358f28f"
Sep 18 19:13:53.311272 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.311254    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 18 19:13:53.311353 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.311277    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tq9fm\""
Sep 18 19:13:53.311353 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.311296    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 18 19:13:53.311782 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.311765    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lrqrg"
Sep 18 19:13:53.311782 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.311778    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 18 19:13:53.313121 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.313102    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.314707 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.314690    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hbbzh"
Sep 18 19:13:53.315790 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.315773    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mx49g"
Sep 18 19:13:53.317034 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.317018    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-s6qfb"
Sep 18 19:13:53.317276 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.317202    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:13:53.317276 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.317256    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 18 19:13:53.317470 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.317446    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 18 19:13:53.317565 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.317475    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 18 19:13:53.317565 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.317493    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 18 19:13:53.317653 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.317638    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 18 19:13:53.317741 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.317724    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-j4h5m\""
Sep 18 19:13:53.317808 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.317793    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 18 19:13:53.318236 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.318220    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8l98r\""
Sep 18 19:13:53.318575 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.318362    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5jzcl"
Sep 18 19:13:53.318575 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.318416    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 18 19:13:53.318575 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.318430    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 18 19:13:53.318575 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.318449    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:13:53.318575 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.318500    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 18 19:13:53.319667 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.319651    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-v7jmf"
Sep 18 19:13:53.320945 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.320930    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fq6nw"
Sep 18 19:13:53.321798 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.321778    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 18 19:13:53.323190 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.323168    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2m5d7"
Sep 18 19:13:53.323924 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:53.323270    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2m5d7" podUID="5ebd2878-a3f1-467a-b25f-29124e9d5e7b"
Sep 18 19:13:53.323924 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.323880    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 18 19:13:53.324123 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.323880    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-2n9bn\""
Sep 18 19:13:53.324173 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.324122    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 18 19:13:53.324220 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.324173    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 18 19:13:53.324293 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.324279    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6qc5d\""
Sep 18 19:13:53.324367 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.324351    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 18 19:13:53.324426 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.324281    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 18 19:13:53.325192 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.324433    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-z8xfg\""
Sep 18 19:13:53.325287 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.324524    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 18 19:13:53.325287 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.324527    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 18 19:13:53.325384 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.325243    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 18 19:13:53.325604 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.324684    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-r9mzq\""
Sep 18 19:13:53.325655 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.324754    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 18 19:13:53.325717 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.324787    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-f67ck\""
Sep 18 19:13:53.325717 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.325692    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-sjwl2\""
Sep 18 19:13:53.325717 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.325706    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 18 19:13:53.325869 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.324876    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 18 19:13:53.325869 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.324623    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 18 19:13:53.325869 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.325011    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 18 19:13:53.331171 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.331154    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9554bf8e-7591-4261-be81-a00cf9629c8b-device-dir\") pod \"aws-ebs-csi-driver-node-xfddm\" (UID: \"9554bf8e-7591-4261-be81-a00cf9629c8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfddm"
Sep 18 19:13:53.331258 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.331182    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e41c46ea-8012-448d-9a65-0c67c54bbad1-host-run-ovn-kubernetes\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.331258 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.331202    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6f05b5b2-52b5-46f0-8003-30f0bd24c71c-host\") pod \"tuned-hbbzh\" (UID: \"6f05b5b2-52b5-46f0-8003-30f0bd24c71c\") " pod="openshift-cluster-node-tuning-operator/tuned-hbbzh"
Sep 18 19:13:53.331258 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.331224    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e41c46ea-8012-448d-9a65-0c67c54bbad1-host-kubelet\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.331374 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.331256    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5ebd2878-a3f1-467a-b25f-29124e9d5e7b-dbus\") pod \"global-pull-secret-syncer-2m5d7\" (UID: \"5ebd2878-a3f1-467a-b25f-29124e9d5e7b\") " pod="kube-system/global-pull-secret-syncer-2m5d7"
Sep 18 19:13:53.331374 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.331316    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a44929a1-5d90-443e-8d18-f31a4e7e9d28-multus-socket-dir-parent\") pod \"multus-v7jmf\" (UID: \"a44929a1-5d90-443e-8d18-f31a4e7e9d28\") " pod="openshift-multus/multus-v7jmf"
Sep 18 19:13:53.331374 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.331346    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a44929a1-5d90-443e-8d18-f31a4e7e9d28-hostroot\") pod \"multus-v7jmf\" (UID: \"a44929a1-5d90-443e-8d18-f31a4e7e9d28\") " pod="openshift-multus/multus-v7jmf"
Sep 18 19:13:53.331374 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.331370    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a44929a1-5d90-443e-8d18-f31a4e7e9d28-multus-conf-dir\") pod \"multus-v7jmf\" (UID: \"a44929a1-5d90-443e-8d18-f31a4e7e9d28\") " pod="openshift-multus/multus-v7jmf"
Sep 18 19:13:53.331535 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.331396    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/18e3a656-c475-4016-a531-8b17164a1692-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fq6nw\" (UID: \"18e3a656-c475-4016-a531-8b17164a1692\") " pod="openshift-multus/multus-additional-cni-plugins-fq6nw"
Sep 18 19:13:53.331535 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.331420    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nmqd\" (UniqueName: \"kubernetes.io/projected/ea018bdf-321c-4368-875d-937e82b2ed3b-kube-api-access-7nmqd\") pod \"node-ca-mx49g\" (UID: \"ea018bdf-321c-4368-875d-937e82b2ed3b\") " pod="openshift-image-registry/node-ca-mx49g"
Sep 18 19:13:53.331535 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.331454    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5ebd2878-a3f1-467a-b25f-29124e9d5e7b-kubelet-config\") pod \"global-pull-secret-syncer-2m5d7\" (UID: \"5ebd2878-a3f1-467a-b25f-29124e9d5e7b\") " pod="kube-system/global-pull-secret-syncer-2m5d7"
Sep 18 19:13:53.331535 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.331477    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9554bf8e-7591-4261-be81-a00cf9629c8b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xfddm\" (UID: \"9554bf8e-7591-4261-be81-a00cf9629c8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfddm"
Sep 18 19:13:53.331535 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.331495    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e41c46ea-8012-448d-9a65-0c67c54bbad1-ovn-node-metrics-cert\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.331535 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.331524    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch6vp\" (UniqueName: \"kubernetes.io/projected/a44929a1-5d90-443e-8d18-f31a4e7e9d28-kube-api-access-ch6vp\") pod \"multus-v7jmf\" (UID: \"a44929a1-5d90-443e-8d18-f31a4e7e9d28\") " pod="openshift-multus/multus-v7jmf"
Sep 18 19:13:53.331788 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.331544    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/18e3a656-c475-4016-a531-8b17164a1692-system-cni-dir\") pod \"multus-additional-cni-plugins-fq6nw\" (UID: \"18e3a656-c475-4016-a531-8b17164a1692\") " pod="openshift-multus/multus-additional-cni-plugins-fq6nw"
Sep 18 19:13:53.331788 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.331563    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9554bf8e-7591-4261-be81-a00cf9629c8b-etc-selinux\") pod \"aws-ebs-csi-driver-node-xfddm\" (UID: \"9554bf8e-7591-4261-be81-a00cf9629c8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfddm"
Sep 18 19:13:53.331788 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.331585    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d6wc\" (UniqueName: \"kubernetes.io/projected/b7797c1c-31e6-4f63-84d2-6f3c99e195ee-kube-api-access-6d6wc\") pod \"node-resolver-5jzcl\" (UID: \"b7797c1c-31e6-4f63-84d2-6f3c99e195ee\") " pod="openshift-dns/node-resolver-5jzcl"
Sep 18 19:13:53.331788 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.331621    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a44929a1-5d90-443e-8d18-f31a4e7e9d28-cnibin\") pod \"multus-v7jmf\" (UID: \"a44929a1-5d90-443e-8d18-f31a4e7e9d28\") " pod="openshift-multus/multus-v7jmf"
Sep 18 19:13:53.331788 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.331651    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e41c46ea-8012-448d-9a65-0c67c54bbad1-node-log\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.331788 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.331675    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6f05b5b2-52b5-46f0-8003-30f0bd24c71c-etc-sysconfig\") pod \"tuned-hbbzh\" (UID: \"6f05b5b2-52b5-46f0-8003-30f0bd24c71c\") " pod="openshift-cluster-node-tuning-operator/tuned-hbbzh"
Sep 18 19:13:53.331788 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.331699    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6f05b5b2-52b5-46f0-8003-30f0bd24c71c-etc-sysctl-d\") pod \"tuned-hbbzh\" (UID: \"6f05b5b2-52b5-46f0-8003-30f0bd24c71c\") " pod="openshift-cluster-node-tuning-operator/tuned-hbbzh"
Sep 18 19:13:53.331788 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.331727    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/18e3a656-c475-4016-a531-8b17164a1692-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fq6nw\" (UID: \"18e3a656-c475-4016-a531-8b17164a1692\") " pod="openshift-multus/multus-additional-cni-plugins-fq6nw"
Sep 18 19:13:53.331788 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.331761    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cb48998e-b58d-4478-91ed-42a78b2ed6f9-iptables-alerter-script\") pod \"iptables-alerter-lrqrg\" (UID: \"cb48998e-b58d-4478-91ed-42a78b2ed6f9\") " pod="openshift-network-operator/iptables-alerter-lrqrg"
Sep 18 19:13:53.331788 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.331787    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ea018bdf-321c-4368-875d-937e82b2ed3b-host\") pod \"node-ca-mx49g\" (UID: \"ea018bdf-321c-4368-875d-937e82b2ed3b\") " pod="openshift-image-registry/node-ca-mx49g"
Sep 18 19:13:53.332094 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.331803    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a44929a1-5d90-443e-8d18-f31a4e7e9d28-etc-kubernetes\") pod \"multus-v7jmf\" (UID: \"a44929a1-5d90-443e-8d18-f31a4e7e9d28\") " pod="openshift-multus/multus-v7jmf"
Sep 18 19:13:53.332094 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.331824    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzfjc\" (UniqueName: \"kubernetes.io/projected/cb48998e-b58d-4478-91ed-42a78b2ed6f9-kube-api-access-pzfjc\") pod \"iptables-alerter-lrqrg\" (UID: \"cb48998e-b58d-4478-91ed-42a78b2ed6f9\") " pod="openshift-network-operator/iptables-alerter-lrqrg"
Sep 18 19:13:53.332094 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.331855    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e41c46ea-8012-448d-9a65-0c67c54bbad1-run-openvswitch\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.332094 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.331885    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6f05b5b2-52b5-46f0-8003-30f0bd24c71c-etc-modprobe-d\") pod \"tuned-hbbzh\" (UID: \"6f05b5b2-52b5-46f0-8003-30f0bd24c71c\") " pod="openshift-cluster-node-tuning-operator/tuned-hbbzh"
Sep 18 19:13:53.332094 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.331909    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f05b5b2-52b5-46f0-8003-30f0bd24c71c-var-lib-kubelet\") pod \"tuned-hbbzh\" (UID: \"6f05b5b2-52b5-46f0-8003-30f0bd24c71c\") " pod="openshift-cluster-node-tuning-operator/tuned-hbbzh"
Sep 18 19:13:53.332094 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.331929    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fd53f3e2-ce65-4c01-9a2b-10803d702989-agent-certs\") pod \"konnectivity-agent-s6qfb\" (UID: \"fd53f3e2-ce65-4c01-9a2b-10803d702989\") " pod="kube-system/konnectivity-agent-s6qfb"
Sep 18 19:13:53.332094 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.331952    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e41c46ea-8012-448d-9a65-0c67c54bbad1-host-run-netns\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.332094 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.331967    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e41c46ea-8012-448d-9a65-0c67c54bbad1-var-lib-openvswitch\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.332094 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.331981    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e41c46ea-8012-448d-9a65-0c67c54bbad1-env-overrides\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.332094 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.332014    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6f05b5b2-52b5-46f0-8003-30f0bd24c71c-etc-sysctl-conf\") pod \"tuned-hbbzh\" (UID: \"6f05b5b2-52b5-46f0-8003-30f0bd24c71c\") " pod="openshift-cluster-node-tuning-operator/tuned-hbbzh"
Sep 18 19:13:53.332094 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.332040    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6f05b5b2-52b5-46f0-8003-30f0bd24c71c-run\") pod \"tuned-hbbzh\" (UID: \"6f05b5b2-52b5-46f0-8003-30f0bd24c71c\") " pod="openshift-cluster-node-tuning-operator/tuned-hbbzh"
Sep 18 19:13:53.332094 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.332063    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0bb7eb5-2fe7-42f3-92a6-00d65358f28f-metrics-certs\") pod \"network-metrics-daemon-l9g57\" (UID: \"b0bb7eb5-2fe7-42f3-92a6-00d65358f28f\") " pod="openshift-multus/network-metrics-daemon-l9g57"
Sep 18 19:13:53.332484 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.332097    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ea018bdf-321c-4368-875d-937e82b2ed3b-serviceca\") pod \"node-ca-mx49g\" (UID: \"ea018bdf-321c-4368-875d-937e82b2ed3b\") " pod="openshift-image-registry/node-ca-mx49g"
Sep 18 19:13:53.332484 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.332120    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a44929a1-5d90-443e-8d18-f31a4e7e9d28-host-var-lib-kubelet\") pod \"multus-v7jmf\" (UID: \"a44929a1-5d90-443e-8d18-f31a4e7e9d28\") " pod="openshift-multus/multus-v7jmf"
Sep 18 19:13:53.332484 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.332138    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46lwh\" (UniqueName: \"kubernetes.io/projected/1bd5d395-d4f3-489a-a20b-1fd771f55da6-kube-api-access-46lwh\") pod \"network-check-target-2fpv8\" (UID: \"1bd5d395-d4f3-489a-a20b-1fd771f55da6\") " pod="openshift-network-diagnostics/network-check-target-2fpv8"
Sep 18 19:13:53.332484 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.332161    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6f05b5b2-52b5-46f0-8003-30f0bd24c71c-sys\") pod \"tuned-hbbzh\" (UID: \"6f05b5b2-52b5-46f0-8003-30f0bd24c71c\") " pod="openshift-cluster-node-tuning-operator/tuned-hbbzh"
Sep 18 19:13:53.332484 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.332178    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6f05b5b2-52b5-46f0-8003-30f0bd24c71c-lib-modules\") pod \"tuned-hbbzh\" (UID: \"6f05b5b2-52b5-46f0-8003-30f0bd24c71c\") " pod="openshift-cluster-node-tuning-operator/tuned-hbbzh"
Sep 18 19:13:53.332484 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.332191    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a44929a1-5d90-443e-8d18-f31a4e7e9d28-multus-cni-dir\") pod \"multus-v7jmf\" (UID: \"a44929a1-5d90-443e-8d18-f31a4e7e9d28\") " pod="openshift-multus/multus-v7jmf"
Sep 18 19:13:53.332484 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.332210    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a44929a1-5d90-443e-8d18-f31a4e7e9d28-cni-binary-copy\") pod \"multus-v7jmf\" (UID: \"a44929a1-5d90-443e-8d18-f31a4e7e9d28\") " pod="openshift-multus/multus-v7jmf"
Sep 18 19:13:53.332484 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.332238    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/18e3a656-c475-4016-a531-8b17164a1692-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fq6nw\" (UID: \"18e3a656-c475-4016-a531-8b17164a1692\") " pod="openshift-multus/multus-additional-cni-plugins-fq6nw"
Sep 18 19:13:53.332484 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.332260    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krtbr\" (UniqueName: \"kubernetes.io/projected/e41c46ea-8012-448d-9a65-0c67c54bbad1-kube-api-access-krtbr\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.332484 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.332275    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f05b5b2-52b5-46f0-8003-30f0bd24c71c-etc-kubernetes\") pod \"tuned-hbbzh\" (UID: \"6f05b5b2-52b5-46f0-8003-30f0bd24c71c\") " pod="openshift-cluster-node-tuning-operator/tuned-hbbzh"
Sep 18 19:13:53.332484 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.332296    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6f05b5b2-52b5-46f0-8003-30f0bd24c71c-etc-tuned\") pod \"tuned-hbbzh\" (UID: \"6f05b5b2-52b5-46f0-8003-30f0bd24c71c\") " pod="openshift-cluster-node-tuning-operator/tuned-hbbzh"
Sep 18 19:13:53.332484 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.332333    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a44929a1-5d90-443e-8d18-f31a4e7e9d28-host-run-k8s-cni-cncf-io\") pod \"multus-v7jmf\" (UID: \"a44929a1-5d90-443e-8d18-f31a4e7e9d28\") " pod="openshift-multus/multus-v7jmf"
Sep 18 19:13:53.332484 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.332371    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a44929a1-5d90-443e-8d18-f31a4e7e9d28-host-run-multus-certs\") pod \"multus-v7jmf\" (UID: \"a44929a1-5d90-443e-8d18-f31a4e7e9d28\") " pod="openshift-multus/multus-v7jmf"
Sep 18 19:13:53.332484 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.332393    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e41c46ea-8012-448d-9a65-0c67c54bbad1-systemd-units\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.332484 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.332407    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6f05b5b2-52b5-46f0-8003-30f0bd24c71c-etc-systemd\") pod \"tuned-hbbzh\" (UID: \"6f05b5b2-52b5-46f0-8003-30f0bd24c71c\") " pod="openshift-cluster-node-tuning-operator/tuned-hbbzh"
Sep 18 19:13:53.332484 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.332440    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sm82\" (UniqueName: \"kubernetes.io/projected/6f05b5b2-52b5-46f0-8003-30f0bd24c71c-kube-api-access-7sm82\") pod \"tuned-hbbzh\" (UID: \"6f05b5b2-52b5-46f0-8003-30f0bd24c71c\") " pod="openshift-cluster-node-tuning-operator/tuned-hbbzh"
Sep 18 19:13:53.332484 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.332460    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a44929a1-5d90-443e-8d18-f31a4e7e9d28-system-cni-dir\") pod \"multus-v7jmf\" (UID: \"a44929a1-5d90-443e-8d18-f31a4e7e9d28\") " pod="openshift-multus/multus-v7jmf"
Sep 18 19:13:53.333089 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.332493    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a44929a1-5d90-443e-8d18-f31a4e7e9d28-host-run-netns\") pod \"multus-v7jmf\" (UID: \"a44929a1-5d90-443e-8d18-f31a4e7e9d28\") " pod="openshift-multus/multus-v7jmf"
Sep 18 19:13:53.333089 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.332522    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a44929a1-5d90-443e-8d18-f31a4e7e9d28-multus-daemon-config\") pod \"multus-v7jmf\" (UID: \"a44929a1-5d90-443e-8d18-f31a4e7e9d28\") " pod="openshift-multus/multus-v7jmf"
Sep 18 19:13:53.333089 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.332543    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb48998e-b58d-4478-91ed-42a78b2ed6f9-host-slash\") pod \"iptables-alerter-lrqrg\" (UID: \"cb48998e-b58d-4478-91ed-42a78b2ed6f9\") " pod="openshift-network-operator/iptables-alerter-lrqrg"
Sep 18 19:13:53.333089 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.332557    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e41c46ea-8012-448d-9a65-0c67c54bbad1-run-ovn\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.333089 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.332575    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e41c46ea-8012-448d-9a65-0c67c54bbad1-host-cni-netd\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.333089 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.332599    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e41c46ea-8012-448d-9a65-0c67c54bbad1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.333089 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.332635    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fd53f3e2-ce65-4c01-9a2b-10803d702989-konnectivity-ca\") pod \"konnectivity-agent-s6qfb\" (UID: \"fd53f3e2-ce65-4c01-9a2b-10803d702989\") " pod="kube-system/konnectivity-agent-s6qfb"
Sep 18 19:13:53.333089 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.332661    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/18e3a656-c475-4016-a531-8b17164a1692-cni-binary-copy\") pod \"multus-additional-cni-plugins-fq6nw\" (UID: \"18e3a656-c475-4016-a531-8b17164a1692\") " pod="openshift-multus/multus-additional-cni-plugins-fq6nw"
Sep 18 19:13:53.333089 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.332694    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9554bf8e-7591-4261-be81-a00cf9629c8b-registration-dir\") pod \"aws-ebs-csi-driver-node-xfddm\" (UID: \"9554bf8e-7591-4261-be81-a00cf9629c8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfddm"
Sep 18 19:13:53.333089 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.332720    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e41c46ea-8012-448d-9a65-0c67c54bbad1-run-systemd\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.333089 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.332750    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e41c46ea-8012-448d-9a65-0c67c54bbad1-host-cni-bin\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.333089 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.332773    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6f05b5b2-52b5-46f0-8003-30f0bd24c71c-tmp\") pod \"tuned-hbbzh\" (UID: \"6f05b5b2-52b5-46f0-8003-30f0bd24c71c\") " pod="openshift-cluster-node-tuning-operator/tuned-hbbzh"
Sep 18 19:13:53.333089 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.332797    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b7797c1c-31e6-4f63-84d2-6f3c99e195ee-hosts-file\") pod \"node-resolver-5jzcl\" (UID: \"b7797c1c-31e6-4f63-84d2-6f3c99e195ee\") " pod="openshift-dns/node-resolver-5jzcl"
Sep 18 19:13:53.333089 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.332817    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a44929a1-5d90-443e-8d18-f31a4e7e9d28-host-var-lib-cni-bin\") pod \"multus-v7jmf\" (UID: \"a44929a1-5d90-443e-8d18-f31a4e7e9d28\") " pod="openshift-multus/multus-v7jmf"
Sep 18 19:13:53.333089 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.332837    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/18e3a656-c475-4016-a531-8b17164a1692-cnibin\") pod \"multus-additional-cni-plugins-fq6nw\" (UID: \"18e3a656-c475-4016-a531-8b17164a1692\") " pod="openshift-multus/multus-additional-cni-plugins-fq6nw"
Sep 18 19:13:53.333089 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.332859    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ebd2878-a3f1-467a-b25f-29124e9d5e7b-original-pull-secret\") pod \"global-pull-secret-syncer-2m5d7\" (UID: \"5ebd2878-a3f1-467a-b25f-29124e9d5e7b\") " pod="kube-system/global-pull-secret-syncer-2m5d7"
Sep 18 19:13:53.333678 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.332878    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9554bf8e-7591-4261-be81-a00cf9629c8b-socket-dir\") pod \"aws-ebs-csi-driver-node-xfddm\" (UID: \"9554bf8e-7591-4261-be81-a00cf9629c8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfddm"
Sep 18 19:13:53.333678 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.332897    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e41c46ea-8012-448d-9a65-0c67c54bbad1-host-slash\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.333678 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.332920    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e41c46ea-8012-448d-9a65-0c67c54bbad1-etc-openvswitch\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.333678 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.332941    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e41c46ea-8012-448d-9a65-0c67c54bbad1-log-socket\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.333678 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.332963    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e41c46ea-8012-448d-9a65-0c67c54bbad1-ovnkube-config\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.333678 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.333053    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blxbd\" (UniqueName: \"kubernetes.io/projected/18e3a656-c475-4016-a531-8b17164a1692-kube-api-access-blxbd\") pod \"multus-additional-cni-plugins-fq6nw\" (UID: \"18e3a656-c475-4016-a531-8b17164a1692\") " pod="openshift-multus/multus-additional-cni-plugins-fq6nw"
Sep 18 19:13:53.333678 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.333078    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9554bf8e-7591-4261-be81-a00cf9629c8b-sys-fs\") pod \"aws-ebs-csi-driver-node-xfddm\" (UID: \"9554bf8e-7591-4261-be81-a00cf9629c8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfddm"
Sep 18 19:13:53.333678 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.333102    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmpp5\" (UniqueName: \"kubernetes.io/projected/9554bf8e-7591-4261-be81-a00cf9629c8b-kube-api-access-tmpp5\") pod \"aws-ebs-csi-driver-node-xfddm\" (UID: \"9554bf8e-7591-4261-be81-a00cf9629c8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfddm"
Sep 18 19:13:53.333678 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.333125    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e41c46ea-8012-448d-9a65-0c67c54bbad1-ovnkube-script-lib\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.333678 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.333144    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vgjv\" (UniqueName: \"kubernetes.io/projected/b0bb7eb5-2fe7-42f3-92a6-00d65358f28f-kube-api-access-6vgjv\") pod \"network-metrics-daemon-l9g57\" (UID: \"b0bb7eb5-2fe7-42f3-92a6-00d65358f28f\") " pod="openshift-multus/network-metrics-daemon-l9g57"
Sep 18 19:13:53.333678 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.333160    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7797c1c-31e6-4f63-84d2-6f3c99e195ee-tmp-dir\") pod \"node-resolver-5jzcl\" (UID: \"b7797c1c-31e6-4f63-84d2-6f3c99e195ee\") " pod="openshift-dns/node-resolver-5jzcl"
Sep 18 19:13:53.333678 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.333183    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a44929a1-5d90-443e-8d18-f31a4e7e9d28-host-var-lib-cni-multus\") pod \"multus-v7jmf\" (UID: \"a44929a1-5d90-443e-8d18-f31a4e7e9d28\") " pod="openshift-multus/multus-v7jmf"
Sep 18 19:13:53.347341 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.347316    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 19:13:53.373907 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.373883    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-7hnf4"
Sep 18 19:13:53.383832 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.383806    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7hnf4"
Sep 18 19:13:53.423005 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.422962    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 18 19:13:53.433387 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.433353    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9554bf8e-7591-4261-be81-a00cf9629c8b-sys-fs\") pod \"aws-ebs-csi-driver-node-xfddm\" (UID: \"9554bf8e-7591-4261-be81-a00cf9629c8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfddm"
Sep 18 19:13:53.433536 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.433408    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmpp5\" (UniqueName: \"kubernetes.io/projected/9554bf8e-7591-4261-be81-a00cf9629c8b-kube-api-access-tmpp5\") pod \"aws-ebs-csi-driver-node-xfddm\" (UID: \"9554bf8e-7591-4261-be81-a00cf9629c8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfddm"
Sep 18 19:13:53.433536 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.433435    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e41c46ea-8012-448d-9a65-0c67c54bbad1-ovnkube-script-lib\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.433536 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.433463    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vgjv\" (UniqueName: \"kubernetes.io/projected/b0bb7eb5-2fe7-42f3-92a6-00d65358f28f-kube-api-access-6vgjv\") pod \"network-metrics-daemon-l9g57\" (UID: \"b0bb7eb5-2fe7-42f3-92a6-00d65358f28f\") " pod="openshift-multus/network-metrics-daemon-l9g57"
Sep 18 19:13:53.433536 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.433489    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7797c1c-31e6-4f63-84d2-6f3c99e195ee-tmp-dir\") pod \"node-resolver-5jzcl\" (UID: \"b7797c1c-31e6-4f63-84d2-6f3c99e195ee\") " pod="openshift-dns/node-resolver-5jzcl"
Sep 18 19:13:53.433536 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.433512    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a44929a1-5d90-443e-8d18-f31a4e7e9d28-host-var-lib-cni-multus\") pod \"multus-v7jmf\" (UID: \"a44929a1-5d90-443e-8d18-f31a4e7e9d28\") " pod="openshift-multus/multus-v7jmf"
Sep 18 19:13:53.433536 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.433536    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9554bf8e-7591-4261-be81-a00cf9629c8b-device-dir\") pod \"aws-ebs-csi-driver-node-xfddm\" (UID: \"9554bf8e-7591-4261-be81-a00cf9629c8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfddm"
Sep 18 19:13:53.433830 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.433561    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e41c46ea-8012-448d-9a65-0c67c54bbad1-host-run-ovn-kubernetes\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.433830 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.433613    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6f05b5b2-52b5-46f0-8003-30f0bd24c71c-host\") pod \"tuned-hbbzh\" (UID: \"6f05b5b2-52b5-46f0-8003-30f0bd24c71c\") " pod="openshift-cluster-node-tuning-operator/tuned-hbbzh"
Sep 18 19:13:53.433830 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.433636    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e41c46ea-8012-448d-9a65-0c67c54bbad1-host-kubelet\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.433830 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.433662    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5ebd2878-a3f1-467a-b25f-29124e9d5e7b-dbus\") pod \"global-pull-secret-syncer-2m5d7\" (UID: \"5ebd2878-a3f1-467a-b25f-29124e9d5e7b\") " pod="kube-system/global-pull-secret-syncer-2m5d7"
Sep 18 19:13:53.433830 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.433689    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a44929a1-5d90-443e-8d18-f31a4e7e9d28-multus-socket-dir-parent\") pod \"multus-v7jmf\" (UID: \"a44929a1-5d90-443e-8d18-f31a4e7e9d28\") " pod="openshift-multus/multus-v7jmf"
Sep 18 19:13:53.433830 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.433715    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a44929a1-5d90-443e-8d18-f31a4e7e9d28-hostroot\") pod \"multus-v7jmf\" (UID: \"a44929a1-5d90-443e-8d18-f31a4e7e9d28\") " pod="openshift-multus/multus-v7jmf"
Sep 18 19:13:53.433830 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.433755    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a44929a1-5d90-443e-8d18-f31a4e7e9d28-multus-conf-dir\") pod \"multus-v7jmf\" (UID: \"a44929a1-5d90-443e-8d18-f31a4e7e9d28\") " pod="openshift-multus/multus-v7jmf"
Sep 18 19:13:53.433830 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.433782    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/18e3a656-c475-4016-a531-8b17164a1692-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fq6nw\" (UID: \"18e3a656-c475-4016-a531-8b17164a1692\") " pod="openshift-multus/multus-additional-cni-plugins-fq6nw"
Sep 18 19:13:53.433830 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.433807    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nmqd\" (UniqueName: \"kubernetes.io/projected/ea018bdf-321c-4368-875d-937e82b2ed3b-kube-api-access-7nmqd\") pod \"node-ca-mx49g\" (UID: \"ea018bdf-321c-4368-875d-937e82b2ed3b\") " pod="openshift-image-registry/node-ca-mx49g"
Sep 18 19:13:53.434271 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.433836    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5ebd2878-a3f1-467a-b25f-29124e9d5e7b-kubelet-config\") pod \"global-pull-secret-syncer-2m5d7\" (UID: \"5ebd2878-a3f1-467a-b25f-29124e9d5e7b\") " pod="kube-system/global-pull-secret-syncer-2m5d7"
Sep 18 19:13:53.434271 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.433861    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9554bf8e-7591-4261-be81-a00cf9629c8b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xfddm\" (UID: \"9554bf8e-7591-4261-be81-a00cf9629c8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfddm"
Sep 18 19:13:53.434271 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.433884    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e41c46ea-8012-448d-9a65-0c67c54bbad1-ovn-node-metrics-cert\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.434271 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.433920    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch6vp\" (UniqueName: \"kubernetes.io/projected/a44929a1-5d90-443e-8d18-f31a4e7e9d28-kube-api-access-ch6vp\") pod \"multus-v7jmf\" (UID: \"a44929a1-5d90-443e-8d18-f31a4e7e9d28\") " pod="openshift-multus/multus-v7jmf"
Sep 18 19:13:53.434271 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.433944    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/18e3a656-c475-4016-a531-8b17164a1692-system-cni-dir\") pod \"multus-additional-cni-plugins-fq6nw\" (UID: \"18e3a656-c475-4016-a531-8b17164a1692\") " pod="openshift-multus/multus-additional-cni-plugins-fq6nw"
Sep 18 19:13:53.434271 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.433969    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9554bf8e-7591-4261-be81-a00cf9629c8b-etc-selinux\") pod \"aws-ebs-csi-driver-node-xfddm\" (UID: \"9554bf8e-7591-4261-be81-a00cf9629c8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfddm"
Sep 18 19:13:53.434271 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.434009    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d6wc\" (UniqueName: \"kubernetes.io/projected/b7797c1c-31e6-4f63-84d2-6f3c99e195ee-kube-api-access-6d6wc\") pod \"node-resolver-5jzcl\" (UID: \"b7797c1c-31e6-4f63-84d2-6f3c99e195ee\") " pod="openshift-dns/node-resolver-5jzcl"
Sep 18 19:13:53.434271 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.434032    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a44929a1-5d90-443e-8d18-f31a4e7e9d28-cnibin\") pod \"multus-v7jmf\" (UID: \"a44929a1-5d90-443e-8d18-f31a4e7e9d28\") " pod="openshift-multus/multus-v7jmf"
Sep 18 19:13:53.434271 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.434056    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e41c46ea-8012-448d-9a65-0c67c54bbad1-node-log\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.434271 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.434084    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7797c1c-31e6-4f63-84d2-6f3c99e195ee-tmp-dir\") pod \"node-resolver-5jzcl\" (UID: \"b7797c1c-31e6-4f63-84d2-6f3c99e195ee\") " pod="openshift-dns/node-resolver-5jzcl"
Sep 18 19:13:53.434271 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.434078    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6f05b5b2-52b5-46f0-8003-30f0bd24c71c-etc-sysconfig\") pod \"tuned-hbbzh\" (UID: \"6f05b5b2-52b5-46f0-8003-30f0bd24c71c\") " pod="openshift-cluster-node-tuning-operator/tuned-hbbzh"
Sep 18 19:13:53.434271 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.434119    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6f05b5b2-52b5-46f0-8003-30f0bd24c71c-etc-sysctl-d\") pod \"tuned-hbbzh\" (UID: \"6f05b5b2-52b5-46f0-8003-30f0bd24c71c\") " pod="openshift-cluster-node-tuning-operator/tuned-hbbzh"
Sep 18 19:13:53.434271 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.434143    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/18e3a656-c475-4016-a531-8b17164a1692-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fq6nw\" (UID: \"18e3a656-c475-4016-a531-8b17164a1692\") " pod="openshift-multus/multus-additional-cni-plugins-fq6nw"
Sep 18 19:13:53.434271 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.434170    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cb48998e-b58d-4478-91ed-42a78b2ed6f9-iptables-alerter-script\") pod \"iptables-alerter-lrqrg\" (UID: \"cb48998e-b58d-4478-91ed-42a78b2ed6f9\") " pod="openshift-network-operator/iptables-alerter-lrqrg"
Sep 18 19:13:53.434271 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.434194    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ea018bdf-321c-4368-875d-937e82b2ed3b-host\") pod \"node-ca-mx49g\" (UID: \"ea018bdf-321c-4368-875d-937e82b2ed3b\") " pod="openshift-image-registry/node-ca-mx49g"
Sep 18 19:13:53.434271 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.434218    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a44929a1-5d90-443e-8d18-f31a4e7e9d28-etc-kubernetes\") pod \"multus-v7jmf\" (UID: \"a44929a1-5d90-443e-8d18-f31a4e7e9d28\") " pod="openshift-multus/multus-v7jmf"
Sep 18 19:13:53.434271 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.434247    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzfjc\" (UniqueName: \"kubernetes.io/projected/cb48998e-b58d-4478-91ed-42a78b2ed6f9-kube-api-access-pzfjc\") pod \"iptables-alerter-lrqrg\" (UID: \"cb48998e-b58d-4478-91ed-42a78b2ed6f9\") " pod="openshift-network-operator/iptables-alerter-lrqrg"
Sep 18 19:13:53.435094 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.434273    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e41c46ea-8012-448d-9a65-0c67c54bbad1-run-openvswitch\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.435094 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.434296    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6f05b5b2-52b5-46f0-8003-30f0bd24c71c-etc-modprobe-d\") pod \"tuned-hbbzh\" (UID: \"6f05b5b2-52b5-46f0-8003-30f0bd24c71c\") " pod="openshift-cluster-node-tuning-operator/tuned-hbbzh"
Sep 18 19:13:53.435094 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.434320    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f05b5b2-52b5-46f0-8003-30f0bd24c71c-var-lib-kubelet\") pod \"tuned-hbbzh\" (UID: \"6f05b5b2-52b5-46f0-8003-30f0bd24c71c\") " pod="openshift-cluster-node-tuning-operator/tuned-hbbzh"
Sep 18 19:13:53.435094 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.434350    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fd53f3e2-ce65-4c01-9a2b-10803d702989-agent-certs\") pod \"konnectivity-agent-s6qfb\" (UID: \"fd53f3e2-ce65-4c01-9a2b-10803d702989\") " pod="kube-system/konnectivity-agent-s6qfb"
Sep 18 19:13:53.435094 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.434374    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e41c46ea-8012-448d-9a65-0c67c54bbad1-host-run-netns\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.435094 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.434400    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e41c46ea-8012-448d-9a65-0c67c54bbad1-var-lib-openvswitch\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.435094 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.434418    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e41c46ea-8012-448d-9a65-0c67c54bbad1-ovnkube-script-lib\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.435094 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.434426    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e41c46ea-8012-448d-9a65-0c67c54bbad1-env-overrides\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.435094 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.434452    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6f05b5b2-52b5-46f0-8003-30f0bd24c71c-etc-sysctl-conf\") pod \"tuned-hbbzh\" (UID: \"6f05b5b2-52b5-46f0-8003-30f0bd24c71c\") " pod="openshift-cluster-node-tuning-operator/tuned-hbbzh"
Sep 18 19:13:53.435094 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.434476    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6f05b5b2-52b5-46f0-8003-30f0bd24c71c-run\") pod \"tuned-hbbzh\" (UID: \"6f05b5b2-52b5-46f0-8003-30f0bd24c71c\") " pod="openshift-cluster-node-tuning-operator/tuned-hbbzh"
Sep 18 19:13:53.435094 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.434489    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e41c46ea-8012-448d-9a65-0c67c54bbad1-host-kubelet\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.435094 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.434500    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0bb7eb5-2fe7-42f3-92a6-00d65358f28f-metrics-certs\") pod \"network-metrics-daemon-l9g57\" (UID: \"b0bb7eb5-2fe7-42f3-92a6-00d65358f28f\") " pod="openshift-multus/network-metrics-daemon-l9g57"
Sep 18 19:13:53.435094 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.434524    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ea018bdf-321c-4368-875d-937e82b2ed3b-serviceca\") pod \"node-ca-mx49g\" (UID: \"ea018bdf-321c-4368-875d-937e82b2ed3b\") " pod="openshift-image-registry/node-ca-mx49g"
Sep 18 19:13:53.435094 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.434522    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9554bf8e-7591-4261-be81-a00cf9629c8b-sys-fs\") pod \"aws-ebs-csi-driver-node-xfddm\" (UID: \"9554bf8e-7591-4261-be81-a00cf9629c8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfddm"
Sep 18 19:13:53.435094 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.434548    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a44929a1-5d90-443e-8d18-f31a4e7e9d28-host-var-lib-kubelet\") pod \"multus-v7jmf\" (UID: \"a44929a1-5d90-443e-8d18-f31a4e7e9d28\") " pod="openshift-multus/multus-v7jmf"
Sep 18 19:13:53.435094 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.434585    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46lwh\" (UniqueName: \"kubernetes.io/projected/1bd5d395-d4f3-489a-a20b-1fd771f55da6-kube-api-access-46lwh\") pod \"network-check-target-2fpv8\" (UID: \"1bd5d395-d4f3-489a-a20b-1fd771f55da6\") " pod="openshift-network-diagnostics/network-check-target-2fpv8"
Sep 18 19:13:53.435094 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.434598    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a44929a1-5d90-443e-8d18-f31a4e7e9d28-host-var-lib-kubelet\") pod \"multus-v7jmf\" (UID: \"a44929a1-5d90-443e-8d18-f31a4e7e9d28\") " pod="openshift-multus/multus-v7jmf"
Sep 18 19:13:53.435962 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.434627    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5ebd2878-a3f1-467a-b25f-29124e9d5e7b-dbus\") pod \"global-pull-secret-syncer-2m5d7\" (UID: \"5ebd2878-a3f1-467a-b25f-29124e9d5e7b\") " pod="kube-system/global-pull-secret-syncer-2m5d7"
Sep 18 19:13:53.435962 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.434659    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5ebd2878-a3f1-467a-b25f-29124e9d5e7b-kubelet-config\") pod \"global-pull-secret-syncer-2m5d7\" (UID: \"5ebd2878-a3f1-467a-b25f-29124e9d5e7b\") " pod="kube-system/global-pull-secret-syncer-2m5d7"
Sep 18 19:13:53.435962 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.434670    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6f05b5b2-52b5-46f0-8003-30f0bd24c71c-sys\") pod \"tuned-hbbzh\" (UID: \"6f05b5b2-52b5-46f0-8003-30f0bd24c71c\") " pod="openshift-cluster-node-tuning-operator/tuned-hbbzh"
Sep 18 19:13:53.435962 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.434678    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a44929a1-5d90-443e-8d18-f31a4e7e9d28-host-var-lib-cni-multus\") pod \"multus-v7jmf\" (UID: \"a44929a1-5d90-443e-8d18-f31a4e7e9d28\") " pod="openshift-multus/multus-v7jmf"
Sep 18 19:13:53.435962 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.434708    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9554bf8e-7591-4261-be81-a00cf9629c8b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xfddm\" (UID: \"9554bf8e-7591-4261-be81-a00cf9629c8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfddm"
Sep 18 19:13:53.435962 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.434714    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a44929a1-5d90-443e-8d18-f31a4e7e9d28-multus-socket-dir-parent\") pod \"multus-v7jmf\" (UID: \"a44929a1-5d90-443e-8d18-f31a4e7e9d28\") " pod="openshift-multus/multus-v7jmf"
Sep 18 19:13:53.435962 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.434734    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a44929a1-5d90-443e-8d18-f31a4e7e9d28-multus-conf-dir\") pod \"multus-v7jmf\" (UID: \"a44929a1-5d90-443e-8d18-f31a4e7e9d28\") " pod="openshift-multus/multus-v7jmf"
Sep 18 19:13:53.435962 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.434783    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a44929a1-5d90-443e-8d18-f31a4e7e9d28-hostroot\") pod \"multus-v7jmf\" (UID: \"a44929a1-5d90-443e-8d18-f31a4e7e9d28\") " pod="openshift-multus/multus-v7jmf"
Sep 18 19:13:53.435962 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.434827    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e41c46ea-8012-448d-9a65-0c67c54bbad1-run-openvswitch\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.435962 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.434612    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6f05b5b2-52b5-46f0-8003-30f0bd24c71c-sys\") pod \"tuned-hbbzh\" (UID: \"6f05b5b2-52b5-46f0-8003-30f0bd24c71c\") " pod="openshift-cluster-node-tuning-operator/tuned-hbbzh"
Sep 18 19:13:53.435962 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.434893    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/18e3a656-c475-4016-a531-8b17164a1692-system-cni-dir\") pod \"multus-additional-cni-plugins-fq6nw\" (UID: \"18e3a656-c475-4016-a531-8b17164a1692\") " pod="openshift-multus/multus-additional-cni-plugins-fq6nw"
Sep 18 19:13:53.435962 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.434897    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6f05b5b2-52b5-46f0-8003-30f0bd24c71c-lib-modules\") pod \"tuned-hbbzh\" (UID: \"6f05b5b2-52b5-46f0-8003-30f0bd24c71c\") " pod="openshift-cluster-node-tuning-operator/tuned-hbbzh"
Sep 18 19:13:53.435962 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.435038    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6f05b5b2-52b5-46f0-8003-30f0bd24c71c-etc-sysctl-d\") pod \"tuned-hbbzh\" (UID: \"6f05b5b2-52b5-46f0-8003-30f0bd24c71c\") " pod="openshift-cluster-node-tuning-operator/tuned-hbbzh"
Sep 18 19:13:53.435962 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.435038    2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 18 19:13:53.435962 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.435077    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a44929a1-5d90-443e-8d18-f31a4e7e9d28-multus-cni-dir\") pod \"multus-v7jmf\" (UID: \"a44929a1-5d90-443e-8d18-f31a4e7e9d28\") " pod="openshift-multus/multus-v7jmf"
Sep 18 19:13:53.435962 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.435088    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6f05b5b2-52b5-46f0-8003-30f0bd24c71c-lib-modules\") pod \"tuned-hbbzh\" (UID: \"6f05b5b2-52b5-46f0-8003-30f0bd24c71c\") " pod="openshift-cluster-node-tuning-operator/tuned-hbbzh"
Sep 18 19:13:53.435962 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.435102    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a44929a1-5d90-443e-8d18-f31a4e7e9d28-cni-binary-copy\") pod \"multus-v7jmf\" (UID: \"a44929a1-5d90-443e-8d18-f31a4e7e9d28\") " pod="openshift-multus/multus-v7jmf"
Sep 18 19:13:53.435962 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.435125    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/18e3a656-c475-4016-a531-8b17164a1692-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fq6nw\" (UID: \"18e3a656-c475-4016-a531-8b17164a1692\") " pod="openshift-multus/multus-additional-cni-plugins-fq6nw"
Sep 18 19:13:53.436820 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.435147    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krtbr\" (UniqueName: \"kubernetes.io/projected/e41c46ea-8012-448d-9a65-0c67c54bbad1-kube-api-access-krtbr\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.436820 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.435201    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f05b5b2-52b5-46f0-8003-30f0bd24c71c-etc-kubernetes\") pod \"tuned-hbbzh\" (UID: \"6f05b5b2-52b5-46f0-8003-30f0bd24c71c\") " pod="openshift-cluster-node-tuning-operator/tuned-hbbzh"
Sep 18 19:13:53.436820 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.435226    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6f05b5b2-52b5-46f0-8003-30f0bd24c71c-etc-tuned\") pod \"tuned-hbbzh\" (UID: \"6f05b5b2-52b5-46f0-8003-30f0bd24c71c\") " pod="openshift-cluster-node-tuning-operator/tuned-hbbzh"
Sep 18 19:13:53.436820 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.435254    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a44929a1-5d90-443e-8d18-f31a4e7e9d28-host-run-k8s-cni-cncf-io\") pod \"multus-v7jmf\" (UID: \"a44929a1-5d90-443e-8d18-f31a4e7e9d28\") " pod="openshift-multus/multus-v7jmf"
Sep 18 19:13:53.436820 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.435279    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a44929a1-5d90-443e-8d18-f31a4e7e9d28-host-run-multus-certs\") pod \"multus-v7jmf\" (UID: \"a44929a1-5d90-443e-8d18-f31a4e7e9d28\") " pod="openshift-multus/multus-v7jmf"
Sep 18 19:13:53.436820 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.435299    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e41c46ea-8012-448d-9a65-0c67c54bbad1-systemd-units\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.436820 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.435298    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/18e3a656-c475-4016-a531-8b17164a1692-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fq6nw\" (UID: \"18e3a656-c475-4016-a531-8b17164a1692\") " pod="openshift-multus/multus-additional-cni-plugins-fq6nw"
Sep 18 19:13:53.436820 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.435323    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6f05b5b2-52b5-46f0-8003-30f0bd24c71c-etc-systemd\") pod \"tuned-hbbzh\" (UID: \"6f05b5b2-52b5-46f0-8003-30f0bd24c71c\") " pod="openshift-cluster-node-tuning-operator/tuned-hbbzh"
Sep 18 19:13:53.436820 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.435349    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sm82\" (UniqueName: \"kubernetes.io/projected/6f05b5b2-52b5-46f0-8003-30f0bd24c71c-kube-api-access-7sm82\") pod \"tuned-hbbzh\" (UID: \"6f05b5b2-52b5-46f0-8003-30f0bd24c71c\") " pod="openshift-cluster-node-tuning-operator/tuned-hbbzh"
Sep 18 19:13:53.436820 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.435374    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e41c46ea-8012-448d-9a65-0c67c54bbad1-host-run-ovn-kubernetes\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.436820 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.435376    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a44929a1-5d90-443e-8d18-f31a4e7e9d28-system-cni-dir\") pod \"multus-v7jmf\" (UID: \"a44929a1-5d90-443e-8d18-f31a4e7e9d28\") " pod="openshift-multus/multus-v7jmf"
Sep 18 19:13:53.436820 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.435442    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a44929a1-5d90-443e-8d18-f31a4e7e9d28-system-cni-dir\") pod \"multus-v7jmf\" (UID: \"a44929a1-5d90-443e-8d18-f31a4e7e9d28\") " pod="openshift-multus/multus-v7jmf"
Sep 18 19:13:53.436820 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.435473    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6f05b5b2-52b5-46f0-8003-30f0bd24c71c-etc-modprobe-d\") pod \"tuned-hbbzh\" (UID: \"6f05b5b2-52b5-46f0-8003-30f0bd24c71c\") " pod="openshift-cluster-node-tuning-operator/tuned-hbbzh"
Sep 18 19:13:53.436820 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.435502    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a44929a1-5d90-443e-8d18-f31a4e7e9d28-host-run-netns\") pod \"multus-v7jmf\" (UID: \"a44929a1-5d90-443e-8d18-f31a4e7e9d28\") " pod="openshift-multus/multus-v7jmf"
Sep 18 19:13:53.436820 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.435500    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f05b5b2-52b5-46f0-8003-30f0bd24c71c-var-lib-kubelet\") pod \"tuned-hbbzh\" (UID: \"6f05b5b2-52b5-46f0-8003-30f0bd24c71c\") " pod="openshift-cluster-node-tuning-operator/tuned-hbbzh"
Sep 18 19:13:53.436820 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.435528    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a44929a1-5d90-443e-8d18-f31a4e7e9d28-multus-daemon-config\") pod \"multus-v7jmf\" (UID: \"a44929a1-5d90-443e-8d18-f31a4e7e9d28\") " pod="openshift-multus/multus-v7jmf"
Sep 18 19:13:53.436820 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.435572    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e41c46ea-8012-448d-9a65-0c67c54bbad1-host-run-netns\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.437685 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.435611    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e41c46ea-8012-448d-9a65-0c67c54bbad1-var-lib-openvswitch\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.437685 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.434398    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9554bf8e-7591-4261-be81-a00cf9629c8b-device-dir\") pod \"aws-ebs-csi-driver-node-xfddm\" (UID: \"9554bf8e-7591-4261-be81-a00cf9629c8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfddm"
Sep 18 19:13:53.437685 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.434453    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6f05b5b2-52b5-46f0-8003-30f0bd24c71c-host\") pod \"tuned-hbbzh\" (UID: \"6f05b5b2-52b5-46f0-8003-30f0bd24c71c\") " pod="openshift-cluster-node-tuning-operator/tuned-hbbzh"
Sep 18 19:13:53.437685 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.435806    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6f05b5b2-52b5-46f0-8003-30f0bd24c71c-etc-sysctl-conf\") pod \"tuned-hbbzh\" (UID: \"6f05b5b2-52b5-46f0-8003-30f0bd24c71c\") " pod="openshift-cluster-node-tuning-operator/tuned-hbbzh"
Sep 18 19:13:53.437685 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.435847    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/18e3a656-c475-4016-a531-8b17164a1692-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fq6nw\" (UID: \"18e3a656-c475-4016-a531-8b17164a1692\") " pod="openshift-multus/multus-additional-cni-plugins-fq6nw"
Sep 18 19:13:53.437685 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.436170    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a44929a1-5d90-443e-8d18-f31a4e7e9d28-etc-kubernetes\") pod \"multus-v7jmf\" (UID: \"a44929a1-5d90-443e-8d18-f31a4e7e9d28\") " pod="openshift-multus/multus-v7jmf"
Sep 18 19:13:53.437685 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.436201    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cb48998e-b58d-4478-91ed-42a78b2ed6f9-iptables-alerter-script\") pod \"iptables-alerter-lrqrg\" (UID: \"cb48998e-b58d-4478-91ed-42a78b2ed6f9\") " pod="openshift-network-operator/iptables-alerter-lrqrg"
Sep 18 19:13:53.437685 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.436247    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e41c46ea-8012-448d-9a65-0c67c54bbad1-env-overrides\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.437685 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.436260    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f05b5b2-52b5-46f0-8003-30f0bd24c71c-etc-kubernetes\") pod \"tuned-hbbzh\" (UID: \"6f05b5b2-52b5-46f0-8003-30f0bd24c71c\") " pod="openshift-cluster-node-tuning-operator/tuned-hbbzh"
Sep 18 19:13:53.437685 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.435850    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ea018bdf-321c-4368-875d-937e82b2ed3b-host\") pod \"node-ca-mx49g\" (UID: \"ea018bdf-321c-4368-875d-937e82b2ed3b\") " pod="openshift-image-registry/node-ca-mx49g"
Sep 18 19:13:53.437685 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.436338    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9554bf8e-7591-4261-be81-a00cf9629c8b-etc-selinux\") pod \"aws-ebs-csi-driver-node-xfddm\" (UID: \"9554bf8e-7591-4261-be81-a00cf9629c8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfddm"
Sep 18 19:13:53.437685 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.436488    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a44929a1-5d90-443e-8d18-f31a4e7e9d28-multus-cni-dir\") pod \"multus-v7jmf\" (UID: \"a44929a1-5d90-443e-8d18-f31a4e7e9d28\") " pod="openshift-multus/multus-v7jmf"
Sep 18 19:13:53.437685 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.436556    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6f05b5b2-52b5-46f0-8003-30f0bd24c71c-etc-sysconfig\") pod \"tuned-hbbzh\" (UID: \"6f05b5b2-52b5-46f0-8003-30f0bd24c71c\") " pod="openshift-cluster-node-tuning-operator/tuned-hbbzh"
Sep 18 19:13:53.437685 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.436565    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a44929a1-5d90-443e-8d18-f31a4e7e9d28-cnibin\") pod \"multus-v7jmf\" (UID: \"a44929a1-5d90-443e-8d18-f31a4e7e9d28\") " pod="openshift-multus/multus-v7jmf"
Sep 18 19:13:53.437685 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.436616    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6f05b5b2-52b5-46f0-8003-30f0bd24c71c-etc-systemd\") pod \"tuned-hbbzh\" (UID: \"6f05b5b2-52b5-46f0-8003-30f0bd24c71c\") " pod="openshift-cluster-node-tuning-operator/tuned-hbbzh"
Sep 18 19:13:53.437685 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:53.436686    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:13:53.437685 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.436901    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6f05b5b2-52b5-46f0-8003-30f0bd24c71c-run\") pod \"tuned-hbbzh\" (UID: \"6f05b5b2-52b5-46f0-8003-30f0bd24c71c\") " pod="openshift-cluster-node-tuning-operator/tuned-hbbzh"
Sep 18 19:13:53.437685 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.437026    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a44929a1-5d90-443e-8d18-f31a4e7e9d28-cni-binary-copy\") pod \"multus-v7jmf\" (UID: \"a44929a1-5d90-443e-8d18-f31a4e7e9d28\") " pod="openshift-multus/multus-v7jmf"
Sep 18 19:13:53.438540 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.437110    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e41c46ea-8012-448d-9a65-0c67c54bbad1-node-log\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.438540 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.437239    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/18e3a656-c475-4016-a531-8b17164a1692-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fq6nw\" (UID: \"18e3a656-c475-4016-a531-8b17164a1692\") " pod="openshift-multus/multus-additional-cni-plugins-fq6nw"
Sep 18 19:13:53.438540 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.437477    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a44929a1-5d90-443e-8d18-f31a4e7e9d28-host-run-netns\") pod \"multus-v7jmf\" (UID: \"a44929a1-5d90-443e-8d18-f31a4e7e9d28\") " pod="openshift-multus/multus-v7jmf"
Sep 18 19:13:53.438540 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.437586    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ea018bdf-321c-4368-875d-937e82b2ed3b-serviceca\") pod \"node-ca-mx49g\" (UID: \"ea018bdf-321c-4368-875d-937e82b2ed3b\") " pod="openshift-image-registry/node-ca-mx49g"
Sep 18 19:13:53.438540 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.437686    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb48998e-b58d-4478-91ed-42a78b2ed6f9-host-slash\") pod \"iptables-alerter-lrqrg\" (UID: \"cb48998e-b58d-4478-91ed-42a78b2ed6f9\") " pod="openshift-network-operator/iptables-alerter-lrqrg"
Sep 18 19:13:53.438540 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.437717    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e41c46ea-8012-448d-9a65-0c67c54bbad1-run-ovn\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.438540 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.437746    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e41c46ea-8012-448d-9a65-0c67c54bbad1-host-cni-netd\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.438540 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.437773    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e41c46ea-8012-448d-9a65-0c67c54bbad1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.438540 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.437799    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fd53f3e2-ce65-4c01-9a2b-10803d702989-konnectivity-ca\") pod \"konnectivity-agent-s6qfb\" (UID: \"fd53f3e2-ce65-4c01-9a2b-10803d702989\") " pod="kube-system/konnectivity-agent-s6qfb"
Sep 18 19:13:53.438540 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.437828    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/18e3a656-c475-4016-a531-8b17164a1692-cni-binary-copy\") pod \"multus-additional-cni-plugins-fq6nw\" (UID: \"18e3a656-c475-4016-a531-8b17164a1692\") " pod="openshift-multus/multus-additional-cni-plugins-fq6nw"
Sep 18 19:13:53.438540 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.437853    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9554bf8e-7591-4261-be81-a00cf9629c8b-registration-dir\") pod \"aws-ebs-csi-driver-node-xfddm\" (UID: \"9554bf8e-7591-4261-be81-a00cf9629c8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfddm"
Sep 18 19:13:53.438540 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.437877    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e41c46ea-8012-448d-9a65-0c67c54bbad1-run-systemd\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.438540 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.437901    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e41c46ea-8012-448d-9a65-0c67c54bbad1-host-cni-bin\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.438540 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.437926    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6f05b5b2-52b5-46f0-8003-30f0bd24c71c-tmp\") pod \"tuned-hbbzh\" (UID: \"6f05b5b2-52b5-46f0-8003-30f0bd24c71c\") " pod="openshift-cluster-node-tuning-operator/tuned-hbbzh"
Sep 18 19:13:53.438540 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.437949    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b7797c1c-31e6-4f63-84d2-6f3c99e195ee-hosts-file\") pod \"node-resolver-5jzcl\" (UID: \"b7797c1c-31e6-4f63-84d2-6f3c99e195ee\") " pod="openshift-dns/node-resolver-5jzcl"
Sep 18 19:13:53.438540 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.437974    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a44929a1-5d90-443e-8d18-f31a4e7e9d28-host-var-lib-cni-bin\") pod \"multus-v7jmf\" (UID: \"a44929a1-5d90-443e-8d18-f31a4e7e9d28\") " pod="openshift-multus/multus-v7jmf"
Sep 18 19:13:53.438540 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.438018    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/18e3a656-c475-4016-a531-8b17164a1692-cnibin\") pod \"multus-additional-cni-plugins-fq6nw\" (UID: \"18e3a656-c475-4016-a531-8b17164a1692\") " pod="openshift-multus/multus-additional-cni-plugins-fq6nw"
Sep 18 19:13:53.439337 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.438044    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ebd2878-a3f1-467a-b25f-29124e9d5e7b-original-pull-secret\") pod \"global-pull-secret-syncer-2m5d7\" (UID: \"5ebd2878-a3f1-467a-b25f-29124e9d5e7b\") " pod="kube-system/global-pull-secret-syncer-2m5d7"
Sep 18 19:13:53.439337 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.438067    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9554bf8e-7591-4261-be81-a00cf9629c8b-socket-dir\") pod \"aws-ebs-csi-driver-node-xfddm\" (UID: \"9554bf8e-7591-4261-be81-a00cf9629c8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfddm"
Sep 18 19:13:53.439337 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.438087    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a44929a1-5d90-443e-8d18-f31a4e7e9d28-multus-daemon-config\") pod \"multus-v7jmf\" (UID: \"a44929a1-5d90-443e-8d18-f31a4e7e9d28\") " pod="openshift-multus/multus-v7jmf"
Sep 18 19:13:53.439337 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.438095    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e41c46ea-8012-448d-9a65-0c67c54bbad1-host-slash\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.439337 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.438149    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e41c46ea-8012-448d-9a65-0c67c54bbad1-run-systemd\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.439337 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.438149    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a44929a1-5d90-443e-8d18-f31a4e7e9d28-host-run-multus-certs\") pod \"multus-v7jmf\" (UID: \"a44929a1-5d90-443e-8d18-f31a4e7e9d28\") " pod="openshift-multus/multus-v7jmf"
Sep 18 19:13:53.439337 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.438151    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e41c46ea-8012-448d-9a65-0c67c54bbad1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.439337 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:53.438173    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0bb7eb5-2fe7-42f3-92a6-00d65358f28f-metrics-certs podName:b0bb7eb5-2fe7-42f3-92a6-00d65358f28f nodeName:}" failed. No retries permitted until 2026-09-18 19:13:53.938139722 +0000 UTC m=+4.790440080 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b0bb7eb5-2fe7-42f3-92a6-00d65358f28f-metrics-certs") pod "network-metrics-daemon-l9g57" (UID: "b0bb7eb5-2fe7-42f3-92a6-00d65358f28f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:13:53.439337 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.438186    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e41c46ea-8012-448d-9a65-0c67c54bbad1-host-slash\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.439337 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.438231    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e41c46ea-8012-448d-9a65-0c67c54bbad1-host-cni-netd\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.439337 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.438238    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b7797c1c-31e6-4f63-84d2-6f3c99e195ee-hosts-file\") pod \"node-resolver-5jzcl\" (UID: \"b7797c1c-31e6-4f63-84d2-6f3c99e195ee\") " pod="openshift-dns/node-resolver-5jzcl"
Sep 18 19:13:53.439337 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.438268    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e41c46ea-8012-448d-9a65-0c67c54bbad1-run-ovn\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.439337 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.438276    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9554bf8e-7591-4261-be81-a00cf9629c8b-registration-dir\") pod \"aws-ebs-csi-driver-node-xfddm\" (UID: \"9554bf8e-7591-4261-be81-a00cf9629c8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfddm"
Sep 18 19:13:53.439337 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.438314    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/18e3a656-c475-4016-a531-8b17164a1692-cnibin\") pod \"multus-additional-cni-plugins-fq6nw\" (UID: \"18e3a656-c475-4016-a531-8b17164a1692\") " pod="openshift-multus/multus-additional-cni-plugins-fq6nw"
Sep 18 19:13:53.439337 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:53.438342    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:13:53.439337 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:53.438387    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ebd2878-a3f1-467a-b25f-29124e9d5e7b-original-pull-secret podName:5ebd2878-a3f1-467a-b25f-29124e9d5e7b nodeName:}" failed. No retries permitted until 2026-09-18 19:13:53.938371174 +0000 UTC m=+4.790671525 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ebd2878-a3f1-467a-b25f-29124e9d5e7b-original-pull-secret") pod "global-pull-secret-syncer-2m5d7" (UID: "5ebd2878-a3f1-467a-b25f-29124e9d5e7b") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:13:53.440098 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.438477    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9554bf8e-7591-4261-be81-a00cf9629c8b-socket-dir\") pod \"aws-ebs-csi-driver-node-xfddm\" (UID: \"9554bf8e-7591-4261-be81-a00cf9629c8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfddm"
Sep 18 19:13:53.440098 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.438508    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e41c46ea-8012-448d-9a65-0c67c54bbad1-etc-openvswitch\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.440098 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.438534    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e41c46ea-8012-448d-9a65-0c67c54bbad1-log-socket\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.440098 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.438575    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e41c46ea-8012-448d-9a65-0c67c54bbad1-ovnkube-config\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.440098 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.438601    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blxbd\" (UniqueName: \"kubernetes.io/projected/18e3a656-c475-4016-a531-8b17164a1692-kube-api-access-blxbd\") pod \"multus-additional-cni-plugins-fq6nw\" (UID: \"18e3a656-c475-4016-a531-8b17164a1692\") " pod="openshift-multus/multus-additional-cni-plugins-fq6nw"
Sep 18 19:13:53.440098 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.438603    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/18e3a656-c475-4016-a531-8b17164a1692-cni-binary-copy\") pod \"multus-additional-cni-plugins-fq6nw\" (UID: \"18e3a656-c475-4016-a531-8b17164a1692\") " pod="openshift-multus/multus-additional-cni-plugins-fq6nw"
Sep 18 19:13:53.440098 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.438654    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e41c46ea-8012-448d-9a65-0c67c54bbad1-host-cni-bin\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.440098 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.438688    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb48998e-b58d-4478-91ed-42a78b2ed6f9-host-slash\") pod \"iptables-alerter-lrqrg\" (UID: \"cb48998e-b58d-4478-91ed-42a78b2ed6f9\") " pod="openshift-network-operator/iptables-alerter-lrqrg"
Sep 18 19:13:53.440098 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.438691    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a44929a1-5d90-443e-8d18-f31a4e7e9d28-host-run-k8s-cni-cncf-io\") pod \"multus-v7jmf\" (UID: \"a44929a1-5d90-443e-8d18-f31a4e7e9d28\") " pod="openshift-multus/multus-v7jmf"
Sep 18 19:13:53.440098 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.438718    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e41c46ea-8012-448d-9a65-0c67c54bbad1-etc-openvswitch\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.440098 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.438747    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e41c46ea-8012-448d-9a65-0c67c54bbad1-log-socket\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.440098 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.438789    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e41c46ea-8012-448d-9a65-0c67c54bbad1-systemd-units\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.440098 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.438830    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a44929a1-5d90-443e-8d18-f31a4e7e9d28-host-var-lib-cni-bin\") pod \"multus-v7jmf\" (UID: \"a44929a1-5d90-443e-8d18-f31a4e7e9d28\") " pod="openshift-multus/multus-v7jmf"
Sep 18 19:13:53.440098 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.438884    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e41c46ea-8012-448d-9a65-0c67c54bbad1-ovn-node-metrics-cert\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.440098 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.439231    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6f05b5b2-52b5-46f0-8003-30f0bd24c71c-etc-tuned\") pod \"tuned-hbbzh\" (UID: \"6f05b5b2-52b5-46f0-8003-30f0bd24c71c\") " pod="openshift-cluster-node-tuning-operator/tuned-hbbzh"
Sep 18 19:13:53.440098 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.439255    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e41c46ea-8012-448d-9a65-0c67c54bbad1-ovnkube-config\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.440098 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.439255    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fd53f3e2-ce65-4c01-9a2b-10803d702989-agent-certs\") pod \"konnectivity-agent-s6qfb\" (UID: \"fd53f3e2-ce65-4c01-9a2b-10803d702989\") " pod="kube-system/konnectivity-agent-s6qfb"
Sep 18 19:13:53.440873 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.439296    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fd53f3e2-ce65-4c01-9a2b-10803d702989-konnectivity-ca\") pod \"konnectivity-agent-s6qfb\" (UID: \"fd53f3e2-ce65-4c01-9a2b-10803d702989\") " pod="kube-system/konnectivity-agent-s6qfb"
Sep 18 19:13:53.440873 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.440489    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6f05b5b2-52b5-46f0-8003-30f0bd24c71c-tmp\") pod \"tuned-hbbzh\" (UID: \"6f05b5b2-52b5-46f0-8003-30f0bd24c71c\") " pod="openshift-cluster-node-tuning-operator/tuned-hbbzh"
Sep 18 19:13:53.440873 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.440711    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-30.ec2.internal" event={"ID":"216eb917577d10dfc85b487495afcc49","Type":"ContainerStarted","Data":"87921ec499739bbc114de95591acf4e769be1f6261d170966df017cb96ecf545"}
Sep 18 19:13:53.442287 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.442258    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-30.ec2.internal" event={"ID":"5e6343628ef74873cac4c6411141685f","Type":"ContainerStarted","Data":"c808c1be34ffbd7ec03dd67e8299e030934c232320be1f6ccfc6d77254e01610"}
Sep 18 19:13:53.452333 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.452314    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vgjv\" (UniqueName: \"kubernetes.io/projected/b0bb7eb5-2fe7-42f3-92a6-00d65358f28f-kube-api-access-6vgjv\") pod \"network-metrics-daemon-l9g57\" (UID: \"b0bb7eb5-2fe7-42f3-92a6-00d65358f28f\") " pod="openshift-multus/network-metrics-daemon-l9g57"
Sep 18 19:13:53.453326 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.453305    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmpp5\" (UniqueName: \"kubernetes.io/projected/9554bf8e-7591-4261-be81-a00cf9629c8b-kube-api-access-tmpp5\") pod \"aws-ebs-csi-driver-node-xfddm\" (UID: \"9554bf8e-7591-4261-be81-a00cf9629c8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfddm"
Sep 18 19:13:53.456692 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.456657    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d6wc\" (UniqueName: \"kubernetes.io/projected/b7797c1c-31e6-4f63-84d2-6f3c99e195ee-kube-api-access-6d6wc\") pod \"node-resolver-5jzcl\" (UID: \"b7797c1c-31e6-4f63-84d2-6f3c99e195ee\") " pod="openshift-dns/node-resolver-5jzcl"
Sep 18 19:13:53.456963 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.456942    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzfjc\" (UniqueName: \"kubernetes.io/projected/cb48998e-b58d-4478-91ed-42a78b2ed6f9-kube-api-access-pzfjc\") pod \"iptables-alerter-lrqrg\" (UID: \"cb48998e-b58d-4478-91ed-42a78b2ed6f9\") " pod="openshift-network-operator/iptables-alerter-lrqrg"
Sep 18 19:13:53.461054 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:53.461030    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:13:53.461150 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:53.461064    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:13:53.461150 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:53.461087    2577 projected.go:196] Error preparing data for projected volume kube-api-access-46lwh for pod openshift-network-diagnostics/network-check-target-2fpv8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:13:53.461832 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:53.461787    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1bd5d395-d4f3-489a-a20b-1fd771f55da6-kube-api-access-46lwh podName:1bd5d395-d4f3-489a-a20b-1fd771f55da6 nodeName:}" failed. No retries permitted until 2026-09-18 19:13:53.96176221 +0000 UTC m=+4.814062580 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-46lwh" (UniqueName: "kubernetes.io/projected/1bd5d395-d4f3-489a-a20b-1fd771f55da6-kube-api-access-46lwh") pod "network-check-target-2fpv8" (UID: "1bd5d395-d4f3-489a-a20b-1fd771f55da6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:13:53.463414 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.463390    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blxbd\" (UniqueName: \"kubernetes.io/projected/18e3a656-c475-4016-a531-8b17164a1692-kube-api-access-blxbd\") pod \"multus-additional-cni-plugins-fq6nw\" (UID: \"18e3a656-c475-4016-a531-8b17164a1692\") " pod="openshift-multus/multus-additional-cni-plugins-fq6nw"
Sep 18 19:13:53.463499 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.463398    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nmqd\" (UniqueName: \"kubernetes.io/projected/ea018bdf-321c-4368-875d-937e82b2ed3b-kube-api-access-7nmqd\") pod \"node-ca-mx49g\" (UID: \"ea018bdf-321c-4368-875d-937e82b2ed3b\") " pod="openshift-image-registry/node-ca-mx49g"
Sep 18 19:13:53.463499 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.463435    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sm82\" (UniqueName: \"kubernetes.io/projected/6f05b5b2-52b5-46f0-8003-30f0bd24c71c-kube-api-access-7sm82\") pod \"tuned-hbbzh\" (UID: \"6f05b5b2-52b5-46f0-8003-30f0bd24c71c\") " pod="openshift-cluster-node-tuning-operator/tuned-hbbzh"
Sep 18 19:13:53.463793 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.463758    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch6vp\" (UniqueName: \"kubernetes.io/projected/a44929a1-5d90-443e-8d18-f31a4e7e9d28-kube-api-access-ch6vp\") pod \"multus-v7jmf\" (UID: \"a44929a1-5d90-443e-8d18-f31a4e7e9d28\") " pod="openshift-multus/multus-v7jmf"
Sep 18 19:13:53.464116 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.464097    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krtbr\" (UniqueName: \"kubernetes.io/projected/e41c46ea-8012-448d-9a65-0c67c54bbad1-kube-api-access-krtbr\") pod \"ovnkube-node-z76lc\" (UID: \"e41c46ea-8012-448d-9a65-0c67c54bbad1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.537218 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.537190    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:13:53.616813 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.616782    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfddm"
Sep 18 19:13:53.625360 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.625332    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lrqrg"
Sep 18 19:13:53.633098 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.633073    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:13:53.636566 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.636543    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hbbzh"
Sep 18 19:13:53.642793 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.642775    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mx49g"
Sep 18 19:13:53.649728 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.649706    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-s6qfb"
Sep 18 19:13:53.656545 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.656517    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5jzcl"
Sep 18 19:13:53.662749 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.662730    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-v7jmf"
Sep 18 19:13:53.668564 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.668538    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fq6nw"
Sep 18 19:13:53.942181 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.942090    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0bb7eb5-2fe7-42f3-92a6-00d65358f28f-metrics-certs\") pod \"network-metrics-daemon-l9g57\" (UID: \"b0bb7eb5-2fe7-42f3-92a6-00d65358f28f\") " pod="openshift-multus/network-metrics-daemon-l9g57"
Sep 18 19:13:53.942181 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:53.942158    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ebd2878-a3f1-467a-b25f-29124e9d5e7b-original-pull-secret\") pod \"global-pull-secret-syncer-2m5d7\" (UID: \"5ebd2878-a3f1-467a-b25f-29124e9d5e7b\") " pod="kube-system/global-pull-secret-syncer-2m5d7"
Sep 18 19:13:53.942408 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:53.942264    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:13:53.942408 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:53.942269    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:13:53.942408 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:53.942325    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ebd2878-a3f1-467a-b25f-29124e9d5e7b-original-pull-secret podName:5ebd2878-a3f1-467a-b25f-29124e9d5e7b nodeName:}" failed. No retries permitted until 2026-09-18 19:13:54.942308821 +0000 UTC m=+5.794609173 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ebd2878-a3f1-467a-b25f-29124e9d5e7b-original-pull-secret") pod "global-pull-secret-syncer-2m5d7" (UID: "5ebd2878-a3f1-467a-b25f-29124e9d5e7b") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:13:53.942408 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:53.942341    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0bb7eb5-2fe7-42f3-92a6-00d65358f28f-metrics-certs podName:b0bb7eb5-2fe7-42f3-92a6-00d65358f28f nodeName:}" failed. No retries permitted until 2026-09-18 19:13:54.942333138 +0000 UTC m=+5.794633490 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b0bb7eb5-2fe7-42f3-92a6-00d65358f28f-metrics-certs") pod "network-metrics-daemon-l9g57" (UID: "b0bb7eb5-2fe7-42f3-92a6-00d65358f28f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:13:54.042977 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:54.042941    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46lwh\" (UniqueName: \"kubernetes.io/projected/1bd5d395-d4f3-489a-a20b-1fd771f55da6-kube-api-access-46lwh\") pod \"network-check-target-2fpv8\" (UID: \"1bd5d395-d4f3-489a-a20b-1fd771f55da6\") " pod="openshift-network-diagnostics/network-check-target-2fpv8"
Sep 18 19:13:54.043164 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:54.043109    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:13:54.043164 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:54.043125    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:13:54.043164 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:54.043138    2577 projected.go:196] Error preparing data for projected volume kube-api-access-46lwh for pod openshift-network-diagnostics/network-check-target-2fpv8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:13:54.043287 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:54.043191    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1bd5d395-d4f3-489a-a20b-1fd771f55da6-kube-api-access-46lwh podName:1bd5d395-d4f3-489a-a20b-1fd771f55da6 nodeName:}" failed. No retries permitted until 2026-09-18 19:13:55.043175329 +0000 UTC m=+5.895475683 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-46lwh" (UniqueName: "kubernetes.io/projected/1bd5d395-d4f3-489a-a20b-1fd771f55da6-kube-api-access-46lwh") pod "network-check-target-2fpv8" (UID: "1bd5d395-d4f3-489a-a20b-1fd771f55da6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:13:54.385793 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:54.385739    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 19:08:53 +0000 UTC" deadline="2028-04-16 08:52:23.672725554 +0000 UTC"
Sep 18 19:13:54.385793 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:54.385786    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13813h38m29.286943174s"
Sep 18 19:13:54.483379 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:54.483354    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7797c1c_31e6_4f63_84d2_6f3c99e195ee.slice/crio-6c847901bb520e7ecc955f0d4b81b29c9cbab36c5baa795ebeba0b1cb31a65f4 WatchSource:0}: Error finding container 6c847901bb520e7ecc955f0d4b81b29c9cbab36c5baa795ebeba0b1cb31a65f4: Status 404 returned error can't find the container with id 6c847901bb520e7ecc955f0d4b81b29c9cbab36c5baa795ebeba0b1cb31a65f4
Sep 18 19:13:54.484614 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:54.484590    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea018bdf_321c_4368_875d_937e82b2ed3b.slice/crio-48224d54872e4ad70d3e708caf51a764b1f78dad984bac65c543c4fdaeb1dedf WatchSource:0}: Error finding container 48224d54872e4ad70d3e708caf51a764b1f78dad984bac65c543c4fdaeb1dedf: Status 404 returned error can't find the container with id 48224d54872e4ad70d3e708caf51a764b1f78dad984bac65c543c4fdaeb1dedf
Sep 18 19:13:54.486066 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:54.486048    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode41c46ea_8012_448d_9a65_0c67c54bbad1.slice/crio-3c94ee6e24b3c9ab2beba631814ed6b55abe6533568eadd10904b2aef94710e2 WatchSource:0}: Error finding container 3c94ee6e24b3c9ab2beba631814ed6b55abe6533568eadd10904b2aef94710e2: Status 404 returned error can't find the container with id 3c94ee6e24b3c9ab2beba631814ed6b55abe6533568eadd10904b2aef94710e2
Sep 18 19:13:54.500558 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:54.500540    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f05b5b2_52b5_46f0_8003_30f0bd24c71c.slice/crio-b1f785ad294f55f1f7131ac2544a322dc5076f568e1c2383bede36960309aab2 WatchSource:0}: Error finding container b1f785ad294f55f1f7131ac2544a322dc5076f568e1c2383bede36960309aab2: Status 404 returned error can't find the container with id b1f785ad294f55f1f7131ac2544a322dc5076f568e1c2383bede36960309aab2
Sep 18 19:13:54.501686 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:54.501666    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda44929a1_5d90_443e_8d18_f31a4e7e9d28.slice/crio-0d7a59422008a670857da1ea03fe6cf2305a338f959a703ded9667c6448f8305 WatchSource:0}: Error finding container 0d7a59422008a670857da1ea03fe6cf2305a338f959a703ded9667c6448f8305: Status 404 returned error can't find the container with id 0d7a59422008a670857da1ea03fe6cf2305a338f959a703ded9667c6448f8305
Sep 18 19:13:54.503019 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:54.502745    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd53f3e2_ce65_4c01_9a2b_10803d702989.slice/crio-303f5e3863e763f1de19c0585597870e7112e33c0bb687546c0bc3e39d2c320e WatchSource:0}: Error finding container 303f5e3863e763f1de19c0585597870e7112e33c0bb687546c0bc3e39d2c320e: Status 404 returned error can't find the container with id 303f5e3863e763f1de19c0585597870e7112e33c0bb687546c0bc3e39d2c320e
Sep 18 19:13:54.505129 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:54.504898    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18e3a656_c475_4016_a531_8b17164a1692.slice/crio-578b6a308169adf596cf8f58964771bdd0f52ff3d19ba6bd6de8c0c13afa7806 WatchSource:0}: Error finding container 578b6a308169adf596cf8f58964771bdd0f52ff3d19ba6bd6de8c0c13afa7806: Status 404 returned error can't find the container with id 578b6a308169adf596cf8f58964771bdd0f52ff3d19ba6bd6de8c0c13afa7806
Sep 18 19:13:54.505520 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:54.505401    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb48998e_b58d_4478_91ed_42a78b2ed6f9.slice/crio-e533813ca5f35629087276b71a6da96e46da33805e91b63c7a54a3d82da10c91 WatchSource:0}: Error finding container e533813ca5f35629087276b71a6da96e46da33805e91b63c7a54a3d82da10c91: Status 404 returned error can't find the container with id e533813ca5f35629087276b71a6da96e46da33805e91b63c7a54a3d82da10c91
Sep 18 19:13:54.507186 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:13:54.506983    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9554bf8e_7591_4261_be81_a00cf9629c8b.slice/crio-33fe960b0885764567535ec4084dcdfc412c4026c95ef8577eb203818fa7774f WatchSource:0}: Error finding container 33fe960b0885764567535ec4084dcdfc412c4026c95ef8577eb203818fa7774f: Status 404 returned error can't find the container with id 33fe960b0885764567535ec4084dcdfc412c4026c95ef8577eb203818fa7774f
Sep 18 19:13:54.949245 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:54.949113    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ebd2878-a3f1-467a-b25f-29124e9d5e7b-original-pull-secret\") pod \"global-pull-secret-syncer-2m5d7\" (UID: \"5ebd2878-a3f1-467a-b25f-29124e9d5e7b\") " pod="kube-system/global-pull-secret-syncer-2m5d7"
Sep 18 19:13:54.949245 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:54.949237    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0bb7eb5-2fe7-42f3-92a6-00d65358f28f-metrics-certs\") pod \"network-metrics-daemon-l9g57\" (UID: \"b0bb7eb5-2fe7-42f3-92a6-00d65358f28f\") " pod="openshift-multus/network-metrics-daemon-l9g57"
Sep 18 19:13:54.949495 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:54.949280    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:13:54.949495 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:54.949356    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:13:54.949495 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:54.949371    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ebd2878-a3f1-467a-b25f-29124e9d5e7b-original-pull-secret podName:5ebd2878-a3f1-467a-b25f-29124e9d5e7b nodeName:}" failed. No retries permitted until 2026-09-18 19:13:56.949333744 +0000 UTC m=+7.801634099 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ebd2878-a3f1-467a-b25f-29124e9d5e7b-original-pull-secret") pod "global-pull-secret-syncer-2m5d7" (UID: "5ebd2878-a3f1-467a-b25f-29124e9d5e7b") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:13:54.949495 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:54.949402    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0bb7eb5-2fe7-42f3-92a6-00d65358f28f-metrics-certs podName:b0bb7eb5-2fe7-42f3-92a6-00d65358f28f nodeName:}" failed. No retries permitted until 2026-09-18 19:13:56.949389319 +0000 UTC m=+7.801689667 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b0bb7eb5-2fe7-42f3-92a6-00d65358f28f-metrics-certs") pod "network-metrics-daemon-l9g57" (UID: "b0bb7eb5-2fe7-42f3-92a6-00d65358f28f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:13:55.050402 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:55.049771    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46lwh\" (UniqueName: \"kubernetes.io/projected/1bd5d395-d4f3-489a-a20b-1fd771f55da6-kube-api-access-46lwh\") pod \"network-check-target-2fpv8\" (UID: \"1bd5d395-d4f3-489a-a20b-1fd771f55da6\") " pod="openshift-network-diagnostics/network-check-target-2fpv8"
Sep 18 19:13:55.050402 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:55.049966    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:13:55.050402 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:55.050003    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:13:55.050402 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:55.050016    2577 projected.go:196] Error preparing data for projected volume kube-api-access-46lwh for pod openshift-network-diagnostics/network-check-target-2fpv8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:13:55.050402 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:55.050069    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1bd5d395-d4f3-489a-a20b-1fd771f55da6-kube-api-access-46lwh podName:1bd5d395-d4f3-489a-a20b-1fd771f55da6 nodeName:}" failed. No retries permitted until 2026-09-18 19:13:57.050051759 +0000 UTC m=+7.902352121 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-46lwh" (UniqueName: "kubernetes.io/projected/1bd5d395-d4f3-489a-a20b-1fd771f55da6-kube-api-access-46lwh") pod "network-check-target-2fpv8" (UID: "1bd5d395-d4f3-489a-a20b-1fd771f55da6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:13:55.386774 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:55.386710    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 19:08:53 +0000 UTC" deadline="2028-06-20 12:52:08.000251228 +0000 UTC"
Sep 18 19:13:55.386774 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:55.386773    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15377h38m12.613482748s"
Sep 18 19:13:55.434148 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:55.434115    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9g57"
Sep 18 19:13:55.434365 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:55.434150    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2m5d7"
Sep 18 19:13:55.434365 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:55.434264    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9g57" podUID="b0bb7eb5-2fe7-42f3-92a6-00d65358f28f"
Sep 18 19:13:55.434365 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:55.434300    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fpv8"
Sep 18 19:13:55.434540 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:55.434391    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fpv8" podUID="1bd5d395-d4f3-489a-a20b-1fd771f55da6"
Sep 18 19:13:55.434540 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:55.434476    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2m5d7" podUID="5ebd2878-a3f1-467a-b25f-29124e9d5e7b"
Sep 18 19:13:55.461417 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:55.461279    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5jzcl" event={"ID":"b7797c1c-31e6-4f63-84d2-6f3c99e195ee","Type":"ContainerStarted","Data":"6c847901bb520e7ecc955f0d4b81b29c9cbab36c5baa795ebeba0b1cb31a65f4"}
Sep 18 19:13:55.489538 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:55.489389    2577 generic.go:356] "Generic (PLEG): container finished" podID="216eb917577d10dfc85b487495afcc49" containerID="f61d3c4294f84699af404edc54fb2ad8be012908c03a4c1be13f603c1da29d35" exitCode=0
Sep 18 19:13:55.490033 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:55.489750    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-30.ec2.internal" event={"ID":"216eb917577d10dfc85b487495afcc49","Type":"ContainerDied","Data":"f61d3c4294f84699af404edc54fb2ad8be012908c03a4c1be13f603c1da29d35"}
Sep 18 19:13:55.493910 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:55.493624    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lrqrg" event={"ID":"cb48998e-b58d-4478-91ed-42a78b2ed6f9","Type":"ContainerStarted","Data":"e533813ca5f35629087276b71a6da96e46da33805e91b63c7a54a3d82da10c91"}
Sep 18 19:13:55.515379 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:55.515127    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-v7jmf" event={"ID":"a44929a1-5d90-443e-8d18-f31a4e7e9d28","Type":"ContainerStarted","Data":"0d7a59422008a670857da1ea03fe6cf2305a338f959a703ded9667c6448f8305"}
Sep 18 19:13:55.524906 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:55.524783    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mx49g" event={"ID":"ea018bdf-321c-4368-875d-937e82b2ed3b","Type":"ContainerStarted","Data":"48224d54872e4ad70d3e708caf51a764b1f78dad984bac65c543c4fdaeb1dedf"}
Sep 18 19:13:55.529835 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:55.529808    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-30.ec2.internal" event={"ID":"5e6343628ef74873cac4c6411141685f","Type":"ContainerStarted","Data":"d15a21e2367b91e124b25b1dcd10f599ebfb2bc2e7bac2fa72e6338583f8b63c"}
Sep 18 19:13:55.551163 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:55.551031    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfddm" event={"ID":"9554bf8e-7591-4261-be81-a00cf9629c8b","Type":"ContainerStarted","Data":"33fe960b0885764567535ec4084dcdfc412c4026c95ef8577eb203818fa7774f"}
Sep 18 19:13:55.561883 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:55.561852    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fq6nw" event={"ID":"18e3a656-c475-4016-a531-8b17164a1692","Type":"ContainerStarted","Data":"578b6a308169adf596cf8f58964771bdd0f52ff3d19ba6bd6de8c0c13afa7806"}
Sep 18 19:13:55.581525 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:55.581440    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-s6qfb" event={"ID":"fd53f3e2-ce65-4c01-9a2b-10803d702989","Type":"ContainerStarted","Data":"303f5e3863e763f1de19c0585597870e7112e33c0bb687546c0bc3e39d2c320e"}
Sep 18 19:13:55.598927 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:55.598891    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hbbzh" event={"ID":"6f05b5b2-52b5-46f0-8003-30f0bd24c71c","Type":"ContainerStarted","Data":"b1f785ad294f55f1f7131ac2544a322dc5076f568e1c2383bede36960309aab2"}
Sep 18 19:13:55.603372 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:55.603263    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z76lc" event={"ID":"e41c46ea-8012-448d-9a65-0c67c54bbad1","Type":"ContainerStarted","Data":"3c94ee6e24b3c9ab2beba631814ed6b55abe6533568eadd10904b2aef94710e2"}
Sep 18 19:13:56.631903 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:56.631822    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-30.ec2.internal" event={"ID":"216eb917577d10dfc85b487495afcc49","Type":"ContainerStarted","Data":"86b2aeed920f541f7d002e2e5d17e19c5980d72d484e575bfaea2f34493edfb8"}
Sep 18 19:13:56.661191 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:56.661130    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-30.ec2.internal" podStartSLOduration=4.661106891 podStartE2EDuration="4.661106891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:13:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:13:55.557478987 +0000 UTC m=+6.409779358" watchObservedRunningTime="2026-09-18 19:13:56.661106891 +0000 UTC m=+7.513407261"
Sep 18 19:13:56.970430 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:56.970334    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ebd2878-a3f1-467a-b25f-29124e9d5e7b-original-pull-secret\") pod \"global-pull-secret-syncer-2m5d7\" (UID: \"5ebd2878-a3f1-467a-b25f-29124e9d5e7b\") " pod="kube-system/global-pull-secret-syncer-2m5d7"
Sep 18 19:13:56.970430 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:56.970415    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0bb7eb5-2fe7-42f3-92a6-00d65358f28f-metrics-certs\") pod \"network-metrics-daemon-l9g57\" (UID: \"b0bb7eb5-2fe7-42f3-92a6-00d65358f28f\") " pod="openshift-multus/network-metrics-daemon-l9g57"
Sep 18 19:13:56.970648 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:56.970550    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:13:56.970648 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:56.970615    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0bb7eb5-2fe7-42f3-92a6-00d65358f28f-metrics-certs podName:b0bb7eb5-2fe7-42f3-92a6-00d65358f28f nodeName:}" failed. No retries permitted until 2026-09-18 19:14:00.97059743 +0000 UTC m=+11.822897784 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b0bb7eb5-2fe7-42f3-92a6-00d65358f28f-metrics-certs") pod "network-metrics-daemon-l9g57" (UID: "b0bb7eb5-2fe7-42f3-92a6-00d65358f28f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:13:56.971035 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:56.971018    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:13:56.971124 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:56.971073    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ebd2878-a3f1-467a-b25f-29124e9d5e7b-original-pull-secret podName:5ebd2878-a3f1-467a-b25f-29124e9d5e7b nodeName:}" failed. No retries permitted until 2026-09-18 19:14:00.97105706 +0000 UTC m=+11.823357427 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ebd2878-a3f1-467a-b25f-29124e9d5e7b-original-pull-secret") pod "global-pull-secret-syncer-2m5d7" (UID: "5ebd2878-a3f1-467a-b25f-29124e9d5e7b") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:13:57.070910 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:57.070873    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46lwh\" (UniqueName: \"kubernetes.io/projected/1bd5d395-d4f3-489a-a20b-1fd771f55da6-kube-api-access-46lwh\") pod \"network-check-target-2fpv8\" (UID: \"1bd5d395-d4f3-489a-a20b-1fd771f55da6\") " pod="openshift-network-diagnostics/network-check-target-2fpv8"
Sep 18 19:13:57.071129 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:57.071097    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:13:57.071129 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:57.071121    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:13:57.071249 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:57.071133    2577 projected.go:196] Error preparing data for projected volume kube-api-access-46lwh for pod openshift-network-diagnostics/network-check-target-2fpv8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:13:57.071249 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:57.071194    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1bd5d395-d4f3-489a-a20b-1fd771f55da6-kube-api-access-46lwh podName:1bd5d395-d4f3-489a-a20b-1fd771f55da6 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:01.071177546 +0000 UTC m=+11.923477907 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-46lwh" (UniqueName: "kubernetes.io/projected/1bd5d395-d4f3-489a-a20b-1fd771f55da6-kube-api-access-46lwh") pod "network-check-target-2fpv8" (UID: "1bd5d395-d4f3-489a-a20b-1fd771f55da6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:13:57.433794 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:57.433758    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fpv8"
Sep 18 19:13:57.433958 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:57.433897    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fpv8" podUID="1bd5d395-d4f3-489a-a20b-1fd771f55da6"
Sep 18 19:13:57.434045 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:57.433998    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9g57"
Sep 18 19:13:57.434224 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:57.434168    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9g57" podUID="b0bb7eb5-2fe7-42f3-92a6-00d65358f28f"
Sep 18 19:13:57.434224 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:57.433758    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2m5d7"
Sep 18 19:13:57.434392 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:57.434274    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2m5d7" podUID="5ebd2878-a3f1-467a-b25f-29124e9d5e7b"
Sep 18 19:13:59.434844 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:59.434112    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9g57"
Sep 18 19:13:59.434844 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:59.434252    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9g57" podUID="b0bb7eb5-2fe7-42f3-92a6-00d65358f28f"
Sep 18 19:13:59.434844 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:59.434610    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2m5d7"
Sep 18 19:13:59.434844 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:59.434701    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2m5d7" podUID="5ebd2878-a3f1-467a-b25f-29124e9d5e7b"
Sep 18 19:13:59.435751 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:13:59.435613    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fpv8"
Sep 18 19:13:59.435751 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:13:59.435714    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fpv8" podUID="1bd5d395-d4f3-489a-a20b-1fd771f55da6"
Sep 18 19:14:01.004682 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:01.004638    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0bb7eb5-2fe7-42f3-92a6-00d65358f28f-metrics-certs\") pod \"network-metrics-daemon-l9g57\" (UID: \"b0bb7eb5-2fe7-42f3-92a6-00d65358f28f\") " pod="openshift-multus/network-metrics-daemon-l9g57"
Sep 18 19:14:01.005194 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:01.004716    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ebd2878-a3f1-467a-b25f-29124e9d5e7b-original-pull-secret\") pod \"global-pull-secret-syncer-2m5d7\" (UID: \"5ebd2878-a3f1-467a-b25f-29124e9d5e7b\") " pod="kube-system/global-pull-secret-syncer-2m5d7"
Sep 18 19:14:01.005194 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:01.004839    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:01.005194 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:01.004908    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ebd2878-a3f1-467a-b25f-29124e9d5e7b-original-pull-secret podName:5ebd2878-a3f1-467a-b25f-29124e9d5e7b nodeName:}" failed. No retries permitted until 2026-09-18 19:14:09.00489149 +0000 UTC m=+19.857191857 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ebd2878-a3f1-467a-b25f-29124e9d5e7b-original-pull-secret") pod "global-pull-secret-syncer-2m5d7" (UID: "5ebd2878-a3f1-467a-b25f-29124e9d5e7b") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:01.005194 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:01.004941    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:01.005194 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:01.005041    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0bb7eb5-2fe7-42f3-92a6-00d65358f28f-metrics-certs podName:b0bb7eb5-2fe7-42f3-92a6-00d65358f28f nodeName:}" failed. No retries permitted until 2026-09-18 19:14:09.005024288 +0000 UTC m=+19.857324648 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b0bb7eb5-2fe7-42f3-92a6-00d65358f28f-metrics-certs") pod "network-metrics-daemon-l9g57" (UID: "b0bb7eb5-2fe7-42f3-92a6-00d65358f28f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:01.105616 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:01.105458    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46lwh\" (UniqueName: \"kubernetes.io/projected/1bd5d395-d4f3-489a-a20b-1fd771f55da6-kube-api-access-46lwh\") pod \"network-check-target-2fpv8\" (UID: \"1bd5d395-d4f3-489a-a20b-1fd771f55da6\") " pod="openshift-network-diagnostics/network-check-target-2fpv8"
Sep 18 19:14:01.105796 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:01.105626    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:14:01.105796 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:01.105646    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:14:01.105796 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:01.105657    2577 projected.go:196] Error preparing data for projected volume kube-api-access-46lwh for pod openshift-network-diagnostics/network-check-target-2fpv8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:01.105796 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:01.105715    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1bd5d395-d4f3-489a-a20b-1fd771f55da6-kube-api-access-46lwh podName:1bd5d395-d4f3-489a-a20b-1fd771f55da6 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:09.105699379 +0000 UTC m=+19.957999731 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-46lwh" (UniqueName: "kubernetes.io/projected/1bd5d395-d4f3-489a-a20b-1fd771f55da6-kube-api-access-46lwh") pod "network-check-target-2fpv8" (UID: "1bd5d395-d4f3-489a-a20b-1fd771f55da6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:01.433934 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:01.433901    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2m5d7"
Sep 18 19:14:01.434115 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:01.434032    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2m5d7" podUID="5ebd2878-a3f1-467a-b25f-29124e9d5e7b"
Sep 18 19:14:01.434379 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:01.434360    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fpv8"
Sep 18 19:14:01.434804 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:01.434777    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fpv8" podUID="1bd5d395-d4f3-489a-a20b-1fd771f55da6"
Sep 18 19:14:01.437375 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:01.435105    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9g57"
Sep 18 19:14:01.437375 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:01.435304    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9g57" podUID="b0bb7eb5-2fe7-42f3-92a6-00d65358f28f"
Sep 18 19:14:03.433400 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:03.433366    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fpv8"
Sep 18 19:14:03.433919 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:03.433511    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fpv8" podUID="1bd5d395-d4f3-489a-a20b-1fd771f55da6"
Sep 18 19:14:03.433919 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:03.433558    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9g57"
Sep 18 19:14:03.433919 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:03.433678    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9g57" podUID="b0bb7eb5-2fe7-42f3-92a6-00d65358f28f"
Sep 18 19:14:03.433919 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:03.433557    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2m5d7"
Sep 18 19:14:03.433919 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:03.433770    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2m5d7" podUID="5ebd2878-a3f1-467a-b25f-29124e9d5e7b"
Sep 18 19:14:05.433189 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:05.433148    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2m5d7"
Sep 18 19:14:05.433642 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:05.433146    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fpv8"
Sep 18 19:14:05.433642 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:05.433308    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2m5d7" podUID="5ebd2878-a3f1-467a-b25f-29124e9d5e7b"
Sep 18 19:14:05.433642 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:05.433148    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9g57"
Sep 18 19:14:05.433642 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:05.433328    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fpv8" podUID="1bd5d395-d4f3-489a-a20b-1fd771f55da6"
Sep 18 19:14:05.433642 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:05.433406    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9g57" podUID="b0bb7eb5-2fe7-42f3-92a6-00d65358f28f"
Sep 18 19:14:07.433368 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:07.433315    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9g57"
Sep 18 19:14:07.433368 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:07.433334    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2m5d7"
Sep 18 19:14:07.433840 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:07.433466    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9g57" podUID="b0bb7eb5-2fe7-42f3-92a6-00d65358f28f"
Sep 18 19:14:07.433840 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:07.433514    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fpv8"
Sep 18 19:14:07.433840 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:07.433623    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2m5d7" podUID="5ebd2878-a3f1-467a-b25f-29124e9d5e7b"
Sep 18 19:14:07.433840 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:07.433694    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fpv8" podUID="1bd5d395-d4f3-489a-a20b-1fd771f55da6"
Sep 18 19:14:09.063733 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:09.063692    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ebd2878-a3f1-467a-b25f-29124e9d5e7b-original-pull-secret\") pod \"global-pull-secret-syncer-2m5d7\" (UID: \"5ebd2878-a3f1-467a-b25f-29124e9d5e7b\") " pod="kube-system/global-pull-secret-syncer-2m5d7"
Sep 18 19:14:09.064191 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:09.063766    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0bb7eb5-2fe7-42f3-92a6-00d65358f28f-metrics-certs\") pod \"network-metrics-daemon-l9g57\" (UID: \"b0bb7eb5-2fe7-42f3-92a6-00d65358f28f\") " pod="openshift-multus/network-metrics-daemon-l9g57"
Sep 18 19:14:09.064191 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:09.063878    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:09.064191 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:09.063962    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ebd2878-a3f1-467a-b25f-29124e9d5e7b-original-pull-secret podName:5ebd2878-a3f1-467a-b25f-29124e9d5e7b nodeName:}" failed. No retries permitted until 2026-09-18 19:14:25.063941151 +0000 UTC m=+35.916241514 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ebd2878-a3f1-467a-b25f-29124e9d5e7b-original-pull-secret") pod "global-pull-secret-syncer-2m5d7" (UID: "5ebd2878-a3f1-467a-b25f-29124e9d5e7b") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:09.064191 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:09.063885    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:09.064191 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:09.064053    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0bb7eb5-2fe7-42f3-92a6-00d65358f28f-metrics-certs podName:b0bb7eb5-2fe7-42f3-92a6-00d65358f28f nodeName:}" failed. No retries permitted until 2026-09-18 19:14:25.064035564 +0000 UTC m=+35.916335933 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b0bb7eb5-2fe7-42f3-92a6-00d65358f28f-metrics-certs") pod "network-metrics-daemon-l9g57" (UID: "b0bb7eb5-2fe7-42f3-92a6-00d65358f28f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:09.164551 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:09.164507    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46lwh\" (UniqueName: \"kubernetes.io/projected/1bd5d395-d4f3-489a-a20b-1fd771f55da6-kube-api-access-46lwh\") pod \"network-check-target-2fpv8\" (UID: \"1bd5d395-d4f3-489a-a20b-1fd771f55da6\") " pod="openshift-network-diagnostics/network-check-target-2fpv8"
Sep 18 19:14:09.164697 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:09.164681    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:14:09.164749 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:09.164705    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:14:09.164749 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:09.164716    2577 projected.go:196] Error preparing data for projected volume kube-api-access-46lwh for pod openshift-network-diagnostics/network-check-target-2fpv8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:09.164833 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:09.164772    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1bd5d395-d4f3-489a-a20b-1fd771f55da6-kube-api-access-46lwh podName:1bd5d395-d4f3-489a-a20b-1fd771f55da6 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:25.164753828 +0000 UTC m=+36.017054197 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-46lwh" (UniqueName: "kubernetes.io/projected/1bd5d395-d4f3-489a-a20b-1fd771f55da6-kube-api-access-46lwh") pod "network-check-target-2fpv8" (UID: "1bd5d395-d4f3-489a-a20b-1fd771f55da6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:09.433177 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:09.433149    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9g57"
Sep 18 19:14:09.433360 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:09.433149    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2m5d7"
Sep 18 19:14:09.433360 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:09.433271    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9g57" podUID="b0bb7eb5-2fe7-42f3-92a6-00d65358f28f"
Sep 18 19:14:09.433360 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:09.433153    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fpv8"
Sep 18 19:14:09.433463 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:09.433355    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2m5d7" podUID="5ebd2878-a3f1-467a-b25f-29124e9d5e7b"
Sep 18 19:14:09.433463 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:09.433435    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fpv8" podUID="1bd5d395-d4f3-489a-a20b-1fd771f55da6"
Sep 18 19:14:11.434583 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:11.434554    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2m5d7"
Sep 18 19:14:11.434874 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:11.434649    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fpv8"
Sep 18 19:14:11.434874 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:11.434662    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2m5d7" podUID="5ebd2878-a3f1-467a-b25f-29124e9d5e7b"
Sep 18 19:14:11.434874 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:11.434758    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9g57"
Sep 18 19:14:11.434874 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:11.434769    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fpv8" podUID="1bd5d395-d4f3-489a-a20b-1fd771f55da6"
Sep 18 19:14:11.434874 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:11.434817    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9g57" podUID="b0bb7eb5-2fe7-42f3-92a6-00d65358f28f"
Sep 18 19:14:12.668761 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:12.668542    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfddm" event={"ID":"9554bf8e-7591-4261-be81-a00cf9629c8b","Type":"ContainerStarted","Data":"957b4f2e3c2b34e81b7ba890ddaa143f350aeb7a5c1803a1ff90cdd866baa057"}
Sep 18 19:14:12.670259 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:12.670227    2577 generic.go:356] "Generic (PLEG): container finished" podID="18e3a656-c475-4016-a531-8b17164a1692" containerID="36fab93e6b551d6387daf2e00389493711a0b5f0519fc98907e6776aa3920203" exitCode=0
Sep 18 19:14:12.670381 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:12.670313    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fq6nw" event={"ID":"18e3a656-c475-4016-a531-8b17164a1692","Type":"ContainerDied","Data":"36fab93e6b551d6387daf2e00389493711a0b5f0519fc98907e6776aa3920203"}
Sep 18 19:14:12.672302 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:12.672112    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-s6qfb" event={"ID":"fd53f3e2-ce65-4c01-9a2b-10803d702989","Type":"ContainerStarted","Data":"4439d17e4280a1e25c633bba339dac657862410ce315880a930aa161009a19b8"}
Sep 18 19:14:12.673815 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:12.673795    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hbbzh" event={"ID":"6f05b5b2-52b5-46f0-8003-30f0bd24c71c","Type":"ContainerStarted","Data":"9658b32d578612e2a03942ec7b0f4f798158a4003afb92ab528969c80e6461c9"}
Sep 18 19:14:12.677237 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:12.677214    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z76lc" event={"ID":"e41c46ea-8012-448d-9a65-0c67c54bbad1","Type":"ContainerStarted","Data":"26456e233d3a68a1aa409964d037dcd12c79b2e6e80fbf2276a89d9c592d4c70"}
Sep 18 19:14:12.677331 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:12.677286    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z76lc" event={"ID":"e41c46ea-8012-448d-9a65-0c67c54bbad1","Type":"ContainerStarted","Data":"2d4cd2b14816649191b37ed5fea01c690ded015be6fc0fb65c4888502572b8ae"}
Sep 18 19:14:12.677381 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:12.677335    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z76lc" event={"ID":"e41c46ea-8012-448d-9a65-0c67c54bbad1","Type":"ContainerStarted","Data":"73438ec498f2b5545abc770d9b13d4af1699fb1747115e6df5c0ba07d3e8f73b"}
Sep 18 19:14:12.677417 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:12.677380    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z76lc" event={"ID":"e41c46ea-8012-448d-9a65-0c67c54bbad1","Type":"ContainerStarted","Data":"ed0a3d02f710e9aa5553067e33d8a993e8ecf2103dc6c253f2ae26a64bb1d167"}
Sep 18 19:14:12.677452 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:12.677426    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z76lc" event={"ID":"e41c46ea-8012-448d-9a65-0c67c54bbad1","Type":"ContainerStarted","Data":"dc3c2844409cc22423e157bd0c5f8bfc1f96079a9c0853bb4cf94028346223d2"}
Sep 18 19:14:12.677498 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:12.677473    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z76lc" event={"ID":"e41c46ea-8012-448d-9a65-0c67c54bbad1","Type":"ContainerStarted","Data":"a93fcf618854208bcc6e2acb643df62317789e367a103c761346a94b9818506a"}
Sep 18 19:14:12.678873 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:12.678848    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5jzcl" event={"ID":"b7797c1c-31e6-4f63-84d2-6f3c99e195ee","Type":"ContainerStarted","Data":"c7b2bea3b104e594249ab61d5589dccd01e60de8f6c053216c73316d7b12117e"}
Sep 18 19:14:12.680411 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:12.680387    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-v7jmf" event={"ID":"a44929a1-5d90-443e-8d18-f31a4e7e9d28","Type":"ContainerStarted","Data":"34909f471400cb3194af6765763a33eb81df17d5fd3d80307f9e45dac09483df"}
Sep 18 19:14:12.681895 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:12.681873    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mx49g" event={"ID":"ea018bdf-321c-4368-875d-937e82b2ed3b","Type":"ContainerStarted","Data":"d07963ef27bff6338fd38d6de9ce0abd601b0cdfb3b569d04e07ac068302ea8e"}
Sep 18 19:14:12.711388 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:12.711343    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-30.ec2.internal" podStartSLOduration=20.711328174 podStartE2EDuration="20.711328174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:13:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:13:56.661685631 +0000 UTC m=+7.513986002" watchObservedRunningTime="2026-09-18 19:14:12.711328174 +0000 UTC m=+23.563628544"
Sep 18 19:14:12.771979 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:12.771930    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-s6qfb" podStartSLOduration=8.67697459 podStartE2EDuration="21.771915059s" totalImagesPullingTime="13.094940469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:13:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:12.735704522 +0000 UTC m=+23.588004896" watchObservedRunningTime="2026-09-18 19:14:12.771915059 +0000 UTC m=+23.624215450"
Sep 18 19:14:12.803316 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:12.803265    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-5jzcl" podStartSLOduration=4.727837442 podStartE2EDuration="21.803249532s" totalImagesPullingTime="17.07541209s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:13:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:12.772152285 +0000 UTC m=+23.624452654" watchObservedRunningTime="2026-09-18 19:14:12.803249532 +0000 UTC m=+23.655549884"
Sep 18 19:14:12.852478 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:12.852386    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-v7jmf" podStartSLOduration=4.707034263 podStartE2EDuration="21.85237004s" totalImagesPullingTime="17.145335777s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:13:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:12.804106013 +0000 UTC m=+23.656406379" watchObservedRunningTime="2026-09-18 19:14:12.85237004 +0000 UTC m=+23.704670410"
Sep 18 19:14:12.852869 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:12.852844    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-hbbzh" podStartSLOduration=4.78791226 podStartE2EDuration="21.852837362s" totalImagesPullingTime="17.064925102s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:13:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:12.852511321 +0000 UTC m=+23.704811691" watchObservedRunningTime="2026-09-18 19:14:12.852837362 +0000 UTC m=+23.705137731"
Sep 18 19:14:13.247806 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:13.247777    2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 18 19:14:13.414072 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:13.413895    2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T19:14:13.247799624Z","UUID":"04b70030-8d2d-407c-89e8-345c05fb7d20","Handler":null,"Name":"","Endpoint":""}
Sep 18 19:14:13.415221 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:13.415203    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 18 19:14:13.415315 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:13.415233    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 18 19:14:13.433514 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:13.433487    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fpv8"
Sep 18 19:14:13.433624 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:13.433577    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fpv8" podUID="1bd5d395-d4f3-489a-a20b-1fd771f55da6"
Sep 18 19:14:13.433675 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:13.433635    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9g57"
Sep 18 19:14:13.433780 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:13.433759    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9g57" podUID="b0bb7eb5-2fe7-42f3-92a6-00d65358f28f"
Sep 18 19:14:13.433821 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:13.433781    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2m5d7"
Sep 18 19:14:13.433859 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:13.433834    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2m5d7" podUID="5ebd2878-a3f1-467a-b25f-29124e9d5e7b"
Sep 18 19:14:13.685357 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:13.685272    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lrqrg" event={"ID":"cb48998e-b58d-4478-91ed-42a78b2ed6f9","Type":"ContainerStarted","Data":"315f413cf07f5254822f221fde5fddfbc85a6f95f3a41da35a5aec1cb29cb5fc"}
Sep 18 19:14:13.687152 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:13.687126    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfddm" event={"ID":"9554bf8e-7591-4261-be81-a00cf9629c8b","Type":"ContainerStarted","Data":"55da251b7fe243cef65b1ee4f1a12ac00f3b906f8da73734d51b52a940888610"}
Sep 18 19:14:13.713490 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:13.713448    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mx49g" podStartSLOduration=5.75872793 podStartE2EDuration="22.713434759s" totalImagesPullingTime="16.954706829s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:13:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:12.894976004 +0000 UTC m=+23.747276373" watchObservedRunningTime="2026-09-18 19:14:13.713434759 +0000 UTC m=+24.565735128"
Sep 18 19:14:13.713816 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:13.713796    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-lrqrg" podStartSLOduration=5.655774807 podStartE2EDuration="22.713789809s" totalImagesPullingTime="17.058015002s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:13:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:13.713744185 +0000 UTC m=+24.566044570" watchObservedRunningTime="2026-09-18 19:14:13.713789809 +0000 UTC m=+24.566090178"
Sep 18 19:14:14.692414 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:14.692378    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfddm" event={"ID":"9554bf8e-7591-4261-be81-a00cf9629c8b","Type":"ContainerStarted","Data":"d76d4126b3a058f5d3da4a50802dbcf142c336541b05a77e61a3221d455536a8"}
Sep 18 19:14:14.696306 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:14.696277    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z76lc" event={"ID":"e41c46ea-8012-448d-9a65-0c67c54bbad1","Type":"ContainerStarted","Data":"2968315cbc01fee95b4d6b5b4a42bb2fcbeaac056f15a413de5c3c56ca244999"}
Sep 18 19:14:15.055186 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:15.055102    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-s6qfb"
Sep 18 19:14:15.055774 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:15.055750    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-s6qfb"
Sep 18 19:14:15.096798 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:15.096752    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfddm" podStartSLOduration=4.709700544 podStartE2EDuration="24.096739043s" totalImagesPullingTime="19.387038499s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:13:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:14.803285348 +0000 UTC m=+25.655585720" watchObservedRunningTime="2026-09-18 19:14:15.096739043 +0000 UTC m=+25.949039410"
Sep 18 19:14:15.433257 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:15.433224    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fpv8"
Sep 18 19:14:15.433418 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:15.433237    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9g57"
Sep 18 19:14:15.433418 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:15.433325    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fpv8" podUID="1bd5d395-d4f3-489a-a20b-1fd771f55da6"
Sep 18 19:14:15.433418 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:15.433237    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2m5d7"
Sep 18 19:14:15.433418 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:15.433403    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9g57" podUID="b0bb7eb5-2fe7-42f3-92a6-00d65358f28f"
Sep 18 19:14:15.433583 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:15.433454    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2m5d7" podUID="5ebd2878-a3f1-467a-b25f-29124e9d5e7b"
Sep 18 19:14:17.433481 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:17.433276    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9g57"
Sep 18 19:14:17.433918 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:17.433284    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2m5d7"
Sep 18 19:14:17.433918 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:17.433591    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9g57" podUID="b0bb7eb5-2fe7-42f3-92a6-00d65358f28f"
Sep 18 19:14:17.433918 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:17.433397    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fpv8"
Sep 18 19:14:17.433918 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:17.433668    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2m5d7" podUID="5ebd2878-a3f1-467a-b25f-29124e9d5e7b"
Sep 18 19:14:17.433918 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:17.433725    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fpv8" podUID="1bd5d395-d4f3-489a-a20b-1fd771f55da6"
Sep 18 19:14:18.706176 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:18.706138    2577 generic.go:356] "Generic (PLEG): container finished" podID="18e3a656-c475-4016-a531-8b17164a1692" containerID="944643d9a0cc6d4ed2ab71b59ba02ad4bf154faf914dbfe3f2d4132d1794c57c" exitCode=0
Sep 18 19:14:18.706936 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:18.706186    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fq6nw" event={"ID":"18e3a656-c475-4016-a531-8b17164a1692","Type":"ContainerDied","Data":"944643d9a0cc6d4ed2ab71b59ba02ad4bf154faf914dbfe3f2d4132d1794c57c"}
Sep 18 19:14:18.710365 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:18.710209    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z76lc" event={"ID":"e41c46ea-8012-448d-9a65-0c67c54bbad1","Type":"ContainerStarted","Data":"145b075112137b25dac1c72626a348da7b0a8b08233f0a4ebc254b838f3ac817"}
Sep 18 19:14:18.710451 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:18.710411    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:14:18.710451 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:18.710447    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:14:18.726251 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:18.726231    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:14:19.433341 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:19.433308    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9g57"
Sep 18 19:14:19.433544 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:19.433433    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9g57" podUID="b0bb7eb5-2fe7-42f3-92a6-00d65358f28f"
Sep 18 19:14:19.433544 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:19.433500    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2m5d7"
Sep 18 19:14:19.433656 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:19.433499    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fpv8"
Sep 18 19:14:19.433656 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:19.433584    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2m5d7" podUID="5ebd2878-a3f1-467a-b25f-29124e9d5e7b"
Sep 18 19:14:19.433762 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:19.433702    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fpv8" podUID="1bd5d395-d4f3-489a-a20b-1fd771f55da6"
Sep 18 19:14:19.712583 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:19.712553    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:14:19.738245 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:19.738222    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:14:19.789835 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:19.789793    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-z76lc" podStartSLOduration=11.719092204 podStartE2EDuration="28.789777031s" totalImagesPullingTime="17.070684827s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:13:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:18.82620495 +0000 UTC m=+29.678505320" watchObservedRunningTime="2026-09-18 19:14:19.789777031 +0000 UTC m=+30.642077400"
Sep 18 19:14:20.206893 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:20.206857    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2fpv8"]
Sep 18 19:14:20.207107 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:20.206976    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fpv8"
Sep 18 19:14:20.207107 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:20.207096    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fpv8" podUID="1bd5d395-d4f3-489a-a20b-1fd771f55da6"
Sep 18 19:14:20.211684 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:20.211659    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-l9g57"]
Sep 18 19:14:20.211798 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:20.211750    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9g57"
Sep 18 19:14:20.211860 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:20.211839    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9g57" podUID="b0bb7eb5-2fe7-42f3-92a6-00d65358f28f"
Sep 18 19:14:20.216005 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:20.215966    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2m5d7"]
Sep 18 19:14:20.216096 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:20.216060    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2m5d7"
Sep 18 19:14:20.216163 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:20.216126    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2m5d7" podUID="5ebd2878-a3f1-467a-b25f-29124e9d5e7b"
Sep 18 19:14:20.469003 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:20.468908    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-s6qfb"
Sep 18 19:14:20.469161 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:20.469061    2577 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 18 19:14:20.469525 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:20.469504    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-s6qfb"
Sep 18 19:14:20.716289 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:20.716255    2577 generic.go:356] "Generic (PLEG): container finished" podID="18e3a656-c475-4016-a531-8b17164a1692" containerID="44111f723f23edba729153cdd9340be7356615844075f11ba3aab43e33b0486d" exitCode=0
Sep 18 19:14:20.716688 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:20.716337    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fq6nw" event={"ID":"18e3a656-c475-4016-a531-8b17164a1692","Type":"ContainerDied","Data":"44111f723f23edba729153cdd9340be7356615844075f11ba3aab43e33b0486d"}
Sep 18 19:14:21.434185 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:21.434153    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9g57"
Sep 18 19:14:21.434357 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:21.434294    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9g57" podUID="b0bb7eb5-2fe7-42f3-92a6-00d65358f28f"
Sep 18 19:14:21.434684 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:21.434658    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2m5d7"
Sep 18 19:14:21.434796 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:21.434777    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2m5d7" podUID="5ebd2878-a3f1-467a-b25f-29124e9d5e7b"
Sep 18 19:14:22.432884 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:22.432856    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fpv8"
Sep 18 19:14:22.433266 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:22.432948    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fpv8" podUID="1bd5d395-d4f3-489a-a20b-1fd771f55da6"
Sep 18 19:14:22.725486 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:22.725401    2577 generic.go:356] "Generic (PLEG): container finished" podID="18e3a656-c475-4016-a531-8b17164a1692" containerID="d9fb304da2774d427e92aeec9f19be4ab265b63ac5da2252c70aad704db8dc17" exitCode=0
Sep 18 19:14:22.725486 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:22.725444    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fq6nw" event={"ID":"18e3a656-c475-4016-a531-8b17164a1692","Type":"ContainerDied","Data":"d9fb304da2774d427e92aeec9f19be4ab265b63ac5da2252c70aad704db8dc17"}
Sep 18 19:14:23.433665 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:23.433630    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2m5d7"
Sep 18 19:14:23.434071 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:23.433740    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2m5d7" podUID="5ebd2878-a3f1-467a-b25f-29124e9d5e7b"
Sep 18 19:14:23.434071 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:23.433888    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9g57"
Sep 18 19:14:23.434071 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:23.434031    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9g57" podUID="b0bb7eb5-2fe7-42f3-92a6-00d65358f28f"
Sep 18 19:14:24.432712 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:24.432683    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fpv8"
Sep 18 19:14:24.432874 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:24.432779    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fpv8" podUID="1bd5d395-d4f3-489a-a20b-1fd771f55da6"
Sep 18 19:14:25.075465 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.075426    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0bb7eb5-2fe7-42f3-92a6-00d65358f28f-metrics-certs\") pod \"network-metrics-daemon-l9g57\" (UID: \"b0bb7eb5-2fe7-42f3-92a6-00d65358f28f\") " pod="openshift-multus/network-metrics-daemon-l9g57"
Sep 18 19:14:25.076007 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.075504    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ebd2878-a3f1-467a-b25f-29124e9d5e7b-original-pull-secret\") pod \"global-pull-secret-syncer-2m5d7\" (UID: \"5ebd2878-a3f1-467a-b25f-29124e9d5e7b\") " pod="kube-system/global-pull-secret-syncer-2m5d7"
Sep 18 19:14:25.076007 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:25.075579    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:25.076007 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:25.075611    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:25.076007 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:25.075656    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0bb7eb5-2fe7-42f3-92a6-00d65358f28f-metrics-certs podName:b0bb7eb5-2fe7-42f3-92a6-00d65358f28f nodeName:}" failed. No retries permitted until 2026-09-18 19:14:57.075635789 +0000 UTC m=+67.927936150 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b0bb7eb5-2fe7-42f3-92a6-00d65358f28f-metrics-certs") pod "network-metrics-daemon-l9g57" (UID: "b0bb7eb5-2fe7-42f3-92a6-00d65358f28f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:25.076007 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:25.075671    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ebd2878-a3f1-467a-b25f-29124e9d5e7b-original-pull-secret podName:5ebd2878-a3f1-467a-b25f-29124e9d5e7b nodeName:}" failed. No retries permitted until 2026-09-18 19:14:57.075664852 +0000 UTC m=+67.927965200 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ebd2878-a3f1-467a-b25f-29124e9d5e7b-original-pull-secret") pod "global-pull-secret-syncer-2m5d7" (UID: "5ebd2878-a3f1-467a-b25f-29124e9d5e7b") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:25.176044 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.176001    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46lwh\" (UniqueName: \"kubernetes.io/projected/1bd5d395-d4f3-489a-a20b-1fd771f55da6-kube-api-access-46lwh\") pod \"network-check-target-2fpv8\" (UID: \"1bd5d395-d4f3-489a-a20b-1fd771f55da6\") " pod="openshift-network-diagnostics/network-check-target-2fpv8"
Sep 18 19:14:25.176204 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:25.176118    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:14:25.176204 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:25.176145    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:14:25.176204 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:25.176156    2577 projected.go:196] Error preparing data for projected volume kube-api-access-46lwh for pod openshift-network-diagnostics/network-check-target-2fpv8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:25.176322 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:25.176213    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1bd5d395-d4f3-489a-a20b-1fd771f55da6-kube-api-access-46lwh podName:1bd5d395-d4f3-489a-a20b-1fd771f55da6 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:57.1761959 +0000 UTC m=+68.028496268 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-46lwh" (UniqueName: "kubernetes.io/projected/1bd5d395-d4f3-489a-a20b-1fd771f55da6-kube-api-access-46lwh") pod "network-check-target-2fpv8" (UID: "1bd5d395-d4f3-489a-a20b-1fd771f55da6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:25.338573 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.338488    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-30.ec2.internal" event="NodeReady"
Sep 18 19:14:25.338739 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.338642    2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 18 19:14:25.403790 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.403723    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-69df585c75-9p6zz"]
Sep 18 19:14:25.416453 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.416428    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69df585c75-9p6zz"
Sep 18 19:14:25.419947 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.419906    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 18 19:14:25.421057 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.421034    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 18 19:14:25.423451 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.423431    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 18 19:14:25.423662 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.423607    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-fkdh2\""
Sep 18 19:14:25.426510 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.426207    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69df585c75-9p6zz"]
Sep 18 19:14:25.428428 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.428407    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 18 19:14:25.432914 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.432895    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9g57"
Sep 18 19:14:25.433026 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.432896    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2m5d7"
Sep 18 19:14:25.436380 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.436130    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-thgh8\""
Sep 18 19:14:25.436492 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.436387    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 19:14:25.437662 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.437643    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 19:14:25.438241 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.438218    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-dg5zx"]
Sep 18 19:14:25.456970 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.456951    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mb6kf"]
Sep 18 19:14:25.457120 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.457103    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dg5zx"
Sep 18 19:14:25.473618 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.473598    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 18 19:14:25.476363 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.476343    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 18 19:14:25.476440 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.476387    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zqv6h\""
Sep 18 19:14:25.478470 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.478451    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dg5zx"]
Sep 18 19:14:25.478569 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.478555    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mb6kf"
Sep 18 19:14:25.478638 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.478604    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-registry-tls\") pod \"image-registry-69df585c75-9p6zz\" (UID: \"7feb1f7b-32c7-4e47-a257-31e42c72cd0f\") " pod="openshift-image-registry/image-registry-69df585c75-9p6zz"
Sep 18 19:14:25.478695 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.478647    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-bound-sa-token\") pod \"image-registry-69df585c75-9p6zz\" (UID: \"7feb1f7b-32c7-4e47-a257-31e42c72cd0f\") " pod="openshift-image-registry/image-registry-69df585c75-9p6zz"
Sep 18 19:14:25.478695 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.478673    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-ca-trust-extracted\") pod \"image-registry-69df585c75-9p6zz\" (UID: \"7feb1f7b-32c7-4e47-a257-31e42c72cd0f\") " pod="openshift-image-registry/image-registry-69df585c75-9p6zz"
Sep 18 19:14:25.478793 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.478764    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-registry-certificates\") pod \"image-registry-69df585c75-9p6zz\" (UID: \"7feb1f7b-32c7-4e47-a257-31e42c72cd0f\") " pod="openshift-image-registry/image-registry-69df585c75-9p6zz"
Sep 18 19:14:25.478844 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.478799    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-trusted-ca\") pod \"image-registry-69df585c75-9p6zz\" (UID: \"7feb1f7b-32c7-4e47-a257-31e42c72cd0f\") " pod="openshift-image-registry/image-registry-69df585c75-9p6zz"
Sep 18 19:14:25.478844 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.478821    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-installation-pull-secrets\") pod \"image-registry-69df585c75-9p6zz\" (UID: \"7feb1f7b-32c7-4e47-a257-31e42c72cd0f\") " pod="openshift-image-registry/image-registry-69df585c75-9p6zz"
Sep 18 19:14:25.478940 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.478879    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qztnv\" (UniqueName: \"kubernetes.io/projected/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-kube-api-access-qztnv\") pod \"image-registry-69df585c75-9p6zz\" (UID: \"7feb1f7b-32c7-4e47-a257-31e42c72cd0f\") " pod="openshift-image-registry/image-registry-69df585c75-9p6zz"
Sep 18 19:14:25.478940 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.478917    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-image-registry-private-configuration\") pod \"image-registry-69df585c75-9p6zz\" (UID: \"7feb1f7b-32c7-4e47-a257-31e42c72cd0f\") " pod="openshift-image-registry/image-registry-69df585c75-9p6zz"
Sep 18 19:14:25.482038 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.482024    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 18 19:14:25.482305 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.482292    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-4mxkp\""
Sep 18 19:14:25.542307 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.542277    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 18 19:14:25.542464 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.542286    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 18 19:14:25.543314 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.543275    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mb6kf"]
Sep 18 19:14:25.579472 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.579442    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-trusted-ca\") pod \"image-registry-69df585c75-9p6zz\" (UID: \"7feb1f7b-32c7-4e47-a257-31e42c72cd0f\") " pod="openshift-image-registry/image-registry-69df585c75-9p6zz"
Sep 18 19:14:25.579644 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.579479    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-installation-pull-secrets\") pod \"image-registry-69df585c75-9p6zz\" (UID: \"7feb1f7b-32c7-4e47-a257-31e42c72cd0f\") " pod="openshift-image-registry/image-registry-69df585c75-9p6zz"
Sep 18 19:14:25.579644 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.579636    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qztnv\" (UniqueName: \"kubernetes.io/projected/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-kube-api-access-qztnv\") pod \"image-registry-69df585c75-9p6zz\" (UID: \"7feb1f7b-32c7-4e47-a257-31e42c72cd0f\") " pod="openshift-image-registry/image-registry-69df585c75-9p6zz"
Sep 18 19:14:25.579791 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.579673    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dbd605d1-eb07-4c9c-aea1-f1a610394801-metrics-tls\") pod \"dns-default-dg5zx\" (UID: \"dbd605d1-eb07-4c9c-aea1-f1a610394801\") " pod="openshift-dns/dns-default-dg5zx"
Sep 18 19:14:25.579791 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.579707    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-image-registry-private-configuration\") pod \"image-registry-69df585c75-9p6zz\" (UID: \"7feb1f7b-32c7-4e47-a257-31e42c72cd0f\") " pod="openshift-image-registry/image-registry-69df585c75-9p6zz"
Sep 18 19:14:25.579791 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.579773    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd605d1-eb07-4c9c-aea1-f1a610394801-config-volume\") pod \"dns-default-dg5zx\" (UID: \"dbd605d1-eb07-4c9c-aea1-f1a610394801\") " pod="openshift-dns/dns-default-dg5zx"
Sep 18 19:14:25.579941 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.579808    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbd605d1-eb07-4c9c-aea1-f1a610394801-tmp-dir\") pod \"dns-default-dg5zx\" (UID: \"dbd605d1-eb07-4c9c-aea1-f1a610394801\") " pod="openshift-dns/dns-default-dg5zx"
Sep 18 19:14:25.579941 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.579840    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-registry-tls\") pod \"image-registry-69df585c75-9p6zz\" (UID: \"7feb1f7b-32c7-4e47-a257-31e42c72cd0f\") " pod="openshift-image-registry/image-registry-69df585c75-9p6zz"
Sep 18 19:14:25.579941 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.579864    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-bound-sa-token\") pod \"image-registry-69df585c75-9p6zz\" (UID: \"7feb1f7b-32c7-4e47-a257-31e42c72cd0f\") " pod="openshift-image-registry/image-registry-69df585c75-9p6zz"
Sep 18 19:14:25.579941 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.579894    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-ca-trust-extracted\") pod \"image-registry-69df585c75-9p6zz\" (UID: \"7feb1f7b-32c7-4e47-a257-31e42c72cd0f\") " pod="openshift-image-registry/image-registry-69df585c75-9p6zz"
Sep 18 19:14:25.580164 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.579961    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bkzg\" (UniqueName: \"kubernetes.io/projected/dbd605d1-eb07-4c9c-aea1-f1a610394801-kube-api-access-5bkzg\") pod \"dns-default-dg5zx\" (UID: \"dbd605d1-eb07-4c9c-aea1-f1a610394801\") " pod="openshift-dns/dns-default-dg5zx"
Sep 18 19:14:25.580164 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:25.580016    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:14:25.580164 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:25.580032    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69df585c75-9p6zz: secret "image-registry-tls" not found
Sep 18 19:14:25.580164 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.580032    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g2jb\" (UniqueName: \"kubernetes.io/projected/44aa71cc-fda2-4a75-a657-23e31f95e973-kube-api-access-8g2jb\") pod \"ingress-canary-mb6kf\" (UID: \"44aa71cc-fda2-4a75-a657-23e31f95e973\") " pod="openshift-ingress-canary/ingress-canary-mb6kf"
Sep 18 19:14:25.580164 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.580070    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-registry-certificates\") pod \"image-registry-69df585c75-9p6zz\" (UID: \"7feb1f7b-32c7-4e47-a257-31e42c72cd0f\") " pod="openshift-image-registry/image-registry-69df585c75-9p6zz"
Sep 18 19:14:25.580164 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:25.580095    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-registry-tls podName:7feb1f7b-32c7-4e47-a257-31e42c72cd0f nodeName:}" failed. No retries permitted until 2026-09-18 19:14:26.080075354 +0000 UTC m=+36.932375720 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-registry-tls") pod "image-registry-69df585c75-9p6zz" (UID: "7feb1f7b-32c7-4e47-a257-31e42c72cd0f") : secret "image-registry-tls" not found
Sep 18 19:14:25.580164 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.580123    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/44aa71cc-fda2-4a75-a657-23e31f95e973-cert\") pod \"ingress-canary-mb6kf\" (UID: \"44aa71cc-fda2-4a75-a657-23e31f95e973\") " pod="openshift-ingress-canary/ingress-canary-mb6kf"
Sep 18 19:14:25.580595 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.580566    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-ca-trust-extracted\") pod \"image-registry-69df585c75-9p6zz\" (UID: \"7feb1f7b-32c7-4e47-a257-31e42c72cd0f\") " pod="openshift-image-registry/image-registry-69df585c75-9p6zz"
Sep 18 19:14:25.580723 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.580701    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-registry-certificates\") pod \"image-registry-69df585c75-9p6zz\" (UID: \"7feb1f7b-32c7-4e47-a257-31e42c72cd0f\") " pod="openshift-image-registry/image-registry-69df585c75-9p6zz"
Sep 18 19:14:25.581207 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.581184    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-trusted-ca\") pod \"image-registry-69df585c75-9p6zz\" (UID: \"7feb1f7b-32c7-4e47-a257-31e42c72cd0f\") " pod="openshift-image-registry/image-registry-69df585c75-9p6zz"
Sep 18 19:14:25.583890 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.583870    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-installation-pull-secrets\") pod \"image-registry-69df585c75-9p6zz\" (UID: \"7feb1f7b-32c7-4e47-a257-31e42c72cd0f\") " pod="openshift-image-registry/image-registry-69df585c75-9p6zz"
Sep 18 19:14:25.583979 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.583916    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-image-registry-private-configuration\") pod \"image-registry-69df585c75-9p6zz\" (UID: \"7feb1f7b-32c7-4e47-a257-31e42c72cd0f\") " pod="openshift-image-registry/image-registry-69df585c75-9p6zz"
Sep 18 19:14:25.607061 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.607005    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qztnv\" (UniqueName: \"kubernetes.io/projected/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-kube-api-access-qztnv\") pod \"image-registry-69df585c75-9p6zz\" (UID: \"7feb1f7b-32c7-4e47-a257-31e42c72cd0f\") " pod="openshift-image-registry/image-registry-69df585c75-9p6zz"
Sep 18 19:14:25.613920 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.613895    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-bound-sa-token\") pod \"image-registry-69df585c75-9p6zz\" (UID: \"7feb1f7b-32c7-4e47-a257-31e42c72cd0f\") " pod="openshift-image-registry/image-registry-69df585c75-9p6zz"
Sep 18 19:14:25.681041 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.681012    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbd605d1-eb07-4c9c-aea1-f1a610394801-tmp-dir\") pod \"dns-default-dg5zx\" (UID: \"dbd605d1-eb07-4c9c-aea1-f1a610394801\") " pod="openshift-dns/dns-default-dg5zx"
Sep 18 19:14:25.681232 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.681086    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bkzg\" (UniqueName: \"kubernetes.io/projected/dbd605d1-eb07-4c9c-aea1-f1a610394801-kube-api-access-5bkzg\") pod \"dns-default-dg5zx\" (UID: \"dbd605d1-eb07-4c9c-aea1-f1a610394801\") " pod="openshift-dns/dns-default-dg5zx"
Sep 18 19:14:25.681232 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.681120    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g2jb\" (UniqueName: \"kubernetes.io/projected/44aa71cc-fda2-4a75-a657-23e31f95e973-kube-api-access-8g2jb\") pod \"ingress-canary-mb6kf\" (UID: \"44aa71cc-fda2-4a75-a657-23e31f95e973\") " pod="openshift-ingress-canary/ingress-canary-mb6kf"
Sep 18 19:14:25.681232 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.681149    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/44aa71cc-fda2-4a75-a657-23e31f95e973-cert\") pod \"ingress-canary-mb6kf\" (UID: \"44aa71cc-fda2-4a75-a657-23e31f95e973\") " pod="openshift-ingress-canary/ingress-canary-mb6kf"
Sep 18 19:14:25.681232 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.681210    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dbd605d1-eb07-4c9c-aea1-f1a610394801-metrics-tls\") pod \"dns-default-dg5zx\" (UID: \"dbd605d1-eb07-4c9c-aea1-f1a610394801\") " pod="openshift-dns/dns-default-dg5zx"
Sep 18 19:14:25.681433 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.681243    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd605d1-eb07-4c9c-aea1-f1a610394801-config-volume\") pod \"dns-default-dg5zx\" (UID: \"dbd605d1-eb07-4c9c-aea1-f1a610394801\") " pod="openshift-dns/dns-default-dg5zx"
Sep 18 19:14:25.681433 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.681388    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbd605d1-eb07-4c9c-aea1-f1a610394801-tmp-dir\") pod \"dns-default-dg5zx\" (UID: \"dbd605d1-eb07-4c9c-aea1-f1a610394801\") " pod="openshift-dns/dns-default-dg5zx"
Sep 18 19:14:25.681534 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:25.681458    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:14:25.681590 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:25.681534    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbd605d1-eb07-4c9c-aea1-f1a610394801-metrics-tls podName:dbd605d1-eb07-4c9c-aea1-f1a610394801 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:26.18151328 +0000 UTC m=+37.033813646 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dbd605d1-eb07-4c9c-aea1-f1a610394801-metrics-tls") pod "dns-default-dg5zx" (UID: "dbd605d1-eb07-4c9c-aea1-f1a610394801") : secret "dns-default-metrics-tls" not found
Sep 18 19:14:25.681834 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.681811    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd605d1-eb07-4c9c-aea1-f1a610394801-config-volume\") pod \"dns-default-dg5zx\" (UID: \"dbd605d1-eb07-4c9c-aea1-f1a610394801\") " pod="openshift-dns/dns-default-dg5zx"
Sep 18 19:14:25.684140 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.684084    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/44aa71cc-fda2-4a75-a657-23e31f95e973-cert\") pod \"ingress-canary-mb6kf\" (UID: \"44aa71cc-fda2-4a75-a657-23e31f95e973\") " pod="openshift-ingress-canary/ingress-canary-mb6kf"
Sep 18 19:14:25.699573 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.699546    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bkzg\" (UniqueName: \"kubernetes.io/projected/dbd605d1-eb07-4c9c-aea1-f1a610394801-kube-api-access-5bkzg\") pod \"dns-default-dg5zx\" (UID: \"dbd605d1-eb07-4c9c-aea1-f1a610394801\") " pod="openshift-dns/dns-default-dg5zx"
Sep 18 19:14:25.701850 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.701822    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g2jb\" (UniqueName: \"kubernetes.io/projected/44aa71cc-fda2-4a75-a657-23e31f95e973-kube-api-access-8g2jb\") pod \"ingress-canary-mb6kf\" (UID: \"44aa71cc-fda2-4a75-a657-23e31f95e973\") " pod="openshift-ingress-canary/ingress-canary-mb6kf"
Sep 18 19:14:25.786537 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:25.786506    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mb6kf"
Sep 18 19:14:26.085864 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:26.085826    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-registry-tls\") pod \"image-registry-69df585c75-9p6zz\" (UID: \"7feb1f7b-32c7-4e47-a257-31e42c72cd0f\") " pod="openshift-image-registry/image-registry-69df585c75-9p6zz"
Sep 18 19:14:26.086545 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:26.085956    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:14:26.086545 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:26.085973    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69df585c75-9p6zz: secret "image-registry-tls" not found
Sep 18 19:14:26.086545 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:26.086037    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-registry-tls podName:7feb1f7b-32c7-4e47-a257-31e42c72cd0f nodeName:}" failed. No retries permitted until 2026-09-18 19:14:27.086023451 +0000 UTC m=+37.938323811 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-registry-tls") pod "image-registry-69df585c75-9p6zz" (UID: "7feb1f7b-32c7-4e47-a257-31e42c72cd0f") : secret "image-registry-tls" not found
Sep 18 19:14:26.186325 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:26.186287    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dbd605d1-eb07-4c9c-aea1-f1a610394801-metrics-tls\") pod \"dns-default-dg5zx\" (UID: \"dbd605d1-eb07-4c9c-aea1-f1a610394801\") " pod="openshift-dns/dns-default-dg5zx"
Sep 18 19:14:26.186498 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:26.186456    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:14:26.186545 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:26.186524    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbd605d1-eb07-4c9c-aea1-f1a610394801-metrics-tls podName:dbd605d1-eb07-4c9c-aea1-f1a610394801 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:27.186506993 +0000 UTC m=+38.038807340 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dbd605d1-eb07-4c9c-aea1-f1a610394801-metrics-tls") pod "dns-default-dg5zx" (UID: "dbd605d1-eb07-4c9c-aea1-f1a610394801") : secret "dns-default-metrics-tls" not found
Sep 18 19:14:26.433396 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:26.433364    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fpv8"
Sep 18 19:14:26.438061 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:26.438034    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 19:14:26.438214 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:26.438091    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 19:14:26.438276 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:26.438237    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4flf6\""
Sep 18 19:14:27.094041 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:27.094005    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-registry-tls\") pod \"image-registry-69df585c75-9p6zz\" (UID: \"7feb1f7b-32c7-4e47-a257-31e42c72cd0f\") " pod="openshift-image-registry/image-registry-69df585c75-9p6zz"
Sep 18 19:14:27.094440 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:27.094179    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:14:27.094440 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:27.094197    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69df585c75-9p6zz: secret "image-registry-tls" not found
Sep 18 19:14:27.094440 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:27.094270    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-registry-tls podName:7feb1f7b-32c7-4e47-a257-31e42c72cd0f nodeName:}" failed. No retries permitted until 2026-09-18 19:14:29.094254006 +0000 UTC m=+39.946554354 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-registry-tls") pod "image-registry-69df585c75-9p6zz" (UID: "7feb1f7b-32c7-4e47-a257-31e42c72cd0f") : secret "image-registry-tls" not found
Sep 18 19:14:27.194799 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:27.194760    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dbd605d1-eb07-4c9c-aea1-f1a610394801-metrics-tls\") pod \"dns-default-dg5zx\" (UID: \"dbd605d1-eb07-4c9c-aea1-f1a610394801\") " pod="openshift-dns/dns-default-dg5zx"
Sep 18 19:14:27.194954 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:27.194909    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:14:27.195057 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:27.195045    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbd605d1-eb07-4c9c-aea1-f1a610394801-metrics-tls podName:dbd605d1-eb07-4c9c-aea1-f1a610394801 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:29.195026847 +0000 UTC m=+40.047327209 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dbd605d1-eb07-4c9c-aea1-f1a610394801-metrics-tls") pod "dns-default-dg5zx" (UID: "dbd605d1-eb07-4c9c-aea1-f1a610394801") : secret "dns-default-metrics-tls" not found
Sep 18 19:14:27.640172 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:27.640143    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mb6kf"]
Sep 18 19:14:27.643693 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:14:27.643669    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44aa71cc_fda2_4a75_a657_23e31f95e973.slice/crio-8540fe83fe7988e125f536e9c9ac951ca5a23ecbebb74f54774d93ab310d7a53 WatchSource:0}: Error finding container 8540fe83fe7988e125f536e9c9ac951ca5a23ecbebb74f54774d93ab310d7a53: Status 404 returned error can't find the container with id 8540fe83fe7988e125f536e9c9ac951ca5a23ecbebb74f54774d93ab310d7a53
Sep 18 19:14:27.739505 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:27.739344    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fq6nw" event={"ID":"18e3a656-c475-4016-a531-8b17164a1692","Type":"ContainerStarted","Data":"7a8fec95791c94533b7ba82db6ec08274e65987fcb8e31798c6c7fc1c3777e18"}
Sep 18 19:14:27.740512 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:27.740488    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mb6kf" event={"ID":"44aa71cc-fda2-4a75-a657-23e31f95e973","Type":"ContainerStarted","Data":"8540fe83fe7988e125f536e9c9ac951ca5a23ecbebb74f54774d93ab310d7a53"}
Sep 18 19:14:28.746398 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:28.746360    2577 generic.go:356] "Generic (PLEG): container finished" podID="18e3a656-c475-4016-a531-8b17164a1692" containerID="7a8fec95791c94533b7ba82db6ec08274e65987fcb8e31798c6c7fc1c3777e18" exitCode=0
Sep 18 19:14:28.747007 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:28.746406    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fq6nw" event={"ID":"18e3a656-c475-4016-a531-8b17164a1692","Type":"ContainerDied","Data":"7a8fec95791c94533b7ba82db6ec08274e65987fcb8e31798c6c7fc1c3777e18"}
Sep 18 19:14:29.110201 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:29.110112    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-registry-tls\") pod \"image-registry-69df585c75-9p6zz\" (UID: \"7feb1f7b-32c7-4e47-a257-31e42c72cd0f\") " pod="openshift-image-registry/image-registry-69df585c75-9p6zz"
Sep 18 19:14:29.110369 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:29.110277    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:14:29.110369 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:29.110300    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69df585c75-9p6zz: secret "image-registry-tls" not found
Sep 18 19:14:29.110480 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:29.110373    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-registry-tls podName:7feb1f7b-32c7-4e47-a257-31e42c72cd0f nodeName:}" failed. No retries permitted until 2026-09-18 19:14:33.110350529 +0000 UTC m=+43.962650891 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-registry-tls") pod "image-registry-69df585c75-9p6zz" (UID: "7feb1f7b-32c7-4e47-a257-31e42c72cd0f") : secret "image-registry-tls" not found
Sep 18 19:14:29.211188 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:29.211156    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dbd605d1-eb07-4c9c-aea1-f1a610394801-metrics-tls\") pod \"dns-default-dg5zx\" (UID: \"dbd605d1-eb07-4c9c-aea1-f1a610394801\") " pod="openshift-dns/dns-default-dg5zx"
Sep 18 19:14:29.211379 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:29.211326    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:14:29.211443 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:29.211402    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbd605d1-eb07-4c9c-aea1-f1a610394801-metrics-tls podName:dbd605d1-eb07-4c9c-aea1-f1a610394801 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:33.211382849 +0000 UTC m=+44.063683204 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dbd605d1-eb07-4c9c-aea1-f1a610394801-metrics-tls") pod "dns-default-dg5zx" (UID: "dbd605d1-eb07-4c9c-aea1-f1a610394801") : secret "dns-default-metrics-tls" not found
Sep 18 19:14:29.752314 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:29.752241    2577 generic.go:356] "Generic (PLEG): container finished" podID="18e3a656-c475-4016-a531-8b17164a1692" containerID="29b5ecbfe08f91f9301f966626443c5ce50b583446abf495862584585a36e132" exitCode=0
Sep 18 19:14:29.752674 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:29.752314    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fq6nw" event={"ID":"18e3a656-c475-4016-a531-8b17164a1692","Type":"ContainerDied","Data":"29b5ecbfe08f91f9301f966626443c5ce50b583446abf495862584585a36e132"}
Sep 18 19:14:30.757822 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:30.757783    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fq6nw" event={"ID":"18e3a656-c475-4016-a531-8b17164a1692","Type":"ContainerStarted","Data":"2ad68cd4f08a4491f3ae95417107c0e8d17a3f0aae40624593cfb3c2582de9f3"}
Sep 18 19:14:30.759237 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:30.759210    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mb6kf" event={"ID":"44aa71cc-fda2-4a75-a657-23e31f95e973","Type":"ContainerStarted","Data":"bdcb19c394ea327857ffdc5789b84fd05b08f97ed0c8a3f8156a424e75a39e73"}
Sep 18 19:14:30.847826 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:30.847774    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-fq6nw" podStartSLOduration=10.768999944 podStartE2EDuration="39.847759993s" totalImagesPullingTime="29.078760049s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:13:51 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:30.847232606 +0000 UTC m=+41.699532975" watchObservedRunningTime="2026-09-18 19:14:30.847759993 +0000 UTC m=+41.700060360"
Sep 18 19:14:30.903046 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:30.902982    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mb6kf" podStartSLOduration=3.5616834429999997 podStartE2EDuration="5.902971196s" totalImagesPullingTime="2.341287753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:14:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:30.901212718 +0000 UTC m=+41.753513087" watchObservedRunningTime="2026-09-18 19:14:30.902971196 +0000 UTC m=+41.755271565"
Sep 18 19:14:33.142515 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:33.142464    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-registry-tls\") pod \"image-registry-69df585c75-9p6zz\" (UID: \"7feb1f7b-32c7-4e47-a257-31e42c72cd0f\") " pod="openshift-image-registry/image-registry-69df585c75-9p6zz"
Sep 18 19:14:33.142912 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:33.142616    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:14:33.142912 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:33.142634    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69df585c75-9p6zz: secret "image-registry-tls" not found
Sep 18 19:14:33.142912 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:33.142687    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-registry-tls podName:7feb1f7b-32c7-4e47-a257-31e42c72cd0f nodeName:}" failed. No retries permitted until 2026-09-18 19:14:41.142670503 +0000 UTC m=+51.994970871 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-registry-tls") pod "image-registry-69df585c75-9p6zz" (UID: "7feb1f7b-32c7-4e47-a257-31e42c72cd0f") : secret "image-registry-tls" not found
Sep 18 19:14:33.243369 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:33.243335    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dbd605d1-eb07-4c9c-aea1-f1a610394801-metrics-tls\") pod \"dns-default-dg5zx\" (UID: \"dbd605d1-eb07-4c9c-aea1-f1a610394801\") " pod="openshift-dns/dns-default-dg5zx"
Sep 18 19:14:33.243542 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:33.243446    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:14:33.243542 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:33.243501    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbd605d1-eb07-4c9c-aea1-f1a610394801-metrics-tls podName:dbd605d1-eb07-4c9c-aea1-f1a610394801 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:41.243485295 +0000 UTC m=+52.095785644 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dbd605d1-eb07-4c9c-aea1-f1a610394801-metrics-tls") pod "dns-default-dg5zx" (UID: "dbd605d1-eb07-4c9c-aea1-f1a610394801") : secret "dns-default-metrics-tls" not found
Sep 18 19:14:41.200190 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:41.200144    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-registry-tls\") pod \"image-registry-69df585c75-9p6zz\" (UID: \"7feb1f7b-32c7-4e47-a257-31e42c72cd0f\") " pod="openshift-image-registry/image-registry-69df585c75-9p6zz"
Sep 18 19:14:41.200596 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:41.200263    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:14:41.200596 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:41.200275    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69df585c75-9p6zz: secret "image-registry-tls" not found
Sep 18 19:14:41.200596 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:41.200325    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-registry-tls podName:7feb1f7b-32c7-4e47-a257-31e42c72cd0f nodeName:}" failed. No retries permitted until 2026-09-18 19:14:57.200312533 +0000 UTC m=+68.052612881 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-registry-tls") pod "image-registry-69df585c75-9p6zz" (UID: "7feb1f7b-32c7-4e47-a257-31e42c72cd0f") : secret "image-registry-tls" not found
Sep 18 19:14:41.300816 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:41.300777    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dbd605d1-eb07-4c9c-aea1-f1a610394801-metrics-tls\") pod \"dns-default-dg5zx\" (UID: \"dbd605d1-eb07-4c9c-aea1-f1a610394801\") " pod="openshift-dns/dns-default-dg5zx"
Sep 18 19:14:41.300958 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:41.300917    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:14:41.301018 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:41.300976    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbd605d1-eb07-4c9c-aea1-f1a610394801-metrics-tls podName:dbd605d1-eb07-4c9c-aea1-f1a610394801 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:57.300961609 +0000 UTC m=+68.153261957 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dbd605d1-eb07-4c9c-aea1-f1a610394801-metrics-tls") pod "dns-default-dg5zx" (UID: "dbd605d1-eb07-4c9c-aea1-f1a610394801") : secret "dns-default-metrics-tls" not found
Sep 18 19:14:51.732214 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:51.732185    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z76lc"
Sep 18 19:14:57.116297 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:57.116251    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ebd2878-a3f1-467a-b25f-29124e9d5e7b-original-pull-secret\") pod \"global-pull-secret-syncer-2m5d7\" (UID: \"5ebd2878-a3f1-467a-b25f-29124e9d5e7b\") " pod="kube-system/global-pull-secret-syncer-2m5d7"
Sep 18 19:14:57.116706 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:57.116331    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0bb7eb5-2fe7-42f3-92a6-00d65358f28f-metrics-certs\") pod \"network-metrics-daemon-l9g57\" (UID: \"b0bb7eb5-2fe7-42f3-92a6-00d65358f28f\") " pod="openshift-multus/network-metrics-daemon-l9g57"
Sep 18 19:14:57.119646 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:57.119628    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 19:14:57.120338 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:57.120325    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 19:14:57.127178 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:57.127158    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 18 19:14:57.127266 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:57.127250    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0bb7eb5-2fe7-42f3-92a6-00d65358f28f-metrics-certs podName:b0bb7eb5-2fe7-42f3-92a6-00d65358f28f nodeName:}" failed. No retries permitted until 2026-09-18 19:16:01.127228003 +0000 UTC m=+131.979528370 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b0bb7eb5-2fe7-42f3-92a6-00d65358f28f-metrics-certs") pod "network-metrics-daemon-l9g57" (UID: "b0bb7eb5-2fe7-42f3-92a6-00d65358f28f") : secret "metrics-daemon-secret" not found
Sep 18 19:14:57.130133 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:57.130111    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ebd2878-a3f1-467a-b25f-29124e9d5e7b-original-pull-secret\") pod \"global-pull-secret-syncer-2m5d7\" (UID: \"5ebd2878-a3f1-467a-b25f-29124e9d5e7b\") " pod="kube-system/global-pull-secret-syncer-2m5d7"
Sep 18 19:14:57.217078 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:57.217033    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46lwh\" (UniqueName: \"kubernetes.io/projected/1bd5d395-d4f3-489a-a20b-1fd771f55da6-kube-api-access-46lwh\") pod \"network-check-target-2fpv8\" (UID: \"1bd5d395-d4f3-489a-a20b-1fd771f55da6\") " pod="openshift-network-diagnostics/network-check-target-2fpv8"
Sep 18 19:14:57.217270 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:57.217131    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-registry-tls\") pod \"image-registry-69df585c75-9p6zz\" (UID: \"7feb1f7b-32c7-4e47-a257-31e42c72cd0f\") " pod="openshift-image-registry/image-registry-69df585c75-9p6zz"
Sep 18 19:14:57.217270 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:57.217219    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:14:57.217270 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:57.217232    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69df585c75-9p6zz: secret "image-registry-tls" not found
Sep 18 19:14:57.217428 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:57.217293    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-registry-tls podName:7feb1f7b-32c7-4e47-a257-31e42c72cd0f nodeName:}" failed. No retries permitted until 2026-09-18 19:15:29.21727749 +0000 UTC m=+100.069577838 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-registry-tls") pod "image-registry-69df585c75-9p6zz" (UID: "7feb1f7b-32c7-4e47-a257-31e42c72cd0f") : secret "image-registry-tls" not found
Sep 18 19:14:57.222111 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:57.222086    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 19:14:57.230884 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:57.230865    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 19:14:57.240355 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:57.240333    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46lwh\" (UniqueName: \"kubernetes.io/projected/1bd5d395-d4f3-489a-a20b-1fd771f55da6-kube-api-access-46lwh\") pod \"network-check-target-2fpv8\" (UID: \"1bd5d395-d4f3-489a-a20b-1fd771f55da6\") " pod="openshift-network-diagnostics/network-check-target-2fpv8"
Sep 18 19:14:57.245134 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:57.245116    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2m5d7"
Sep 18 19:14:57.317714 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:57.317682    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dbd605d1-eb07-4c9c-aea1-f1a610394801-metrics-tls\") pod \"dns-default-dg5zx\" (UID: \"dbd605d1-eb07-4c9c-aea1-f1a610394801\") " pod="openshift-dns/dns-default-dg5zx"
Sep 18 19:14:57.317853 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:57.317833    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:14:57.317933 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:14:57.317923    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbd605d1-eb07-4c9c-aea1-f1a610394801-metrics-tls podName:dbd605d1-eb07-4c9c-aea1-f1a610394801 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:29.317905159 +0000 UTC m=+100.170205512 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dbd605d1-eb07-4c9c-aea1-f1a610394801-metrics-tls") pod "dns-default-dg5zx" (UID: "dbd605d1-eb07-4c9c-aea1-f1a610394801") : secret "dns-default-metrics-tls" not found
Sep 18 19:14:57.349091 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:57.349046    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4flf6\""
Sep 18 19:14:57.353844 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:57.353829    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fpv8"
Sep 18 19:14:57.516809 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:57.516776    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2m5d7"]
Sep 18 19:14:57.519670 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:14:57.519645    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ebd2878_a3f1_467a_b25f_29124e9d5e7b.slice/crio-665a9df894df806afc751d04456da7b30396f37cd58e984ac2a702af77946c30 WatchSource:0}: Error finding container 665a9df894df806afc751d04456da7b30396f37cd58e984ac2a702af77946c30: Status 404 returned error can't find the container with id 665a9df894df806afc751d04456da7b30396f37cd58e984ac2a702af77946c30
Sep 18 19:14:57.630817 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:57.630787    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2fpv8"]
Sep 18 19:14:57.633404 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:14:57.633381    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bd5d395_d4f3_489a_a20b_1fd771f55da6.slice/crio-c4bd1a38375cfdbfe7abf261271642998733d51f1570d190161d03cf1d4908b2 WatchSource:0}: Error finding container c4bd1a38375cfdbfe7abf261271642998733d51f1570d190161d03cf1d4908b2: Status 404 returned error can't find the container with id c4bd1a38375cfdbfe7abf261271642998733d51f1570d190161d03cf1d4908b2
Sep 18 19:14:57.824427 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:57.824334    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2fpv8" event={"ID":"1bd5d395-d4f3-489a-a20b-1fd771f55da6","Type":"ContainerStarted","Data":"c4bd1a38375cfdbfe7abf261271642998733d51f1570d190161d03cf1d4908b2"}
Sep 18 19:14:57.825416 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:14:57.825392    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2m5d7" event={"ID":"5ebd2878-a3f1-467a-b25f-29124e9d5e7b","Type":"ContainerStarted","Data":"665a9df894df806afc751d04456da7b30396f37cd58e984ac2a702af77946c30"}
Sep 18 19:15:02.840325 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:02.840288    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2fpv8" event={"ID":"1bd5d395-d4f3-489a-a20b-1fd771f55da6","Type":"ContainerStarted","Data":"e4f5044788f262ba92d88411f3f438adcdc5e35da530cb70301be81e0adbdfa5"}
Sep 18 19:15:02.840745 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:02.840412    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2fpv8"
Sep 18 19:15:02.841848 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:02.841825    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2m5d7" event={"ID":"5ebd2878-a3f1-467a-b25f-29124e9d5e7b","Type":"ContainerStarted","Data":"f168f9af4f4d1e2e2c43066aa9cef3fa9763a404426aca7c5b09f917f7711058"}
Sep 18 19:15:02.858896 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:02.858850    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2fpv8" podStartSLOduration=67.342687984 podStartE2EDuration="1m11.858836508s" totalImagesPullingTime="4.516148524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:13:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:02.85786931 +0000 UTC m=+73.710169679" watchObservedRunningTime="2026-09-18 19:15:02.858836508 +0000 UTC m=+73.711136878"
Sep 18 19:15:02.879745 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:02.879673    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-2m5d7" podStartSLOduration=66.247063746 podStartE2EDuration="1m10.87965787s" totalImagesPullingTime="4.632594124s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:13:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:02.879257223 +0000 UTC m=+73.731557599" watchObservedRunningTime="2026-09-18 19:15:02.87965787 +0000 UTC m=+73.731958243"
Sep 18 19:15:20.259729 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:20.259690    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2mhnp"]
Sep 18 19:15:20.262808 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:20.262780    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2mhnp"
Sep 18 19:15:20.267669 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:20.267640    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 18 19:15:20.267781 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:20.267650    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 18 19:15:20.269189 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:20.268960    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 18 19:15:20.269189 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:20.269081    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-d2glk\""
Sep 18 19:15:20.269323 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:20.269219    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 18 19:15:20.283834 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:20.283812    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2mhnp"]
Sep 18 19:15:20.379219 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:20.379181    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4b200649-8dc1-4a78-9d23-4f0c0ff3fc08-data-volume\") pod \"insights-runtime-extractor-2mhnp\" (UID: \"4b200649-8dc1-4a78-9d23-4f0c0ff3fc08\") " pod="openshift-insights/insights-runtime-extractor-2mhnp"
Sep 18 19:15:20.379219 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:20.379223    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4b200649-8dc1-4a78-9d23-4f0c0ff3fc08-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2mhnp\" (UID: \"4b200649-8dc1-4a78-9d23-4f0c0ff3fc08\") " pod="openshift-insights/insights-runtime-extractor-2mhnp"
Sep 18 19:15:20.379442 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:20.379298    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4b200649-8dc1-4a78-9d23-4f0c0ff3fc08-crio-socket\") pod \"insights-runtime-extractor-2mhnp\" (UID: \"4b200649-8dc1-4a78-9d23-4f0c0ff3fc08\") " pod="openshift-insights/insights-runtime-extractor-2mhnp"
Sep 18 19:15:20.379442 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:20.379342    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4b200649-8dc1-4a78-9d23-4f0c0ff3fc08-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2mhnp\" (UID: \"4b200649-8dc1-4a78-9d23-4f0c0ff3fc08\") " pod="openshift-insights/insights-runtime-extractor-2mhnp"
Sep 18 19:15:20.379442 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:20.379370    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjcww\" (UniqueName: \"kubernetes.io/projected/4b200649-8dc1-4a78-9d23-4f0c0ff3fc08-kube-api-access-xjcww\") pod \"insights-runtime-extractor-2mhnp\" (UID: \"4b200649-8dc1-4a78-9d23-4f0c0ff3fc08\") " pod="openshift-insights/insights-runtime-extractor-2mhnp"
Sep 18 19:15:20.480051 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:20.480017    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4b200649-8dc1-4a78-9d23-4f0c0ff3fc08-crio-socket\") pod \"insights-runtime-extractor-2mhnp\" (UID: \"4b200649-8dc1-4a78-9d23-4f0c0ff3fc08\") " pod="openshift-insights/insights-runtime-extractor-2mhnp"
Sep 18 19:15:20.480213 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:20.480070    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4b200649-8dc1-4a78-9d23-4f0c0ff3fc08-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2mhnp\" (UID: \"4b200649-8dc1-4a78-9d23-4f0c0ff3fc08\") " pod="openshift-insights/insights-runtime-extractor-2mhnp"
Sep 18 19:15:20.480213 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:20.480105    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjcww\" (UniqueName: \"kubernetes.io/projected/4b200649-8dc1-4a78-9d23-4f0c0ff3fc08-kube-api-access-xjcww\") pod \"insights-runtime-extractor-2mhnp\" (UID: \"4b200649-8dc1-4a78-9d23-4f0c0ff3fc08\") " pod="openshift-insights/insights-runtime-extractor-2mhnp"
Sep 18 19:15:20.480213 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:20.480128    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4b200649-8dc1-4a78-9d23-4f0c0ff3fc08-crio-socket\") pod \"insights-runtime-extractor-2mhnp\" (UID: \"4b200649-8dc1-4a78-9d23-4f0c0ff3fc08\") " pod="openshift-insights/insights-runtime-extractor-2mhnp"
Sep 18 19:15:20.480213 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:20.480159    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4b200649-8dc1-4a78-9d23-4f0c0ff3fc08-data-volume\") pod \"insights-runtime-extractor-2mhnp\" (UID: \"4b200649-8dc1-4a78-9d23-4f0c0ff3fc08\") " pod="openshift-insights/insights-runtime-extractor-2mhnp"
Sep 18 19:15:20.480213 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:20.480188    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4b200649-8dc1-4a78-9d23-4f0c0ff3fc08-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2mhnp\" (UID: \"4b200649-8dc1-4a78-9d23-4f0c0ff3fc08\") " pod="openshift-insights/insights-runtime-extractor-2mhnp"
Sep 18 19:15:20.480213 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:15:20.480199    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:15:20.480483 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:15:20.480281    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4b200649-8dc1-4a78-9d23-4f0c0ff3fc08-kube-rbac-proxy-cm podName:4b200649-8dc1-4a78-9d23-4f0c0ff3fc08 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:20.980262259 +0000 UTC m=+91.832562622 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4b200649-8dc1-4a78-9d23-4f0c0ff3fc08-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2mhnp" (UID: "4b200649-8dc1-4a78-9d23-4f0c0ff3fc08") : configmap "kube-rbac-proxy" not found
Sep 18 19:15:20.480483 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:15:20.480287    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:20.480483 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:15:20.480329    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b200649-8dc1-4a78-9d23-4f0c0ff3fc08-insights-runtime-extractor-tls podName:4b200649-8dc1-4a78-9d23-4f0c0ff3fc08 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:20.980319443 +0000 UTC m=+91.832619791 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4b200649-8dc1-4a78-9d23-4f0c0ff3fc08-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2mhnp" (UID: "4b200649-8dc1-4a78-9d23-4f0c0ff3fc08") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:20.480483 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:20.480446    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4b200649-8dc1-4a78-9d23-4f0c0ff3fc08-data-volume\") pod \"insights-runtime-extractor-2mhnp\" (UID: \"4b200649-8dc1-4a78-9d23-4f0c0ff3fc08\") " pod="openshift-insights/insights-runtime-extractor-2mhnp"
Sep 18 19:15:20.489332 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:20.489312    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjcww\" (UniqueName: \"kubernetes.io/projected/4b200649-8dc1-4a78-9d23-4f0c0ff3fc08-kube-api-access-xjcww\") pod \"insights-runtime-extractor-2mhnp\" (UID: \"4b200649-8dc1-4a78-9d23-4f0c0ff3fc08\") " pod="openshift-insights/insights-runtime-extractor-2mhnp"
Sep 18 19:15:20.984167 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:20.984116    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4b200649-8dc1-4a78-9d23-4f0c0ff3fc08-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2mhnp\" (UID: \"4b200649-8dc1-4a78-9d23-4f0c0ff3fc08\") " pod="openshift-insights/insights-runtime-extractor-2mhnp"
Sep 18 19:15:20.984352 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:20.984200    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4b200649-8dc1-4a78-9d23-4f0c0ff3fc08-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2mhnp\" (UID: \"4b200649-8dc1-4a78-9d23-4f0c0ff3fc08\") " pod="openshift-insights/insights-runtime-extractor-2mhnp"
Sep 18 19:15:20.984352 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:15:20.984228    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:15:20.984352 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:15:20.984280    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4b200649-8dc1-4a78-9d23-4f0c0ff3fc08-kube-rbac-proxy-cm podName:4b200649-8dc1-4a78-9d23-4f0c0ff3fc08 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:21.984267076 +0000 UTC m=+92.836567425 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4b200649-8dc1-4a78-9d23-4f0c0ff3fc08-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2mhnp" (UID: "4b200649-8dc1-4a78-9d23-4f0c0ff3fc08") : configmap "kube-rbac-proxy" not found
Sep 18 19:15:20.984352 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:15:20.984308    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:20.984503 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:15:20.984367    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b200649-8dc1-4a78-9d23-4f0c0ff3fc08-insights-runtime-extractor-tls podName:4b200649-8dc1-4a78-9d23-4f0c0ff3fc08 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:21.984353929 +0000 UTC m=+92.836654277 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4b200649-8dc1-4a78-9d23-4f0c0ff3fc08-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2mhnp" (UID: "4b200649-8dc1-4a78-9d23-4f0c0ff3fc08") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:21.976539 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:21.976509    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5jzcl_b7797c1c-31e6-4f63-84d2-6f3c99e195ee/dns-node-resolver/0.log"
Sep 18 19:15:21.991757 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:21.991729    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4b200649-8dc1-4a78-9d23-4f0c0ff3fc08-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2mhnp\" (UID: \"4b200649-8dc1-4a78-9d23-4f0c0ff3fc08\") " pod="openshift-insights/insights-runtime-extractor-2mhnp"
Sep 18 19:15:21.991846 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:21.991785    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4b200649-8dc1-4a78-9d23-4f0c0ff3fc08-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2mhnp\" (UID: \"4b200649-8dc1-4a78-9d23-4f0c0ff3fc08\") " pod="openshift-insights/insights-runtime-extractor-2mhnp"
Sep 18 19:15:21.991902 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:15:21.991867    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:15:21.991962 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:15:21.991896    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:21.991962 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:15:21.991915    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4b200649-8dc1-4a78-9d23-4f0c0ff3fc08-kube-rbac-proxy-cm podName:4b200649-8dc1-4a78-9d23-4f0c0ff3fc08 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:23.991902846 +0000 UTC m=+94.844203194 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4b200649-8dc1-4a78-9d23-4f0c0ff3fc08-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2mhnp" (UID: "4b200649-8dc1-4a78-9d23-4f0c0ff3fc08") : configmap "kube-rbac-proxy" not found
Sep 18 19:15:21.991962 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:15:21.991956    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b200649-8dc1-4a78-9d23-4f0c0ff3fc08-insights-runtime-extractor-tls podName:4b200649-8dc1-4a78-9d23-4f0c0ff3fc08 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:23.991938684 +0000 UTC m=+94.844239046 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4b200649-8dc1-4a78-9d23-4f0c0ff3fc08-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2mhnp" (UID: "4b200649-8dc1-4a78-9d23-4f0c0ff3fc08") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:22.037594 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:22.037563    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-p7mxq"]
Sep 18 19:15:22.039878 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:22.039863    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-p7mxq"
Sep 18 19:15:22.055550 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:22.055523    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:15:22.055550 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:22.055543    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-fsdmn\""
Sep 18 19:15:22.055732 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:22.055657    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 18 19:15:22.059197 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:22.059171    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-p7mxq"]
Sep 18 19:15:22.092933 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:22.092899    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8svh8\" (UniqueName: \"kubernetes.io/projected/2767860b-c421-4310-8990-99b908e03b08-kube-api-access-8svh8\") pod \"volume-data-source-validator-5c4db8bdf9-p7mxq\" (UID: \"2767860b-c421-4310-8990-99b908e03b08\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-p7mxq"
Sep 18 19:15:22.193937 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:22.193904    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8svh8\" (UniqueName: \"kubernetes.io/projected/2767860b-c421-4310-8990-99b908e03b08-kube-api-access-8svh8\") pod \"volume-data-source-validator-5c4db8bdf9-p7mxq\" (UID: \"2767860b-c421-4310-8990-99b908e03b08\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-p7mxq"
Sep 18 19:15:22.203269 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:22.203237    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8svh8\" (UniqueName: \"kubernetes.io/projected/2767860b-c421-4310-8990-99b908e03b08-kube-api-access-8svh8\") pod \"volume-data-source-validator-5c4db8bdf9-p7mxq\" (UID: \"2767860b-c421-4310-8990-99b908e03b08\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-p7mxq"
Sep 18 19:15:22.352183 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:22.352099    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-p7mxq"
Sep 18 19:15:22.620503 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:22.620474    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-p7mxq"]
Sep 18 19:15:22.623502 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:15:22.623478    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2767860b_c421_4310_8990_99b908e03b08.slice/crio-3b79220ded3d283ece0613c81d0bd3654ac00918aa899264052b322f350ce9c1 WatchSource:0}: Error finding container 3b79220ded3d283ece0613c81d0bd3654ac00918aa899264052b322f350ce9c1: Status 404 returned error can't find the container with id 3b79220ded3d283ece0613c81d0bd3654ac00918aa899264052b322f350ce9c1
Sep 18 19:15:22.776113 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:22.776089    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mx49g_ea018bdf-321c-4368-875d-937e82b2ed3b/node-ca/0.log"
Sep 18 19:15:22.892020 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:22.891921    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-p7mxq" event={"ID":"2767860b-c421-4310-8990-99b908e03b08","Type":"ContainerStarted","Data":"3b79220ded3d283ece0613c81d0bd3654ac00918aa899264052b322f350ce9c1"}
Sep 18 19:15:23.978722 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:23.978693    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mb6kf_44aa71cc-fda2-4a75-a657-23e31f95e973/serve-healthcheck-canary/0.log"
Sep 18 19:15:24.007638 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:24.007609    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4b200649-8dc1-4a78-9d23-4f0c0ff3fc08-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2mhnp\" (UID: \"4b200649-8dc1-4a78-9d23-4f0c0ff3fc08\") " pod="openshift-insights/insights-runtime-extractor-2mhnp"
Sep 18 19:15:24.007725 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:24.007673    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4b200649-8dc1-4a78-9d23-4f0c0ff3fc08-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2mhnp\" (UID: \"4b200649-8dc1-4a78-9d23-4f0c0ff3fc08\") " pod="openshift-insights/insights-runtime-extractor-2mhnp"
Sep 18 19:15:24.007768 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:15:24.007728    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:15:24.007803 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:15:24.007777    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:24.007803 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:15:24.007785    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4b200649-8dc1-4a78-9d23-4f0c0ff3fc08-kube-rbac-proxy-cm podName:4b200649-8dc1-4a78-9d23-4f0c0ff3fc08 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:28.007771687 +0000 UTC m=+98.860072039 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4b200649-8dc1-4a78-9d23-4f0c0ff3fc08-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2mhnp" (UID: "4b200649-8dc1-4a78-9d23-4f0c0ff3fc08") : configmap "kube-rbac-proxy" not found
Sep 18 19:15:24.007879 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:15:24.007817    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b200649-8dc1-4a78-9d23-4f0c0ff3fc08-insights-runtime-extractor-tls podName:4b200649-8dc1-4a78-9d23-4f0c0ff3fc08 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:28.00780675 +0000 UTC m=+98.860107097 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4b200649-8dc1-4a78-9d23-4f0c0ff3fc08-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2mhnp" (UID: "4b200649-8dc1-4a78-9d23-4f0c0ff3fc08") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:24.899418 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:24.899384    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-p7mxq" event={"ID":"2767860b-c421-4310-8990-99b908e03b08","Type":"ContainerStarted","Data":"6921e9b7466b337053140562308b1f398f9ba0442cef1467d8485e7040cc98ec"}
Sep 18 19:15:24.918899 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:24.918853    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-p7mxq" podStartSLOduration=1.6603192199999999 podStartE2EDuration="2.918837587s" totalImagesPullingTime="1.258518367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:15:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:24.917549291 +0000 UTC m=+95.769849657" watchObservedRunningTime="2026-09-18 19:15:24.918837587 +0000 UTC m=+95.771138000"
Sep 18 19:15:28.036251 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:28.036196    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4b200649-8dc1-4a78-9d23-4f0c0ff3fc08-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2mhnp\" (UID: \"4b200649-8dc1-4a78-9d23-4f0c0ff3fc08\") " pod="openshift-insights/insights-runtime-extractor-2mhnp"
Sep 18 19:15:28.036615 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:28.036277    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4b200649-8dc1-4a78-9d23-4f0c0ff3fc08-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2mhnp\" (UID: \"4b200649-8dc1-4a78-9d23-4f0c0ff3fc08\") " pod="openshift-insights/insights-runtime-extractor-2mhnp"
Sep 18 19:15:28.036615 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:15:28.036349    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:15:28.036615 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:15:28.036367    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:28.036615 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:15:28.036401    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4b200649-8dc1-4a78-9d23-4f0c0ff3fc08-kube-rbac-proxy-cm podName:4b200649-8dc1-4a78-9d23-4f0c0ff3fc08 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:36.036388442 +0000 UTC m=+106.888688790 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4b200649-8dc1-4a78-9d23-4f0c0ff3fc08-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2mhnp" (UID: "4b200649-8dc1-4a78-9d23-4f0c0ff3fc08") : configmap "kube-rbac-proxy" not found
Sep 18 19:15:28.036615 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:15:28.036424    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b200649-8dc1-4a78-9d23-4f0c0ff3fc08-insights-runtime-extractor-tls podName:4b200649-8dc1-4a78-9d23-4f0c0ff3fc08 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:36.036410403 +0000 UTC m=+106.888710774 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4b200649-8dc1-4a78-9d23-4f0c0ff3fc08-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2mhnp" (UID: "4b200649-8dc1-4a78-9d23-4f0c0ff3fc08") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:29.246930 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:29.246891    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-registry-tls\") pod \"image-registry-69df585c75-9p6zz\" (UID: \"7feb1f7b-32c7-4e47-a257-31e42c72cd0f\") " pod="openshift-image-registry/image-registry-69df585c75-9p6zz"
Sep 18 19:15:29.247325 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:15:29.247030    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:15:29.247325 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:15:29.247042    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69df585c75-9p6zz: secret "image-registry-tls" not found
Sep 18 19:15:29.247325 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:15:29.247095    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-registry-tls podName:7feb1f7b-32c7-4e47-a257-31e42c72cd0f nodeName:}" failed. No retries permitted until 2026-09-18 19:16:33.247079821 +0000 UTC m=+164.099380190 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-registry-tls") pod "image-registry-69df585c75-9p6zz" (UID: "7feb1f7b-32c7-4e47-a257-31e42c72cd0f") : secret "image-registry-tls" not found
Sep 18 19:15:29.347575 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:29.347535    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dbd605d1-eb07-4c9c-aea1-f1a610394801-metrics-tls\") pod \"dns-default-dg5zx\" (UID: \"dbd605d1-eb07-4c9c-aea1-f1a610394801\") " pod="openshift-dns/dns-default-dg5zx"
Sep 18 19:15:29.347746 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:15:29.347662    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:15:29.347746 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:15:29.347718    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbd605d1-eb07-4c9c-aea1-f1a610394801-metrics-tls podName:dbd605d1-eb07-4c9c-aea1-f1a610394801 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:33.347701734 +0000 UTC m=+164.200002101 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dbd605d1-eb07-4c9c-aea1-f1a610394801-metrics-tls") pod "dns-default-dg5zx" (UID: "dbd605d1-eb07-4c9c-aea1-f1a610394801") : secret "dns-default-metrics-tls" not found
Sep 18 19:15:29.854811 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:29.854730    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-mzfv6"]
Sep 18 19:15:29.856662 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:29.856645    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-mzfv6"
Sep 18 19:15:29.859400 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:29.859374    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-w6mgf\""
Sep 18 19:15:29.867949 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:29.867925    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-mzfv6"]
Sep 18 19:15:29.952073 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:29.952032    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjtv7\" (UniqueName: \"kubernetes.io/projected/47535649-18ed-4ff9-830a-5915c298edde-kube-api-access-jjtv7\") pod \"network-check-source-7cbc45fddc-mzfv6\" (UID: \"47535649-18ed-4ff9-830a-5915c298edde\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-mzfv6"
Sep 18 19:15:30.052716 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:30.052683    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjtv7\" (UniqueName: \"kubernetes.io/projected/47535649-18ed-4ff9-830a-5915c298edde-kube-api-access-jjtv7\") pod \"network-check-source-7cbc45fddc-mzfv6\" (UID: \"47535649-18ed-4ff9-830a-5915c298edde\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-mzfv6"
Sep 18 19:15:30.062009 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:30.061965    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjtv7\" (UniqueName: \"kubernetes.io/projected/47535649-18ed-4ff9-830a-5915c298edde-kube-api-access-jjtv7\") pod \"network-check-source-7cbc45fddc-mzfv6\" (UID: \"47535649-18ed-4ff9-830a-5915c298edde\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-mzfv6"
Sep 18 19:15:30.165051 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:30.165009    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-mzfv6"
Sep 18 19:15:30.444519 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:30.444444    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-mzfv6"]
Sep 18 19:15:30.447387 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:15:30.447362    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47535649_18ed_4ff9_830a_5915c298edde.slice/crio-94e6d1e9ae48a44c6da869dc8f3659e21b6b0f3891ff4e869d5076d9311f028c WatchSource:0}: Error finding container 94e6d1e9ae48a44c6da869dc8f3659e21b6b0f3891ff4e869d5076d9311f028c: Status 404 returned error can't find the container with id 94e6d1e9ae48a44c6da869dc8f3659e21b6b0f3891ff4e869d5076d9311f028c
Sep 18 19:15:30.915949 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:30.915916    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-mzfv6" event={"ID":"47535649-18ed-4ff9-830a-5915c298edde","Type":"ContainerStarted","Data":"e141d8eec91ba85c5793fe34aca6a55ae803fb7f213568e0073919a50899454b"}
Sep 18 19:15:30.916228 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:30.915964    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-mzfv6" event={"ID":"47535649-18ed-4ff9-830a-5915c298edde","Type":"ContainerStarted","Data":"94e6d1e9ae48a44c6da869dc8f3659e21b6b0f3891ff4e869d5076d9311f028c"}
Sep 18 19:15:30.937190 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:30.937145    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-mzfv6" podStartSLOduration=1.9371306339999999 podStartE2EDuration="1.937130634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:15:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:30.937063685 +0000 UTC m=+101.789364057" watchObservedRunningTime="2026-09-18 19:15:30.937130634 +0000 UTC m=+101.789431003"
Sep 18 19:15:32.948034 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:32.948002    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-92v7w"]
Sep 18 19:15:32.951108 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:32.951092    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-92v7w"
Sep 18 19:15:32.955715 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:32.955697    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-6tfvl\""
Sep 18 19:15:32.956004 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:32.955969    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 18 19:15:32.956802 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:32.956784    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 18 19:15:32.962984 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:32.962966    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-92v7w"]
Sep 18 19:15:33.074654 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:33.074621    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bf41ad7a-3615-4981-8a3e-125f973295aa-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-92v7w\" (UID: \"bf41ad7a-3615-4981-8a3e-125f973295aa\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-92v7w"
Sep 18 19:15:33.074654 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:33.074656    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bf41ad7a-3615-4981-8a3e-125f973295aa-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-92v7w\" (UID: \"bf41ad7a-3615-4981-8a3e-125f973295aa\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-92v7w"
Sep 18 19:15:33.175393 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:33.175360    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bf41ad7a-3615-4981-8a3e-125f973295aa-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-92v7w\" (UID: \"bf41ad7a-3615-4981-8a3e-125f973295aa\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-92v7w"
Sep 18 19:15:33.175393 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:33.175395    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bf41ad7a-3615-4981-8a3e-125f973295aa-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-92v7w\" (UID: \"bf41ad7a-3615-4981-8a3e-125f973295aa\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-92v7w"
Sep 18 19:15:33.175579 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:15:33.175525    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 18 19:15:33.175614 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:15:33.175584    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf41ad7a-3615-4981-8a3e-125f973295aa-networking-console-plugin-cert podName:bf41ad7a-3615-4981-8a3e-125f973295aa nodeName:}" failed. No retries permitted until 2026-09-18 19:15:33.675571502 +0000 UTC m=+104.527871855 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bf41ad7a-3615-4981-8a3e-125f973295aa-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-92v7w" (UID: "bf41ad7a-3615-4981-8a3e-125f973295aa") : secret "networking-console-plugin-cert" not found
Sep 18 19:15:33.175960 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:33.175941    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bf41ad7a-3615-4981-8a3e-125f973295aa-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-92v7w\" (UID: \"bf41ad7a-3615-4981-8a3e-125f973295aa\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-92v7w"
Sep 18 19:15:33.678130 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:33.678099    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bf41ad7a-3615-4981-8a3e-125f973295aa-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-92v7w\" (UID: \"bf41ad7a-3615-4981-8a3e-125f973295aa\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-92v7w"
Sep 18 19:15:33.678344 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:15:33.678263    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 18 19:15:33.678344 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:15:33.678339    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf41ad7a-3615-4981-8a3e-125f973295aa-networking-console-plugin-cert podName:bf41ad7a-3615-4981-8a3e-125f973295aa nodeName:}" failed. No retries permitted until 2026-09-18 19:15:34.678319515 +0000 UTC m=+105.530619877 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bf41ad7a-3615-4981-8a3e-125f973295aa-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-92v7w" (UID: "bf41ad7a-3615-4981-8a3e-125f973295aa") : secret "networking-console-plugin-cert" not found
Sep 18 19:15:33.846949 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:33.846921    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2fpv8"
Sep 18 19:15:34.687809 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:34.687770    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bf41ad7a-3615-4981-8a3e-125f973295aa-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-92v7w\" (UID: \"bf41ad7a-3615-4981-8a3e-125f973295aa\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-92v7w"
Sep 18 19:15:34.688320 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:15:34.687943    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 18 19:15:34.688320 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:15:34.688050    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf41ad7a-3615-4981-8a3e-125f973295aa-networking-console-plugin-cert podName:bf41ad7a-3615-4981-8a3e-125f973295aa nodeName:}" failed. No retries permitted until 2026-09-18 19:15:36.688028896 +0000 UTC m=+107.540329262 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bf41ad7a-3615-4981-8a3e-125f973295aa-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-92v7w" (UID: "bf41ad7a-3615-4981-8a3e-125f973295aa") : secret "networking-console-plugin-cert" not found
Sep 18 19:15:36.099785 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:36.099746    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4b200649-8dc1-4a78-9d23-4f0c0ff3fc08-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2mhnp\" (UID: \"4b200649-8dc1-4a78-9d23-4f0c0ff3fc08\") " pod="openshift-insights/insights-runtime-extractor-2mhnp"
Sep 18 19:15:36.100201 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:36.099839    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4b200649-8dc1-4a78-9d23-4f0c0ff3fc08-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2mhnp\" (UID: \"4b200649-8dc1-4a78-9d23-4f0c0ff3fc08\") " pod="openshift-insights/insights-runtime-extractor-2mhnp"
Sep 18 19:15:36.100201 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:15:36.099950    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:36.100201 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:15:36.100046    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b200649-8dc1-4a78-9d23-4f0c0ff3fc08-insights-runtime-extractor-tls podName:4b200649-8dc1-4a78-9d23-4f0c0ff3fc08 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:52.10002772 +0000 UTC m=+122.952328082 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4b200649-8dc1-4a78-9d23-4f0c0ff3fc08-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2mhnp" (UID: "4b200649-8dc1-4a78-9d23-4f0c0ff3fc08") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:36.100433 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:36.100412    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4b200649-8dc1-4a78-9d23-4f0c0ff3fc08-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2mhnp\" (UID: \"4b200649-8dc1-4a78-9d23-4f0c0ff3fc08\") " pod="openshift-insights/insights-runtime-extractor-2mhnp"
Sep 18 19:15:36.704505 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:36.704464    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bf41ad7a-3615-4981-8a3e-125f973295aa-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-92v7w\" (UID: \"bf41ad7a-3615-4981-8a3e-125f973295aa\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-92v7w"
Sep 18 19:15:36.704694 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:15:36.704605    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 18 19:15:36.704694 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:15:36.704671    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf41ad7a-3615-4981-8a3e-125f973295aa-networking-console-plugin-cert podName:bf41ad7a-3615-4981-8a3e-125f973295aa nodeName:}" failed. No retries permitted until 2026-09-18 19:15:40.704657781 +0000 UTC m=+111.556958133 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bf41ad7a-3615-4981-8a3e-125f973295aa-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-92v7w" (UID: "bf41ad7a-3615-4981-8a3e-125f973295aa") : secret "networking-console-plugin-cert" not found
Sep 18 19:15:40.734610 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:40.734560    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bf41ad7a-3615-4981-8a3e-125f973295aa-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-92v7w\" (UID: \"bf41ad7a-3615-4981-8a3e-125f973295aa\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-92v7w"
Sep 18 19:15:40.735018 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:15:40.734720    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 18 19:15:40.735018 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:15:40.734793    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf41ad7a-3615-4981-8a3e-125f973295aa-networking-console-plugin-cert podName:bf41ad7a-3615-4981-8a3e-125f973295aa nodeName:}" failed. No retries permitted until 2026-09-18 19:15:48.734777517 +0000 UTC m=+119.587077865 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bf41ad7a-3615-4981-8a3e-125f973295aa-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-92v7w" (UID: "bf41ad7a-3615-4981-8a3e-125f973295aa") : secret "networking-console-plugin-cert" not found
Sep 18 19:15:48.796063 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:48.796006    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bf41ad7a-3615-4981-8a3e-125f973295aa-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-92v7w\" (UID: \"bf41ad7a-3615-4981-8a3e-125f973295aa\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-92v7w"
Sep 18 19:15:48.798416 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:48.798393    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bf41ad7a-3615-4981-8a3e-125f973295aa-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-92v7w\" (UID: \"bf41ad7a-3615-4981-8a3e-125f973295aa\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-92v7w"
Sep 18 19:15:48.859634 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:48.859588    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-92v7w"
Sep 18 19:15:49.154914 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:49.154876    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-92v7w"]
Sep 18 19:15:49.160797 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:15:49.160772    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf41ad7a_3615_4981_8a3e_125f973295aa.slice/crio-0450fb69b235d37b79f01d09739f5cd806ac3e7a363ac21c48a004795a8edb33 WatchSource:0}: Error finding container 0450fb69b235d37b79f01d09739f5cd806ac3e7a363ac21c48a004795a8edb33: Status 404 returned error can't find the container with id 0450fb69b235d37b79f01d09739f5cd806ac3e7a363ac21c48a004795a8edb33
Sep 18 19:15:49.962569 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:49.962537    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-92v7w" event={"ID":"bf41ad7a-3615-4981-8a3e-125f973295aa","Type":"ContainerStarted","Data":"0450fb69b235d37b79f01d09739f5cd806ac3e7a363ac21c48a004795a8edb33"}
Sep 18 19:15:50.966708 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:50.966665    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-92v7w" event={"ID":"bf41ad7a-3615-4981-8a3e-125f973295aa","Type":"ContainerStarted","Data":"73302b40aeb38167383ff2521a7fa079e7318c45d40143c3741858bf55939d6b"}
Sep 18 19:15:51.012198 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:51.012145    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6446c8d75d-92v7w" podStartSLOduration=17.905539624 podStartE2EDuration="19.012127583s" totalImagesPullingTime="1.106587959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:15:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:51.011502962 +0000 UTC m=+121.863803342" watchObservedRunningTime="2026-09-18 19:15:51.012127583 +0000 UTC m=+121.864427953"
Sep 18 19:15:52.119835 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:52.119796    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4b200649-8dc1-4a78-9d23-4f0c0ff3fc08-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2mhnp\" (UID: \"4b200649-8dc1-4a78-9d23-4f0c0ff3fc08\") " pod="openshift-insights/insights-runtime-extractor-2mhnp"
Sep 18 19:15:52.122031 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:52.122011    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4b200649-8dc1-4a78-9d23-4f0c0ff3fc08-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2mhnp\" (UID: \"4b200649-8dc1-4a78-9d23-4f0c0ff3fc08\") " pod="openshift-insights/insights-runtime-extractor-2mhnp"
Sep 18 19:15:52.374349 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:52.374271    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-d2glk\""
Sep 18 19:15:52.381757 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:52.381741    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2mhnp"
Sep 18 19:15:52.655540 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:52.655455    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2mhnp"]
Sep 18 19:15:52.658694 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:15:52.658665    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b200649_8dc1_4a78_9d23_4f0c0ff3fc08.slice/crio-14d5a3ee0729d09b966313999362a965afe5cd0ed5ec45830bde7d4dcfd208c4 WatchSource:0}: Error finding container 14d5a3ee0729d09b966313999362a965afe5cd0ed5ec45830bde7d4dcfd208c4: Status 404 returned error can't find the container with id 14d5a3ee0729d09b966313999362a965afe5cd0ed5ec45830bde7d4dcfd208c4
Sep 18 19:15:52.867399 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:52.867370    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-86c6cc8688-pln5k"]
Sep 18 19:15:52.869345 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:52.869325    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86c6cc8688-pln5k"
Sep 18 19:15:52.875901 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:52.875767    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 18 19:15:52.875901 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:52.875844    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 18 19:15:52.876045 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:52.876028    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 18 19:15:52.876847 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:52.876826    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 18 19:15:52.892312 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:52.892278    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d7657945b-djv9d"]
Sep 18 19:15:52.894669 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:52.894650    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d7657945b-djv9d"
Sep 18 19:15:52.896290 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:52.896270    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-86c6cc8688-pln5k"]
Sep 18 19:15:52.897157 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:52.897139    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-wslqh\""
Sep 18 19:15:52.897373 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:52.897357    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 18 19:15:52.916280 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:52.916231    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d7657945b-djv9d"]
Sep 18 19:15:52.974690 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:52.974653    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2mhnp" event={"ID":"4b200649-8dc1-4a78-9d23-4f0c0ff3fc08","Type":"ContainerStarted","Data":"74c685602ec4ead9ec9ee70ffdd609a7e66e71ade5b24d1be3916607dd23e293"}
Sep 18 19:15:52.974836 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:52.974703    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2mhnp" event={"ID":"4b200649-8dc1-4a78-9d23-4f0c0ff3fc08","Type":"ContainerStarted","Data":"14d5a3ee0729d09b966313999362a965afe5cd0ed5ec45830bde7d4dcfd208c4"}
Sep 18 19:15:53.026559 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:53.026521    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgg7f\" (UniqueName: \"kubernetes.io/projected/3403d83c-b655-4a44-8728-a27ea452fab9-kube-api-access-qgg7f\") pod \"managed-serviceaccount-addon-agent-5d7657945b-djv9d\" (UID: \"3403d83c-b655-4a44-8728-a27ea452fab9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d7657945b-djv9d"
Sep 18 19:15:53.026559 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:53.026557    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzqp2\" (UniqueName: \"kubernetes.io/projected/24b8d378-f15d-474e-95ce-c723c128eba4-kube-api-access-wzqp2\") pod \"klusterlet-addon-workmgr-86c6cc8688-pln5k\" (UID: \"24b8d378-f15d-474e-95ce-c723c128eba4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86c6cc8688-pln5k"
Sep 18 19:15:53.026747 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:53.026588    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3403d83c-b655-4a44-8728-a27ea452fab9-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5d7657945b-djv9d\" (UID: \"3403d83c-b655-4a44-8728-a27ea452fab9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d7657945b-djv9d"
Sep 18 19:15:53.026747 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:53.026646    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24b8d378-f15d-474e-95ce-c723c128eba4-tmp\") pod \"klusterlet-addon-workmgr-86c6cc8688-pln5k\" (UID: \"24b8d378-f15d-474e-95ce-c723c128eba4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86c6cc8688-pln5k"
Sep 18 19:15:53.026817 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:53.026750    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/24b8d378-f15d-474e-95ce-c723c128eba4-klusterlet-config\") pod \"klusterlet-addon-workmgr-86c6cc8688-pln5k\" (UID: \"24b8d378-f15d-474e-95ce-c723c128eba4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86c6cc8688-pln5k"
Sep 18 19:15:53.127771 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:53.127735    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/24b8d378-f15d-474e-95ce-c723c128eba4-klusterlet-config\") pod \"klusterlet-addon-workmgr-86c6cc8688-pln5k\" (UID: \"24b8d378-f15d-474e-95ce-c723c128eba4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86c6cc8688-pln5k"
Sep 18 19:15:53.127771 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:53.127773    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgg7f\" (UniqueName: \"kubernetes.io/projected/3403d83c-b655-4a44-8728-a27ea452fab9-kube-api-access-qgg7f\") pod \"managed-serviceaccount-addon-agent-5d7657945b-djv9d\" (UID: \"3403d83c-b655-4a44-8728-a27ea452fab9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d7657945b-djv9d"
Sep 18 19:15:53.128284 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:53.127789    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzqp2\" (UniqueName: \"kubernetes.io/projected/24b8d378-f15d-474e-95ce-c723c128eba4-kube-api-access-wzqp2\") pod \"klusterlet-addon-workmgr-86c6cc8688-pln5k\" (UID: \"24b8d378-f15d-474e-95ce-c723c128eba4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86c6cc8688-pln5k"
Sep 18 19:15:53.128284 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:53.128021    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3403d83c-b655-4a44-8728-a27ea452fab9-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5d7657945b-djv9d\" (UID: \"3403d83c-b655-4a44-8728-a27ea452fab9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d7657945b-djv9d"
Sep 18 19:15:53.128284 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:53.128082    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24b8d378-f15d-474e-95ce-c723c128eba4-tmp\") pod \"klusterlet-addon-workmgr-86c6cc8688-pln5k\" (UID: \"24b8d378-f15d-474e-95ce-c723c128eba4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86c6cc8688-pln5k"
Sep 18 19:15:53.128477 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:53.128453    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24b8d378-f15d-474e-95ce-c723c128eba4-tmp\") pod \"klusterlet-addon-workmgr-86c6cc8688-pln5k\" (UID: \"24b8d378-f15d-474e-95ce-c723c128eba4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86c6cc8688-pln5k"
Sep 18 19:15:53.130771 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:53.130746    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/24b8d378-f15d-474e-95ce-c723c128eba4-klusterlet-config\") pod \"klusterlet-addon-workmgr-86c6cc8688-pln5k\" (UID: \"24b8d378-f15d-474e-95ce-c723c128eba4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86c6cc8688-pln5k"
Sep 18 19:15:53.130870 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:53.130801    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3403d83c-b655-4a44-8728-a27ea452fab9-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5d7657945b-djv9d\" (UID: \"3403d83c-b655-4a44-8728-a27ea452fab9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d7657945b-djv9d"
Sep 18 19:15:53.151694 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:53.151666    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzqp2\" (UniqueName: \"kubernetes.io/projected/24b8d378-f15d-474e-95ce-c723c128eba4-kube-api-access-wzqp2\") pod \"klusterlet-addon-workmgr-86c6cc8688-pln5k\" (UID: \"24b8d378-f15d-474e-95ce-c723c128eba4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86c6cc8688-pln5k"
Sep 18 19:15:53.152814 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:53.152791    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgg7f\" (UniqueName: \"kubernetes.io/projected/3403d83c-b655-4a44-8728-a27ea452fab9-kube-api-access-qgg7f\") pod \"managed-serviceaccount-addon-agent-5d7657945b-djv9d\" (UID: \"3403d83c-b655-4a44-8728-a27ea452fab9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d7657945b-djv9d"
Sep 18 19:15:53.178863 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:53.178780    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86c6cc8688-pln5k"
Sep 18 19:15:53.213352 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:53.213319    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d7657945b-djv9d"
Sep 18 19:15:53.466726 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:53.466659    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-86c6cc8688-pln5k"]
Sep 18 19:15:53.469739 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:15:53.469715    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24b8d378_f15d_474e_95ce_c723c128eba4.slice/crio-967e386007d6c94bdfc92e572828a7222dd0033082bb23f3973a2a9daff9adcd WatchSource:0}: Error finding container 967e386007d6c94bdfc92e572828a7222dd0033082bb23f3973a2a9daff9adcd: Status 404 returned error can't find the container with id 967e386007d6c94bdfc92e572828a7222dd0033082bb23f3973a2a9daff9adcd
Sep 18 19:15:53.569933 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:53.569892    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d7657945b-djv9d"]
Sep 18 19:15:53.572582 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:15:53.572555    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3403d83c_b655_4a44_8728_a27ea452fab9.slice/crio-1f798bb3c3f45e72ee86d949debdb4290f3fd874d1afce486f7abc9cab358f68 WatchSource:0}: Error finding container 1f798bb3c3f45e72ee86d949debdb4290f3fd874d1afce486f7abc9cab358f68: Status 404 returned error can't find the container with id 1f798bb3c3f45e72ee86d949debdb4290f3fd874d1afce486f7abc9cab358f68
Sep 18 19:15:53.980472 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:53.980426    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86c6cc8688-pln5k" event={"ID":"24b8d378-f15d-474e-95ce-c723c128eba4","Type":"ContainerStarted","Data":"967e386007d6c94bdfc92e572828a7222dd0033082bb23f3973a2a9daff9adcd"}
Sep 18 19:15:53.982254 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:53.982223    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2mhnp" event={"ID":"4b200649-8dc1-4a78-9d23-4f0c0ff3fc08","Type":"ContainerStarted","Data":"fe5fce190a298feef095facdb25ba6c24bd61f20cfcde9b1ec453743b9eb06c5"}
Sep 18 19:15:53.983148 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:53.983128    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d7657945b-djv9d" event={"ID":"3403d83c-b655-4a44-8728-a27ea452fab9","Type":"ContainerStarted","Data":"1f798bb3c3f45e72ee86d949debdb4290f3fd874d1afce486f7abc9cab358f68"}
Sep 18 19:15:55.998373 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:55.998334    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2mhnp" event={"ID":"4b200649-8dc1-4a78-9d23-4f0c0ff3fc08","Type":"ContainerStarted","Data":"13661107ac99a6ffdad4e0cd618b19f0ad0540c25fceda768dff53ca7ba71b29"}
Sep 18 19:15:56.057150 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:56.057086    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2mhnp" podStartSLOduration=33.549549604 podStartE2EDuration="36.057065146s" totalImagesPullingTime="2.507515542s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:15:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:56.055855755 +0000 UTC m=+126.908156121" watchObservedRunningTime="2026-09-18 19:15:56.057065146 +0000 UTC m=+126.909365521"
Sep 18 19:15:56.234323 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:56.234290    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-w7p4q"]
Sep 18 19:15:56.236851 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:56.236826    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-w7p4q"
Sep 18 19:15:56.242357 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:56.242317    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 18 19:15:56.243547 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:56.243500    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 18 19:15:56.243677 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:56.243570    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-4hdvq\""
Sep 18 19:15:56.243806 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:56.243784    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 18 19:15:56.243890 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:56.243868    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 18 19:15:56.244086 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:56.243978    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 18 19:15:56.254459 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:56.254401    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-w7p4q"]
Sep 18 19:15:56.357771 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:56.357734    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/83e874e2-103a-4097-b485-253beb0f1590-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-w7p4q\" (UID: \"83e874e2-103a-4097-b485-253beb0f1590\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-w7p4q"
Sep 18 19:15:56.357954 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:56.357813    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83e874e2-103a-4097-b485-253beb0f1590-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-w7p4q\" (UID: \"83e874e2-103a-4097-b485-253beb0f1590\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-w7p4q"
Sep 18 19:15:56.357954 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:56.357928    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snddh\" (UniqueName: \"kubernetes.io/projected/83e874e2-103a-4097-b485-253beb0f1590-kube-api-access-snddh\") pod \"prometheus-operator-7bd5d56d6-w7p4q\" (UID: \"83e874e2-103a-4097-b485-253beb0f1590\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-w7p4q"
Sep 18 19:15:56.358098 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:56.357970    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/83e874e2-103a-4097-b485-253beb0f1590-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-w7p4q\" (UID: \"83e874e2-103a-4097-b485-253beb0f1590\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-w7p4q"
Sep 18 19:15:56.458717 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:56.458675    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snddh\" (UniqueName: \"kubernetes.io/projected/83e874e2-103a-4097-b485-253beb0f1590-kube-api-access-snddh\") pod \"prometheus-operator-7bd5d56d6-w7p4q\" (UID: \"83e874e2-103a-4097-b485-253beb0f1590\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-w7p4q"
Sep 18 19:15:56.458717 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:56.458721    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/83e874e2-103a-4097-b485-253beb0f1590-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-w7p4q\" (UID: \"83e874e2-103a-4097-b485-253beb0f1590\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-w7p4q"
Sep 18 19:15:56.458961 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:56.458768    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/83e874e2-103a-4097-b485-253beb0f1590-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-w7p4q\" (UID: \"83e874e2-103a-4097-b485-253beb0f1590\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-w7p4q"
Sep 18 19:15:56.458961 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:56.458813    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83e874e2-103a-4097-b485-253beb0f1590-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-w7p4q\" (UID: \"83e874e2-103a-4097-b485-253beb0f1590\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-w7p4q"
Sep 18 19:15:56.459511 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:56.459484    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83e874e2-103a-4097-b485-253beb0f1590-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-w7p4q\" (UID: \"83e874e2-103a-4097-b485-253beb0f1590\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-w7p4q"
Sep 18 19:15:56.461292 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:56.461274    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/83e874e2-103a-4097-b485-253beb0f1590-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-w7p4q\" (UID: \"83e874e2-103a-4097-b485-253beb0f1590\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-w7p4q"
Sep 18 19:15:56.461410 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:56.461378    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/83e874e2-103a-4097-b485-253beb0f1590-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-w7p4q\" (UID: \"83e874e2-103a-4097-b485-253beb0f1590\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-w7p4q"
Sep 18 19:15:56.472505 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:56.472485    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snddh\" (UniqueName: \"kubernetes.io/projected/83e874e2-103a-4097-b485-253beb0f1590-kube-api-access-snddh\") pod \"prometheus-operator-7bd5d56d6-w7p4q\" (UID: \"83e874e2-103a-4097-b485-253beb0f1590\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-w7p4q"
Sep 18 19:15:56.548096 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:56.548029    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-w7p4q"
Sep 18 19:15:58.631021 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:58.630970    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-w7p4q"]
Sep 18 19:15:58.634055 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:15:58.634026    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83e874e2_103a_4097_b485_253beb0f1590.slice/crio-c0aa383425352cee48aea1350ded51bcdb3d6c896c122f49493c0a40e5edb387 WatchSource:0}: Error finding container c0aa383425352cee48aea1350ded51bcdb3d6c896c122f49493c0a40e5edb387: Status 404 returned error can't find the container with id c0aa383425352cee48aea1350ded51bcdb3d6c896c122f49493c0a40e5edb387
Sep 18 19:15:59.012464 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:59.012373    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-w7p4q" event={"ID":"83e874e2-103a-4097-b485-253beb0f1590","Type":"ContainerStarted","Data":"c0aa383425352cee48aea1350ded51bcdb3d6c896c122f49493c0a40e5edb387"}
Sep 18 19:15:59.013876 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:59.013843    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d7657945b-djv9d" event={"ID":"3403d83c-b655-4a44-8728-a27ea452fab9","Type":"ContainerStarted","Data":"6b82d85b86ad5875637f2a9c773619b82f40ffbd92cee2cf1cd3b0001a45749f"}
Sep 18 19:15:59.015317 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:59.015294    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86c6cc8688-pln5k" event={"ID":"24b8d378-f15d-474e-95ce-c723c128eba4","Type":"ContainerStarted","Data":"d7138af00016b89c034ef9d4c15f527002128b964ec128e84d0b0a162799a722"}
Sep 18 19:15:59.015494 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:59.015483    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86c6cc8688-pln5k"
Sep 18 19:15:59.017089 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:59.017070    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86c6cc8688-pln5k"
Sep 18 19:15:59.044200 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:59.044161    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d7657945b-djv9d" podStartSLOduration=2.268852848 podStartE2EDuration="7.044146567s" totalImagesPullingTime="4.775293719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:15:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:59.043607607 +0000 UTC m=+129.895907977" watchObservedRunningTime="2026-09-18 19:15:59.044146567 +0000 UTC m=+129.896446931"
Sep 18 19:15:59.087714 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:15:59.087660    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86c6cc8688-pln5k" podStartSLOduration=2.166035935 podStartE2EDuration="7.087645591s" totalImagesPullingTime="4.921609656s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:15:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:59.086908165 +0000 UTC m=+129.939208535" watchObservedRunningTime="2026-09-18 19:15:59.087645591 +0000 UTC m=+129.939945968"
Sep 18 19:16:01.023392 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:01.023359    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-w7p4q" event={"ID":"83e874e2-103a-4097-b485-253beb0f1590","Type":"ContainerStarted","Data":"c85bfd84e692c3359b6157fef98d7c404997875396832ef1c0e37c2445ac2518"}
Sep 18 19:16:01.023881 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:01.023415    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-w7p4q" event={"ID":"83e874e2-103a-4097-b485-253beb0f1590","Type":"ContainerStarted","Data":"d85921848fd7aa7f547d3317ccf330c967317e7e53cf29f671e20b18f2cc26e2"}
Sep 18 19:16:01.045408 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:01.045361    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-w7p4q" podStartSLOduration=3.649289159 podStartE2EDuration="5.04534534s" totalImagesPullingTime="1.396056181s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:15:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:16:01.043455513 +0000 UTC m=+131.895755882" watchObservedRunningTime="2026-09-18 19:16:01.04534534 +0000 UTC m=+131.897645711"
Sep 18 19:16:01.199379 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:01.199342    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0bb7eb5-2fe7-42f3-92a6-00d65358f28f-metrics-certs\") pod \"network-metrics-daemon-l9g57\" (UID: \"b0bb7eb5-2fe7-42f3-92a6-00d65358f28f\") " pod="openshift-multus/network-metrics-daemon-l9g57"
Sep 18 19:16:01.201698 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:01.201675    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0bb7eb5-2fe7-42f3-92a6-00d65358f28f-metrics-certs\") pod \"network-metrics-daemon-l9g57\" (UID: \"b0bb7eb5-2fe7-42f3-92a6-00d65358f28f\") " pod="openshift-multus/network-metrics-daemon-l9g57"
Sep 18 19:16:01.453291 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:01.453263    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-thgh8\""
Sep 18 19:16:01.460767 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:01.460749    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9g57"
Sep 18 19:16:01.738565 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:01.738488    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-l9g57"]
Sep 18 19:16:01.743532 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:16:01.743506    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0bb7eb5_2fe7_42f3_92a6_00d65358f28f.slice/crio-74316787101fbe1a8e3cd0fbeb018e6aba75db7808ae3afafdf2d6c86614eebd WatchSource:0}: Error finding container 74316787101fbe1a8e3cd0fbeb018e6aba75db7808ae3afafdf2d6c86614eebd: Status 404 returned error can't find the container with id 74316787101fbe1a8e3cd0fbeb018e6aba75db7808ae3afafdf2d6c86614eebd
Sep 18 19:16:02.027815 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:02.027734    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l9g57" event={"ID":"b0bb7eb5-2fe7-42f3-92a6-00d65358f28f","Type":"ContainerStarted","Data":"74316787101fbe1a8e3cd0fbeb018e6aba75db7808ae3afafdf2d6c86614eebd"}
Sep 18 19:16:02.637384 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:02.637356    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9xg4r"]
Sep 18 19:16:02.639941 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:02.639911    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9xg4r"
Sep 18 19:16:02.645949 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:02.645911    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 18 19:16:02.645949 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:02.645947    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 18 19:16:02.646287 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:02.646028    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-45754\""
Sep 18 19:16:02.646287 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:02.646069    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 18 19:16:02.812467 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:02.812433    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nr24\" (UniqueName: \"kubernetes.io/projected/58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd-kube-api-access-9nr24\") pod \"node-exporter-9xg4r\" (UID: \"58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd\") " pod="openshift-monitoring/node-exporter-9xg4r"
Sep 18 19:16:02.812583 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:02.812481    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd-root\") pod \"node-exporter-9xg4r\" (UID: \"58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd\") " pod="openshift-monitoring/node-exporter-9xg4r"
Sep 18 19:16:02.812649 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:02.812604    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd-sys\") pod \"node-exporter-9xg4r\" (UID: \"58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd\") " pod="openshift-monitoring/node-exporter-9xg4r"
Sep 18 19:16:02.812649 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:02.812630    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd-node-exporter-wtmp\") pod \"node-exporter-9xg4r\" (UID: \"58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd\") " pod="openshift-monitoring/node-exporter-9xg4r"
Sep 18 19:16:02.812731 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:02.812651    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9xg4r\" (UID: \"58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd\") " pod="openshift-monitoring/node-exporter-9xg4r"
Sep 18 19:16:02.812731 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:02.812680    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd-node-exporter-accelerators-collector-config\") pod \"node-exporter-9xg4r\" (UID: \"58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd\") " pod="openshift-monitoring/node-exporter-9xg4r"
Sep 18 19:16:02.812731 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:02.812721    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd-metrics-client-ca\") pod \"node-exporter-9xg4r\" (UID: \"58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd\") " pod="openshift-monitoring/node-exporter-9xg4r"
Sep 18 19:16:02.812840 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:02.812790    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd-node-exporter-tls\") pod \"node-exporter-9xg4r\" (UID: \"58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd\") " pod="openshift-monitoring/node-exporter-9xg4r"
Sep 18 19:16:02.812874 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:02.812834    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd-node-exporter-textfile\") pod \"node-exporter-9xg4r\" (UID: \"58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd\") " pod="openshift-monitoring/node-exporter-9xg4r"
Sep 18 19:16:02.914165 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:02.914087    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nr24\" (UniqueName: \"kubernetes.io/projected/58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd-kube-api-access-9nr24\") pod \"node-exporter-9xg4r\" (UID: \"58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd\") " pod="openshift-monitoring/node-exporter-9xg4r"
Sep 18 19:16:02.914165 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:02.914117    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd-root\") pod \"node-exporter-9xg4r\" (UID: \"58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd\") " pod="openshift-monitoring/node-exporter-9xg4r"
Sep 18 19:16:02.914165 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:02.914164    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd-sys\") pod \"node-exporter-9xg4r\" (UID: \"58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd\") " pod="openshift-monitoring/node-exporter-9xg4r"
Sep 18 19:16:02.914414 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:02.914182    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd-node-exporter-wtmp\") pod \"node-exporter-9xg4r\" (UID: \"58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd\") " pod="openshift-monitoring/node-exporter-9xg4r"
Sep 18 19:16:02.914414 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:02.914197    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9xg4r\" (UID: \"58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd\") " pod="openshift-monitoring/node-exporter-9xg4r"
Sep 18 19:16:02.914414 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:02.914213    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd-node-exporter-accelerators-collector-config\") pod \"node-exporter-9xg4r\" (UID: \"58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd\") " pod="openshift-monitoring/node-exporter-9xg4r"
Sep 18 19:16:02.914414 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:02.914242    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd-metrics-client-ca\") pod \"node-exporter-9xg4r\" (UID: \"58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd\") " pod="openshift-monitoring/node-exporter-9xg4r"
Sep 18 19:16:02.914414 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:02.914268    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd-sys\") pod \"node-exporter-9xg4r\" (UID: \"58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd\") " pod="openshift-monitoring/node-exporter-9xg4r"
Sep 18 19:16:02.914414 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:02.914307    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd-root\") pod \"node-exporter-9xg4r\" (UID: \"58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd\") " pod="openshift-monitoring/node-exporter-9xg4r"
Sep 18 19:16:02.914414 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:02.914309    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd-node-exporter-tls\") pod \"node-exporter-9xg4r\" (UID: \"58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd\") " pod="openshift-monitoring/node-exporter-9xg4r"
Sep 18 19:16:02.914414 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:02.914350    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd-node-exporter-wtmp\") pod \"node-exporter-9xg4r\" (UID: \"58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd\") " pod="openshift-monitoring/node-exporter-9xg4r"
Sep 18 19:16:02.914414 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:02.914367    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd-node-exporter-textfile\") pod \"node-exporter-9xg4r\" (UID: \"58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd\") " pod="openshift-monitoring/node-exporter-9xg4r"
Sep 18 19:16:02.914809 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:02.914736    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd-node-exporter-textfile\") pod \"node-exporter-9xg4r\" (UID: \"58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd\") " pod="openshift-monitoring/node-exporter-9xg4r"
Sep 18 19:16:02.914876 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:02.914855    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd-node-exporter-accelerators-collector-config\") pod \"node-exporter-9xg4r\" (UID: \"58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd\") " pod="openshift-monitoring/node-exporter-9xg4r"
Sep 18 19:16:02.914929 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:02.914915    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd-metrics-client-ca\") pod \"node-exporter-9xg4r\" (UID: \"58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd\") " pod="openshift-monitoring/node-exporter-9xg4r"
Sep 18 19:16:02.916481 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:02.916460    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9xg4r\" (UID: \"58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd\") " pod="openshift-monitoring/node-exporter-9xg4r"
Sep 18 19:16:02.916610 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:02.916593    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd-node-exporter-tls\") pod \"node-exporter-9xg4r\" (UID: \"58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd\") " pod="openshift-monitoring/node-exporter-9xg4r"
Sep 18 19:16:02.925230 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:02.925210    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nr24\" (UniqueName: \"kubernetes.io/projected/58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd-kube-api-access-9nr24\") pod \"node-exporter-9xg4r\" (UID: \"58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd\") " pod="openshift-monitoring/node-exporter-9xg4r"
Sep 18 19:16:02.951298 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:02.951278    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9xg4r"
Sep 18 19:16:02.960490 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:16:02.960460    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58eefe10_4fdc_4fff_a8b1_1ccb6f1744fd.slice/crio-cb4ae085c8d4110f2b9fc52e3994367f0b004c587c82c55c33f800502212a163 WatchSource:0}: Error finding container cb4ae085c8d4110f2b9fc52e3994367f0b004c587c82c55c33f800502212a163: Status 404 returned error can't find the container with id cb4ae085c8d4110f2b9fc52e3994367f0b004c587c82c55c33f800502212a163
Sep 18 19:16:03.032340 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:03.032306    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9xg4r" event={"ID":"58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd","Type":"ContainerStarted","Data":"cb4ae085c8d4110f2b9fc52e3994367f0b004c587c82c55c33f800502212a163"}
Sep 18 19:16:03.034067 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:03.034040    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l9g57" event={"ID":"b0bb7eb5-2fe7-42f3-92a6-00d65358f28f","Type":"ContainerStarted","Data":"ff7bc962c33a7af5f478ae5af1f4e3cef72636e9d45177570cff6155e15dbb52"}
Sep 18 19:16:03.034170 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:03.034089    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l9g57" event={"ID":"b0bb7eb5-2fe7-42f3-92a6-00d65358f28f","Type":"ContainerStarted","Data":"ebef8c425530ec9fa4dfd54a0a85e428cc60fae7e5ac70ba65542affdb47c41f"}
Sep 18 19:16:03.063767 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:03.063719    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-l9g57" podStartSLOduration=131.087380327 podStartE2EDuration="2m12.063704178s" totalImagesPullingTime="976.323851ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:13:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:16:03.06219326 +0000 UTC m=+133.914493623" watchObservedRunningTime="2026-09-18 19:16:03.063704178 +0000 UTC m=+133.916004548"
Sep 18 19:16:03.715131 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:03.715093    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 19:16:03.717951 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:03.717927    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:03.726812 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:03.726792    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 18 19:16:03.726812 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:03.726803    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 18 19:16:03.727527 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:03.727511    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 18 19:16:03.727623 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:03.727557    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 18 19:16:03.727769 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:03.727755    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 18 19:16:03.730009 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:03.729925    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-drzx7\""
Sep 18 19:16:03.730009 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:03.729978    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 18 19:16:03.730151 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:03.730045    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 18 19:16:03.730214 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:03.730183    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 18 19:16:03.730372 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:03.730356    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 18 19:16:03.745745 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:03.745710    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 19:16:03.820520 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:03.820485    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1c0d88df-3d51-4e0e-883a-294ca50af402-web-config\") pod \"alertmanager-main-0\" (UID: \"1c0d88df-3d51-4e0e-883a-294ca50af402\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:03.820692 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:03.820539    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1c0d88df-3d51-4e0e-883a-294ca50af402-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1c0d88df-3d51-4e0e-883a-294ca50af402\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:03.820692 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:03.820571    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1c0d88df-3d51-4e0e-883a-294ca50af402-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1c0d88df-3d51-4e0e-883a-294ca50af402\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:03.820692 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:03.820620    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1c0d88df-3d51-4e0e-883a-294ca50af402-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1c0d88df-3d51-4e0e-883a-294ca50af402\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:03.820692 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:03.820650    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1c0d88df-3d51-4e0e-883a-294ca50af402-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1c0d88df-3d51-4e0e-883a-294ca50af402\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:03.820692 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:03.820672    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1c0d88df-3d51-4e0e-883a-294ca50af402-config-out\") pod \"alertmanager-main-0\" (UID: \"1c0d88df-3d51-4e0e-883a-294ca50af402\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:03.820952 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:03.820693    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c0d88df-3d51-4e0e-883a-294ca50af402-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1c0d88df-3d51-4e0e-883a-294ca50af402\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:03.820952 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:03.820726    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1c0d88df-3d51-4e0e-883a-294ca50af402-config-volume\") pod \"alertmanager-main-0\" (UID: \"1c0d88df-3d51-4e0e-883a-294ca50af402\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:03.820952 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:03.820805    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c0d88df-3d51-4e0e-883a-294ca50af402-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1c0d88df-3d51-4e0e-883a-294ca50af402\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:03.820952 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:03.820889    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1c0d88df-3d51-4e0e-883a-294ca50af402-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1c0d88df-3d51-4e0e-883a-294ca50af402\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:03.821178 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:03.820967    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1c0d88df-3d51-4e0e-883a-294ca50af402-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1c0d88df-3d51-4e0e-883a-294ca50af402\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:03.821178 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:03.821030    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt4lr\" (UniqueName: \"kubernetes.io/projected/1c0d88df-3d51-4e0e-883a-294ca50af402-kube-api-access-rt4lr\") pod \"alertmanager-main-0\" (UID: \"1c0d88df-3d51-4e0e-883a-294ca50af402\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:03.821178 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:03.821059    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1c0d88df-3d51-4e0e-883a-294ca50af402-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1c0d88df-3d51-4e0e-883a-294ca50af402\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:03.921438 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:03.921404    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1c0d88df-3d51-4e0e-883a-294ca50af402-web-config\") pod \"alertmanager-main-0\" (UID: \"1c0d88df-3d51-4e0e-883a-294ca50af402\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:03.921438 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:03.921448    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1c0d88df-3d51-4e0e-883a-294ca50af402-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1c0d88df-3d51-4e0e-883a-294ca50af402\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:03.921660 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:03.921476    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1c0d88df-3d51-4e0e-883a-294ca50af402-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1c0d88df-3d51-4e0e-883a-294ca50af402\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:03.921660 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:03.921547    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1c0d88df-3d51-4e0e-883a-294ca50af402-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1c0d88df-3d51-4e0e-883a-294ca50af402\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:03.921660 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:03.921577    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1c0d88df-3d51-4e0e-883a-294ca50af402-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1c0d88df-3d51-4e0e-883a-294ca50af402\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:03.921660 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:03.921593    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1c0d88df-3d51-4e0e-883a-294ca50af402-config-out\") pod \"alertmanager-main-0\" (UID: \"1c0d88df-3d51-4e0e-883a-294ca50af402\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:03.921660 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:03.921614    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c0d88df-3d51-4e0e-883a-294ca50af402-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1c0d88df-3d51-4e0e-883a-294ca50af402\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:03.921971 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:03.921943    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1c0d88df-3d51-4e0e-883a-294ca50af402-config-volume\") pod \"alertmanager-main-0\" (UID: \"1c0d88df-3d51-4e0e-883a-294ca50af402\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:03.922071 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:03.922025    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c0d88df-3d51-4e0e-883a-294ca50af402-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1c0d88df-3d51-4e0e-883a-294ca50af402\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:03.922071 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:03.922059    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1c0d88df-3d51-4e0e-883a-294ca50af402-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1c0d88df-3d51-4e0e-883a-294ca50af402\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:03.922178 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:03.922088    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1c0d88df-3d51-4e0e-883a-294ca50af402-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1c0d88df-3d51-4e0e-883a-294ca50af402\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:03.922178 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:03.922118    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt4lr\" (UniqueName: \"kubernetes.io/projected/1c0d88df-3d51-4e0e-883a-294ca50af402-kube-api-access-rt4lr\") pod \"alertmanager-main-0\" (UID: \"1c0d88df-3d51-4e0e-883a-294ca50af402\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:03.922178 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:03.922145    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1c0d88df-3d51-4e0e-883a-294ca50af402-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1c0d88df-3d51-4e0e-883a-294ca50af402\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:03.923191 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:03.922414    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c0d88df-3d51-4e0e-883a-294ca50af402-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1c0d88df-3d51-4e0e-883a-294ca50af402\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:03.923191 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:03.922690    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1c0d88df-3d51-4e0e-883a-294ca50af402-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1c0d88df-3d51-4e0e-883a-294ca50af402\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:03.924631 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:03.924605    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1c0d88df-3d51-4e0e-883a-294ca50af402-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1c0d88df-3d51-4e0e-883a-294ca50af402\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:03.925239 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:03.924896    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1c0d88df-3d51-4e0e-883a-294ca50af402-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1c0d88df-3d51-4e0e-883a-294ca50af402\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:03.925239 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:03.924969    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1c0d88df-3d51-4e0e-883a-294ca50af402-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1c0d88df-3d51-4e0e-883a-294ca50af402\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:03.925445 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:03.925421    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c0d88df-3d51-4e0e-883a-294ca50af402-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1c0d88df-3d51-4e0e-883a-294ca50af402\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:03.925789 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:03.925748    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1c0d88df-3d51-4e0e-883a-294ca50af402-config-out\") pod \"alertmanager-main-0\" (UID: \"1c0d88df-3d51-4e0e-883a-294ca50af402\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:03.926059 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:03.926027    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1c0d88df-3d51-4e0e-883a-294ca50af402-web-config\") pod \"alertmanager-main-0\" (UID: \"1c0d88df-3d51-4e0e-883a-294ca50af402\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:03.926740 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:03.926447    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1c0d88df-3d51-4e0e-883a-294ca50af402-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1c0d88df-3d51-4e0e-883a-294ca50af402\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:03.927087 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:03.927068    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1c0d88df-3d51-4e0e-883a-294ca50af402-config-volume\") pod \"alertmanager-main-0\" (UID: \"1c0d88df-3d51-4e0e-883a-294ca50af402\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:03.927178 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:03.927151    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1c0d88df-3d51-4e0e-883a-294ca50af402-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1c0d88df-3d51-4e0e-883a-294ca50af402\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:03.927469 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:03.927450    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1c0d88df-3d51-4e0e-883a-294ca50af402-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1c0d88df-3d51-4e0e-883a-294ca50af402\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:03.937362 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:03.937342    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt4lr\" (UniqueName: \"kubernetes.io/projected/1c0d88df-3d51-4e0e-883a-294ca50af402-kube-api-access-rt4lr\") pod \"alertmanager-main-0\" (UID: \"1c0d88df-3d51-4e0e-883a-294ca50af402\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:04.028732 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:04.028705    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:16:04.039377 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:04.039348    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9xg4r" event={"ID":"58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd","Type":"ContainerStarted","Data":"bed7873c9ee0b5869425791c56e944ced13dc304b7095bc64f3cfee9c2ee03be"}
Sep 18 19:16:04.323227 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:04.323140    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 19:16:04.326759 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:16:04.326737    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c0d88df_3d51_4e0e_883a_294ca50af402.slice/crio-cd43e72d1a43bdbaf0c5cc0c6c36282d8bca9ded0e6dfaad95dfe600a6dad155 WatchSource:0}: Error finding container cd43e72d1a43bdbaf0c5cc0c6c36282d8bca9ded0e6dfaad95dfe600a6dad155: Status 404 returned error can't find the container with id cd43e72d1a43bdbaf0c5cc0c6c36282d8bca9ded0e6dfaad95dfe600a6dad155
Sep 18 19:16:05.044735 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:05.044659    2577 generic.go:356] "Generic (PLEG): container finished" podID="58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd" containerID="bed7873c9ee0b5869425791c56e944ced13dc304b7095bc64f3cfee9c2ee03be" exitCode=0
Sep 18 19:16:05.045162 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:05.044755    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9xg4r" event={"ID":"58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd","Type":"ContainerDied","Data":"bed7873c9ee0b5869425791c56e944ced13dc304b7095bc64f3cfee9c2ee03be"}
Sep 18 19:16:05.046009 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:05.045973    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c0d88df-3d51-4e0e-883a-294ca50af402","Type":"ContainerStarted","Data":"cd43e72d1a43bdbaf0c5cc0c6c36282d8bca9ded0e6dfaad95dfe600a6dad155"}
Sep 18 19:16:06.051711 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:06.051672    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9xg4r" event={"ID":"58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd","Type":"ContainerStarted","Data":"4cb6465d734dc61e29c5486cd1ea02d199f81dcc89f4156297b86c2b61ef05fe"}
Sep 18 19:16:06.052168 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:06.051726    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9xg4r" event={"ID":"58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd","Type":"ContainerStarted","Data":"478426bd5f3528827dcd94bbad97b28980dfa6f4ef67d2442c32d18f4bb3540a"}
Sep 18 19:16:06.053266 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:06.053237    2577 generic.go:356] "Generic (PLEG): container finished" podID="1c0d88df-3d51-4e0e-883a-294ca50af402" containerID="b33a28be48026c6aa525eb08702db590e11dddbb04b3dd81e24c77ce01ac66aa" exitCode=0
Sep 18 19:16:06.053388 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:06.053289    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c0d88df-3d51-4e0e-883a-294ca50af402","Type":"ContainerDied","Data":"b33a28be48026c6aa525eb08702db590e11dddbb04b3dd81e24c77ce01ac66aa"}
Sep 18 19:16:06.088686 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:06.088636    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9xg4r" podStartSLOduration=3.007577868 podStartE2EDuration="4.08862018s" totalImagesPullingTime="1.00690342s" totalInitContainerRuntime="74.138892ms" isStatefulPod=true podCreationTimestamp="2026-09-18 19:16:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:16:06.086561256 +0000 UTC m=+136.938861626" watchObservedRunningTime="2026-09-18 19:16:06.08862018 +0000 UTC m=+136.940920576"
Sep 18 19:16:07.048718 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:07.048680    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-9b97dfbcc-xz9kd"]
Sep 18 19:16:07.051326 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:07.051301    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-9b97dfbcc-xz9kd"
Sep 18 19:16:07.056151 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:16:07.056119    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"metrics-server-5t3an5lfbv4ob\" is forbidden: User \"system:node:ip-10-0-129-30.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-129-30.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"metrics-server-5t3an5lfbv4ob\"" type="*v1.Secret"
Sep 18 19:16:07.056557 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:07.056155    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 18 19:16:07.056557 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:07.056194    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 18 19:16:07.057593 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:07.057416    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-jpsxw\""
Sep 18 19:16:07.057593 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:07.057435    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 18 19:16:07.062001 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:07.061966    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 18 19:16:07.089692 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:07.089668    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-9b97dfbcc-xz9kd"]
Sep 18 19:16:07.150601 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:07.150567    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-569mn\" (UniqueName: \"kubernetes.io/projected/e612908b-7c17-483f-8c5d-9480888ceddf-kube-api-access-569mn\") pod \"metrics-server-9b97dfbcc-xz9kd\" (UID: \"e612908b-7c17-483f-8c5d-9480888ceddf\") " pod="openshift-monitoring/metrics-server-9b97dfbcc-xz9kd"
Sep 18 19:16:07.150747 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:07.150651    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e612908b-7c17-483f-8c5d-9480888ceddf-audit-log\") pod \"metrics-server-9b97dfbcc-xz9kd\" (UID: \"e612908b-7c17-483f-8c5d-9480888ceddf\") " pod="openshift-monitoring/metrics-server-9b97dfbcc-xz9kd"
Sep 18 19:16:07.150747 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:07.150689    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e612908b-7c17-483f-8c5d-9480888ceddf-secret-metrics-server-client-certs\") pod \"metrics-server-9b97dfbcc-xz9kd\" (UID: \"e612908b-7c17-483f-8c5d-9480888ceddf\") " pod="openshift-monitoring/metrics-server-9b97dfbcc-xz9kd"
Sep 18 19:16:07.150845 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:07.150755    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e612908b-7c17-483f-8c5d-9480888ceddf-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-9b97dfbcc-xz9kd\" (UID: \"e612908b-7c17-483f-8c5d-9480888ceddf\") " pod="openshift-monitoring/metrics-server-9b97dfbcc-xz9kd"
Sep 18 19:16:07.150845 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:07.150820    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e612908b-7c17-483f-8c5d-9480888ceddf-client-ca-bundle\") pod \"metrics-server-9b97dfbcc-xz9kd\" (UID: \"e612908b-7c17-483f-8c5d-9480888ceddf\") " pod="openshift-monitoring/metrics-server-9b97dfbcc-xz9kd"
Sep 18 19:16:07.150939 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:07.150916    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e612908b-7c17-483f-8c5d-9480888ceddf-metrics-server-audit-profiles\") pod \"metrics-server-9b97dfbcc-xz9kd\" (UID: \"e612908b-7c17-483f-8c5d-9480888ceddf\") " pod="openshift-monitoring/metrics-server-9b97dfbcc-xz9kd"
Sep 18 19:16:07.151010 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:07.150947    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e612908b-7c17-483f-8c5d-9480888ceddf-secret-metrics-server-tls\") pod \"metrics-server-9b97dfbcc-xz9kd\" (UID: \"e612908b-7c17-483f-8c5d-9480888ceddf\") " pod="openshift-monitoring/metrics-server-9b97dfbcc-xz9kd"
Sep 18 19:16:07.252171 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:07.252136    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e612908b-7c17-483f-8c5d-9480888ceddf-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-9b97dfbcc-xz9kd\" (UID: \"e612908b-7c17-483f-8c5d-9480888ceddf\") " pod="openshift-monitoring/metrics-server-9b97dfbcc-xz9kd"
Sep 18 19:16:07.252361 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:07.252189    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e612908b-7c17-483f-8c5d-9480888ceddf-client-ca-bundle\") pod \"metrics-server-9b97dfbcc-xz9kd\" (UID: \"e612908b-7c17-483f-8c5d-9480888ceddf\") " pod="openshift-monitoring/metrics-server-9b97dfbcc-xz9kd"
Sep 18 19:16:07.252361 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:07.252227    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e612908b-7c17-483f-8c5d-9480888ceddf-metrics-server-audit-profiles\") pod \"metrics-server-9b97dfbcc-xz9kd\" (UID: \"e612908b-7c17-483f-8c5d-9480888ceddf\") " pod="openshift-monitoring/metrics-server-9b97dfbcc-xz9kd"
Sep 18 19:16:07.252480 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:07.252390    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e612908b-7c17-483f-8c5d-9480888ceddf-secret-metrics-server-tls\") pod \"metrics-server-9b97dfbcc-xz9kd\" (UID: \"e612908b-7c17-483f-8c5d-9480888ceddf\") " pod="openshift-monitoring/metrics-server-9b97dfbcc-xz9kd"
Sep 18 19:16:07.252480 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:07.252466    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-569mn\" (UniqueName: \"kubernetes.io/projected/e612908b-7c17-483f-8c5d-9480888ceddf-kube-api-access-569mn\") pod \"metrics-server-9b97dfbcc-xz9kd\" (UID: \"e612908b-7c17-483f-8c5d-9480888ceddf\") " pod="openshift-monitoring/metrics-server-9b97dfbcc-xz9kd"
Sep 18 19:16:07.252585 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:07.252510    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e612908b-7c17-483f-8c5d-9480888ceddf-audit-log\") pod \"metrics-server-9b97dfbcc-xz9kd\" (UID: \"e612908b-7c17-483f-8c5d-9480888ceddf\") " pod="openshift-monitoring/metrics-server-9b97dfbcc-xz9kd"
Sep 18 19:16:07.252585 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:07.252537    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e612908b-7c17-483f-8c5d-9480888ceddf-secret-metrics-server-client-certs\") pod \"metrics-server-9b97dfbcc-xz9kd\" (UID: \"e612908b-7c17-483f-8c5d-9480888ceddf\") " pod="openshift-monitoring/metrics-server-9b97dfbcc-xz9kd"
Sep 18 19:16:07.252867 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:07.252825    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e612908b-7c17-483f-8c5d-9480888ceddf-audit-log\") pod \"metrics-server-9b97dfbcc-xz9kd\" (UID: \"e612908b-7c17-483f-8c5d-9480888ceddf\") " pod="openshift-monitoring/metrics-server-9b97dfbcc-xz9kd"
Sep 18 19:16:07.253013 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:07.252980    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e612908b-7c17-483f-8c5d-9480888ceddf-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-9b97dfbcc-xz9kd\" (UID: \"e612908b-7c17-483f-8c5d-9480888ceddf\") " pod="openshift-monitoring/metrics-server-9b97dfbcc-xz9kd"
Sep 18 19:16:07.253167 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:07.253146    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e612908b-7c17-483f-8c5d-9480888ceddf-metrics-server-audit-profiles\") pod \"metrics-server-9b97dfbcc-xz9kd\" (UID: \"e612908b-7c17-483f-8c5d-9480888ceddf\") " pod="openshift-monitoring/metrics-server-9b97dfbcc-xz9kd"
Sep 18 19:16:07.254783 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:07.254752    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e612908b-7c17-483f-8c5d-9480888ceddf-secret-metrics-server-client-certs\") pod \"metrics-server-9b97dfbcc-xz9kd\" (UID: \"e612908b-7c17-483f-8c5d-9480888ceddf\") " pod="openshift-monitoring/metrics-server-9b97dfbcc-xz9kd"
Sep 18 19:16:07.254900 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:07.254850    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e612908b-7c17-483f-8c5d-9480888ceddf-secret-metrics-server-tls\") pod \"metrics-server-9b97dfbcc-xz9kd\" (UID: \"e612908b-7c17-483f-8c5d-9480888ceddf\") " pod="openshift-monitoring/metrics-server-9b97dfbcc-xz9kd"
Sep 18 19:16:07.268171 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:07.268150    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-569mn\" (UniqueName: \"kubernetes.io/projected/e612908b-7c17-483f-8c5d-9480888ceddf-kube-api-access-569mn\") pod \"metrics-server-9b97dfbcc-xz9kd\" (UID: \"e612908b-7c17-483f-8c5d-9480888ceddf\") " pod="openshift-monitoring/metrics-server-9b97dfbcc-xz9kd"
Sep 18 19:16:08.064882 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:08.064844    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c0d88df-3d51-4e0e-883a-294ca50af402","Type":"ContainerStarted","Data":"4327994fdd2c8981fb587400225e2acac010d03f54ee4791b336305228cf1fd0"}
Sep 18 19:16:08.065252 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:08.064910    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c0d88df-3d51-4e0e-883a-294ca50af402","Type":"ContainerStarted","Data":"1afe39e5df2a249eb8e863429f84b2ac8e04234d477149e3dafe26c8b4009985"}
Sep 18 19:16:08.065252 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:08.064935    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c0d88df-3d51-4e0e-883a-294ca50af402","Type":"ContainerStarted","Data":"dc0391340ea0f0329ea62a20f899bbb284ee640667f4229d48b837481616deeb"}
Sep 18 19:16:08.065252 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:08.064960    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c0d88df-3d51-4e0e-883a-294ca50af402","Type":"ContainerStarted","Data":"19e6a608bf6ab74b0cfa191186b666e45288adc5c20c18b40539dccc98d0811b"}
Sep 18 19:16:08.065252 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:08.064984    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c0d88df-3d51-4e0e-883a-294ca50af402","Type":"ContainerStarted","Data":"8e836dd7d3715a756b4a0de45b63b8339f110a35e5116ed2f6db31ffaa0eeef1"}
Sep 18 19:16:08.142971 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:08.142943    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-5t3an5lfbv4ob\""
Sep 18 19:16:08.155026 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:08.155006    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e612908b-7c17-483f-8c5d-9480888ceddf-client-ca-bundle\") pod \"metrics-server-9b97dfbcc-xz9kd\" (UID: \"e612908b-7c17-483f-8c5d-9480888ceddf\") " pod="openshift-monitoring/metrics-server-9b97dfbcc-xz9kd"
Sep 18 19:16:08.263162 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:08.263066    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-9b97dfbcc-xz9kd"
Sep 18 19:16:08.540061 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:08.539972    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-9b97dfbcc-xz9kd"]
Sep 18 19:16:08.543503 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:16:08.543475    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode612908b_7c17_483f_8c5d_9480888ceddf.slice/crio-e5066c1cfe4c049a981a87703d76a54d95e5e6f37a16db62f4fa319beb05ba24 WatchSource:0}: Error finding container e5066c1cfe4c049a981a87703d76a54d95e5e6f37a16db62f4fa319beb05ba24: Status 404 returned error can't find the container with id e5066c1cfe4c049a981a87703d76a54d95e5e6f37a16db62f4fa319beb05ba24
Sep 18 19:16:09.072817 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.072778    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c0d88df-3d51-4e0e-883a-294ca50af402","Type":"ContainerStarted","Data":"d6d48b0b05201deae9fa6353f7b8cb1d5c462297ac399dc8c4fb4391b6b6c65a"}
Sep 18 19:16:09.074147 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.074119    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-9b97dfbcc-xz9kd" event={"ID":"e612908b-7c17-483f-8c5d-9480888ceddf","Type":"ContainerStarted","Data":"e5066c1cfe4c049a981a87703d76a54d95e5e6f37a16db62f4fa319beb05ba24"}
Sep 18 19:16:09.111391 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.111278    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.337806004 podStartE2EDuration="6.111253862s" totalImagesPullingTime="3.755211547s" totalInitContainerRuntime="18.236311ms" isStatefulPod=false podCreationTimestamp="2026-09-18 19:16:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:16:09.108508795 +0000 UTC m=+139.960809202" watchObservedRunningTime="2026-09-18 19:16:09.111253862 +0000 UTC m=+139.963554224"
Sep 18 19:16:09.414942 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.414850    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 19:16:09.418473 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.418443    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.422498 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.422469    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 18 19:16:09.423001 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.422816    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 18 19:16:09.423547 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.423528    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 18 19:16:09.424405 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.424388    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 18 19:16:09.424761 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.424737    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-24qpqpko3d561\""
Sep 18 19:16:09.424977 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.424832    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 18 19:16:09.425202 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.425146    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 18 19:16:09.425202 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.425183    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 18 19:16:09.425418 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.425404    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 18 19:16:09.426425 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.426401    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 18 19:16:09.427496 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.427467    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 18 19:16:09.427606 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.427549    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-sncd5\""
Sep 18 19:16:09.431371 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.431318    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 18 19:16:09.432283 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.432262    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 18 19:16:09.433741 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.433716    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 18 19:16:09.441589 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.441564    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 18 19:16:09.473689 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.473650    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/83f44b93-4402-4869-91ec-6fd23c938443-config\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.473689 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.473688    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/83f44b93-4402-4869-91ec-6fd23c938443-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.473858 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.473722    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/83f44b93-4402-4869-91ec-6fd23c938443-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.473858 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.473771    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/83f44b93-4402-4869-91ec-6fd23c938443-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.473858 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.473801    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83f44b93-4402-4869-91ec-6fd23c938443-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.473968 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.473874    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/83f44b93-4402-4869-91ec-6fd23c938443-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.473968 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.473916    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2trp\" (UniqueName: \"kubernetes.io/projected/83f44b93-4402-4869-91ec-6fd23c938443-kube-api-access-b2trp\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.473968 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.473947    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/83f44b93-4402-4869-91ec-6fd23c938443-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.474118 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.474009    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/83f44b93-4402-4869-91ec-6fd23c938443-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.474118 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.474031    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83f44b93-4402-4869-91ec-6fd23c938443-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.474118 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.474067    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83f44b93-4402-4869-91ec-6fd23c938443-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.474267 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.474137    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/83f44b93-4402-4869-91ec-6fd23c938443-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.474267 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.474180    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/83f44b93-4402-4869-91ec-6fd23c938443-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.474267 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.474210    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/83f44b93-4402-4869-91ec-6fd23c938443-config-out\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.474267 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.474235    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/83f44b93-4402-4869-91ec-6fd23c938443-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.474267 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.474255    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83f44b93-4402-4869-91ec-6fd23c938443-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.474496 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.474277    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/83f44b93-4402-4869-91ec-6fd23c938443-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.474496 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.474295    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/83f44b93-4402-4869-91ec-6fd23c938443-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.474496 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.474365    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/83f44b93-4402-4869-91ec-6fd23c938443-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.474496 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.474421    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/83f44b93-4402-4869-91ec-6fd23c938443-web-config\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.476862 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.476836    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 19:16:09.574799 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.574765    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83f44b93-4402-4869-91ec-6fd23c938443-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.574799 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.574800    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/83f44b93-4402-4869-91ec-6fd23c938443-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.575032 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.574817    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2trp\" (UniqueName: \"kubernetes.io/projected/83f44b93-4402-4869-91ec-6fd23c938443-kube-api-access-b2trp\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.575032 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.574840    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/83f44b93-4402-4869-91ec-6fd23c938443-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.575032 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.574869    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/83f44b93-4402-4869-91ec-6fd23c938443-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.575032 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.574899    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83f44b93-4402-4869-91ec-6fd23c938443-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.575032 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.574943    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83f44b93-4402-4869-91ec-6fd23c938443-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.575032 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.575019    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/83f44b93-4402-4869-91ec-6fd23c938443-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.575326 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.575047    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/83f44b93-4402-4869-91ec-6fd23c938443-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.575326 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.575078    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/83f44b93-4402-4869-91ec-6fd23c938443-config-out\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.575326 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.575104    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/83f44b93-4402-4869-91ec-6fd23c938443-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.575326 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.575127    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83f44b93-4402-4869-91ec-6fd23c938443-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.575326 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.575154    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/83f44b93-4402-4869-91ec-6fd23c938443-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.575326 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.575177    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/83f44b93-4402-4869-91ec-6fd23c938443-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.575326 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.575216    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/83f44b93-4402-4869-91ec-6fd23c938443-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.575326 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.575225    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/83f44b93-4402-4869-91ec-6fd23c938443-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.575326 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.575243    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/83f44b93-4402-4869-91ec-6fd23c938443-web-config\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.575326 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.575286    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/83f44b93-4402-4869-91ec-6fd23c938443-config\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.575326 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.575310    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/83f44b93-4402-4869-91ec-6fd23c938443-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.575834 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.575348    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/83f44b93-4402-4869-91ec-6fd23c938443-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.575834 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.575378    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/83f44b93-4402-4869-91ec-6fd23c938443-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.575834 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.575706    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83f44b93-4402-4869-91ec-6fd23c938443-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.576005 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.575877    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83f44b93-4402-4869-91ec-6fd23c938443-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.576005 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.575947    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83f44b93-4402-4869-91ec-6fd23c938443-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.576583 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.576356    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83f44b93-4402-4869-91ec-6fd23c938443-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.578273 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.578243    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/83f44b93-4402-4869-91ec-6fd23c938443-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.578514 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.578490    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/83f44b93-4402-4869-91ec-6fd23c938443-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.578935 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.578622    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/83f44b93-4402-4869-91ec-6fd23c938443-web-config\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.578935 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.578683    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/83f44b93-4402-4869-91ec-6fd23c938443-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.578935 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.578802    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/83f44b93-4402-4869-91ec-6fd23c938443-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.579215 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.579185    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/83f44b93-4402-4869-91ec-6fd23c938443-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.579377 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.579321    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/83f44b93-4402-4869-91ec-6fd23c938443-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.579377 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.579319    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/83f44b93-4402-4869-91ec-6fd23c938443-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.579377 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.579363    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/83f44b93-4402-4869-91ec-6fd23c938443-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.580043 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.580023    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/83f44b93-4402-4869-91ec-6fd23c938443-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.580611 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.580595    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/83f44b93-4402-4869-91ec-6fd23c938443-config\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.581064 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.581043    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/83f44b93-4402-4869-91ec-6fd23c938443-config-out\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.581316 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.581297    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/83f44b93-4402-4869-91ec-6fd23c938443-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.581590 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.581573    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/83f44b93-4402-4869-91ec-6fd23c938443-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.602495 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.602467    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2trp\" (UniqueName: \"kubernetes.io/projected/83f44b93-4402-4869-91ec-6fd23c938443-kube-api-access-b2trp\") pod \"prometheus-k8s-0\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:09.730974 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:09.730341    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:10.320544 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:10.320519    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 19:16:10.322039 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:16:10.322008    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83f44b93_4402_4869_91ec_6fd23c938443.slice/crio-cabc9e21b6a0d68f5743f0527cf4ef8c97aa678f6212b4e7724edb2ab2cd91f7 WatchSource:0}: Error finding container cabc9e21b6a0d68f5743f0527cf4ef8c97aa678f6212b4e7724edb2ab2cd91f7: Status 404 returned error can't find the container with id cabc9e21b6a0d68f5743f0527cf4ef8c97aa678f6212b4e7724edb2ab2cd91f7
Sep 18 19:16:11.083973 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:11.083938    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-9b97dfbcc-xz9kd" event={"ID":"e612908b-7c17-483f-8c5d-9480888ceddf","Type":"ContainerStarted","Data":"59ca9d698b856647cfcb2bef89fc4e3978adb572eccc54225d66483e0ad5c706"}
Sep 18 19:16:11.085491 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:11.085462    2577 generic.go:356] "Generic (PLEG): container finished" podID="83f44b93-4402-4869-91ec-6fd23c938443" containerID="1c935190ec508e926e04c5b084b4581799040e7bbe94ce5e5b72d36076e64daa" exitCode=0
Sep 18 19:16:11.085595 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:11.085520    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83f44b93-4402-4869-91ec-6fd23c938443","Type":"ContainerDied","Data":"1c935190ec508e926e04c5b084b4581799040e7bbe94ce5e5b72d36076e64daa"}
Sep 18 19:16:11.085595 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:11.085579    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83f44b93-4402-4869-91ec-6fd23c938443","Type":"ContainerStarted","Data":"cabc9e21b6a0d68f5743f0527cf4ef8c97aa678f6212b4e7724edb2ab2cd91f7"}
Sep 18 19:16:11.116747 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:11.116700    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-9b97dfbcc-xz9kd" podStartSLOduration=2.6175617879999997 podStartE2EDuration="4.116683396s" totalImagesPullingTime="1.499121608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:16:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:16:11.116099166 +0000 UTC m=+141.968399536" watchObservedRunningTime="2026-09-18 19:16:11.116683396 +0000 UTC m=+141.968983766"
Sep 18 19:16:14.946109 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:14.946074    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-69df585c75-9p6zz"]
Sep 18 19:16:14.946481 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:16:14.946334    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-69df585c75-9p6zz" podUID="7feb1f7b-32c7-4e47-a257-31e42c72cd0f"
Sep 18 19:16:15.103236 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:15.103211    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69df585c75-9p6zz"
Sep 18 19:16:15.103401 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:15.103203    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83f44b93-4402-4869-91ec-6fd23c938443","Type":"ContainerStarted","Data":"7b4ce9377d3cd59952c0dec7066b62da90ceaa80018a94755c3f411ae9810f4e"}
Sep 18 19:16:15.103401 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:15.103386    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83f44b93-4402-4869-91ec-6fd23c938443","Type":"ContainerStarted","Data":"623c43a362fdd42fbd934d3e390aff7323d886b56800a72f871d9496a91c93a9"}
Sep 18 19:16:15.109944 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:15.109923    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69df585c75-9p6zz"
Sep 18 19:16:15.234013 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:15.233916    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-registry-certificates\") pod \"7feb1f7b-32c7-4e47-a257-31e42c72cd0f\" (UID: \"7feb1f7b-32c7-4e47-a257-31e42c72cd0f\") "
Sep 18 19:16:15.234013 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:15.233973    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-image-registry-private-configuration\") pod \"7feb1f7b-32c7-4e47-a257-31e42c72cd0f\" (UID: \"7feb1f7b-32c7-4e47-a257-31e42c72cd0f\") "
Sep 18 19:16:15.234251 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:15.234145    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-installation-pull-secrets\") pod \"7feb1f7b-32c7-4e47-a257-31e42c72cd0f\" (UID: \"7feb1f7b-32c7-4e47-a257-31e42c72cd0f\") "
Sep 18 19:16:15.234251 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:15.234203    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-ca-trust-extracted\") pod \"7feb1f7b-32c7-4e47-a257-31e42c72cd0f\" (UID: \"7feb1f7b-32c7-4e47-a257-31e42c72cd0f\") "
Sep 18 19:16:15.234251 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:15.234237    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-trusted-ca\") pod \"7feb1f7b-32c7-4e47-a257-31e42c72cd0f\" (UID: \"7feb1f7b-32c7-4e47-a257-31e42c72cd0f\") "
Sep 18 19:16:15.234410 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:15.234263    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qztnv\" (UniqueName: \"kubernetes.io/projected/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-kube-api-access-qztnv\") pod \"7feb1f7b-32c7-4e47-a257-31e42c72cd0f\" (UID: \"7feb1f7b-32c7-4e47-a257-31e42c72cd0f\") "
Sep 18 19:16:15.234410 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:15.234300    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-bound-sa-token\") pod \"7feb1f7b-32c7-4e47-a257-31e42c72cd0f\" (UID: \"7feb1f7b-32c7-4e47-a257-31e42c72cd0f\") "
Sep 18 19:16:15.234410 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:15.234339    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "7feb1f7b-32c7-4e47-a257-31e42c72cd0f" (UID: "7feb1f7b-32c7-4e47-a257-31e42c72cd0f"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:16:15.234544 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:15.234458    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "7feb1f7b-32c7-4e47-a257-31e42c72cd0f" (UID: "7feb1f7b-32c7-4e47-a257-31e42c72cd0f"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:16:15.234669 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:15.234606    2577 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-registry-certificates\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:16:15.234669 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:15.234630    2577 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-ca-trust-extracted\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:16:15.234845 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:15.234824    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "7feb1f7b-32c7-4e47-a257-31e42c72cd0f" (UID: "7feb1f7b-32c7-4e47-a257-31e42c72cd0f"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:16:15.236967 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:15.236938    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "7feb1f7b-32c7-4e47-a257-31e42c72cd0f" (UID: "7feb1f7b-32c7-4e47-a257-31e42c72cd0f"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:16:15.237106 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:15.236977    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "7feb1f7b-32c7-4e47-a257-31e42c72cd0f" (UID: "7feb1f7b-32c7-4e47-a257-31e42c72cd0f"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:16:15.237178 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:15.237135    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-kube-api-access-qztnv" (OuterVolumeSpecName: "kube-api-access-qztnv") pod "7feb1f7b-32c7-4e47-a257-31e42c72cd0f" (UID: "7feb1f7b-32c7-4e47-a257-31e42c72cd0f"). InnerVolumeSpecName "kube-api-access-qztnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:16:15.237235 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:15.237164    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "7feb1f7b-32c7-4e47-a257-31e42c72cd0f" (UID: "7feb1f7b-32c7-4e47-a257-31e42c72cd0f"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:16:15.335907 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:15.335873    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-trusted-ca\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:16:15.335907 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:15.335901    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qztnv\" (UniqueName: \"kubernetes.io/projected/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-kube-api-access-qztnv\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:16:15.335907 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:15.335910    2577 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-bound-sa-token\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:16:15.336151 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:15.335923    2577 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-image-registry-private-configuration\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:16:15.336151 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:15.335937    2577 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-installation-pull-secrets\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:16:16.106891 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:16.106857    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69df585c75-9p6zz"
Sep 18 19:16:16.169252 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:16.169208    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-69df585c75-9p6zz"]
Sep 18 19:16:16.187675 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:16.187636    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-69df585c75-9p6zz"]
Sep 18 19:16:16.247010 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:16.246967    2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7feb1f7b-32c7-4e47-a257-31e42c72cd0f-registry-tls\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:16:17.114629 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:17.114595    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83f44b93-4402-4869-91ec-6fd23c938443","Type":"ContainerStarted","Data":"5514363ae6531cf5e1818a035dcd73688ca2f21dbd973fed1f2355c9906efa75"}
Sep 18 19:16:17.115025 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:17.114660    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83f44b93-4402-4869-91ec-6fd23c938443","Type":"ContainerStarted","Data":"db7145413295c0ef5d2d57f4e8fb8aeb0e9b3aef0dad53f21664a7e6abb4c5c2"}
Sep 18 19:16:17.115025 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:17.114686    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83f44b93-4402-4869-91ec-6fd23c938443","Type":"ContainerStarted","Data":"7c509504133689d7581b3cfacdb76fe7f180d8f6424bdf3f525e32d1eca275a3"}
Sep 18 19:16:17.115025 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:17.114711    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83f44b93-4402-4869-91ec-6fd23c938443","Type":"ContainerStarted","Data":"3233e21243a7da879fd5deee2efa5c041a24efa428c4cc29ae956b85e3f64ed8"}
Sep 18 19:16:17.438045 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:17.437956    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7feb1f7b-32c7-4e47-a257-31e42c72cd0f" path="/var/lib/kubelet/pods/7feb1f7b-32c7-4e47-a257-31e42c72cd0f/volumes"
Sep 18 19:16:17.558110 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:17.558047    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.503307265 podStartE2EDuration="8.558024427s" totalImagesPullingTime="5.033119139s" totalInitContainerRuntime="21.598023ms" isStatefulPod=false podCreationTimestamp="2026-09-18 19:16:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:16:17.555345428 +0000 UTC m=+148.407645800" watchObservedRunningTime="2026-09-18 19:16:17.558024427 +0000 UTC m=+148.410324792"
Sep 18 19:16:19.731137 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:19.731091    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:16:28.264160 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:28.264126    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-9b97dfbcc-xz9kd"
Sep 18 19:16:28.264652 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:28.264222    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-9b97dfbcc-xz9kd"
Sep 18 19:16:28.467735 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:16:28.467693    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-dg5zx" podUID="dbd605d1-eb07-4c9c-aea1-f1a610394801"
Sep 18 19:16:29.162670 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:29.162639    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dg5zx"
Sep 18 19:16:33.394504 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:33.394416    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dbd605d1-eb07-4c9c-aea1-f1a610394801-metrics-tls\") pod \"dns-default-dg5zx\" (UID: \"dbd605d1-eb07-4c9c-aea1-f1a610394801\") " pod="openshift-dns/dns-default-dg5zx"
Sep 18 19:16:33.396747 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:33.396727    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dbd605d1-eb07-4c9c-aea1-f1a610394801-metrics-tls\") pod \"dns-default-dg5zx\" (UID: \"dbd605d1-eb07-4c9c-aea1-f1a610394801\") " pod="openshift-dns/dns-default-dg5zx"
Sep 18 19:16:33.666717 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:33.666643    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zqv6h\""
Sep 18 19:16:33.674052 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:33.674036    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dg5zx"
Sep 18 19:16:33.986137 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:33.986060    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dg5zx"]
Sep 18 19:16:33.989055 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:16:33.989026    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbd605d1_eb07_4c9c_aea1_f1a610394801.slice/crio-ea1c6c4e16114cd35ec85af6c8ca2acf068ed3acb6297c97ea87e632172933bd WatchSource:0}: Error finding container ea1c6c4e16114cd35ec85af6c8ca2acf068ed3acb6297c97ea87e632172933bd: Status 404 returned error can't find the container with id ea1c6c4e16114cd35ec85af6c8ca2acf068ed3acb6297c97ea87e632172933bd
Sep 18 19:16:34.182967 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:34.182929    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dg5zx" event={"ID":"dbd605d1-eb07-4c9c-aea1-f1a610394801","Type":"ContainerStarted","Data":"ea1c6c4e16114cd35ec85af6c8ca2acf068ed3acb6297c97ea87e632172933bd"}
Sep 18 19:16:36.193202 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:36.193167    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dg5zx" event={"ID":"dbd605d1-eb07-4c9c-aea1-f1a610394801","Type":"ContainerStarted","Data":"155277d467324a302515793186a52a2991c263b3737164fffb7265bed9961990"}
Sep 18 19:16:36.193588 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:36.193228    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dg5zx" event={"ID":"dbd605d1-eb07-4c9c-aea1-f1a610394801","Type":"ContainerStarted","Data":"b2954696c788d1e905a27304bc128a42e39b87ff14ffbd109a3c533c748f8f99"}
Sep 18 19:16:36.193588 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:36.193252    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-dg5zx"
Sep 18 19:16:36.215145 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:36.215095    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-dg5zx" podStartSLOduration=129.649400254 podStartE2EDuration="2m11.215080561s" totalImagesPullingTime="1.565680307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:14:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:16:36.214754582 +0000 UTC m=+167.067054951" watchObservedRunningTime="2026-09-18 19:16:36.215080561 +0000 UTC m=+167.067380933"
Sep 18 19:16:46.199538 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:46.199505    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dg5zx"
Sep 18 19:16:46.808928 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:46.808900    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1c0d88df-3d51-4e0e-883a-294ca50af402/init-config-reloader/0.log"
Sep 18 19:16:46.857506 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:46.857471    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1c0d88df-3d51-4e0e-883a-294ca50af402/alertmanager/0.log"
Sep 18 19:16:46.883590 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:46.883569    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1c0d88df-3d51-4e0e-883a-294ca50af402/config-reloader/0.log"
Sep 18 19:16:46.899696 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:46.899648    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1c0d88df-3d51-4e0e-883a-294ca50af402/kube-rbac-proxy-web/0.log"
Sep 18 19:16:47.049515 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:47.049491    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1c0d88df-3d51-4e0e-883a-294ca50af402/kube-rbac-proxy/0.log"
Sep 18 19:16:47.230576 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:47.230545    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1c0d88df-3d51-4e0e-883a-294ca50af402/kube-rbac-proxy-metric/0.log"
Sep 18 19:16:47.430803 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:47.430775    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1c0d88df-3d51-4e0e-883a-294ca50af402/prom-label-proxy/0.log"
Sep 18 19:16:48.272713 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:48.272679    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-9b97dfbcc-xz9kd"
Sep 18 19:16:48.276871 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:48.276842    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-9b97dfbcc-xz9kd"
Sep 18 19:16:48.432915 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:48.432886    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-9b97dfbcc-xz9kd_e612908b-7c17-483f-8c5d-9480888ceddf/metrics-server/0.log"
Sep 18 19:16:49.417670 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:49.417648    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9xg4r_58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd/init-textfile/0.log"
Sep 18 19:16:49.633691 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:49.633664    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9xg4r_58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd/node-exporter/0.log"
Sep 18 19:16:49.824170 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:49.824087    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9xg4r_58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd/kube-rbac-proxy/0.log"
Sep 18 19:16:51.229956 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:51.229925    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_83f44b93-4402-4869-91ec-6fd23c938443/init-config-reloader/0.log"
Sep 18 19:16:51.440467 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:51.440442    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_83f44b93-4402-4869-91ec-6fd23c938443/prometheus/0.log"
Sep 18 19:16:51.634471 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:51.634446    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_83f44b93-4402-4869-91ec-6fd23c938443/config-reloader/0.log"
Sep 18 19:16:51.815560 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:51.815529    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_83f44b93-4402-4869-91ec-6fd23c938443/thanos-sidecar/0.log"
Sep 18 19:16:52.015830 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:52.015764    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_83f44b93-4402-4869-91ec-6fd23c938443/kube-rbac-proxy-web/0.log"
Sep 18 19:16:52.214865 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:52.214836    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_83f44b93-4402-4869-91ec-6fd23c938443/kube-rbac-proxy/0.log"
Sep 18 19:16:52.414617 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:52.414589    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_83f44b93-4402-4869-91ec-6fd23c938443/kube-rbac-proxy-thanos/0.log"
Sep 18 19:16:52.615001 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:52.614967    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-w7p4q_83e874e2-103a-4097-b485-253beb0f1590/prometheus-operator/0.log"
Sep 18 19:16:52.815877 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:52.815808    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-w7p4q_83e874e2-103a-4097-b485-253beb0f1590/kube-rbac-proxy/0.log"
Sep 18 19:16:54.416817 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:16:54.416783    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6446c8d75d-92v7w_bf41ad7a-3615-4981-8a3e-125f973295aa/networking-console-plugin/0.log"
Sep 18 19:17:09.730788 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:09.730756    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:09.749284 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:09.749254    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:10.338078 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:10.338054    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:23.535379 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:23.535338    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 19:17:23.535949 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:23.535637    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1c0d88df-3d51-4e0e-883a-294ca50af402" containerName="alertmanager" containerID="cri-o://8e836dd7d3715a756b4a0de45b63b8339f110a35e5116ed2f6db31ffaa0eeef1" gracePeriod=120
Sep 18 19:17:23.535949 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:23.535699    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1c0d88df-3d51-4e0e-883a-294ca50af402" containerName="kube-rbac-proxy-metric" containerID="cri-o://4327994fdd2c8981fb587400225e2acac010d03f54ee4791b336305228cf1fd0" gracePeriod=120
Sep 18 19:17:23.535949 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:23.535730    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1c0d88df-3d51-4e0e-883a-294ca50af402" containerName="kube-rbac-proxy-web" containerID="cri-o://dc0391340ea0f0329ea62a20f899bbb284ee640667f4229d48b837481616deeb" gracePeriod=120
Sep 18 19:17:23.535949 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:23.535768    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1c0d88df-3d51-4e0e-883a-294ca50af402" containerName="kube-rbac-proxy" containerID="cri-o://1afe39e5df2a249eb8e863429f84b2ac8e04234d477149e3dafe26c8b4009985" gracePeriod=120
Sep 18 19:17:23.535949 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:23.535777    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1c0d88df-3d51-4e0e-883a-294ca50af402" containerName="prom-label-proxy" containerID="cri-o://d6d48b0b05201deae9fa6353f7b8cb1d5c462297ac399dc8c4fb4391b6b6c65a" gracePeriod=120
Sep 18 19:17:23.535949 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:23.535792    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1c0d88df-3d51-4e0e-883a-294ca50af402" containerName="config-reloader" containerID="cri-o://19e6a608bf6ab74b0cfa191186b666e45288adc5c20c18b40539dccc98d0811b" gracePeriod=120
Sep 18 19:17:24.380108 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:24.380078    2577 generic.go:356] "Generic (PLEG): container finished" podID="1c0d88df-3d51-4e0e-883a-294ca50af402" containerID="d6d48b0b05201deae9fa6353f7b8cb1d5c462297ac399dc8c4fb4391b6b6c65a" exitCode=0
Sep 18 19:17:24.380108 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:24.380101    2577 generic.go:356] "Generic (PLEG): container finished" podID="1c0d88df-3d51-4e0e-883a-294ca50af402" containerID="1afe39e5df2a249eb8e863429f84b2ac8e04234d477149e3dafe26c8b4009985" exitCode=0
Sep 18 19:17:24.380108 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:24.380110    2577 generic.go:356] "Generic (PLEG): container finished" podID="1c0d88df-3d51-4e0e-883a-294ca50af402" containerID="19e6a608bf6ab74b0cfa191186b666e45288adc5c20c18b40539dccc98d0811b" exitCode=0
Sep 18 19:17:24.380108 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:24.380117    2577 generic.go:356] "Generic (PLEG): container finished" podID="1c0d88df-3d51-4e0e-883a-294ca50af402" containerID="8e836dd7d3715a756b4a0de45b63b8339f110a35e5116ed2f6db31ffaa0eeef1" exitCode=0
Sep 18 19:17:24.380384 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:24.380156    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c0d88df-3d51-4e0e-883a-294ca50af402","Type":"ContainerDied","Data":"d6d48b0b05201deae9fa6353f7b8cb1d5c462297ac399dc8c4fb4391b6b6c65a"}
Sep 18 19:17:24.380384 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:24.380227    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c0d88df-3d51-4e0e-883a-294ca50af402","Type":"ContainerDied","Data":"1afe39e5df2a249eb8e863429f84b2ac8e04234d477149e3dafe26c8b4009985"}
Sep 18 19:17:24.380384 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:24.380254    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c0d88df-3d51-4e0e-883a-294ca50af402","Type":"ContainerDied","Data":"19e6a608bf6ab74b0cfa191186b666e45288adc5c20c18b40539dccc98d0811b"}
Sep 18 19:17:24.380384 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:24.380278    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c0d88df-3d51-4e0e-883a-294ca50af402","Type":"ContainerDied","Data":"8e836dd7d3715a756b4a0de45b63b8339f110a35e5116ed2f6db31ffaa0eeef1"}
Sep 18 19:17:24.767975 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:24.767953    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:24.935186 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:24.935105    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1c0d88df-3d51-4e0e-883a-294ca50af402-cluster-tls-config\") pod \"1c0d88df-3d51-4e0e-883a-294ca50af402\" (UID: \"1c0d88df-3d51-4e0e-883a-294ca50af402\") "
Sep 18 19:17:24.935186 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:24.935149    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1c0d88df-3d51-4e0e-883a-294ca50af402-config-out\") pod \"1c0d88df-3d51-4e0e-883a-294ca50af402\" (UID: \"1c0d88df-3d51-4e0e-883a-294ca50af402\") "
Sep 18 19:17:24.935186 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:24.935171    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rt4lr\" (UniqueName: \"kubernetes.io/projected/1c0d88df-3d51-4e0e-883a-294ca50af402-kube-api-access-rt4lr\") pod \"1c0d88df-3d51-4e0e-883a-294ca50af402\" (UID: \"1c0d88df-3d51-4e0e-883a-294ca50af402\") "
Sep 18 19:17:24.935432 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:24.935196    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1c0d88df-3d51-4e0e-883a-294ca50af402-alertmanager-main-db\") pod \"1c0d88df-3d51-4e0e-883a-294ca50af402\" (UID: \"1c0d88df-3d51-4e0e-883a-294ca50af402\") "
Sep 18 19:17:24.935432 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:24.935251    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1c0d88df-3d51-4e0e-883a-294ca50af402-web-config\") pod \"1c0d88df-3d51-4e0e-883a-294ca50af402\" (UID: \"1c0d88df-3d51-4e0e-883a-294ca50af402\") "
Sep 18 19:17:24.935432 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:24.935267    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1c0d88df-3d51-4e0e-883a-294ca50af402-secret-alertmanager-kube-rbac-proxy-metric\") pod \"1c0d88df-3d51-4e0e-883a-294ca50af402\" (UID: \"1c0d88df-3d51-4e0e-883a-294ca50af402\") "
Sep 18 19:17:24.935432 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:24.935308    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1c0d88df-3d51-4e0e-883a-294ca50af402-secret-alertmanager-kube-rbac-proxy\") pod \"1c0d88df-3d51-4e0e-883a-294ca50af402\" (UID: \"1c0d88df-3d51-4e0e-883a-294ca50af402\") "
Sep 18 19:17:24.935432 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:24.935346    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1c0d88df-3d51-4e0e-883a-294ca50af402-config-volume\") pod \"1c0d88df-3d51-4e0e-883a-294ca50af402\" (UID: \"1c0d88df-3d51-4e0e-883a-294ca50af402\") "
Sep 18 19:17:24.935432 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:24.935382    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1c0d88df-3d51-4e0e-883a-294ca50af402-tls-assets\") pod \"1c0d88df-3d51-4e0e-883a-294ca50af402\" (UID: \"1c0d88df-3d51-4e0e-883a-294ca50af402\") "
Sep 18 19:17:24.935432 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:24.935416    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c0d88df-3d51-4e0e-883a-294ca50af402-alertmanager-trusted-ca-bundle\") pod \"1c0d88df-3d51-4e0e-883a-294ca50af402\" (UID: \"1c0d88df-3d51-4e0e-883a-294ca50af402\") "
Sep 18 19:17:24.935810 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:24.935456    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1c0d88df-3d51-4e0e-883a-294ca50af402-secret-alertmanager-kube-rbac-proxy-web\") pod \"1c0d88df-3d51-4e0e-883a-294ca50af402\" (UID: \"1c0d88df-3d51-4e0e-883a-294ca50af402\") "
Sep 18 19:17:24.935810 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:24.935488    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1c0d88df-3d51-4e0e-883a-294ca50af402-secret-alertmanager-main-tls\") pod \"1c0d88df-3d51-4e0e-883a-294ca50af402\" (UID: \"1c0d88df-3d51-4e0e-883a-294ca50af402\") "
Sep 18 19:17:24.935810 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:24.935518    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c0d88df-3d51-4e0e-883a-294ca50af402-metrics-client-ca\") pod \"1c0d88df-3d51-4e0e-883a-294ca50af402\" (UID: \"1c0d88df-3d51-4e0e-883a-294ca50af402\") "
Sep 18 19:17:24.935810 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:24.935796    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c0d88df-3d51-4e0e-883a-294ca50af402-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "1c0d88df-3d51-4e0e-883a-294ca50af402" (UID: "1c0d88df-3d51-4e0e-883a-294ca50af402"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:17:24.936359 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:24.936100    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c0d88df-3d51-4e0e-883a-294ca50af402-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "1c0d88df-3d51-4e0e-883a-294ca50af402" (UID: "1c0d88df-3d51-4e0e-883a-294ca50af402"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:17:24.936956 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:24.936669    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c0d88df-3d51-4e0e-883a-294ca50af402-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "1c0d88df-3d51-4e0e-883a-294ca50af402" (UID: "1c0d88df-3d51-4e0e-883a-294ca50af402"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:17:24.938159 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:24.938135    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c0d88df-3d51-4e0e-883a-294ca50af402-kube-api-access-rt4lr" (OuterVolumeSpecName: "kube-api-access-rt4lr") pod "1c0d88df-3d51-4e0e-883a-294ca50af402" (UID: "1c0d88df-3d51-4e0e-883a-294ca50af402"). InnerVolumeSpecName "kube-api-access-rt4lr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:17:24.938653 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:24.938377    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c0d88df-3d51-4e0e-883a-294ca50af402-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "1c0d88df-3d51-4e0e-883a-294ca50af402" (UID: "1c0d88df-3d51-4e0e-883a-294ca50af402"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:17:24.938653 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:24.938616    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c0d88df-3d51-4e0e-883a-294ca50af402-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "1c0d88df-3d51-4e0e-883a-294ca50af402" (UID: "1c0d88df-3d51-4e0e-883a-294ca50af402"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:17:24.938789 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:24.938755    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c0d88df-3d51-4e0e-883a-294ca50af402-config-out" (OuterVolumeSpecName: "config-out") pod "1c0d88df-3d51-4e0e-883a-294ca50af402" (UID: "1c0d88df-3d51-4e0e-883a-294ca50af402"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:17:24.939037 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:24.939015    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c0d88df-3d51-4e0e-883a-294ca50af402-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "1c0d88df-3d51-4e0e-883a-294ca50af402" (UID: "1c0d88df-3d51-4e0e-883a-294ca50af402"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:17:24.939214 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:24.939195    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c0d88df-3d51-4e0e-883a-294ca50af402-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "1c0d88df-3d51-4e0e-883a-294ca50af402" (UID: "1c0d88df-3d51-4e0e-883a-294ca50af402"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:17:24.939907 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:24.939888    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c0d88df-3d51-4e0e-883a-294ca50af402-config-volume" (OuterVolumeSpecName: "config-volume") pod "1c0d88df-3d51-4e0e-883a-294ca50af402" (UID: "1c0d88df-3d51-4e0e-883a-294ca50af402"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:17:24.940634 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:24.940614    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c0d88df-3d51-4e0e-883a-294ca50af402-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "1c0d88df-3d51-4e0e-883a-294ca50af402" (UID: "1c0d88df-3d51-4e0e-883a-294ca50af402"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:17:24.942704 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:24.942678    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c0d88df-3d51-4e0e-883a-294ca50af402-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "1c0d88df-3d51-4e0e-883a-294ca50af402" (UID: "1c0d88df-3d51-4e0e-883a-294ca50af402"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:17:24.948949 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:24.948931    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c0d88df-3d51-4e0e-883a-294ca50af402-web-config" (OuterVolumeSpecName: "web-config") pod "1c0d88df-3d51-4e0e-883a-294ca50af402" (UID: "1c0d88df-3d51-4e0e-883a-294ca50af402"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:17:25.037080 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.037045    2577 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1c0d88df-3d51-4e0e-883a-294ca50af402-tls-assets\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:17:25.037080 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.037076    2577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c0d88df-3d51-4e0e-883a-294ca50af402-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:17:25.037288 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.037091    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1c0d88df-3d51-4e0e-883a-294ca50af402-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:17:25.037288 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.037105    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1c0d88df-3d51-4e0e-883a-294ca50af402-secret-alertmanager-main-tls\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:17:25.037288 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.037117    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c0d88df-3d51-4e0e-883a-294ca50af402-metrics-client-ca\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:17:25.037288 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.037128    2577 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1c0d88df-3d51-4e0e-883a-294ca50af402-cluster-tls-config\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:17:25.037288 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.037138    2577 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1c0d88df-3d51-4e0e-883a-294ca50af402-config-out\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:17:25.037288 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.037150    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt4lr\" (UniqueName: \"kubernetes.io/projected/1c0d88df-3d51-4e0e-883a-294ca50af402-kube-api-access-rt4lr\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:17:25.037288 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.037160    2577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1c0d88df-3d51-4e0e-883a-294ca50af402-alertmanager-main-db\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:17:25.037288 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.037174    2577 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1c0d88df-3d51-4e0e-883a-294ca50af402-web-config\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:17:25.037288 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.037187    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1c0d88df-3d51-4e0e-883a-294ca50af402-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:17:25.037288 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.037200    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1c0d88df-3d51-4e0e-883a-294ca50af402-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:17:25.037288 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.037212    2577 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1c0d88df-3d51-4e0e-883a-294ca50af402-config-volume\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:17:25.387533 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.387502    2577 generic.go:356] "Generic (PLEG): container finished" podID="1c0d88df-3d51-4e0e-883a-294ca50af402" containerID="4327994fdd2c8981fb587400225e2acac010d03f54ee4791b336305228cf1fd0" exitCode=0
Sep 18 19:17:25.387533 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.387529    2577 generic.go:356] "Generic (PLEG): container finished" podID="1c0d88df-3d51-4e0e-883a-294ca50af402" containerID="dc0391340ea0f0329ea62a20f899bbb284ee640667f4229d48b837481616deeb" exitCode=0
Sep 18 19:17:25.387726 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.387574    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:25.387726 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.387577    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c0d88df-3d51-4e0e-883a-294ca50af402","Type":"ContainerDied","Data":"4327994fdd2c8981fb587400225e2acac010d03f54ee4791b336305228cf1fd0"}
Sep 18 19:17:25.387726 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.387640    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c0d88df-3d51-4e0e-883a-294ca50af402","Type":"ContainerDied","Data":"dc0391340ea0f0329ea62a20f899bbb284ee640667f4229d48b837481616deeb"}
Sep 18 19:17:25.387726 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.387668    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c0d88df-3d51-4e0e-883a-294ca50af402","Type":"ContainerDied","Data":"cd43e72d1a43bdbaf0c5cc0c6c36282d8bca9ded0e6dfaad95dfe600a6dad155"}
Sep 18 19:17:25.387726 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.387697    2577 scope.go:117] "RemoveContainer" containerID="d6d48b0b05201deae9fa6353f7b8cb1d5c462297ac399dc8c4fb4391b6b6c65a"
Sep 18 19:17:25.395414 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.395274    2577 scope.go:117] "RemoveContainer" containerID="4327994fdd2c8981fb587400225e2acac010d03f54ee4791b336305228cf1fd0"
Sep 18 19:17:25.405915 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.405896    2577 scope.go:117] "RemoveContainer" containerID="1afe39e5df2a249eb8e863429f84b2ac8e04234d477149e3dafe26c8b4009985"
Sep 18 19:17:25.413206 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.413181    2577 scope.go:117] "RemoveContainer" containerID="dc0391340ea0f0329ea62a20f899bbb284ee640667f4229d48b837481616deeb"
Sep 18 19:17:25.417280 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.417256    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 19:17:25.440617 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.440587    2577 scope.go:117] "RemoveContainer" containerID="19e6a608bf6ab74b0cfa191186b666e45288adc5c20c18b40539dccc98d0811b"
Sep 18 19:17:25.445583 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.445562    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 19:17:25.450101 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.450082    2577 scope.go:117] "RemoveContainer" containerID="8e836dd7d3715a756b4a0de45b63b8339f110a35e5116ed2f6db31ffaa0eeef1"
Sep 18 19:17:25.457094 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.457069    2577 scope.go:117] "RemoveContainer" containerID="b33a28be48026c6aa525eb08702db590e11dddbb04b3dd81e24c77ce01ac66aa"
Sep 18 19:17:25.463817 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.463802    2577 scope.go:117] "RemoveContainer" containerID="d6d48b0b05201deae9fa6353f7b8cb1d5c462297ac399dc8c4fb4391b6b6c65a"
Sep 18 19:17:25.464098 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:17:25.464080    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6d48b0b05201deae9fa6353f7b8cb1d5c462297ac399dc8c4fb4391b6b6c65a\": container with ID starting with d6d48b0b05201deae9fa6353f7b8cb1d5c462297ac399dc8c4fb4391b6b6c65a not found: ID does not exist" containerID="d6d48b0b05201deae9fa6353f7b8cb1d5c462297ac399dc8c4fb4391b6b6c65a"
Sep 18 19:17:25.464150 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.464109    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6d48b0b05201deae9fa6353f7b8cb1d5c462297ac399dc8c4fb4391b6b6c65a"} err="failed to get container status \"d6d48b0b05201deae9fa6353f7b8cb1d5c462297ac399dc8c4fb4391b6b6c65a\": rpc error: code = NotFound desc = could not find container \"d6d48b0b05201deae9fa6353f7b8cb1d5c462297ac399dc8c4fb4391b6b6c65a\": container with ID starting with d6d48b0b05201deae9fa6353f7b8cb1d5c462297ac399dc8c4fb4391b6b6c65a not found: ID does not exist"
Sep 18 19:17:25.464150 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.464140    2577 scope.go:117] "RemoveContainer" containerID="4327994fdd2c8981fb587400225e2acac010d03f54ee4791b336305228cf1fd0"
Sep 18 19:17:25.464393 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:17:25.464375    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4327994fdd2c8981fb587400225e2acac010d03f54ee4791b336305228cf1fd0\": container with ID starting with 4327994fdd2c8981fb587400225e2acac010d03f54ee4791b336305228cf1fd0 not found: ID does not exist" containerID="4327994fdd2c8981fb587400225e2acac010d03f54ee4791b336305228cf1fd0"
Sep 18 19:17:25.464432 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.464400    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4327994fdd2c8981fb587400225e2acac010d03f54ee4791b336305228cf1fd0"} err="failed to get container status \"4327994fdd2c8981fb587400225e2acac010d03f54ee4791b336305228cf1fd0\": rpc error: code = NotFound desc = could not find container \"4327994fdd2c8981fb587400225e2acac010d03f54ee4791b336305228cf1fd0\": container with ID starting with 4327994fdd2c8981fb587400225e2acac010d03f54ee4791b336305228cf1fd0 not found: ID does not exist"
Sep 18 19:17:25.464432 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.464415    2577 scope.go:117] "RemoveContainer" containerID="1afe39e5df2a249eb8e863429f84b2ac8e04234d477149e3dafe26c8b4009985"
Sep 18 19:17:25.464647 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:17:25.464630    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1afe39e5df2a249eb8e863429f84b2ac8e04234d477149e3dafe26c8b4009985\": container with ID starting with 1afe39e5df2a249eb8e863429f84b2ac8e04234d477149e3dafe26c8b4009985 not found: ID does not exist" containerID="1afe39e5df2a249eb8e863429f84b2ac8e04234d477149e3dafe26c8b4009985"
Sep 18 19:17:25.464716 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.464655    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1afe39e5df2a249eb8e863429f84b2ac8e04234d477149e3dafe26c8b4009985"} err="failed to get container status \"1afe39e5df2a249eb8e863429f84b2ac8e04234d477149e3dafe26c8b4009985\": rpc error: code = NotFound desc = could not find container \"1afe39e5df2a249eb8e863429f84b2ac8e04234d477149e3dafe26c8b4009985\": container with ID starting with 1afe39e5df2a249eb8e863429f84b2ac8e04234d477149e3dafe26c8b4009985 not found: ID does not exist"
Sep 18 19:17:25.464716 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.464676    2577 scope.go:117] "RemoveContainer" containerID="dc0391340ea0f0329ea62a20f899bbb284ee640667f4229d48b837481616deeb"
Sep 18 19:17:25.464916 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:17:25.464902    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc0391340ea0f0329ea62a20f899bbb284ee640667f4229d48b837481616deeb\": container with ID starting with dc0391340ea0f0329ea62a20f899bbb284ee640667f4229d48b837481616deeb not found: ID does not exist" containerID="dc0391340ea0f0329ea62a20f899bbb284ee640667f4229d48b837481616deeb"
Sep 18 19:17:25.464958 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.464919    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc0391340ea0f0329ea62a20f899bbb284ee640667f4229d48b837481616deeb"} err="failed to get container status \"dc0391340ea0f0329ea62a20f899bbb284ee640667f4229d48b837481616deeb\": rpc error: code = NotFound desc = could not find container \"dc0391340ea0f0329ea62a20f899bbb284ee640667f4229d48b837481616deeb\": container with ID starting with dc0391340ea0f0329ea62a20f899bbb284ee640667f4229d48b837481616deeb not found: ID does not exist"
Sep 18 19:17:25.464958 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.464930    2577 scope.go:117] "RemoveContainer" containerID="19e6a608bf6ab74b0cfa191186b666e45288adc5c20c18b40539dccc98d0811b"
Sep 18 19:17:25.465148 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:17:25.465131    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19e6a608bf6ab74b0cfa191186b666e45288adc5c20c18b40539dccc98d0811b\": container with ID starting with 19e6a608bf6ab74b0cfa191186b666e45288adc5c20c18b40539dccc98d0811b not found: ID does not exist" containerID="19e6a608bf6ab74b0cfa191186b666e45288adc5c20c18b40539dccc98d0811b"
Sep 18 19:17:25.465214 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.465156    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19e6a608bf6ab74b0cfa191186b666e45288adc5c20c18b40539dccc98d0811b"} err="failed to get container status \"19e6a608bf6ab74b0cfa191186b666e45288adc5c20c18b40539dccc98d0811b\": rpc error: code = NotFound desc = could not find container \"19e6a608bf6ab74b0cfa191186b666e45288adc5c20c18b40539dccc98d0811b\": container with ID starting with 19e6a608bf6ab74b0cfa191186b666e45288adc5c20c18b40539dccc98d0811b not found: ID does not exist"
Sep 18 19:17:25.465214 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.465175    2577 scope.go:117] "RemoveContainer" containerID="8e836dd7d3715a756b4a0de45b63b8339f110a35e5116ed2f6db31ffaa0eeef1"
Sep 18 19:17:25.465393 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:17:25.465374    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e836dd7d3715a756b4a0de45b63b8339f110a35e5116ed2f6db31ffaa0eeef1\": container with ID starting with 8e836dd7d3715a756b4a0de45b63b8339f110a35e5116ed2f6db31ffaa0eeef1 not found: ID does not exist" containerID="8e836dd7d3715a756b4a0de45b63b8339f110a35e5116ed2f6db31ffaa0eeef1"
Sep 18 19:17:25.465430 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.465397    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e836dd7d3715a756b4a0de45b63b8339f110a35e5116ed2f6db31ffaa0eeef1"} err="failed to get container status \"8e836dd7d3715a756b4a0de45b63b8339f110a35e5116ed2f6db31ffaa0eeef1\": rpc error: code = NotFound desc = could not find container \"8e836dd7d3715a756b4a0de45b63b8339f110a35e5116ed2f6db31ffaa0eeef1\": container with ID starting with 8e836dd7d3715a756b4a0de45b63b8339f110a35e5116ed2f6db31ffaa0eeef1 not found: ID does not exist"
Sep 18 19:17:25.465430 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.465409    2577 scope.go:117] "RemoveContainer" containerID="b33a28be48026c6aa525eb08702db590e11dddbb04b3dd81e24c77ce01ac66aa"
Sep 18 19:17:25.465617 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:17:25.465602    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b33a28be48026c6aa525eb08702db590e11dddbb04b3dd81e24c77ce01ac66aa\": container with ID starting with b33a28be48026c6aa525eb08702db590e11dddbb04b3dd81e24c77ce01ac66aa not found: ID does not exist" containerID="b33a28be48026c6aa525eb08702db590e11dddbb04b3dd81e24c77ce01ac66aa"
Sep 18 19:17:25.465675 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.465622    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b33a28be48026c6aa525eb08702db590e11dddbb04b3dd81e24c77ce01ac66aa"} err="failed to get container status \"b33a28be48026c6aa525eb08702db590e11dddbb04b3dd81e24c77ce01ac66aa\": rpc error: code = NotFound desc = could not find container \"b33a28be48026c6aa525eb08702db590e11dddbb04b3dd81e24c77ce01ac66aa\": container with ID starting with b33a28be48026c6aa525eb08702db590e11dddbb04b3dd81e24c77ce01ac66aa not found: ID does not exist"
Sep 18 19:17:25.465675 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.465642    2577 scope.go:117] "RemoveContainer" containerID="d6d48b0b05201deae9fa6353f7b8cb1d5c462297ac399dc8c4fb4391b6b6c65a"
Sep 18 19:17:25.465871 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.465855    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6d48b0b05201deae9fa6353f7b8cb1d5c462297ac399dc8c4fb4391b6b6c65a"} err="failed to get container status \"d6d48b0b05201deae9fa6353f7b8cb1d5c462297ac399dc8c4fb4391b6b6c65a\": rpc error: code = NotFound desc = could not find container \"d6d48b0b05201deae9fa6353f7b8cb1d5c462297ac399dc8c4fb4391b6b6c65a\": container with ID starting with d6d48b0b05201deae9fa6353f7b8cb1d5c462297ac399dc8c4fb4391b6b6c65a not found: ID does not exist"
Sep 18 19:17:25.465871 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.465870    2577 scope.go:117] "RemoveContainer" containerID="4327994fdd2c8981fb587400225e2acac010d03f54ee4791b336305228cf1fd0"
Sep 18 19:17:25.466148 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.466128    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4327994fdd2c8981fb587400225e2acac010d03f54ee4791b336305228cf1fd0"} err="failed to get container status \"4327994fdd2c8981fb587400225e2acac010d03f54ee4791b336305228cf1fd0\": rpc error: code = NotFound desc = could not find container \"4327994fdd2c8981fb587400225e2acac010d03f54ee4791b336305228cf1fd0\": container with ID starting with 4327994fdd2c8981fb587400225e2acac010d03f54ee4791b336305228cf1fd0 not found: ID does not exist"
Sep 18 19:17:25.466203 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.466149    2577 scope.go:117] "RemoveContainer" containerID="1afe39e5df2a249eb8e863429f84b2ac8e04234d477149e3dafe26c8b4009985"
Sep 18 19:17:25.466376 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.466361    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1afe39e5df2a249eb8e863429f84b2ac8e04234d477149e3dafe26c8b4009985"} err="failed to get container status \"1afe39e5df2a249eb8e863429f84b2ac8e04234d477149e3dafe26c8b4009985\": rpc error: code = NotFound desc = could not find container \"1afe39e5df2a249eb8e863429f84b2ac8e04234d477149e3dafe26c8b4009985\": container with ID starting with 1afe39e5df2a249eb8e863429f84b2ac8e04234d477149e3dafe26c8b4009985 not found: ID does not exist"
Sep 18 19:17:25.466420 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.466377    2577 scope.go:117] "RemoveContainer" containerID="dc0391340ea0f0329ea62a20f899bbb284ee640667f4229d48b837481616deeb"
Sep 18 19:17:25.466598 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.466580    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc0391340ea0f0329ea62a20f899bbb284ee640667f4229d48b837481616deeb"} err="failed to get container status \"dc0391340ea0f0329ea62a20f899bbb284ee640667f4229d48b837481616deeb\": rpc error: code = NotFound desc = could not find container \"dc0391340ea0f0329ea62a20f899bbb284ee640667f4229d48b837481616deeb\": container with ID starting with dc0391340ea0f0329ea62a20f899bbb284ee640667f4229d48b837481616deeb not found: ID does not exist"
Sep 18 19:17:25.466643 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.466599    2577 scope.go:117] "RemoveContainer" containerID="19e6a608bf6ab74b0cfa191186b666e45288adc5c20c18b40539dccc98d0811b"
Sep 18 19:17:25.466824 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.466804    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19e6a608bf6ab74b0cfa191186b666e45288adc5c20c18b40539dccc98d0811b"} err="failed to get container status \"19e6a608bf6ab74b0cfa191186b666e45288adc5c20c18b40539dccc98d0811b\": rpc error: code = NotFound desc = could not find container \"19e6a608bf6ab74b0cfa191186b666e45288adc5c20c18b40539dccc98d0811b\": container with ID starting with 19e6a608bf6ab74b0cfa191186b666e45288adc5c20c18b40539dccc98d0811b not found: ID does not exist"
Sep 18 19:17:25.466870 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.466826    2577 scope.go:117] "RemoveContainer" containerID="8e836dd7d3715a756b4a0de45b63b8339f110a35e5116ed2f6db31ffaa0eeef1"
Sep 18 19:17:25.467068 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.467047    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e836dd7d3715a756b4a0de45b63b8339f110a35e5116ed2f6db31ffaa0eeef1"} err="failed to get container status \"8e836dd7d3715a756b4a0de45b63b8339f110a35e5116ed2f6db31ffaa0eeef1\": rpc error: code = NotFound desc = could not find container \"8e836dd7d3715a756b4a0de45b63b8339f110a35e5116ed2f6db31ffaa0eeef1\": container with ID starting with 8e836dd7d3715a756b4a0de45b63b8339f110a35e5116ed2f6db31ffaa0eeef1 not found: ID does not exist"
Sep 18 19:17:25.467068 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.467066    2577 scope.go:117] "RemoveContainer" containerID="b33a28be48026c6aa525eb08702db590e11dddbb04b3dd81e24c77ce01ac66aa"
Sep 18 19:17:25.467420 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.467394    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b33a28be48026c6aa525eb08702db590e11dddbb04b3dd81e24c77ce01ac66aa"} err="failed to get container status \"b33a28be48026c6aa525eb08702db590e11dddbb04b3dd81e24c77ce01ac66aa\": rpc error: code = NotFound desc = could not find container \"b33a28be48026c6aa525eb08702db590e11dddbb04b3dd81e24c77ce01ac66aa\": container with ID starting with b33a28be48026c6aa525eb08702db590e11dddbb04b3dd81e24c77ce01ac66aa not found: ID does not exist"
Sep 18 19:17:25.469217 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.469199    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 19:17:25.469446 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.469435    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c0d88df-3d51-4e0e-883a-294ca50af402" containerName="kube-rbac-proxy"
Sep 18 19:17:25.469490 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.469448    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c0d88df-3d51-4e0e-883a-294ca50af402" containerName="kube-rbac-proxy"
Sep 18 19:17:25.469490 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.469458    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c0d88df-3d51-4e0e-883a-294ca50af402" containerName="config-reloader"
Sep 18 19:17:25.469490 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.469463    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c0d88df-3d51-4e0e-883a-294ca50af402" containerName="config-reloader"
Sep 18 19:17:25.469490 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.469470    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c0d88df-3d51-4e0e-883a-294ca50af402" containerName="alertmanager"
Sep 18 19:17:25.469490 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.469474    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c0d88df-3d51-4e0e-883a-294ca50af402" containerName="alertmanager"
Sep 18 19:17:25.469490 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.469481    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c0d88df-3d51-4e0e-883a-294ca50af402" containerName="kube-rbac-proxy-metric"
Sep 18 19:17:25.469490 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.469486    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c0d88df-3d51-4e0e-883a-294ca50af402" containerName="kube-rbac-proxy-metric"
Sep 18 19:17:25.469490 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.469492    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c0d88df-3d51-4e0e-883a-294ca50af402" containerName="prom-label-proxy"
Sep 18 19:17:25.469714 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.469497    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c0d88df-3d51-4e0e-883a-294ca50af402" containerName="prom-label-proxy"
Sep 18 19:17:25.469714 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.469511    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c0d88df-3d51-4e0e-883a-294ca50af402" containerName="init-config-reloader"
Sep 18 19:17:25.469714 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.469519    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c0d88df-3d51-4e0e-883a-294ca50af402" containerName="init-config-reloader"
Sep 18 19:17:25.469714 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.469527    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c0d88df-3d51-4e0e-883a-294ca50af402" containerName="kube-rbac-proxy-web"
Sep 18 19:17:25.469714 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.469534    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c0d88df-3d51-4e0e-883a-294ca50af402" containerName="kube-rbac-proxy-web"
Sep 18 19:17:25.469714 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.469588    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c0d88df-3d51-4e0e-883a-294ca50af402" containerName="alertmanager"
Sep 18 19:17:25.469714 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.469595    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c0d88df-3d51-4e0e-883a-294ca50af402" containerName="kube-rbac-proxy-metric"
Sep 18 19:17:25.469714 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.469601    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c0d88df-3d51-4e0e-883a-294ca50af402" containerName="config-reloader"
Sep 18 19:17:25.469714 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.469606    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c0d88df-3d51-4e0e-883a-294ca50af402" containerName="prom-label-proxy"
Sep 18 19:17:25.469714 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.469612    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c0d88df-3d51-4e0e-883a-294ca50af402" containerName="kube-rbac-proxy-web"
Sep 18 19:17:25.469714 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.469617    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c0d88df-3d51-4e0e-883a-294ca50af402" containerName="kube-rbac-proxy"
Sep 18 19:17:25.473144 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.473130    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:25.476632 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.476613    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 18 19:17:25.476798 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.476778    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 18 19:17:25.476798 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.476795    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 18 19:17:25.476945 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.476808    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 18 19:17:25.476945 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.476816    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 18 19:17:25.477252 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.477236    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 18 19:17:25.477317 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.477300    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 18 19:17:25.477467 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.477453    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 18 19:17:25.479004 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.478970    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-drzx7\""
Sep 18 19:17:25.483803 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.483787    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 18 19:17:25.496389 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.496369    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 19:17:25.642552 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.642491    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b682942d-7069-404b-918d-e83dfeb960cf-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b682942d-7069-404b-918d-e83dfeb960cf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:25.642552 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.642521    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b682942d-7069-404b-918d-e83dfeb960cf-config-out\") pod \"alertmanager-main-0\" (UID: \"b682942d-7069-404b-918d-e83dfeb960cf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:25.642552 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.642540    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b682942d-7069-404b-918d-e83dfeb960cf-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b682942d-7069-404b-918d-e83dfeb960cf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:25.642733 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.642631    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b682942d-7069-404b-918d-e83dfeb960cf-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b682942d-7069-404b-918d-e83dfeb960cf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:25.642733 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.642687    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b682942d-7069-404b-918d-e83dfeb960cf-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b682942d-7069-404b-918d-e83dfeb960cf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:25.642733 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.642714    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b682942d-7069-404b-918d-e83dfeb960cf-web-config\") pod \"alertmanager-main-0\" (UID: \"b682942d-7069-404b-918d-e83dfeb960cf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:25.642733 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.642728    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b682942d-7069-404b-918d-e83dfeb960cf-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b682942d-7069-404b-918d-e83dfeb960cf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:25.642980 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.642745    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b682942d-7069-404b-918d-e83dfeb960cf-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b682942d-7069-404b-918d-e83dfeb960cf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:25.642980 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.642808    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b682942d-7069-404b-918d-e83dfeb960cf-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b682942d-7069-404b-918d-e83dfeb960cf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:25.642980 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.642848    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrhzc\" (UniqueName: \"kubernetes.io/projected/b682942d-7069-404b-918d-e83dfeb960cf-kube-api-access-lrhzc\") pod \"alertmanager-main-0\" (UID: \"b682942d-7069-404b-918d-e83dfeb960cf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:25.642980 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.642887    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b682942d-7069-404b-918d-e83dfeb960cf-config-volume\") pod \"alertmanager-main-0\" (UID: \"b682942d-7069-404b-918d-e83dfeb960cf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:25.642980 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.642901    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b682942d-7069-404b-918d-e83dfeb960cf-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b682942d-7069-404b-918d-e83dfeb960cf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:25.642980 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.642921    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b682942d-7069-404b-918d-e83dfeb960cf-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b682942d-7069-404b-918d-e83dfeb960cf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:25.743654 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.743617    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b682942d-7069-404b-918d-e83dfeb960cf-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b682942d-7069-404b-918d-e83dfeb960cf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:25.743654 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.743658    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b682942d-7069-404b-918d-e83dfeb960cf-web-config\") pod \"alertmanager-main-0\" (UID: \"b682942d-7069-404b-918d-e83dfeb960cf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:25.743878 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.743677    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b682942d-7069-404b-918d-e83dfeb960cf-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b682942d-7069-404b-918d-e83dfeb960cf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:25.743878 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.743798    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b682942d-7069-404b-918d-e83dfeb960cf-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b682942d-7069-404b-918d-e83dfeb960cf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:25.743878 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.743842    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b682942d-7069-404b-918d-e83dfeb960cf-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b682942d-7069-404b-918d-e83dfeb960cf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:25.743878 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.743868    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrhzc\" (UniqueName: \"kubernetes.io/projected/b682942d-7069-404b-918d-e83dfeb960cf-kube-api-access-lrhzc\") pod \"alertmanager-main-0\" (UID: \"b682942d-7069-404b-918d-e83dfeb960cf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:25.744100 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.743913    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b682942d-7069-404b-918d-e83dfeb960cf-config-volume\") pod \"alertmanager-main-0\" (UID: \"b682942d-7069-404b-918d-e83dfeb960cf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:25.744100 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.743938    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b682942d-7069-404b-918d-e83dfeb960cf-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b682942d-7069-404b-918d-e83dfeb960cf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:25.744100 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.743967    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b682942d-7069-404b-918d-e83dfeb960cf-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b682942d-7069-404b-918d-e83dfeb960cf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:25.744100 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.744033    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b682942d-7069-404b-918d-e83dfeb960cf-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b682942d-7069-404b-918d-e83dfeb960cf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:25.744100 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.744062    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b682942d-7069-404b-918d-e83dfeb960cf-config-out\") pod \"alertmanager-main-0\" (UID: \"b682942d-7069-404b-918d-e83dfeb960cf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:25.744345 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.744103    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b682942d-7069-404b-918d-e83dfeb960cf-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b682942d-7069-404b-918d-e83dfeb960cf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:25.744345 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.744158    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b682942d-7069-404b-918d-e83dfeb960cf-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b682942d-7069-404b-918d-e83dfeb960cf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:25.744829 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.744738    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b682942d-7069-404b-918d-e83dfeb960cf-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b682942d-7069-404b-918d-e83dfeb960cf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:25.745104 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.745087    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b682942d-7069-404b-918d-e83dfeb960cf-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b682942d-7069-404b-918d-e83dfeb960cf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:25.745229 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.745169    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b682942d-7069-404b-918d-e83dfeb960cf-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b682942d-7069-404b-918d-e83dfeb960cf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:25.746972 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.746903    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b682942d-7069-404b-918d-e83dfeb960cf-web-config\") pod \"alertmanager-main-0\" (UID: \"b682942d-7069-404b-918d-e83dfeb960cf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:25.746972 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.746903    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b682942d-7069-404b-918d-e83dfeb960cf-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b682942d-7069-404b-918d-e83dfeb960cf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:25.746972 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.746952    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b682942d-7069-404b-918d-e83dfeb960cf-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b682942d-7069-404b-918d-e83dfeb960cf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:25.747351 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.747328    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b682942d-7069-404b-918d-e83dfeb960cf-config-volume\") pod \"alertmanager-main-0\" (UID: \"b682942d-7069-404b-918d-e83dfeb960cf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:25.747627 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.747601    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b682942d-7069-404b-918d-e83dfeb960cf-config-out\") pod \"alertmanager-main-0\" (UID: \"b682942d-7069-404b-918d-e83dfeb960cf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:25.747758 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.747739    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b682942d-7069-404b-918d-e83dfeb960cf-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b682942d-7069-404b-918d-e83dfeb960cf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:25.747883 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.747860    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b682942d-7069-404b-918d-e83dfeb960cf-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b682942d-7069-404b-918d-e83dfeb960cf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:25.748362 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.748346    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b682942d-7069-404b-918d-e83dfeb960cf-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b682942d-7069-404b-918d-e83dfeb960cf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:25.748656 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.748638    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b682942d-7069-404b-918d-e83dfeb960cf-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b682942d-7069-404b-918d-e83dfeb960cf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:25.754708 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.754686    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrhzc\" (UniqueName: \"kubernetes.io/projected/b682942d-7069-404b-918d-e83dfeb960cf-kube-api-access-lrhzc\") pod \"alertmanager-main-0\" (UID: \"b682942d-7069-404b-918d-e83dfeb960cf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:25.783136 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:25.783116    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:26.069550 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:26.069473    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 19:17:26.073193 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:17:26.073166    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb682942d_7069_404b_918d_e83dfeb960cf.slice/crio-a689dfa2826c540f844fdf70bd435ec6aa8779ec604b5457477027b262cc3c7e WatchSource:0}: Error finding container a689dfa2826c540f844fdf70bd435ec6aa8779ec604b5457477027b262cc3c7e: Status 404 returned error can't find the container with id a689dfa2826c540f844fdf70bd435ec6aa8779ec604b5457477027b262cc3c7e
Sep 18 19:17:26.393465 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:26.393435    2577 generic.go:356] "Generic (PLEG): container finished" podID="b682942d-7069-404b-918d-e83dfeb960cf" containerID="537a123c24ec06f214e03012ec2ac93ef23da36e7bbc4c9a35d40c74f8496bc4" exitCode=0
Sep 18 19:17:26.393465 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:26.393469    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b682942d-7069-404b-918d-e83dfeb960cf","Type":"ContainerDied","Data":"537a123c24ec06f214e03012ec2ac93ef23da36e7bbc4c9a35d40c74f8496bc4"}
Sep 18 19:17:26.393667 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:26.393515    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b682942d-7069-404b-918d-e83dfeb960cf","Type":"ContainerStarted","Data":"a689dfa2826c540f844fdf70bd435ec6aa8779ec604b5457477027b262cc3c7e"}
Sep 18 19:17:27.401327 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:27.401292    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b682942d-7069-404b-918d-e83dfeb960cf","Type":"ContainerStarted","Data":"8ef1c9494129fe0d753e1aa6eb3414e7b44bf6be53ae0de0a4bc34473fb99959"}
Sep 18 19:17:27.401700 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:27.401392    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b682942d-7069-404b-918d-e83dfeb960cf","Type":"ContainerStarted","Data":"940d52bada457dc3fa712a5f801f4821b31d5e7e9fb5ed1ddbbbd88eb2c8842d"}
Sep 18 19:17:27.401700 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:27.401419    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b682942d-7069-404b-918d-e83dfeb960cf","Type":"ContainerStarted","Data":"4a0e5d76555429b00586d81d7016055f5e31a6975daed2159cb33883da682edd"}
Sep 18 19:17:27.401700 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:27.401446    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b682942d-7069-404b-918d-e83dfeb960cf","Type":"ContainerStarted","Data":"357ca915b9546a2e910622eb5f24da66debbf365ddeb95ad7e48a8cd7dd4ff76"}
Sep 18 19:17:27.401700 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:27.401483    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b682942d-7069-404b-918d-e83dfeb960cf","Type":"ContainerStarted","Data":"2a92a64f9b864243f6746680195625383403c7d081117ca7901f627fafab0928"}
Sep 18 19:17:27.401700 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:27.401520    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b682942d-7069-404b-918d-e83dfeb960cf","Type":"ContainerStarted","Data":"a131399dbe1f8988e64d9b4f39713a774850d2b485682b72fef1b6320e9f1a76"}
Sep 18 19:17:27.435189 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:27.434917    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.420196683 podStartE2EDuration="2.434896414s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.699731ms" isStatefulPod=false podCreationTimestamp="2026-09-18 19:17:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:17:27.432736181 +0000 UTC m=+218.285036551" watchObservedRunningTime="2026-09-18 19:17:27.434896414 +0000 UTC m=+218.287196785"
Sep 18 19:17:27.438556 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:27.438533    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c0d88df-3d51-4e0e-883a-294ca50af402" path="/var/lib/kubelet/pods/1c0d88df-3d51-4e0e-883a-294ca50af402/volumes"
Sep 18 19:17:28.039355 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.039315    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 19:17:28.040706 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.039954    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="83f44b93-4402-4869-91ec-6fd23c938443" containerName="kube-rbac-proxy-thanos" containerID="cri-o://5514363ae6531cf5e1818a035dcd73688ca2f21dbd973fed1f2355c9906efa75" gracePeriod=600
Sep 18 19:17:28.040706 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.040354    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="83f44b93-4402-4869-91ec-6fd23c938443" containerName="kube-rbac-proxy-web" containerID="cri-o://7c509504133689d7581b3cfacdb76fe7f180d8f6424bdf3f525e32d1eca275a3" gracePeriod=600
Sep 18 19:17:28.040706 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.040459    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="83f44b93-4402-4869-91ec-6fd23c938443" containerName="kube-rbac-proxy" containerID="cri-o://db7145413295c0ef5d2d57f4e8fb8aeb0e9b3aef0dad53f21664a7e6abb4c5c2" gracePeriod=600
Sep 18 19:17:28.040706 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.040539    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="83f44b93-4402-4869-91ec-6fd23c938443" containerName="thanos-sidecar" containerID="cri-o://3233e21243a7da879fd5deee2efa5c041a24efa428c4cc29ae956b85e3f64ed8" gracePeriod=600
Sep 18 19:17:28.040706 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.040622    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="83f44b93-4402-4869-91ec-6fd23c938443" containerName="config-reloader" containerID="cri-o://7b4ce9377d3cd59952c0dec7066b62da90ceaa80018a94755c3f411ae9810f4e" gracePeriod=600
Sep 18 19:17:28.041050 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.040873    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="83f44b93-4402-4869-91ec-6fd23c938443" containerName="prometheus" containerID="cri-o://623c43a362fdd42fbd934d3e390aff7323d886b56800a72f871d9496a91c93a9" gracePeriod=600
Sep 18 19:17:28.269177 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.269151    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.370387 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.370337    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83f44b93-4402-4869-91ec-6fd23c938443-prometheus-trusted-ca-bundle\") pod \"83f44b93-4402-4869-91ec-6fd23c938443\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") "
Sep 18 19:17:28.370588 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.370399    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/83f44b93-4402-4869-91ec-6fd23c938443-prometheus-k8s-rulefiles-0\") pod \"83f44b93-4402-4869-91ec-6fd23c938443\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") "
Sep 18 19:17:28.370588 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.370441    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/83f44b93-4402-4869-91ec-6fd23c938443-prometheus-k8s-db\") pod \"83f44b93-4402-4869-91ec-6fd23c938443\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") "
Sep 18 19:17:28.370588 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.370464    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83f44b93-4402-4869-91ec-6fd23c938443-configmap-metrics-client-ca\") pod \"83f44b93-4402-4869-91ec-6fd23c938443\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") "
Sep 18 19:17:28.370588 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.370481    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/83f44b93-4402-4869-91ec-6fd23c938443-prometheus-k8s-rulefiles-1\") pod \"83f44b93-4402-4869-91ec-6fd23c938443\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") "
Sep 18 19:17:28.370588 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.370518    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/83f44b93-4402-4869-91ec-6fd23c938443-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"83f44b93-4402-4869-91ec-6fd23c938443\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") "
Sep 18 19:17:28.370588 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.370543    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83f44b93-4402-4869-91ec-6fd23c938443-configmap-kubelet-serving-ca-bundle\") pod \"83f44b93-4402-4869-91ec-6fd23c938443\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") "
Sep 18 19:17:28.370588 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.370578    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/83f44b93-4402-4869-91ec-6fd23c938443-secret-kube-rbac-proxy\") pod \"83f44b93-4402-4869-91ec-6fd23c938443\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") "
Sep 18 19:17:28.370941 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.370609    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/83f44b93-4402-4869-91ec-6fd23c938443-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"83f44b93-4402-4869-91ec-6fd23c938443\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") "
Sep 18 19:17:28.370941 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.370637    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/83f44b93-4402-4869-91ec-6fd23c938443-config\") pod \"83f44b93-4402-4869-91ec-6fd23c938443\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") "
Sep 18 19:17:28.370941 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.370659    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/83f44b93-4402-4869-91ec-6fd23c938443-secret-prometheus-k8s-tls\") pod \"83f44b93-4402-4869-91ec-6fd23c938443\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") "
Sep 18 19:17:28.370941 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.370682    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/83f44b93-4402-4869-91ec-6fd23c938443-thanos-prometheus-http-client-file\") pod \"83f44b93-4402-4869-91ec-6fd23c938443\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") "
Sep 18 19:17:28.370941 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.370698    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/83f44b93-4402-4869-91ec-6fd23c938443-config-out\") pod \"83f44b93-4402-4869-91ec-6fd23c938443\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") "
Sep 18 19:17:28.370941 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.370722    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83f44b93-4402-4869-91ec-6fd23c938443-configmap-serving-certs-ca-bundle\") pod \"83f44b93-4402-4869-91ec-6fd23c938443\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") "
Sep 18 19:17:28.370941 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.370743    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b2trp\" (UniqueName: \"kubernetes.io/projected/83f44b93-4402-4869-91ec-6fd23c938443-kube-api-access-b2trp\") pod \"83f44b93-4402-4869-91ec-6fd23c938443\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") "
Sep 18 19:17:28.370941 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.370744    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83f44b93-4402-4869-91ec-6fd23c938443-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "83f44b93-4402-4869-91ec-6fd23c938443" (UID: "83f44b93-4402-4869-91ec-6fd23c938443"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:17:28.370941 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.370758    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/83f44b93-4402-4869-91ec-6fd23c938443-secret-grpc-tls\") pod \"83f44b93-4402-4869-91ec-6fd23c938443\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") "
Sep 18 19:17:28.370941 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.370815    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/83f44b93-4402-4869-91ec-6fd23c938443-tls-assets\") pod \"83f44b93-4402-4869-91ec-6fd23c938443\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") "
Sep 18 19:17:28.370941 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.370858    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/83f44b93-4402-4869-91ec-6fd23c938443-prometheus-k8s-rulefiles-2\") pod \"83f44b93-4402-4869-91ec-6fd23c938443\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") "
Sep 18 19:17:28.370941 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.370909    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/83f44b93-4402-4869-91ec-6fd23c938443-secret-metrics-client-certs\") pod \"83f44b93-4402-4869-91ec-6fd23c938443\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") "
Sep 18 19:17:28.370941 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.370936    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/83f44b93-4402-4869-91ec-6fd23c938443-web-config\") pod \"83f44b93-4402-4869-91ec-6fd23c938443\" (UID: \"83f44b93-4402-4869-91ec-6fd23c938443\") "
Sep 18 19:17:28.371596 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.371199    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83f44b93-4402-4869-91ec-6fd23c938443-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "83f44b93-4402-4869-91ec-6fd23c938443" (UID: "83f44b93-4402-4869-91ec-6fd23c938443"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:17:28.371596 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.371251    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83f44b93-4402-4869-91ec-6fd23c938443-prometheus-trusted-ca-bundle\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:17:28.372097 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.372064    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83f44b93-4402-4869-91ec-6fd23c938443-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "83f44b93-4402-4869-91ec-6fd23c938443" (UID: "83f44b93-4402-4869-91ec-6fd23c938443"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:17:28.372097 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.372081    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83f44b93-4402-4869-91ec-6fd23c938443-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "83f44b93-4402-4869-91ec-6fd23c938443" (UID: "83f44b93-4402-4869-91ec-6fd23c938443"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:17:28.372662 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.372412    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83f44b93-4402-4869-91ec-6fd23c938443-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "83f44b93-4402-4869-91ec-6fd23c938443" (UID: "83f44b93-4402-4869-91ec-6fd23c938443"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:17:28.372662 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.372463    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83f44b93-4402-4869-91ec-6fd23c938443-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "83f44b93-4402-4869-91ec-6fd23c938443" (UID: "83f44b93-4402-4869-91ec-6fd23c938443"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:17:28.373434 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.373400    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83f44b93-4402-4869-91ec-6fd23c938443-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "83f44b93-4402-4869-91ec-6fd23c938443" (UID: "83f44b93-4402-4869-91ec-6fd23c938443"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:17:28.375376 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.375266    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83f44b93-4402-4869-91ec-6fd23c938443-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "83f44b93-4402-4869-91ec-6fd23c938443" (UID: "83f44b93-4402-4869-91ec-6fd23c938443"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:17:28.375376 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.375282    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83f44b93-4402-4869-91ec-6fd23c938443-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "83f44b93-4402-4869-91ec-6fd23c938443" (UID: "83f44b93-4402-4869-91ec-6fd23c938443"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:17:28.375376 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.375333    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83f44b93-4402-4869-91ec-6fd23c938443-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "83f44b93-4402-4869-91ec-6fd23c938443" (UID: "83f44b93-4402-4869-91ec-6fd23c938443"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:17:28.375376 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.375351    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83f44b93-4402-4869-91ec-6fd23c938443-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "83f44b93-4402-4869-91ec-6fd23c938443" (UID: "83f44b93-4402-4869-91ec-6fd23c938443"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:17:28.375964 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.375708    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83f44b93-4402-4869-91ec-6fd23c938443-config" (OuterVolumeSpecName: "config") pod "83f44b93-4402-4869-91ec-6fd23c938443" (UID: "83f44b93-4402-4869-91ec-6fd23c938443"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:17:28.375964 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.375787    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83f44b93-4402-4869-91ec-6fd23c938443-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "83f44b93-4402-4869-91ec-6fd23c938443" (UID: "83f44b93-4402-4869-91ec-6fd23c938443"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:17:28.375964 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.375936    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83f44b93-4402-4869-91ec-6fd23c938443-config-out" (OuterVolumeSpecName: "config-out") pod "83f44b93-4402-4869-91ec-6fd23c938443" (UID: "83f44b93-4402-4869-91ec-6fd23c938443"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:17:28.376289 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.376074    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83f44b93-4402-4869-91ec-6fd23c938443-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "83f44b93-4402-4869-91ec-6fd23c938443" (UID: "83f44b93-4402-4869-91ec-6fd23c938443"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:17:28.376583 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.376553    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83f44b93-4402-4869-91ec-6fd23c938443-kube-api-access-b2trp" (OuterVolumeSpecName: "kube-api-access-b2trp") pod "83f44b93-4402-4869-91ec-6fd23c938443" (UID: "83f44b93-4402-4869-91ec-6fd23c938443"). InnerVolumeSpecName "kube-api-access-b2trp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:17:28.376691 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.376672    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83f44b93-4402-4869-91ec-6fd23c938443-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "83f44b93-4402-4869-91ec-6fd23c938443" (UID: "83f44b93-4402-4869-91ec-6fd23c938443"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:17:28.377069 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.377042    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83f44b93-4402-4869-91ec-6fd23c938443-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "83f44b93-4402-4869-91ec-6fd23c938443" (UID: "83f44b93-4402-4869-91ec-6fd23c938443"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:17:28.377233 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.377216    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83f44b93-4402-4869-91ec-6fd23c938443-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "83f44b93-4402-4869-91ec-6fd23c938443" (UID: "83f44b93-4402-4869-91ec-6fd23c938443"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:17:28.385896 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.385875    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83f44b93-4402-4869-91ec-6fd23c938443-web-config" (OuterVolumeSpecName: "web-config") pod "83f44b93-4402-4869-91ec-6fd23c938443" (UID: "83f44b93-4402-4869-91ec-6fd23c938443"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:17:28.408328 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.408305    2577 generic.go:356] "Generic (PLEG): container finished" podID="83f44b93-4402-4869-91ec-6fd23c938443" containerID="5514363ae6531cf5e1818a035dcd73688ca2f21dbd973fed1f2355c9906efa75" exitCode=0
Sep 18 19:17:28.408328 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.408325    2577 generic.go:356] "Generic (PLEG): container finished" podID="83f44b93-4402-4869-91ec-6fd23c938443" containerID="db7145413295c0ef5d2d57f4e8fb8aeb0e9b3aef0dad53f21664a7e6abb4c5c2" exitCode=0
Sep 18 19:17:28.408328 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.408331    2577 generic.go:356] "Generic (PLEG): container finished" podID="83f44b93-4402-4869-91ec-6fd23c938443" containerID="7c509504133689d7581b3cfacdb76fe7f180d8f6424bdf3f525e32d1eca275a3" exitCode=0
Sep 18 19:17:28.408781 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.408337    2577 generic.go:356] "Generic (PLEG): container finished" podID="83f44b93-4402-4869-91ec-6fd23c938443" containerID="3233e21243a7da879fd5deee2efa5c041a24efa428c4cc29ae956b85e3f64ed8" exitCode=0
Sep 18 19:17:28.408781 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.408342    2577 generic.go:356] "Generic (PLEG): container finished" podID="83f44b93-4402-4869-91ec-6fd23c938443" containerID="7b4ce9377d3cd59952c0dec7066b62da90ceaa80018a94755c3f411ae9810f4e" exitCode=0
Sep 18 19:17:28.408781 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.408347    2577 generic.go:356] "Generic (PLEG): container finished" podID="83f44b93-4402-4869-91ec-6fd23c938443" containerID="623c43a362fdd42fbd934d3e390aff7323d886b56800a72f871d9496a91c93a9" exitCode=0
Sep 18 19:17:28.408781 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.408443    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.408781 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.408478    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83f44b93-4402-4869-91ec-6fd23c938443","Type":"ContainerDied","Data":"5514363ae6531cf5e1818a035dcd73688ca2f21dbd973fed1f2355c9906efa75"}
Sep 18 19:17:28.408781 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.408570    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83f44b93-4402-4869-91ec-6fd23c938443","Type":"ContainerDied","Data":"db7145413295c0ef5d2d57f4e8fb8aeb0e9b3aef0dad53f21664a7e6abb4c5c2"}
Sep 18 19:17:28.408781 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.408618    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83f44b93-4402-4869-91ec-6fd23c938443","Type":"ContainerDied","Data":"7c509504133689d7581b3cfacdb76fe7f180d8f6424bdf3f525e32d1eca275a3"}
Sep 18 19:17:28.408781 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.408639    2577 scope.go:117] "RemoveContainer" containerID="5514363ae6531cf5e1818a035dcd73688ca2f21dbd973fed1f2355c9906efa75"
Sep 18 19:17:28.408781 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.408665    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83f44b93-4402-4869-91ec-6fd23c938443","Type":"ContainerDied","Data":"3233e21243a7da879fd5deee2efa5c041a24efa428c4cc29ae956b85e3f64ed8"}
Sep 18 19:17:28.408781 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.408705    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83f44b93-4402-4869-91ec-6fd23c938443","Type":"ContainerDied","Data":"7b4ce9377d3cd59952c0dec7066b62da90ceaa80018a94755c3f411ae9810f4e"}
Sep 18 19:17:28.408781 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.408746    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83f44b93-4402-4869-91ec-6fd23c938443","Type":"ContainerDied","Data":"623c43a362fdd42fbd934d3e390aff7323d886b56800a72f871d9496a91c93a9"}
Sep 18 19:17:28.409269 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.408792    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83f44b93-4402-4869-91ec-6fd23c938443","Type":"ContainerDied","Data":"cabc9e21b6a0d68f5743f0527cf4ef8c97aa678f6212b4e7724edb2ab2cd91f7"}
Sep 18 19:17:28.416102 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.416081    2577 scope.go:117] "RemoveContainer" containerID="db7145413295c0ef5d2d57f4e8fb8aeb0e9b3aef0dad53f21664a7e6abb4c5c2"
Sep 18 19:17:28.424204 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.424187    2577 scope.go:117] "RemoveContainer" containerID="7c509504133689d7581b3cfacdb76fe7f180d8f6424bdf3f525e32d1eca275a3"
Sep 18 19:17:28.430756 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.430739    2577 scope.go:117] "RemoveContainer" containerID="3233e21243a7da879fd5deee2efa5c041a24efa428c4cc29ae956b85e3f64ed8"
Sep 18 19:17:28.437634 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.437617    2577 scope.go:117] "RemoveContainer" containerID="7b4ce9377d3cd59952c0dec7066b62da90ceaa80018a94755c3f411ae9810f4e"
Sep 18 19:17:28.438946 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.438929    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 19:17:28.444700 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.444678    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 19:17:28.445343 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.445328    2577 scope.go:117] "RemoveContainer" containerID="623c43a362fdd42fbd934d3e390aff7323d886b56800a72f871d9496a91c93a9"
Sep 18 19:17:28.452120 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.452105    2577 scope.go:117] "RemoveContainer" containerID="1c935190ec508e926e04c5b084b4581799040e7bbe94ce5e5b72d36076e64daa"
Sep 18 19:17:28.459782 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.459756    2577 scope.go:117] "RemoveContainer" containerID="5514363ae6531cf5e1818a035dcd73688ca2f21dbd973fed1f2355c9906efa75"
Sep 18 19:17:28.460045 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:17:28.460027    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5514363ae6531cf5e1818a035dcd73688ca2f21dbd973fed1f2355c9906efa75\": container with ID starting with 5514363ae6531cf5e1818a035dcd73688ca2f21dbd973fed1f2355c9906efa75 not found: ID does not exist" containerID="5514363ae6531cf5e1818a035dcd73688ca2f21dbd973fed1f2355c9906efa75"
Sep 18 19:17:28.460088 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.460055    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5514363ae6531cf5e1818a035dcd73688ca2f21dbd973fed1f2355c9906efa75"} err="failed to get container status \"5514363ae6531cf5e1818a035dcd73688ca2f21dbd973fed1f2355c9906efa75\": rpc error: code = NotFound desc = could not find container \"5514363ae6531cf5e1818a035dcd73688ca2f21dbd973fed1f2355c9906efa75\": container with ID starting with 5514363ae6531cf5e1818a035dcd73688ca2f21dbd973fed1f2355c9906efa75 not found: ID does not exist"
Sep 18 19:17:28.460088 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.460073    2577 scope.go:117] "RemoveContainer" containerID="db7145413295c0ef5d2d57f4e8fb8aeb0e9b3aef0dad53f21664a7e6abb4c5c2"
Sep 18 19:17:28.460328 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:17:28.460313    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db7145413295c0ef5d2d57f4e8fb8aeb0e9b3aef0dad53f21664a7e6abb4c5c2\": container with ID starting with db7145413295c0ef5d2d57f4e8fb8aeb0e9b3aef0dad53f21664a7e6abb4c5c2 not found: ID does not exist" containerID="db7145413295c0ef5d2d57f4e8fb8aeb0e9b3aef0dad53f21664a7e6abb4c5c2"
Sep 18 19:17:28.460371 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.460331    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db7145413295c0ef5d2d57f4e8fb8aeb0e9b3aef0dad53f21664a7e6abb4c5c2"} err="failed to get container status \"db7145413295c0ef5d2d57f4e8fb8aeb0e9b3aef0dad53f21664a7e6abb4c5c2\": rpc error: code = NotFound desc = could not find container \"db7145413295c0ef5d2d57f4e8fb8aeb0e9b3aef0dad53f21664a7e6abb4c5c2\": container with ID starting with db7145413295c0ef5d2d57f4e8fb8aeb0e9b3aef0dad53f21664a7e6abb4c5c2 not found: ID does not exist"
Sep 18 19:17:28.460371 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.460344    2577 scope.go:117] "RemoveContainer" containerID="7c509504133689d7581b3cfacdb76fe7f180d8f6424bdf3f525e32d1eca275a3"
Sep 18 19:17:28.460589 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:17:28.460571    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c509504133689d7581b3cfacdb76fe7f180d8f6424bdf3f525e32d1eca275a3\": container with ID starting with 7c509504133689d7581b3cfacdb76fe7f180d8f6424bdf3f525e32d1eca275a3 not found: ID does not exist" containerID="7c509504133689d7581b3cfacdb76fe7f180d8f6424bdf3f525e32d1eca275a3"
Sep 18 19:17:28.460656 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.460599    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c509504133689d7581b3cfacdb76fe7f180d8f6424bdf3f525e32d1eca275a3"} err="failed to get container status \"7c509504133689d7581b3cfacdb76fe7f180d8f6424bdf3f525e32d1eca275a3\": rpc error: code = NotFound desc = could not find container \"7c509504133689d7581b3cfacdb76fe7f180d8f6424bdf3f525e32d1eca275a3\": container with ID starting with 7c509504133689d7581b3cfacdb76fe7f180d8f6424bdf3f525e32d1eca275a3 not found: ID does not exist"
Sep 18 19:17:28.460656 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.460621    2577 scope.go:117] "RemoveContainer" containerID="3233e21243a7da879fd5deee2efa5c041a24efa428c4cc29ae956b85e3f64ed8"
Sep 18 19:17:28.460889 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:17:28.460873    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3233e21243a7da879fd5deee2efa5c041a24efa428c4cc29ae956b85e3f64ed8\": container with ID starting with 3233e21243a7da879fd5deee2efa5c041a24efa428c4cc29ae956b85e3f64ed8 not found: ID does not exist" containerID="3233e21243a7da879fd5deee2efa5c041a24efa428c4cc29ae956b85e3f64ed8"
Sep 18 19:17:28.460923 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.460893    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3233e21243a7da879fd5deee2efa5c041a24efa428c4cc29ae956b85e3f64ed8"} err="failed to get container status \"3233e21243a7da879fd5deee2efa5c041a24efa428c4cc29ae956b85e3f64ed8\": rpc error: code = NotFound desc = could not find container \"3233e21243a7da879fd5deee2efa5c041a24efa428c4cc29ae956b85e3f64ed8\": container with ID starting with 3233e21243a7da879fd5deee2efa5c041a24efa428c4cc29ae956b85e3f64ed8 not found: ID does not exist"
Sep 18 19:17:28.460923 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.460906    2577 scope.go:117] "RemoveContainer" containerID="7b4ce9377d3cd59952c0dec7066b62da90ceaa80018a94755c3f411ae9810f4e"
Sep 18 19:17:28.461145 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:17:28.461129    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b4ce9377d3cd59952c0dec7066b62da90ceaa80018a94755c3f411ae9810f4e\": container with ID starting with 7b4ce9377d3cd59952c0dec7066b62da90ceaa80018a94755c3f411ae9810f4e not found: ID does not exist" containerID="7b4ce9377d3cd59952c0dec7066b62da90ceaa80018a94755c3f411ae9810f4e"
Sep 18 19:17:28.461204 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.461152    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b4ce9377d3cd59952c0dec7066b62da90ceaa80018a94755c3f411ae9810f4e"} err="failed to get container status \"7b4ce9377d3cd59952c0dec7066b62da90ceaa80018a94755c3f411ae9810f4e\": rpc error: code = NotFound desc = could not find container \"7b4ce9377d3cd59952c0dec7066b62da90ceaa80018a94755c3f411ae9810f4e\": container with ID starting with 7b4ce9377d3cd59952c0dec7066b62da90ceaa80018a94755c3f411ae9810f4e not found: ID does not exist"
Sep 18 19:17:28.461204 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.461170    2577 scope.go:117] "RemoveContainer" containerID="623c43a362fdd42fbd934d3e390aff7323d886b56800a72f871d9496a91c93a9"
Sep 18 19:17:28.461428 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:17:28.461412    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"623c43a362fdd42fbd934d3e390aff7323d886b56800a72f871d9496a91c93a9\": container with ID starting with 623c43a362fdd42fbd934d3e390aff7323d886b56800a72f871d9496a91c93a9 not found: ID does not exist" containerID="623c43a362fdd42fbd934d3e390aff7323d886b56800a72f871d9496a91c93a9"
Sep 18 19:17:28.461463 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.461431    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"623c43a362fdd42fbd934d3e390aff7323d886b56800a72f871d9496a91c93a9"} err="failed to get container status \"623c43a362fdd42fbd934d3e390aff7323d886b56800a72f871d9496a91c93a9\": rpc error: code = NotFound desc = could not find container \"623c43a362fdd42fbd934d3e390aff7323d886b56800a72f871d9496a91c93a9\": container with ID starting with 623c43a362fdd42fbd934d3e390aff7323d886b56800a72f871d9496a91c93a9 not found: ID does not exist"
Sep 18 19:17:28.461463 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.461444    2577 scope.go:117] "RemoveContainer" containerID="1c935190ec508e926e04c5b084b4581799040e7bbe94ce5e5b72d36076e64daa"
Sep 18 19:17:28.461658 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:17:28.461641    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c935190ec508e926e04c5b084b4581799040e7bbe94ce5e5b72d36076e64daa\": container with ID starting with 1c935190ec508e926e04c5b084b4581799040e7bbe94ce5e5b72d36076e64daa not found: ID does not exist" containerID="1c935190ec508e926e04c5b084b4581799040e7bbe94ce5e5b72d36076e64daa"
Sep 18 19:17:28.461700 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.461663    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c935190ec508e926e04c5b084b4581799040e7bbe94ce5e5b72d36076e64daa"} err="failed to get container status \"1c935190ec508e926e04c5b084b4581799040e7bbe94ce5e5b72d36076e64daa\": rpc error: code = NotFound desc = could not find container \"1c935190ec508e926e04c5b084b4581799040e7bbe94ce5e5b72d36076e64daa\": container with ID starting with 1c935190ec508e926e04c5b084b4581799040e7bbe94ce5e5b72d36076e64daa not found: ID does not exist"
Sep 18 19:17:28.461700 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.461678    2577 scope.go:117] "RemoveContainer" containerID="5514363ae6531cf5e1818a035dcd73688ca2f21dbd973fed1f2355c9906efa75"
Sep 18 19:17:28.461889 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.461869    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5514363ae6531cf5e1818a035dcd73688ca2f21dbd973fed1f2355c9906efa75"} err="failed to get container status \"5514363ae6531cf5e1818a035dcd73688ca2f21dbd973fed1f2355c9906efa75\": rpc error: code = NotFound desc = could not find container \"5514363ae6531cf5e1818a035dcd73688ca2f21dbd973fed1f2355c9906efa75\": container with ID starting with 5514363ae6531cf5e1818a035dcd73688ca2f21dbd973fed1f2355c9906efa75 not found: ID does not exist"
Sep 18 19:17:28.461889 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.461889    2577 scope.go:117] "RemoveContainer" containerID="db7145413295c0ef5d2d57f4e8fb8aeb0e9b3aef0dad53f21664a7e6abb4c5c2"
Sep 18 19:17:28.462122 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.462105    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db7145413295c0ef5d2d57f4e8fb8aeb0e9b3aef0dad53f21664a7e6abb4c5c2"} err="failed to get container status \"db7145413295c0ef5d2d57f4e8fb8aeb0e9b3aef0dad53f21664a7e6abb4c5c2\": rpc error: code = NotFound desc = could not find container \"db7145413295c0ef5d2d57f4e8fb8aeb0e9b3aef0dad53f21664a7e6abb4c5c2\": container with ID starting with db7145413295c0ef5d2d57f4e8fb8aeb0e9b3aef0dad53f21664a7e6abb4c5c2 not found: ID does not exist"
Sep 18 19:17:28.462122 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.462122    2577 scope.go:117] "RemoveContainer" containerID="7c509504133689d7581b3cfacdb76fe7f180d8f6424bdf3f525e32d1eca275a3"
Sep 18 19:17:28.462350 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.462330    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c509504133689d7581b3cfacdb76fe7f180d8f6424bdf3f525e32d1eca275a3"} err="failed to get container status \"7c509504133689d7581b3cfacdb76fe7f180d8f6424bdf3f525e32d1eca275a3\": rpc error: code = NotFound desc = could not find container \"7c509504133689d7581b3cfacdb76fe7f180d8f6424bdf3f525e32d1eca275a3\": container with ID starting with 7c509504133689d7581b3cfacdb76fe7f180d8f6424bdf3f525e32d1eca275a3 not found: ID does not exist"
Sep 18 19:17:28.462350 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.462349    2577 scope.go:117] "RemoveContainer" containerID="3233e21243a7da879fd5deee2efa5c041a24efa428c4cc29ae956b85e3f64ed8"
Sep 18 19:17:28.462590 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.462572    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3233e21243a7da879fd5deee2efa5c041a24efa428c4cc29ae956b85e3f64ed8"} err="failed to get container status \"3233e21243a7da879fd5deee2efa5c041a24efa428c4cc29ae956b85e3f64ed8\": rpc error: code = NotFound desc = could not find container \"3233e21243a7da879fd5deee2efa5c041a24efa428c4cc29ae956b85e3f64ed8\": container with ID starting with 3233e21243a7da879fd5deee2efa5c041a24efa428c4cc29ae956b85e3f64ed8 not found: ID does not exist"
Sep 18 19:17:28.462631 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.462592    2577 scope.go:117] "RemoveContainer" containerID="7b4ce9377d3cd59952c0dec7066b62da90ceaa80018a94755c3f411ae9810f4e"
Sep 18 19:17:28.462806 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.462791    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b4ce9377d3cd59952c0dec7066b62da90ceaa80018a94755c3f411ae9810f4e"} err="failed to get container status \"7b4ce9377d3cd59952c0dec7066b62da90ceaa80018a94755c3f411ae9810f4e\": rpc error: code = NotFound desc = could not find container \"7b4ce9377d3cd59952c0dec7066b62da90ceaa80018a94755c3f411ae9810f4e\": container with ID starting with 7b4ce9377d3cd59952c0dec7066b62da90ceaa80018a94755c3f411ae9810f4e not found: ID does not exist"
Sep 18 19:17:28.462857 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.462808    2577 scope.go:117] "RemoveContainer" containerID="623c43a362fdd42fbd934d3e390aff7323d886b56800a72f871d9496a91c93a9"
Sep 18 19:17:28.463051 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.463034    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"623c43a362fdd42fbd934d3e390aff7323d886b56800a72f871d9496a91c93a9"} err="failed to get container status \"623c43a362fdd42fbd934d3e390aff7323d886b56800a72f871d9496a91c93a9\": rpc error: code = NotFound desc = could not find container \"623c43a362fdd42fbd934d3e390aff7323d886b56800a72f871d9496a91c93a9\": container with ID starting with 623c43a362fdd42fbd934d3e390aff7323d886b56800a72f871d9496a91c93a9 not found: ID does not exist"
Sep 18 19:17:28.463051 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.463050    2577 scope.go:117] "RemoveContainer" containerID="1c935190ec508e926e04c5b084b4581799040e7bbe94ce5e5b72d36076e64daa"
Sep 18 19:17:28.465703 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.463596    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c935190ec508e926e04c5b084b4581799040e7bbe94ce5e5b72d36076e64daa"} err="failed to get container status \"1c935190ec508e926e04c5b084b4581799040e7bbe94ce5e5b72d36076e64daa\": rpc error: code = NotFound desc = could not find container \"1c935190ec508e926e04c5b084b4581799040e7bbe94ce5e5b72d36076e64daa\": container with ID starting with 1c935190ec508e926e04c5b084b4581799040e7bbe94ce5e5b72d36076e64daa not found: ID does not exist"
Sep 18 19:17:28.465703 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.463618    2577 scope.go:117] "RemoveContainer" containerID="5514363ae6531cf5e1818a035dcd73688ca2f21dbd973fed1f2355c9906efa75"
Sep 18 19:17:28.465703 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.463921    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5514363ae6531cf5e1818a035dcd73688ca2f21dbd973fed1f2355c9906efa75"} err="failed to get container status \"5514363ae6531cf5e1818a035dcd73688ca2f21dbd973fed1f2355c9906efa75\": rpc error: code = NotFound desc = could not find container \"5514363ae6531cf5e1818a035dcd73688ca2f21dbd973fed1f2355c9906efa75\": container with ID starting with 5514363ae6531cf5e1818a035dcd73688ca2f21dbd973fed1f2355c9906efa75 not found: ID does not exist"
Sep 18 19:17:28.465703 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.463948    2577 scope.go:117] "RemoveContainer" containerID="db7145413295c0ef5d2d57f4e8fb8aeb0e9b3aef0dad53f21664a7e6abb4c5c2"
Sep 18 19:17:28.465703 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.464274    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db7145413295c0ef5d2d57f4e8fb8aeb0e9b3aef0dad53f21664a7e6abb4c5c2"} err="failed to get container status \"db7145413295c0ef5d2d57f4e8fb8aeb0e9b3aef0dad53f21664a7e6abb4c5c2\": rpc error: code = NotFound desc = could not find container \"db7145413295c0ef5d2d57f4e8fb8aeb0e9b3aef0dad53f21664a7e6abb4c5c2\": container with ID starting with db7145413295c0ef5d2d57f4e8fb8aeb0e9b3aef0dad53f21664a7e6abb4c5c2 not found: ID does not exist"
Sep 18 19:17:28.465703 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.464293    2577 scope.go:117] "RemoveContainer" containerID="7c509504133689d7581b3cfacdb76fe7f180d8f6424bdf3f525e32d1eca275a3"
Sep 18 19:17:28.465703 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.464585    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c509504133689d7581b3cfacdb76fe7f180d8f6424bdf3f525e32d1eca275a3"} err="failed to get container status \"7c509504133689d7581b3cfacdb76fe7f180d8f6424bdf3f525e32d1eca275a3\": rpc error: code = NotFound desc = could not find container \"7c509504133689d7581b3cfacdb76fe7f180d8f6424bdf3f525e32d1eca275a3\": container with ID starting with 7c509504133689d7581b3cfacdb76fe7f180d8f6424bdf3f525e32d1eca275a3 not found: ID does not exist"
Sep 18 19:17:28.465703 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.464603    2577 scope.go:117] "RemoveContainer" containerID="3233e21243a7da879fd5deee2efa5c041a24efa428c4cc29ae956b85e3f64ed8"
Sep 18 19:17:28.465703 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.464874    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3233e21243a7da879fd5deee2efa5c041a24efa428c4cc29ae956b85e3f64ed8"} err="failed to get container status \"3233e21243a7da879fd5deee2efa5c041a24efa428c4cc29ae956b85e3f64ed8\": rpc error: code = NotFound desc = could not find container \"3233e21243a7da879fd5deee2efa5c041a24efa428c4cc29ae956b85e3f64ed8\": container with ID starting with 3233e21243a7da879fd5deee2efa5c041a24efa428c4cc29ae956b85e3f64ed8 not found: ID does not exist"
Sep 18 19:17:28.465703 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.464892    2577 scope.go:117] "RemoveContainer" containerID="7b4ce9377d3cd59952c0dec7066b62da90ceaa80018a94755c3f411ae9810f4e"
Sep 18 19:17:28.465703 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.465291    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b4ce9377d3cd59952c0dec7066b62da90ceaa80018a94755c3f411ae9810f4e"} err="failed to get container status \"7b4ce9377d3cd59952c0dec7066b62da90ceaa80018a94755c3f411ae9810f4e\": rpc error: code = NotFound desc = could not find container \"7b4ce9377d3cd59952c0dec7066b62da90ceaa80018a94755c3f411ae9810f4e\": container with ID starting with 7b4ce9377d3cd59952c0dec7066b62da90ceaa80018a94755c3f411ae9810f4e not found: ID does not exist"
Sep 18 19:17:28.465703 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.465312    2577 scope.go:117] "RemoveContainer" containerID="623c43a362fdd42fbd934d3e390aff7323d886b56800a72f871d9496a91c93a9"
Sep 18 19:17:28.465703 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.465625    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"623c43a362fdd42fbd934d3e390aff7323d886b56800a72f871d9496a91c93a9"} err="failed to get container status \"623c43a362fdd42fbd934d3e390aff7323d886b56800a72f871d9496a91c93a9\": rpc error: code = NotFound desc = could not find container \"623c43a362fdd42fbd934d3e390aff7323d886b56800a72f871d9496a91c93a9\": container with ID starting with 623c43a362fdd42fbd934d3e390aff7323d886b56800a72f871d9496a91c93a9 not found: ID does not exist"
Sep 18 19:17:28.465703 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.465643    2577 scope.go:117] "RemoveContainer" containerID="1c935190ec508e926e04c5b084b4581799040e7bbe94ce5e5b72d36076e64daa"
Sep 18 19:17:28.466335 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.466208    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c935190ec508e926e04c5b084b4581799040e7bbe94ce5e5b72d36076e64daa"} err="failed to get container status \"1c935190ec508e926e04c5b084b4581799040e7bbe94ce5e5b72d36076e64daa\": rpc error: code = NotFound desc = could not find container \"1c935190ec508e926e04c5b084b4581799040e7bbe94ce5e5b72d36076e64daa\": container with ID starting with 1c935190ec508e926e04c5b084b4581799040e7bbe94ce5e5b72d36076e64daa not found: ID does not exist"
Sep 18 19:17:28.466335 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.466230    2577 scope.go:117] "RemoveContainer" containerID="5514363ae6531cf5e1818a035dcd73688ca2f21dbd973fed1f2355c9906efa75"
Sep 18 19:17:28.466475 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.466457    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5514363ae6531cf5e1818a035dcd73688ca2f21dbd973fed1f2355c9906efa75"} err="failed to get container status \"5514363ae6531cf5e1818a035dcd73688ca2f21dbd973fed1f2355c9906efa75\": rpc error: code = NotFound desc = could not find container \"5514363ae6531cf5e1818a035dcd73688ca2f21dbd973fed1f2355c9906efa75\": container with ID starting with 5514363ae6531cf5e1818a035dcd73688ca2f21dbd973fed1f2355c9906efa75 not found: ID does not exist"
Sep 18 19:17:28.466520 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.466475    2577 scope.go:117] "RemoveContainer" containerID="db7145413295c0ef5d2d57f4e8fb8aeb0e9b3aef0dad53f21664a7e6abb4c5c2"
Sep 18 19:17:28.466698 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.466678    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db7145413295c0ef5d2d57f4e8fb8aeb0e9b3aef0dad53f21664a7e6abb4c5c2"} err="failed to get container status \"db7145413295c0ef5d2d57f4e8fb8aeb0e9b3aef0dad53f21664a7e6abb4c5c2\": rpc error: code = NotFound desc = could not find container \"db7145413295c0ef5d2d57f4e8fb8aeb0e9b3aef0dad53f21664a7e6abb4c5c2\": container with ID starting with db7145413295c0ef5d2d57f4e8fb8aeb0e9b3aef0dad53f21664a7e6abb4c5c2 not found: ID does not exist"
Sep 18 19:17:28.466698 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.466697    2577 scope.go:117] "RemoveContainer" containerID="7c509504133689d7581b3cfacdb76fe7f180d8f6424bdf3f525e32d1eca275a3"
Sep 18 19:17:28.466906 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.466888    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c509504133689d7581b3cfacdb76fe7f180d8f6424bdf3f525e32d1eca275a3"} err="failed to get container status \"7c509504133689d7581b3cfacdb76fe7f180d8f6424bdf3f525e32d1eca275a3\": rpc error: code = NotFound desc = could not find container \"7c509504133689d7581b3cfacdb76fe7f180d8f6424bdf3f525e32d1eca275a3\": container with ID starting with 7c509504133689d7581b3cfacdb76fe7f180d8f6424bdf3f525e32d1eca275a3 not found: ID does not exist"
Sep 18 19:17:28.466951 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.466907    2577 scope.go:117] "RemoveContainer" containerID="3233e21243a7da879fd5deee2efa5c041a24efa428c4cc29ae956b85e3f64ed8"
Sep 18 19:17:28.467142 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.467126    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3233e21243a7da879fd5deee2efa5c041a24efa428c4cc29ae956b85e3f64ed8"} err="failed to get container status \"3233e21243a7da879fd5deee2efa5c041a24efa428c4cc29ae956b85e3f64ed8\": rpc error: code = NotFound desc = could not find container \"3233e21243a7da879fd5deee2efa5c041a24efa428c4cc29ae956b85e3f64ed8\": container with ID starting with 3233e21243a7da879fd5deee2efa5c041a24efa428c4cc29ae956b85e3f64ed8 not found: ID does not exist"
Sep 18 19:17:28.467195 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.467142    2577 scope.go:117] "RemoveContainer" containerID="7b4ce9377d3cd59952c0dec7066b62da90ceaa80018a94755c3f411ae9810f4e"
Sep 18 19:17:28.467375 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.467357    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b4ce9377d3cd59952c0dec7066b62da90ceaa80018a94755c3f411ae9810f4e"} err="failed to get container status \"7b4ce9377d3cd59952c0dec7066b62da90ceaa80018a94755c3f411ae9810f4e\": rpc error: code = NotFound desc = could not find container \"7b4ce9377d3cd59952c0dec7066b62da90ceaa80018a94755c3f411ae9810f4e\": container with ID starting with 7b4ce9377d3cd59952c0dec7066b62da90ceaa80018a94755c3f411ae9810f4e not found: ID does not exist"
Sep 18 19:17:28.467443 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.467376    2577 scope.go:117] "RemoveContainer" containerID="623c43a362fdd42fbd934d3e390aff7323d886b56800a72f871d9496a91c93a9"
Sep 18 19:17:28.467603 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.467586    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"623c43a362fdd42fbd934d3e390aff7323d886b56800a72f871d9496a91c93a9"} err="failed to get container status \"623c43a362fdd42fbd934d3e390aff7323d886b56800a72f871d9496a91c93a9\": rpc error: code = NotFound desc = could not find container \"623c43a362fdd42fbd934d3e390aff7323d886b56800a72f871d9496a91c93a9\": container with ID starting with 623c43a362fdd42fbd934d3e390aff7323d886b56800a72f871d9496a91c93a9 not found: ID does not exist"
Sep 18 19:17:28.467652 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.467603    2577 scope.go:117] "RemoveContainer" containerID="1c935190ec508e926e04c5b084b4581799040e7bbe94ce5e5b72d36076e64daa"
Sep 18 19:17:28.467834 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.467817    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c935190ec508e926e04c5b084b4581799040e7bbe94ce5e5b72d36076e64daa"} err="failed to get container status \"1c935190ec508e926e04c5b084b4581799040e7bbe94ce5e5b72d36076e64daa\": rpc error: code = NotFound desc = could not find container \"1c935190ec508e926e04c5b084b4581799040e7bbe94ce5e5b72d36076e64daa\": container with ID starting with 1c935190ec508e926e04c5b084b4581799040e7bbe94ce5e5b72d36076e64daa not found: ID does not exist"
Sep 18 19:17:28.467879 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.467835    2577 scope.go:117] "RemoveContainer" containerID="5514363ae6531cf5e1818a035dcd73688ca2f21dbd973fed1f2355c9906efa75"
Sep 18 19:17:28.468064 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.468042    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5514363ae6531cf5e1818a035dcd73688ca2f21dbd973fed1f2355c9906efa75"} err="failed to get container status \"5514363ae6531cf5e1818a035dcd73688ca2f21dbd973fed1f2355c9906efa75\": rpc error: code = NotFound desc = could not find container \"5514363ae6531cf5e1818a035dcd73688ca2f21dbd973fed1f2355c9906efa75\": container with ID starting with 5514363ae6531cf5e1818a035dcd73688ca2f21dbd973fed1f2355c9906efa75 not found: ID does not exist"
Sep 18 19:17:28.468064 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.468063    2577 scope.go:117] "RemoveContainer" containerID="db7145413295c0ef5d2d57f4e8fb8aeb0e9b3aef0dad53f21664a7e6abb4c5c2"
Sep 18 19:17:28.468277 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.468260    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db7145413295c0ef5d2d57f4e8fb8aeb0e9b3aef0dad53f21664a7e6abb4c5c2"} err="failed to get container status \"db7145413295c0ef5d2d57f4e8fb8aeb0e9b3aef0dad53f21664a7e6abb4c5c2\": rpc error: code = NotFound desc = could not find container \"db7145413295c0ef5d2d57f4e8fb8aeb0e9b3aef0dad53f21664a7e6abb4c5c2\": container with ID starting with db7145413295c0ef5d2d57f4e8fb8aeb0e9b3aef0dad53f21664a7e6abb4c5c2 not found: ID does not exist"
Sep 18 19:17:28.468318 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.468277    2577 scope.go:117] "RemoveContainer" containerID="7c509504133689d7581b3cfacdb76fe7f180d8f6424bdf3f525e32d1eca275a3"
Sep 18 19:17:28.468468 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.468452    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c509504133689d7581b3cfacdb76fe7f180d8f6424bdf3f525e32d1eca275a3"} err="failed to get container status \"7c509504133689d7581b3cfacdb76fe7f180d8f6424bdf3f525e32d1eca275a3\": rpc error: code = NotFound desc = could not find container \"7c509504133689d7581b3cfacdb76fe7f180d8f6424bdf3f525e32d1eca275a3\": container with ID starting with 7c509504133689d7581b3cfacdb76fe7f180d8f6424bdf3f525e32d1eca275a3 not found: ID does not exist"
Sep 18 19:17:28.468510 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.468468    2577 scope.go:117] "RemoveContainer" containerID="3233e21243a7da879fd5deee2efa5c041a24efa428c4cc29ae956b85e3f64ed8"
Sep 18 19:17:28.468692 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.468675    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3233e21243a7da879fd5deee2efa5c041a24efa428c4cc29ae956b85e3f64ed8"} err="failed to get container status \"3233e21243a7da879fd5deee2efa5c041a24efa428c4cc29ae956b85e3f64ed8\": rpc error: code = NotFound desc = could not find container \"3233e21243a7da879fd5deee2efa5c041a24efa428c4cc29ae956b85e3f64ed8\": container with ID starting with 3233e21243a7da879fd5deee2efa5c041a24efa428c4cc29ae956b85e3f64ed8 not found: ID does not exist"
Sep 18 19:17:28.468692 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.468691    2577 scope.go:117] "RemoveContainer" containerID="7b4ce9377d3cd59952c0dec7066b62da90ceaa80018a94755c3f411ae9810f4e"
Sep 18 19:17:28.468882 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.468866    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b4ce9377d3cd59952c0dec7066b62da90ceaa80018a94755c3f411ae9810f4e"} err="failed to get container status \"7b4ce9377d3cd59952c0dec7066b62da90ceaa80018a94755c3f411ae9810f4e\": rpc error: code = NotFound desc = could not find container \"7b4ce9377d3cd59952c0dec7066b62da90ceaa80018a94755c3f411ae9810f4e\": container with ID starting with 7b4ce9377d3cd59952c0dec7066b62da90ceaa80018a94755c3f411ae9810f4e not found: ID does not exist"
Sep 18 19:17:28.468924 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.468882    2577 scope.go:117] "RemoveContainer" containerID="623c43a362fdd42fbd934d3e390aff7323d886b56800a72f871d9496a91c93a9"
Sep 18 19:17:28.469089 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.469073    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"623c43a362fdd42fbd934d3e390aff7323d886b56800a72f871d9496a91c93a9"} err="failed to get container status \"623c43a362fdd42fbd934d3e390aff7323d886b56800a72f871d9496a91c93a9\": rpc error: code = NotFound desc = could not find container \"623c43a362fdd42fbd934d3e390aff7323d886b56800a72f871d9496a91c93a9\": container with ID starting with 623c43a362fdd42fbd934d3e390aff7323d886b56800a72f871d9496a91c93a9 not found: ID does not exist"
Sep 18 19:17:28.469153 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.469089    2577 scope.go:117] "RemoveContainer" containerID="1c935190ec508e926e04c5b084b4581799040e7bbe94ce5e5b72d36076e64daa"
Sep 18 19:17:28.469318 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.469303    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c935190ec508e926e04c5b084b4581799040e7bbe94ce5e5b72d36076e64daa"} err="failed to get container status \"1c935190ec508e926e04c5b084b4581799040e7bbe94ce5e5b72d36076e64daa\": rpc error: code = NotFound desc = could not find container \"1c935190ec508e926e04c5b084b4581799040e7bbe94ce5e5b72d36076e64daa\": container with ID starting with 1c935190ec508e926e04c5b084b4581799040e7bbe94ce5e5b72d36076e64daa not found: ID does not exist"
Sep 18 19:17:28.469318 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.469317    2577 scope.go:117] "RemoveContainer" containerID="5514363ae6531cf5e1818a035dcd73688ca2f21dbd973fed1f2355c9906efa75"
Sep 18 19:17:28.469507 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.469492    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5514363ae6531cf5e1818a035dcd73688ca2f21dbd973fed1f2355c9906efa75"} err="failed to get container status \"5514363ae6531cf5e1818a035dcd73688ca2f21dbd973fed1f2355c9906efa75\": rpc error: code = NotFound desc = could not find container \"5514363ae6531cf5e1818a035dcd73688ca2f21dbd973fed1f2355c9906efa75\": container with ID starting with 5514363ae6531cf5e1818a035dcd73688ca2f21dbd973fed1f2355c9906efa75 not found: ID does not exist"
Sep 18 19:17:28.469583 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.469508    2577 scope.go:117] "RemoveContainer" containerID="db7145413295c0ef5d2d57f4e8fb8aeb0e9b3aef0dad53f21664a7e6abb4c5c2"
Sep 18 19:17:28.469717 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.469703    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db7145413295c0ef5d2d57f4e8fb8aeb0e9b3aef0dad53f21664a7e6abb4c5c2"} err="failed to get container status \"db7145413295c0ef5d2d57f4e8fb8aeb0e9b3aef0dad53f21664a7e6abb4c5c2\": rpc error: code = NotFound desc = could not find container \"db7145413295c0ef5d2d57f4e8fb8aeb0e9b3aef0dad53f21664a7e6abb4c5c2\": container with ID starting with db7145413295c0ef5d2d57f4e8fb8aeb0e9b3aef0dad53f21664a7e6abb4c5c2 not found: ID does not exist"
Sep 18 19:17:28.469717 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.469716    2577 scope.go:117] "RemoveContainer" containerID="7c509504133689d7581b3cfacdb76fe7f180d8f6424bdf3f525e32d1eca275a3"
Sep 18 19:17:28.469894 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.469878    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c509504133689d7581b3cfacdb76fe7f180d8f6424bdf3f525e32d1eca275a3"} err="failed to get container status \"7c509504133689d7581b3cfacdb76fe7f180d8f6424bdf3f525e32d1eca275a3\": rpc error: code = NotFound desc = could not find container \"7c509504133689d7581b3cfacdb76fe7f180d8f6424bdf3f525e32d1eca275a3\": container with ID starting with 7c509504133689d7581b3cfacdb76fe7f180d8f6424bdf3f525e32d1eca275a3 not found: ID does not exist"
Sep 18 19:17:28.469936 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.469895    2577 scope.go:117] "RemoveContainer" containerID="3233e21243a7da879fd5deee2efa5c041a24efa428c4cc29ae956b85e3f64ed8"
Sep 18 19:17:28.470098 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.470076    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3233e21243a7da879fd5deee2efa5c041a24efa428c4cc29ae956b85e3f64ed8"} err="failed to get container status \"3233e21243a7da879fd5deee2efa5c041a24efa428c4cc29ae956b85e3f64ed8\": rpc error: code = NotFound desc = could not find container \"3233e21243a7da879fd5deee2efa5c041a24efa428c4cc29ae956b85e3f64ed8\": container with ID starting with 3233e21243a7da879fd5deee2efa5c041a24efa428c4cc29ae956b85e3f64ed8 not found: ID does not exist"
Sep 18 19:17:28.470098 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.470098    2577 scope.go:117] "RemoveContainer" containerID="7b4ce9377d3cd59952c0dec7066b62da90ceaa80018a94755c3f411ae9810f4e"
Sep 18 19:17:28.470348 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.470331    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b4ce9377d3cd59952c0dec7066b62da90ceaa80018a94755c3f411ae9810f4e"} err="failed to get container status \"7b4ce9377d3cd59952c0dec7066b62da90ceaa80018a94755c3f411ae9810f4e\": rpc error: code = NotFound desc = could not find container \"7b4ce9377d3cd59952c0dec7066b62da90ceaa80018a94755c3f411ae9810f4e\": container with ID starting with 7b4ce9377d3cd59952c0dec7066b62da90ceaa80018a94755c3f411ae9810f4e not found: ID does not exist"
Sep 18 19:17:28.470348 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.470349    2577 scope.go:117] "RemoveContainer" containerID="623c43a362fdd42fbd934d3e390aff7323d886b56800a72f871d9496a91c93a9"
Sep 18 19:17:28.470550 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.470534    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"623c43a362fdd42fbd934d3e390aff7323d886b56800a72f871d9496a91c93a9"} err="failed to get container status \"623c43a362fdd42fbd934d3e390aff7323d886b56800a72f871d9496a91c93a9\": rpc error: code = NotFound desc = could not find container \"623c43a362fdd42fbd934d3e390aff7323d886b56800a72f871d9496a91c93a9\": container with ID starting with 623c43a362fdd42fbd934d3e390aff7323d886b56800a72f871d9496a91c93a9 not found: ID does not exist"
Sep 18 19:17:28.470596 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.470551    2577 scope.go:117] "RemoveContainer" containerID="1c935190ec508e926e04c5b084b4581799040e7bbe94ce5e5b72d36076e64daa"
Sep 18 19:17:28.470767 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.470752    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c935190ec508e926e04c5b084b4581799040e7bbe94ce5e5b72d36076e64daa"} err="failed to get container status \"1c935190ec508e926e04c5b084b4581799040e7bbe94ce5e5b72d36076e64daa\": rpc error: code = NotFound desc = could not find container \"1c935190ec508e926e04c5b084b4581799040e7bbe94ce5e5b72d36076e64daa\": container with ID starting with 1c935190ec508e926e04c5b084b4581799040e7bbe94ce5e5b72d36076e64daa not found: ID does not exist"
Sep 18 19:17:28.472067 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.472048    2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/83f44b93-4402-4869-91ec-6fd23c938443-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:17:28.472067 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.472066    2577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/83f44b93-4402-4869-91ec-6fd23c938443-config\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:17:28.472172 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.472076    2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/83f44b93-4402-4869-91ec-6fd23c938443-secret-prometheus-k8s-tls\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:17:28.472172 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.472085    2577 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/83f44b93-4402-4869-91ec-6fd23c938443-thanos-prometheus-http-client-file\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:17:28.472172 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.472095    2577 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/83f44b93-4402-4869-91ec-6fd23c938443-config-out\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:17:28.472172 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.472103    2577 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83f44b93-4402-4869-91ec-6fd23c938443-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:17:28.472172 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.472111    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2trp\" (UniqueName: \"kubernetes.io/projected/83f44b93-4402-4869-91ec-6fd23c938443-kube-api-access-b2trp\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:17:28.472172 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.472119    2577 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/83f44b93-4402-4869-91ec-6fd23c938443-secret-grpc-tls\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:17:28.472172 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.472127    2577 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/83f44b93-4402-4869-91ec-6fd23c938443-tls-assets\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:17:28.472172 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.472135    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/83f44b93-4402-4869-91ec-6fd23c938443-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:17:28.472172 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.472143    2577 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/83f44b93-4402-4869-91ec-6fd23c938443-secret-metrics-client-certs\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:17:28.472172 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.472150    2577 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/83f44b93-4402-4869-91ec-6fd23c938443-web-config\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:17:28.472172 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.472159    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/83f44b93-4402-4869-91ec-6fd23c938443-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:17:28.472172 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.472168    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/83f44b93-4402-4869-91ec-6fd23c938443-prometheus-k8s-db\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:17:28.472172 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.472176    2577 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83f44b93-4402-4869-91ec-6fd23c938443-configmap-metrics-client-ca\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:17:28.472520 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.472184    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/83f44b93-4402-4869-91ec-6fd23c938443-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:17:28.472520 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.472192    2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/83f44b93-4402-4869-91ec-6fd23c938443-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:17:28.472520 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.472200    2577 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83f44b93-4402-4869-91ec-6fd23c938443-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:17:28.472520 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.472208    2577 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/83f44b93-4402-4869-91ec-6fd23c938443-secret-kube-rbac-proxy\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:17:28.482220 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.482201    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 19:17:28.482482 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.482471    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83f44b93-4402-4869-91ec-6fd23c938443" containerName="kube-rbac-proxy"
Sep 18 19:17:28.482519 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.482486    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="83f44b93-4402-4869-91ec-6fd23c938443" containerName="kube-rbac-proxy"
Sep 18 19:17:28.482519 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.482499    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83f44b93-4402-4869-91ec-6fd23c938443" containerName="config-reloader"
Sep 18 19:17:28.482519 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.482506    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="83f44b93-4402-4869-91ec-6fd23c938443" containerName="config-reloader"
Sep 18 19:17:28.482519 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.482516    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83f44b93-4402-4869-91ec-6fd23c938443" containerName="kube-rbac-proxy-web"
Sep 18 19:17:28.482637 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.482525    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="83f44b93-4402-4869-91ec-6fd23c938443" containerName="kube-rbac-proxy-web"
Sep 18 19:17:28.482637 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.482534    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83f44b93-4402-4869-91ec-6fd23c938443" containerName="kube-rbac-proxy-thanos"
Sep 18 19:17:28.482637 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.482541    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="83f44b93-4402-4869-91ec-6fd23c938443" containerName="kube-rbac-proxy-thanos"
Sep 18 19:17:28.482637 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.482552    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83f44b93-4402-4869-91ec-6fd23c938443" containerName="init-config-reloader"
Sep 18 19:17:28.482637 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.482557    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="83f44b93-4402-4869-91ec-6fd23c938443" containerName="init-config-reloader"
Sep 18 19:17:28.482637 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.482563    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83f44b93-4402-4869-91ec-6fd23c938443" containerName="thanos-sidecar"
Sep 18 19:17:28.482637 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.482568    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="83f44b93-4402-4869-91ec-6fd23c938443" containerName="thanos-sidecar"
Sep 18 19:17:28.482637 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.482575    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83f44b93-4402-4869-91ec-6fd23c938443" containerName="prometheus"
Sep 18 19:17:28.482637 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.482579    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="83f44b93-4402-4869-91ec-6fd23c938443" containerName="prometheus"
Sep 18 19:17:28.482637 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.482625    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="83f44b93-4402-4869-91ec-6fd23c938443" containerName="config-reloader"
Sep 18 19:17:28.482637 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.482632    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="83f44b93-4402-4869-91ec-6fd23c938443" containerName="kube-rbac-proxy-web"
Sep 18 19:17:28.482637 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.482638    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="83f44b93-4402-4869-91ec-6fd23c938443" containerName="prometheus"
Sep 18 19:17:28.482977 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.482643    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="83f44b93-4402-4869-91ec-6fd23c938443" containerName="thanos-sidecar"
Sep 18 19:17:28.482977 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.482648    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="83f44b93-4402-4869-91ec-6fd23c938443" containerName="kube-rbac-proxy-thanos"
Sep 18 19:17:28.482977 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.482655    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="83f44b93-4402-4869-91ec-6fd23c938443" containerName="kube-rbac-proxy"
Sep 18 19:17:28.486373 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.486357    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.489787 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.489766    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 18 19:17:28.490070 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.490056    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 18 19:17:28.490342 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.490307    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 18 19:17:28.490342 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.490329    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 18 19:17:28.490514 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.490343    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 18 19:17:28.490592 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.490576    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 18 19:17:28.490674 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.490660    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 18 19:17:28.490757 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.490739    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-24qpqpko3d561\""
Sep 18 19:17:28.490882 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.490868    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-sncd5\""
Sep 18 19:17:28.491027 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.491007    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 18 19:17:28.491136 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.491029    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 18 19:17:28.491136 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.491068    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 18 19:17:28.491271 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.491257    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 18 19:17:28.491350 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.491334    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 18 19:17:28.492619 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.492601    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 18 19:17:28.500239 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.500219    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 18 19:17:28.529449 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.529410    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 19:17:28.572675 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.572638    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.572675 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.572677    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.572937 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.572762    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.572937 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.572842    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.572937 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.572870    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.573121 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.573073    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.573181 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.573120    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.573181 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.573140    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-web-config\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.573181 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.573163    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-config-out\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.573275 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.573231    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.573358 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.573344    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.573405 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.573389    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.573451 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.573409    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.573504 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.573458    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.573555 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.573514    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.573555 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.573543    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.573691 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.573586    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.573691 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.573620    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9qrg\" (UniqueName: \"kubernetes.io/projected/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-kube-api-access-p9qrg\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.573691 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.573650    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-config\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.573691 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.573665    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.674971 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.674886    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9qrg\" (UniqueName: \"kubernetes.io/projected/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-kube-api-access-p9qrg\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.674971 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.674925    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-config\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.674971 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.674942    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.674971 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.674970    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.675314 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.675013    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.675314 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.675046    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.675314 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.675078    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.675314 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.675101    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.675314 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.675130    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.675314 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.675153    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.675314 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.675174    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-web-config\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.675314 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.675197    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-config-out\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.675314 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.675218    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.675314 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.675269    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.675780 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.675756    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.675839 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.675756    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.675895 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.675861    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.675963 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.675936    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.676038 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.675976    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.676038 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.676027    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.676155 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.676055    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.676155 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.676083    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.676155 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.676112    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.676892 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.676834    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.678875 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.677905    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.678875 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.678353    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.678875 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.678431    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.678875 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.678826    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.679186 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.678889    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-web-config\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.679186 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.679050    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.679723 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.679681    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.679823 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.679775    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-config-out\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.680036 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.680011    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.680239 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.680222    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.680320 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.680292    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.680320 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.680300    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.680580 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.680554    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.681231 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.681214    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-config\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.681307 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.681290    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.684966 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.684950    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9qrg\" (UniqueName: \"kubernetes.io/projected/c0b6d69b-f7ce-4b85-84d5-dc6440246e3d-kube-api-access-p9qrg\") pod \"prometheus-k8s-0\" (UID: \"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:28.799434 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:28.799401    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:29.120258 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:29.120223    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 19:17:29.125296 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:17:29.125257    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0b6d69b_f7ce_4b85_84d5_dc6440246e3d.slice/crio-fd9c9c7d2711346b95eb859acbfafa01612af5e4e3567fd9cb265d7ea5f13494 WatchSource:0}: Error finding container fd9c9c7d2711346b95eb859acbfafa01612af5e4e3567fd9cb265d7ea5f13494: Status 404 returned error can't find the container with id fd9c9c7d2711346b95eb859acbfafa01612af5e4e3567fd9cb265d7ea5f13494
Sep 18 19:17:29.414245 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:29.414216    2577 generic.go:356] "Generic (PLEG): container finished" podID="c0b6d69b-f7ce-4b85-84d5-dc6440246e3d" containerID="03c44334bb757a41e9c14eee564a4282ef2731c53c9ef292d1a2a5d833a4f316" exitCode=0
Sep 18 19:17:29.414614 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:29.414293    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d","Type":"ContainerDied","Data":"03c44334bb757a41e9c14eee564a4282ef2731c53c9ef292d1a2a5d833a4f316"}
Sep 18 19:17:29.414614 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:29.414347    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d","Type":"ContainerStarted","Data":"fd9c9c7d2711346b95eb859acbfafa01612af5e4e3567fd9cb265d7ea5f13494"}
Sep 18 19:17:29.437371 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:29.437349    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83f44b93-4402-4869-91ec-6fd23c938443" path="/var/lib/kubelet/pods/83f44b93-4402-4869-91ec-6fd23c938443/volumes"
Sep 18 19:17:30.421673 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:30.421638    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d","Type":"ContainerStarted","Data":"bc9b3b0f20713debd9d0215a949a8cc87c5d31fcf9170ff4a509f92017e53eda"}
Sep 18 19:17:30.422158 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:30.421700    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d","Type":"ContainerStarted","Data":"ffe30ed30d04d82a27125d7ca65b44d4f5afb90736d7d0d57dea818ba2971f15"}
Sep 18 19:17:30.422158 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:30.421725    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d","Type":"ContainerStarted","Data":"e55aebc6df42d2a541a4453c8c418f589d73d0fd7a235fd2274a837966f20512"}
Sep 18 19:17:30.422158 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:30.421749    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d","Type":"ContainerStarted","Data":"885f2d748f236325ac8133bbe54b10b8bbb8600724b1894832dfd0252e45a6e1"}
Sep 18 19:17:30.422158 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:30.421772    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d","Type":"ContainerStarted","Data":"85badc6994b68fce7c53fc5a4fcf75a594d8066a4e9c81927731efe396694264"}
Sep 18 19:17:30.422158 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:30.421794    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0b6d69b-f7ce-4b85-84d5-dc6440246e3d","Type":"ContainerStarted","Data":"3e2e19f2e20f41a4cab7f79b9256420293bd96d86cb98395081fe037c5beca86"}
Sep 18 19:17:30.455406 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:30.455350    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.438112677 podStartE2EDuration="2.455331344s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.218667ms" isStatefulPod=false podCreationTimestamp="2026-09-18 19:17:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:17:30.453472425 +0000 UTC m=+221.305772833" watchObservedRunningTime="2026-09-18 19:17:30.455331344 +0000 UTC m=+221.307631727"
Sep 18 19:17:33.800041 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:17:33.800000    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:28.799659 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:18:28.799579    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:28.814682 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:18:28.814659    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:29.658284 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:18:29.658259    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:51.352115 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:18:51.352090    2577 kubelet.go:1648] "Image garbage collection succeeded"
Sep 18 19:23:33.100228 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:23:33.100194    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2kj8w"]
Sep 18 19:23:33.103290 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:23:33.103274    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2kj8w"
Sep 18 19:23:33.109543 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:23:33.109283    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-7tpk5\""
Sep 18 19:23:33.109543 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:23:33.109319    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 18 19:23:33.109543 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:23:33.109370    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 18 19:23:33.126624 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:23:33.126594    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2kj8w"]
Sep 18 19:23:33.245077 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:23:33.245043    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72vpv\" (UniqueName: \"kubernetes.io/projected/1114ff07-8c74-4c64-bcdd-87b38f67eea5-kube-api-access-72vpv\") pod \"cert-manager-cainjector-559cdd8c87-2kj8w\" (UID: \"1114ff07-8c74-4c64-bcdd-87b38f67eea5\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2kj8w"
Sep 18 19:23:33.245250 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:23:33.245102    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1114ff07-8c74-4c64-bcdd-87b38f67eea5-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2kj8w\" (UID: \"1114ff07-8c74-4c64-bcdd-87b38f67eea5\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2kj8w"
Sep 18 19:23:33.345521 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:23:33.345484    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72vpv\" (UniqueName: \"kubernetes.io/projected/1114ff07-8c74-4c64-bcdd-87b38f67eea5-kube-api-access-72vpv\") pod \"cert-manager-cainjector-559cdd8c87-2kj8w\" (UID: \"1114ff07-8c74-4c64-bcdd-87b38f67eea5\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2kj8w"
Sep 18 19:23:33.345681 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:23:33.345537    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1114ff07-8c74-4c64-bcdd-87b38f67eea5-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2kj8w\" (UID: \"1114ff07-8c74-4c64-bcdd-87b38f67eea5\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2kj8w"
Sep 18 19:23:33.355256 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:23:33.355199    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1114ff07-8c74-4c64-bcdd-87b38f67eea5-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2kj8w\" (UID: \"1114ff07-8c74-4c64-bcdd-87b38f67eea5\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2kj8w"
Sep 18 19:23:33.355371 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:23:33.355330    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72vpv\" (UniqueName: \"kubernetes.io/projected/1114ff07-8c74-4c64-bcdd-87b38f67eea5-kube-api-access-72vpv\") pod \"cert-manager-cainjector-559cdd8c87-2kj8w\" (UID: \"1114ff07-8c74-4c64-bcdd-87b38f67eea5\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2kj8w"
Sep 18 19:23:33.412459 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:23:33.412425    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2kj8w"
Sep 18 19:23:33.686548 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:23:33.686522    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2kj8w"]
Sep 18 19:23:33.688931 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:23:33.688904    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1114ff07_8c74_4c64_bcdd_87b38f67eea5.slice/crio-e9e11d3ff28ee67239a64cdfb42dec931daea896e721a51c2fda0f6d58dbfe97 WatchSource:0}: Error finding container e9e11d3ff28ee67239a64cdfb42dec931daea896e721a51c2fda0f6d58dbfe97: Status 404 returned error can't find the container with id e9e11d3ff28ee67239a64cdfb42dec931daea896e721a51c2fda0f6d58dbfe97
Sep 18 19:23:33.690807 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:23:33.690793    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:23:33.744279 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:23:33.744244    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2kj8w" event={"ID":"1114ff07-8c74-4c64-bcdd-87b38f67eea5","Type":"ContainerStarted","Data":"e9e11d3ff28ee67239a64cdfb42dec931daea896e721a51c2fda0f6d58dbfe97"}
Sep 18 19:23:37.763178 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:23:37.763136    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2kj8w" event={"ID":"1114ff07-8c74-4c64-bcdd-87b38f67eea5","Type":"ContainerStarted","Data":"6629eb8ab88c7d8f5ef4662c03a467f7776541396068b1d2e7c33a9e28bc258a"}
Sep 18 19:23:37.789719 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:23:37.789665    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2kj8w" podStartSLOduration=1.313722101 podStartE2EDuration="4.789651032s" totalImagesPullingTime="3.475928931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:23:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:23:37.789566076 +0000 UTC m=+588.641866450" watchObservedRunningTime="2026-09-18 19:23:37.789651032 +0000 UTC m=+588.641951401"
Sep 18 19:23:45.188545 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:23:45.188505    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cfrgg"]
Sep 18 19:23:45.192060 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:23:45.192038    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cfrgg"
Sep 18 19:23:45.195031 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:23:45.195009    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 18 19:23:45.195158 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:23:45.195140    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:23:45.196331 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:23:45.196308    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-ljrg4\""
Sep 18 19:23:45.205933 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:23:45.205911    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cfrgg"]
Sep 18 19:23:45.353736 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:23:45.353696    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdtnh\" (UniqueName: \"kubernetes.io/projected/7b44ab54-569e-4c59-ae22-24d2eeafb924-kube-api-access-sdtnh\") pod \"openshift-lws-operator-fd8ccff4c-cfrgg\" (UID: \"7b44ab54-569e-4c59-ae22-24d2eeafb924\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cfrgg"
Sep 18 19:23:45.353908 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:23:45.353751    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b44ab54-569e-4c59-ae22-24d2eeafb924-tmp\") pod \"openshift-lws-operator-fd8ccff4c-cfrgg\" (UID: \"7b44ab54-569e-4c59-ae22-24d2eeafb924\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cfrgg"
Sep 18 19:23:45.454513 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:23:45.454434    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdtnh\" (UniqueName: \"kubernetes.io/projected/7b44ab54-569e-4c59-ae22-24d2eeafb924-kube-api-access-sdtnh\") pod \"openshift-lws-operator-fd8ccff4c-cfrgg\" (UID: \"7b44ab54-569e-4c59-ae22-24d2eeafb924\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cfrgg"
Sep 18 19:23:45.454513 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:23:45.454482    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b44ab54-569e-4c59-ae22-24d2eeafb924-tmp\") pod \"openshift-lws-operator-fd8ccff4c-cfrgg\" (UID: \"7b44ab54-569e-4c59-ae22-24d2eeafb924\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cfrgg"
Sep 18 19:23:45.454810 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:23:45.454794    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b44ab54-569e-4c59-ae22-24d2eeafb924-tmp\") pod \"openshift-lws-operator-fd8ccff4c-cfrgg\" (UID: \"7b44ab54-569e-4c59-ae22-24d2eeafb924\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cfrgg"
Sep 18 19:23:45.463793 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:23:45.463753    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdtnh\" (UniqueName: \"kubernetes.io/projected/7b44ab54-569e-4c59-ae22-24d2eeafb924-kube-api-access-sdtnh\") pod \"openshift-lws-operator-fd8ccff4c-cfrgg\" (UID: \"7b44ab54-569e-4c59-ae22-24d2eeafb924\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cfrgg"
Sep 18 19:23:45.500673 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:23:45.500642    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cfrgg"
Sep 18 19:23:45.776241 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:23:45.776178    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cfrgg"]
Sep 18 19:23:45.779196 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:23:45.779171    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b44ab54_569e_4c59_ae22_24d2eeafb924.slice/crio-e67f56aa96c257ad0dfa6d70807502049f8b539a955898e3d1f4dce4e8c451a8 WatchSource:0}: Error finding container e67f56aa96c257ad0dfa6d70807502049f8b539a955898e3d1f4dce4e8c451a8: Status 404 returned error can't find the container with id e67f56aa96c257ad0dfa6d70807502049f8b539a955898e3d1f4dce4e8c451a8
Sep 18 19:23:45.794570 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:23:45.794540    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cfrgg" event={"ID":"7b44ab54-569e-4c59-ae22-24d2eeafb924","Type":"ContainerStarted","Data":"e67f56aa96c257ad0dfa6d70807502049f8b539a955898e3d1f4dce4e8c451a8"}
Sep 18 19:23:58.846136 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:23:58.846056    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cfrgg" event={"ID":"7b44ab54-569e-4c59-ae22-24d2eeafb924","Type":"ContainerStarted","Data":"77014738166dddab80d170814187790448846aca71d5c3e4eadf881e4ba35af1"}
Sep 18 19:23:58.868967 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:23:58.868918    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cfrgg" podStartSLOduration=1.252468152 podStartE2EDuration="13.868902602s" totalImagesPullingTime="12.61643445s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:23:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:23:58.867478678 +0000 UTC m=+609.719779049" watchObservedRunningTime="2026-09-18 19:23:58.868902602 +0000 UTC m=+609.721202972"
Sep 18 19:24:07.489354 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:07.489278    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-85dd87cf5-zqskb"]
Sep 18 19:24:07.492359 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:07.492338    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-85dd87cf5-zqskb"
Sep 18 19:24:07.497463 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:07.497443    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-bd5t7\""
Sep 18 19:24:07.497572 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:07.497532    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 18 19:24:07.497807 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:07.497789    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 18 19:24:07.497910 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:07.497809    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 18 19:24:07.508957 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:07.508937    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-85dd87cf5-zqskb"]
Sep 18 19:24:07.538979 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:07.538955    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1b15394-b52c-4899-a52a-1dd0ea6a4823-cert\") pod \"lws-controller-manager-85dd87cf5-zqskb\" (UID: \"c1b15394-b52c-4899-a52a-1dd0ea6a4823\") " pod="openshift-lws-operator/lws-controller-manager-85dd87cf5-zqskb"
Sep 18 19:24:07.539106 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:07.538981    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c1b15394-b52c-4899-a52a-1dd0ea6a4823-metrics-cert\") pod \"lws-controller-manager-85dd87cf5-zqskb\" (UID: \"c1b15394-b52c-4899-a52a-1dd0ea6a4823\") " pod="openshift-lws-operator/lws-controller-manager-85dd87cf5-zqskb"
Sep 18 19:24:07.539106 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:07.539035    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c1b15394-b52c-4899-a52a-1dd0ea6a4823-manager-config\") pod \"lws-controller-manager-85dd87cf5-zqskb\" (UID: \"c1b15394-b52c-4899-a52a-1dd0ea6a4823\") " pod="openshift-lws-operator/lws-controller-manager-85dd87cf5-zqskb"
Sep 18 19:24:07.539106 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:07.539054    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dlwx\" (UniqueName: \"kubernetes.io/projected/c1b15394-b52c-4899-a52a-1dd0ea6a4823-kube-api-access-9dlwx\") pod \"lws-controller-manager-85dd87cf5-zqskb\" (UID: \"c1b15394-b52c-4899-a52a-1dd0ea6a4823\") " pod="openshift-lws-operator/lws-controller-manager-85dd87cf5-zqskb"
Sep 18 19:24:07.640371 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:07.640339    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1b15394-b52c-4899-a52a-1dd0ea6a4823-cert\") pod \"lws-controller-manager-85dd87cf5-zqskb\" (UID: \"c1b15394-b52c-4899-a52a-1dd0ea6a4823\") " pod="openshift-lws-operator/lws-controller-manager-85dd87cf5-zqskb"
Sep 18 19:24:07.640371 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:07.640371    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c1b15394-b52c-4899-a52a-1dd0ea6a4823-metrics-cert\") pod \"lws-controller-manager-85dd87cf5-zqskb\" (UID: \"c1b15394-b52c-4899-a52a-1dd0ea6a4823\") " pod="openshift-lws-operator/lws-controller-manager-85dd87cf5-zqskb"
Sep 18 19:24:07.640599 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:07.640408    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c1b15394-b52c-4899-a52a-1dd0ea6a4823-manager-config\") pod \"lws-controller-manager-85dd87cf5-zqskb\" (UID: \"c1b15394-b52c-4899-a52a-1dd0ea6a4823\") " pod="openshift-lws-operator/lws-controller-manager-85dd87cf5-zqskb"
Sep 18 19:24:07.640599 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:07.640430    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dlwx\" (UniqueName: \"kubernetes.io/projected/c1b15394-b52c-4899-a52a-1dd0ea6a4823-kube-api-access-9dlwx\") pod \"lws-controller-manager-85dd87cf5-zqskb\" (UID: \"c1b15394-b52c-4899-a52a-1dd0ea6a4823\") " pod="openshift-lws-operator/lws-controller-manager-85dd87cf5-zqskb"
Sep 18 19:24:07.641182 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:07.641156    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c1b15394-b52c-4899-a52a-1dd0ea6a4823-manager-config\") pod \"lws-controller-manager-85dd87cf5-zqskb\" (UID: \"c1b15394-b52c-4899-a52a-1dd0ea6a4823\") " pod="openshift-lws-operator/lws-controller-manager-85dd87cf5-zqskb"
Sep 18 19:24:07.642813 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:07.642796    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c1b15394-b52c-4899-a52a-1dd0ea6a4823-metrics-cert\") pod \"lws-controller-manager-85dd87cf5-zqskb\" (UID: \"c1b15394-b52c-4899-a52a-1dd0ea6a4823\") " pod="openshift-lws-operator/lws-controller-manager-85dd87cf5-zqskb"
Sep 18 19:24:07.642926 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:07.642909    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1b15394-b52c-4899-a52a-1dd0ea6a4823-cert\") pod \"lws-controller-manager-85dd87cf5-zqskb\" (UID: \"c1b15394-b52c-4899-a52a-1dd0ea6a4823\") " pod="openshift-lws-operator/lws-controller-manager-85dd87cf5-zqskb"
Sep 18 19:24:07.654048 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:07.654022    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dlwx\" (UniqueName: \"kubernetes.io/projected/c1b15394-b52c-4899-a52a-1dd0ea6a4823-kube-api-access-9dlwx\") pod \"lws-controller-manager-85dd87cf5-zqskb\" (UID: \"c1b15394-b52c-4899-a52a-1dd0ea6a4823\") " pod="openshift-lws-operator/lws-controller-manager-85dd87cf5-zqskb"
Sep 18 19:24:07.801861 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:07.801771    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-85dd87cf5-zqskb"
Sep 18 19:24:08.096006 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:08.095916    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-85dd87cf5-zqskb"]
Sep 18 19:24:08.098849 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:24:08.098819    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1b15394_b52c_4899_a52a_1dd0ea6a4823.slice/crio-bf182c7f69d12a32eb34e2d4624d532b8bbe8143026d383a583a19d500e8e864 WatchSource:0}: Error finding container bf182c7f69d12a32eb34e2d4624d532b8bbe8143026d383a583a19d500e8e864: Status 404 returned error can't find the container with id bf182c7f69d12a32eb34e2d4624d532b8bbe8143026d383a583a19d500e8e864
Sep 18 19:24:08.886355 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:08.886317    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-85dd87cf5-zqskb" event={"ID":"c1b15394-b52c-4899-a52a-1dd0ea6a4823","Type":"ContainerStarted","Data":"bf182c7f69d12a32eb34e2d4624d532b8bbe8143026d383a583a19d500e8e864"}
Sep 18 19:24:09.892474 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:09.892443    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-85dd87cf5-zqskb" event={"ID":"c1b15394-b52c-4899-a52a-1dd0ea6a4823","Type":"ContainerStarted","Data":"10ed2dee0ee4d4295792ba36e0e42f8301fc40e13d7400cf0b7e38d6a8ca7b75"}
Sep 18 19:24:09.892877 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:09.892552    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-85dd87cf5-zqskb"
Sep 18 19:24:09.925181 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:09.925126    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-85dd87cf5-zqskb" podStartSLOduration=1.308348938 podStartE2EDuration="2.925111679s" totalImagesPullingTime="1.616762741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:24:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:24:09.924762001 +0000 UTC m=+620.777062372" watchObservedRunningTime="2026-09-18 19:24:09.925111679 +0000 UTC m=+620.777412049"
Sep 18 19:24:18.478962 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:18.478928    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx"]
Sep 18 19:24:18.482731 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:18.482704    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx"
Sep 18 19:24:18.486007 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:18.485974    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 18 19:24:18.486272 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:18.486254    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 18 19:24:18.486367 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:18.486281    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-6hrs5\""
Sep 18 19:24:18.486367 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:18.486304    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 18 19:24:18.506658 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:18.506634    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx"]
Sep 18 19:24:18.530477 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:18.530456    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/625e8845-a6ab-4281-838a-b4051ae7ad5d-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx\" (UID: \"625e8845-a6ab-4281-838a-b4051ae7ad5d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx"
Sep 18 19:24:18.530574 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:18.530488    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kwng\" (UniqueName: \"kubernetes.io/projected/625e8845-a6ab-4281-838a-b4051ae7ad5d-kube-api-access-5kwng\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx\" (UID: \"625e8845-a6ab-4281-838a-b4051ae7ad5d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx"
Sep 18 19:24:18.530574 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:18.530508    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/625e8845-a6ab-4281-838a-b4051ae7ad5d-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx\" (UID: \"625e8845-a6ab-4281-838a-b4051ae7ad5d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx"
Sep 18 19:24:18.530574 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:18.530524    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/625e8845-a6ab-4281-838a-b4051ae7ad5d-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx\" (UID: \"625e8845-a6ab-4281-838a-b4051ae7ad5d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx"
Sep 18 19:24:18.530678 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:18.530641    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/625e8845-a6ab-4281-838a-b4051ae7ad5d-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx\" (UID: \"625e8845-a6ab-4281-838a-b4051ae7ad5d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx"
Sep 18 19:24:18.530712 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:18.530680    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/625e8845-a6ab-4281-838a-b4051ae7ad5d-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx\" (UID: \"625e8845-a6ab-4281-838a-b4051ae7ad5d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx"
Sep 18 19:24:18.530805 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:18.530786    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/625e8845-a6ab-4281-838a-b4051ae7ad5d-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx\" (UID: \"625e8845-a6ab-4281-838a-b4051ae7ad5d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx"
Sep 18 19:24:18.530842 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:18.530822    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/625e8845-a6ab-4281-838a-b4051ae7ad5d-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx\" (UID: \"625e8845-a6ab-4281-838a-b4051ae7ad5d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx"
Sep 18 19:24:18.530905 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:18.530892    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/625e8845-a6ab-4281-838a-b4051ae7ad5d-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx\" (UID: \"625e8845-a6ab-4281-838a-b4051ae7ad5d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx"
Sep 18 19:24:18.632177 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:18.632139    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/625e8845-a6ab-4281-838a-b4051ae7ad5d-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx\" (UID: \"625e8845-a6ab-4281-838a-b4051ae7ad5d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx"
Sep 18 19:24:18.632177 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:18.632184    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/625e8845-a6ab-4281-838a-b4051ae7ad5d-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx\" (UID: \"625e8845-a6ab-4281-838a-b4051ae7ad5d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx"
Sep 18 19:24:18.632412 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:18.632210    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/625e8845-a6ab-4281-838a-b4051ae7ad5d-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx\" (UID: \"625e8845-a6ab-4281-838a-b4051ae7ad5d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx"
Sep 18 19:24:18.632412 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:18.632234    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/625e8845-a6ab-4281-838a-b4051ae7ad5d-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx\" (UID: \"625e8845-a6ab-4281-838a-b4051ae7ad5d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx"
Sep 18 19:24:18.632412 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:18.632283    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/625e8845-a6ab-4281-838a-b4051ae7ad5d-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx\" (UID: \"625e8845-a6ab-4281-838a-b4051ae7ad5d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx"
Sep 18 19:24:18.632412 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:18.632319    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/625e8845-a6ab-4281-838a-b4051ae7ad5d-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx\" (UID: \"625e8845-a6ab-4281-838a-b4051ae7ad5d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx"
Sep 18 19:24:18.632412 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:18.632348    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kwng\" (UniqueName: \"kubernetes.io/projected/625e8845-a6ab-4281-838a-b4051ae7ad5d-kube-api-access-5kwng\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx\" (UID: \"625e8845-a6ab-4281-838a-b4051ae7ad5d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx"
Sep 18 19:24:18.632412 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:18.632373    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/625e8845-a6ab-4281-838a-b4051ae7ad5d-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx\" (UID: \"625e8845-a6ab-4281-838a-b4051ae7ad5d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx"
Sep 18 19:24:18.632412 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:18.632394    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/625e8845-a6ab-4281-838a-b4051ae7ad5d-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx\" (UID: \"625e8845-a6ab-4281-838a-b4051ae7ad5d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx"
Sep 18 19:24:18.632729 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:18.632534    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/625e8845-a6ab-4281-838a-b4051ae7ad5d-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx\" (UID: \"625e8845-a6ab-4281-838a-b4051ae7ad5d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx"
Sep 18 19:24:18.632955 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:18.632934    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/625e8845-a6ab-4281-838a-b4051ae7ad5d-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx\" (UID: \"625e8845-a6ab-4281-838a-b4051ae7ad5d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx"
Sep 18 19:24:18.633043 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:18.633014    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/625e8845-a6ab-4281-838a-b4051ae7ad5d-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx\" (UID: \"625e8845-a6ab-4281-838a-b4051ae7ad5d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx"
Sep 18 19:24:18.633125 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:18.633106    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/625e8845-a6ab-4281-838a-b4051ae7ad5d-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx\" (UID: \"625e8845-a6ab-4281-838a-b4051ae7ad5d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx"
Sep 18 19:24:18.633228 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:18.633206    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/625e8845-a6ab-4281-838a-b4051ae7ad5d-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx\" (UID: \"625e8845-a6ab-4281-838a-b4051ae7ad5d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx"
Sep 18 19:24:18.634521 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:18.634494    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/625e8845-a6ab-4281-838a-b4051ae7ad5d-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx\" (UID: \"625e8845-a6ab-4281-838a-b4051ae7ad5d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx"
Sep 18 19:24:18.634730 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:18.634714    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/625e8845-a6ab-4281-838a-b4051ae7ad5d-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx\" (UID: \"625e8845-a6ab-4281-838a-b4051ae7ad5d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx"
Sep 18 19:24:18.643126 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:18.643107    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/625e8845-a6ab-4281-838a-b4051ae7ad5d-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx\" (UID: \"625e8845-a6ab-4281-838a-b4051ae7ad5d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx"
Sep 18 19:24:18.643250 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:18.643234    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kwng\" (UniqueName: \"kubernetes.io/projected/625e8845-a6ab-4281-838a-b4051ae7ad5d-kube-api-access-5kwng\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx\" (UID: \"625e8845-a6ab-4281-838a-b4051ae7ad5d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx"
Sep 18 19:24:18.793774 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:18.793691    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx"
Sep 18 19:24:19.080494 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:19.080421    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx"]
Sep 18 19:24:19.084665 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:24:19.084638    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod625e8845_a6ab_4281_838a_b4051ae7ad5d.slice/crio-f1f9674bdd0ce014863bf7a5cc183bce8d427f96cec16895a2f055d5b417939b WatchSource:0}: Error finding container f1f9674bdd0ce014863bf7a5cc183bce8d427f96cec16895a2f055d5b417939b: Status 404 returned error can't find the container with id f1f9674bdd0ce014863bf7a5cc183bce8d427f96cec16895a2f055d5b417939b
Sep 18 19:24:19.930557 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:19.930522    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx" event={"ID":"625e8845-a6ab-4281-838a-b4051ae7ad5d","Type":"ContainerStarted","Data":"f1f9674bdd0ce014863bf7a5cc183bce8d427f96cec16895a2f055d5b417939b"}
Sep 18 19:24:20.899144 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:20.899112    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-85dd87cf5-zqskb"
Sep 18 19:24:22.836505 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:22.836466    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 19:24:22.836788 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:22.836545    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 19:24:22.836788 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:22.836574    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 19:24:22.953430 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:22.953394    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx" event={"ID":"625e8845-a6ab-4281-838a-b4051ae7ad5d","Type":"ContainerStarted","Data":"50bb98618193209f8ae7e062ff0fa3b8642799a55b6a03aa2a364a9267234442"}
Sep 18 19:24:22.978950 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:22.978896    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx" podStartSLOduration=1.229309376 podStartE2EDuration="4.978880521s" totalImagesPullingTime="3.749571145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:24:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:24:22.978334079 +0000 UTC m=+633.830634477" watchObservedRunningTime="2026-09-18 19:24:22.978880521 +0000 UTC m=+633.831180892"
Sep 18 19:24:23.794596 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:23.794553    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx"
Sep 18 19:24:23.799321 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:23.799292    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx"
Sep 18 19:24:23.958053 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:23.958021    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx"
Sep 18 19:24:23.958904 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:23.958883    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx"
Sep 18 19:24:57.201364 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:57.201326    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-zx9fc"]
Sep 18 19:24:57.210444 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:57.210427    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zx9fc"
Sep 18 19:24:57.213203 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:57.213178    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 18 19:24:57.213340 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:57.213322    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 18 19:24:57.213408 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:57.213349    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 18 19:24:57.214418 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:57.214401    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-zlhpm\""
Sep 18 19:24:57.214513 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:57.214403    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 18 19:24:57.217897 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:57.217878    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-zx9fc"]
Sep 18 19:24:57.270187 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:57.270150    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2733a01c-8640-4c51-a877-3ee099deed3b-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-zx9fc\" (UID: \"2733a01c-8640-4c51-a877-3ee099deed3b\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zx9fc"
Sep 18 19:24:57.270349 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:57.270269    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2733a01c-8640-4c51-a877-3ee099deed3b-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-zx9fc\" (UID: \"2733a01c-8640-4c51-a877-3ee099deed3b\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zx9fc"
Sep 18 19:24:57.270349 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:57.270311    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2lsq\" (UniqueName: \"kubernetes.io/projected/2733a01c-8640-4c51-a877-3ee099deed3b-kube-api-access-t2lsq\") pod \"kuadrant-console-plugin-fbdd4f445-zx9fc\" (UID: \"2733a01c-8640-4c51-a877-3ee099deed3b\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zx9fc"
Sep 18 19:24:57.371674 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:57.371631    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2733a01c-8640-4c51-a877-3ee099deed3b-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-zx9fc\" (UID: \"2733a01c-8640-4c51-a877-3ee099deed3b\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zx9fc"
Sep 18 19:24:57.371674 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:57.371679    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2lsq\" (UniqueName: \"kubernetes.io/projected/2733a01c-8640-4c51-a877-3ee099deed3b-kube-api-access-t2lsq\") pod \"kuadrant-console-plugin-fbdd4f445-zx9fc\" (UID: \"2733a01c-8640-4c51-a877-3ee099deed3b\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zx9fc"
Sep 18 19:24:57.371892 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:57.371794    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2733a01c-8640-4c51-a877-3ee099deed3b-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-zx9fc\" (UID: \"2733a01c-8640-4c51-a877-3ee099deed3b\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zx9fc"
Sep 18 19:24:57.371957 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:24:57.371937    2577 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 18 19:24:57.372066 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:24:57.372054    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2733a01c-8640-4c51-a877-3ee099deed3b-plugin-serving-cert podName:2733a01c-8640-4c51-a877-3ee099deed3b nodeName:}" failed. No retries permitted until 2026-09-18 19:24:57.872030421 +0000 UTC m=+668.724330784 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/2733a01c-8640-4c51-a877-3ee099deed3b-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-zx9fc" (UID: "2733a01c-8640-4c51-a877-3ee099deed3b") : secret "plugin-serving-cert" not found
Sep 18 19:24:57.372444 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:57.372426    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2733a01c-8640-4c51-a877-3ee099deed3b-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-zx9fc\" (UID: \"2733a01c-8640-4c51-a877-3ee099deed3b\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zx9fc"
Sep 18 19:24:57.384590 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:57.384562    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2lsq\" (UniqueName: \"kubernetes.io/projected/2733a01c-8640-4c51-a877-3ee099deed3b-kube-api-access-t2lsq\") pod \"kuadrant-console-plugin-fbdd4f445-zx9fc\" (UID: \"2733a01c-8640-4c51-a877-3ee099deed3b\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zx9fc"
Sep 18 19:24:57.877455 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:57.877420    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2733a01c-8640-4c51-a877-3ee099deed3b-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-zx9fc\" (UID: \"2733a01c-8640-4c51-a877-3ee099deed3b\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zx9fc"
Sep 18 19:24:57.877629 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:24:57.877558    2577 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 18 19:24:57.877629 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:24:57.877617    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2733a01c-8640-4c51-a877-3ee099deed3b-plugin-serving-cert podName:2733a01c-8640-4c51-a877-3ee099deed3b nodeName:}" failed. No retries permitted until 2026-09-18 19:24:58.877603536 +0000 UTC m=+669.729903888 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/2733a01c-8640-4c51-a877-3ee099deed3b-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-zx9fc" (UID: "2733a01c-8640-4c51-a877-3ee099deed3b") : secret "plugin-serving-cert" not found
Sep 18 19:24:58.886822 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:58.886788    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2733a01c-8640-4c51-a877-3ee099deed3b-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-zx9fc\" (UID: \"2733a01c-8640-4c51-a877-3ee099deed3b\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zx9fc"
Sep 18 19:24:58.889133 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:58.889105    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2733a01c-8640-4c51-a877-3ee099deed3b-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-zx9fc\" (UID: \"2733a01c-8640-4c51-a877-3ee099deed3b\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zx9fc"
Sep 18 19:24:59.019969 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:59.019936    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zx9fc"
Sep 18 19:24:59.303020 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:24:59.302975    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-zx9fc"]
Sep 18 19:24:59.305588 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:24:59.305557    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2733a01c_8640_4c51_a877_3ee099deed3b.slice/crio-e8963f695fe3482bbdf9361d9e813c409c17c158264c32fd935485e3f6823e6a WatchSource:0}: Error finding container e8963f695fe3482bbdf9361d9e813c409c17c158264c32fd935485e3f6823e6a: Status 404 returned error can't find the container with id e8963f695fe3482bbdf9361d9e813c409c17c158264c32fd935485e3f6823e6a
Sep 18 19:25:00.115106 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:25:00.115069    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zx9fc" event={"ID":"2733a01c-8640-4c51-a877-3ee099deed3b","Type":"ContainerStarted","Data":"e8963f695fe3482bbdf9361d9e813c409c17c158264c32fd935485e3f6823e6a"}
Sep 18 19:25:04.134710 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:25:04.134676    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zx9fc" event={"ID":"2733a01c-8640-4c51-a877-3ee099deed3b","Type":"ContainerStarted","Data":"2003f0af71d115de52c088f59ac9f07d54280e31d37b2f945b53b29344780dfe"}
Sep 18 19:25:04.154918 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:25:04.154862    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zx9fc" podStartSLOduration=2.734951796 podStartE2EDuration="7.154845733s" totalImagesPullingTime="4.419893937s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:24:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:25:04.153315612 +0000 UTC m=+675.005615981" watchObservedRunningTime="2026-09-18 19:25:04.154845733 +0000 UTC m=+675.007146123"
Sep 18 19:25:42.376387 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:25:42.376304    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hq9st"]
Sep 18 19:25:42.391565 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:25:42.391532    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hq9st"]
Sep 18 19:25:42.391722 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:25:42.391630    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-hq9st"
Sep 18 19:25:42.394164 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:25:42.394135    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 18 19:25:42.411564 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:25:42.411538    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hq9st"]
Sep 18 19:25:42.474547 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:25:42.474512    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzs4p\" (UniqueName: \"kubernetes.io/projected/6aae2a34-975a-4871-b0a8-c1e7326f5d12-kube-api-access-dzs4p\") pod \"limitador-limitador-84994f7ffc-hq9st\" (UID: \"6aae2a34-975a-4871-b0a8-c1e7326f5d12\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-hq9st"
Sep 18 19:25:42.474716 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:25:42.474585    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6aae2a34-975a-4871-b0a8-c1e7326f5d12-config-file\") pod \"limitador-limitador-84994f7ffc-hq9st\" (UID: \"6aae2a34-975a-4871-b0a8-c1e7326f5d12\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-hq9st"
Sep 18 19:25:42.575485 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:25:42.575451    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6aae2a34-975a-4871-b0a8-c1e7326f5d12-config-file\") pod \"limitador-limitador-84994f7ffc-hq9st\" (UID: \"6aae2a34-975a-4871-b0a8-c1e7326f5d12\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-hq9st"
Sep 18 19:25:42.575650 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:25:42.575545    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzs4p\" (UniqueName: \"kubernetes.io/projected/6aae2a34-975a-4871-b0a8-c1e7326f5d12-kube-api-access-dzs4p\") pod \"limitador-limitador-84994f7ffc-hq9st\" (UID: \"6aae2a34-975a-4871-b0a8-c1e7326f5d12\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-hq9st"
Sep 18 19:25:42.576119 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:25:42.576098    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6aae2a34-975a-4871-b0a8-c1e7326f5d12-config-file\") pod \"limitador-limitador-84994f7ffc-hq9st\" (UID: \"6aae2a34-975a-4871-b0a8-c1e7326f5d12\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-hq9st"
Sep 18 19:25:42.585408 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:25:42.585390    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzs4p\" (UniqueName: \"kubernetes.io/projected/6aae2a34-975a-4871-b0a8-c1e7326f5d12-kube-api-access-dzs4p\") pod \"limitador-limitador-84994f7ffc-hq9st\" (UID: \"6aae2a34-975a-4871-b0a8-c1e7326f5d12\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-hq9st"
Sep 18 19:25:42.701144 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:25:42.701054    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-hq9st"
Sep 18 19:25:42.973177 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:25:42.973104    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hq9st"]
Sep 18 19:25:42.976093 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:25:42.976065    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aae2a34_975a_4871_b0a8_c1e7326f5d12.slice/crio-4399a2cb4373ae3d673aa8c2d955580f3ea5bbb1f68470fd8f1041587b37e01a WatchSource:0}: Error finding container 4399a2cb4373ae3d673aa8c2d955580f3ea5bbb1f68470fd8f1041587b37e01a: Status 404 returned error can't find the container with id 4399a2cb4373ae3d673aa8c2d955580f3ea5bbb1f68470fd8f1041587b37e01a
Sep 18 19:25:43.303428 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:25:43.303345    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-hq9st" event={"ID":"6aae2a34-975a-4871-b0a8-c1e7326f5d12","Type":"ContainerStarted","Data":"4399a2cb4373ae3d673aa8c2d955580f3ea5bbb1f68470fd8f1041587b37e01a"}
Sep 18 19:25:45.313876 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:25:45.313840    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-hq9st" event={"ID":"6aae2a34-975a-4871-b0a8-c1e7326f5d12","Type":"ContainerStarted","Data":"3e1a394f82b587edd1d74e23f667df9c6fbe26b2799999cfbbd0979e1f44ba1d"}
Sep 18 19:25:45.314247 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:25:45.313906    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-hq9st"
Sep 18 19:25:45.331771 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:25:45.331719    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-hq9st" podStartSLOduration=1.202545229 podStartE2EDuration="3.331703616s" totalImagesPullingTime="2.129158387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:25:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:25:45.330732977 +0000 UTC m=+716.183033350" watchObservedRunningTime="2026-09-18 19:25:45.331703616 +0000 UTC m=+716.184004028"
Sep 18 19:25:56.319058 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:25:56.319022    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-hq9st"
Sep 18 19:30:32.118744 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:30:32.118707    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fvlgm4"]
Sep 18 19:30:32.121144 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:30:32.121127    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fvlgm4"
Sep 18 19:30:32.124586 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:30:32.124558    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Sep 18 19:30:32.124708 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:30:32.124590    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Sep 18 19:30:32.125978 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:30:32.125962    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Sep 18 19:30:32.126076 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:30:32.125962    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-24jtl\""
Sep 18 19:30:32.139247 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:30:32.139228    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fvlgm4"]
Sep 18 19:30:32.177854 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:30:32.177822    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dd6ef4a-6e2e-48d6-a401-196df9fac5b4-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fvlgm4\" (UID: \"1dd6ef4a-6e2e-48d6-a401-196df9fac5b4\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fvlgm4"
Sep 18 19:30:32.178023 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:30:32.177860    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dd6ef4a-6e2e-48d6-a401-196df9fac5b4-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fvlgm4\" (UID: \"1dd6ef4a-6e2e-48d6-a401-196df9fac5b4\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fvlgm4"
Sep 18 19:30:32.178023 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:30:32.177936    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dd6ef4a-6e2e-48d6-a401-196df9fac5b4-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fvlgm4\" (UID: \"1dd6ef4a-6e2e-48d6-a401-196df9fac5b4\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fvlgm4"
Sep 18 19:30:32.178023 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:30:32.177964    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrxp2\" (UniqueName: \"kubernetes.io/projected/1dd6ef4a-6e2e-48d6-a401-196df9fac5b4-kube-api-access-hrxp2\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fvlgm4\" (UID: \"1dd6ef4a-6e2e-48d6-a401-196df9fac5b4\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fvlgm4"
Sep 18 19:30:32.178206 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:30:32.178087    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dd6ef4a-6e2e-48d6-a401-196df9fac5b4-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fvlgm4\" (UID: \"1dd6ef4a-6e2e-48d6-a401-196df9fac5b4\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fvlgm4"
Sep 18 19:30:32.178206 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:30:32.178149    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dd6ef4a-6e2e-48d6-a401-196df9fac5b4-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fvlgm4\" (UID: \"1dd6ef4a-6e2e-48d6-a401-196df9fac5b4\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fvlgm4"
Sep 18 19:30:32.278998 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:30:32.278963    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dd6ef4a-6e2e-48d6-a401-196df9fac5b4-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fvlgm4\" (UID: \"1dd6ef4a-6e2e-48d6-a401-196df9fac5b4\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fvlgm4"
Sep 18 19:30:32.279160 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:30:32.279019    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrxp2\" (UniqueName: \"kubernetes.io/projected/1dd6ef4a-6e2e-48d6-a401-196df9fac5b4-kube-api-access-hrxp2\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fvlgm4\" (UID: \"1dd6ef4a-6e2e-48d6-a401-196df9fac5b4\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fvlgm4"
Sep 18 19:30:32.279160 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:30:32.279061    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dd6ef4a-6e2e-48d6-a401-196df9fac5b4-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fvlgm4\" (UID: \"1dd6ef4a-6e2e-48d6-a401-196df9fac5b4\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fvlgm4"
Sep 18 19:30:32.279160 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:30:32.279095    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dd6ef4a-6e2e-48d6-a401-196df9fac5b4-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fvlgm4\" (UID: \"1dd6ef4a-6e2e-48d6-a401-196df9fac5b4\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fvlgm4"
Sep 18 19:30:32.279160 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:30:32.279130    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dd6ef4a-6e2e-48d6-a401-196df9fac5b4-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fvlgm4\" (UID: \"1dd6ef4a-6e2e-48d6-a401-196df9fac5b4\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fvlgm4"
Sep 18 19:30:32.279357 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:30:32.279243    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dd6ef4a-6e2e-48d6-a401-196df9fac5b4-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fvlgm4\" (UID: \"1dd6ef4a-6e2e-48d6-a401-196df9fac5b4\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fvlgm4"
Sep 18 19:30:32.279545 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:30:32.279520    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dd6ef4a-6e2e-48d6-a401-196df9fac5b4-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fvlgm4\" (UID: \"1dd6ef4a-6e2e-48d6-a401-196df9fac5b4\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fvlgm4"
Sep 18 19:30:32.279658 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:30:32.279638    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dd6ef4a-6e2e-48d6-a401-196df9fac5b4-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fvlgm4\" (UID: \"1dd6ef4a-6e2e-48d6-a401-196df9fac5b4\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fvlgm4"
Sep 18 19:30:32.279702 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:30:32.279675    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dd6ef4a-6e2e-48d6-a401-196df9fac5b4-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fvlgm4\" (UID: \"1dd6ef4a-6e2e-48d6-a401-196df9fac5b4\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fvlgm4"
Sep 18 19:30:32.281327 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:30:32.281311    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dd6ef4a-6e2e-48d6-a401-196df9fac5b4-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fvlgm4\" (UID: \"1dd6ef4a-6e2e-48d6-a401-196df9fac5b4\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fvlgm4"
Sep 18 19:30:32.281445 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:30:32.281426    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dd6ef4a-6e2e-48d6-a401-196df9fac5b4-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fvlgm4\" (UID: \"1dd6ef4a-6e2e-48d6-a401-196df9fac5b4\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fvlgm4"
Sep 18 19:30:32.288453 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:30:32.288429    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrxp2\" (UniqueName: \"kubernetes.io/projected/1dd6ef4a-6e2e-48d6-a401-196df9fac5b4-kube-api-access-hrxp2\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fvlgm4\" (UID: \"1dd6ef4a-6e2e-48d6-a401-196df9fac5b4\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fvlgm4"
Sep 18 19:30:32.430681 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:30:32.430608    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fvlgm4"
Sep 18 19:30:32.714914 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:30:32.714839    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fvlgm4"]
Sep 18 19:30:32.718086 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:30:32.718059    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dd6ef4a_6e2e_48d6_a401_196df9fac5b4.slice/crio-345d4ccaddd807dbf73a1f468532c0599da92bc327470be9be7f49b332edd352 WatchSource:0}: Error finding container 345d4ccaddd807dbf73a1f468532c0599da92bc327470be9be7f49b332edd352: Status 404 returned error can't find the container with id 345d4ccaddd807dbf73a1f468532c0599da92bc327470be9be7f49b332edd352
Sep 18 19:30:32.720030 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:30:32.720009    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:30:33.481108 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:30:33.481065    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fvlgm4" event={"ID":"1dd6ef4a-6e2e-48d6-a401-196df9fac5b4","Type":"ContainerStarted","Data":"345d4ccaddd807dbf73a1f468532c0599da92bc327470be9be7f49b332edd352"}
Sep 18 19:30:34.492911 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:30:34.492879    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fvlgm4" event={"ID":"1dd6ef4a-6e2e-48d6-a401-196df9fac5b4","Type":"ContainerStarted","Data":"f40ab73c3f283f1b087dcd6c25f59c1883ad567e69ba9ef7ee9c26fbac4b4453"}
Sep 18 19:30:34.515646 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:30:34.515591    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fvlgm4" podStartSLOduration=0.905611108 podStartE2EDuration="2.515576038s" totalImagesPullingTime="1.60996493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:30:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:30:34.51278689 +0000 UTC m=+1005.365087262" watchObservedRunningTime="2026-09-18 19:30:34.515576038 +0000 UTC m=+1005.367876408"
Sep 18 19:30:35.431432 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:30:35.431389    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fvlgm4"
Sep 18 19:30:42.431171 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:30:42.431131    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fvlgm4"
Sep 18 19:30:42.443506 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:30:42.443479    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fvlgm4"
Sep 18 19:30:42.530347 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:30:42.530315    2577 generic.go:356] "Generic (PLEG): container finished" podID="625e8845-a6ab-4281-838a-b4051ae7ad5d" containerID="50bb98618193209f8ae7e062ff0fa3b8642799a55b6a03aa2a364a9267234442" exitCode=0
Sep 18 19:30:42.530504 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:30:42.530391    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx" event={"ID":"625e8845-a6ab-4281-838a-b4051ae7ad5d","Type":"ContainerDied","Data":"50bb98618193209f8ae7e062ff0fa3b8642799a55b6a03aa2a364a9267234442"}
Sep 18 19:30:42.530838 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:30:42.530818    2577 scope.go:117] "RemoveContainer" containerID="50bb98618193209f8ae7e062ff0fa3b8642799a55b6a03aa2a364a9267234442"
Sep 18 19:30:42.531908 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:30:42.531883    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 19:30:42.531975 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:30:42.531942    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 19:30:42.532030 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:30:42.531975    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 19:30:42.542305 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:30:42.542283    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fvlgm4"
Sep 18 19:30:43.542366 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:30:43.541618    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx" event={"ID":"625e8845-a6ab-4281-838a-b4051ae7ad5d","Type":"ContainerStarted","Data":"93c951d492bbdf7d606cc6eb14d8ee5154bd0cf90c9fbd7938628f2f8227ae2f"}
Sep 18 19:30:43.793944 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:30:43.793860    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx"
Sep 18 19:30:43.794108 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:30:43.794059    2577 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.24:15021/healthz/ready\": dial tcp 10.132.0.24:15021: connect: connection refused" start-of-body=
Sep 18 19:30:43.794157 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:30:43.794113    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx" podUID="625e8845-a6ab-4281-838a-b4051ae7ad5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.24:15021/healthz/ready\": dial tcp 10.132.0.24:15021: connect: connection refused"
Sep 18 19:30:44.794920 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:30:44.794884    2577 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.24:15021/healthz/ready\": dial tcp 10.132.0.24:15021: connect: connection refused" start-of-body=
Sep 18 19:30:44.795394 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:30:44.794943    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx" podUID="625e8845-a6ab-4281-838a-b4051ae7ad5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.24:15021/healthz/ready\": dial tcp 10.132.0.24:15021: connect: connection refused"
Sep 18 19:30:45.794678 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:30:45.794646    2577 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.24:15021/healthz/ready\": dial tcp 10.132.0.24:15021: connect: connection refused" start-of-body=
Sep 18 19:30:45.794849 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:30:45.794698    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx" podUID="625e8845-a6ab-4281-838a-b4051ae7ad5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.24:15021/healthz/ready\": dial tcp 10.132.0.24:15021: connect: connection refused"
Sep 18 19:30:46.794490 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:30:46.794453    2577 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.24:15021/healthz/ready\": dial tcp 10.132.0.24:15021: connect: connection refused" start-of-body=
Sep 18 19:30:46.794960 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:30:46.794514    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx" podUID="625e8845-a6ab-4281-838a-b4051ae7ad5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.24:15021/healthz/ready\": dial tcp 10.132.0.24:15021: connect: connection refused"
Sep 18 19:30:47.795006 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:30:47.794958    2577 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.24:15021/healthz/ready\": dial tcp 10.132.0.24:15021: connect: connection refused" start-of-body=
Sep 18 19:30:47.795392 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:30:47.795036    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx" podUID="625e8845-a6ab-4281-838a-b4051ae7ad5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.24:15021/healthz/ready\": dial tcp 10.132.0.24:15021: connect: connection refused"
Sep 18 19:30:48.797971 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:30:48.797941    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx"
Sep 18 19:30:48.798363 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:30:48.798294    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx"
Sep 18 19:30:48.798915 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:30:48.798896    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx"
Sep 18 19:31:08.146179 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:31:08.146144    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fvlgm4"]
Sep 18 19:31:08.155710 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:31:08.155670    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 19:31:08.155710 ip-10-0-129-30 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:31:08.155710 ip-10-0-129-30 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 18 19:31:08.155710 ip-10-0-129-30 kubenswrapper[2577]:  > containerID="f40ab73c3f283f1b087dcd6c25f59c1883ad567e69ba9ef7ee9c26fbac4b4453" cmd=["/bin/sleep","15"]
Sep 18 19:31:08.155959 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:31:08.155733    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fvlgm4" message="" err=<
Sep 18 19:31:08.155959 ip-10-0-129-30 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:31:08.155959 ip-10-0-129-30 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 18 19:31:08.155959 ip-10-0-129-30 kubenswrapper[2577]:  >
Sep 18 19:31:08.155959 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:31:08.155745    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 19:31:08.155959 ip-10-0-129-30 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:31:08.155959 ip-10-0-129-30 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 18 19:31:08.155959 ip-10-0-129-30 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fvlgm4" podUID="1dd6ef4a-6e2e-48d6-a401-196df9fac5b4" containerName="main" containerID="cri-o://f40ab73c3f283f1b087dcd6c25f59c1883ad567e69ba9ef7ee9c26fbac4b4453"
Sep 18 19:31:08.155959 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:31:08.155794    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fvlgm4" podUID="1dd6ef4a-6e2e-48d6-a401-196df9fac5b4" containerName="main" containerID="cri-o://f40ab73c3f283f1b087dcd6c25f59c1883ad567e69ba9ef7ee9c26fbac4b4453" gracePeriod=60
Sep 18 19:31:08.513905 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:31:08.513867    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fvlgm4"
Sep 18 19:31:08.659302 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:31:08.659263    2577 generic.go:356] "Generic (PLEG): container finished" podID="1dd6ef4a-6e2e-48d6-a401-196df9fac5b4" containerID="f40ab73c3f283f1b087dcd6c25f59c1883ad567e69ba9ef7ee9c26fbac4b4453" exitCode=0
Sep 18 19:31:08.659503 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:31:08.659310    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fvlgm4"
Sep 18 19:31:08.659503 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:31:08.659326    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fvlgm4" event={"ID":"1dd6ef4a-6e2e-48d6-a401-196df9fac5b4","Type":"ContainerDied","Data":"f40ab73c3f283f1b087dcd6c25f59c1883ad567e69ba9ef7ee9c26fbac4b4453"}
Sep 18 19:31:08.659503 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:31:08.659379    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fvlgm4" event={"ID":"1dd6ef4a-6e2e-48d6-a401-196df9fac5b4","Type":"ContainerDied","Data":"345d4ccaddd807dbf73a1f468532c0599da92bc327470be9be7f49b332edd352"}
Sep 18 19:31:08.659503 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:31:08.659409    2577 scope.go:117] "RemoveContainer" containerID="f40ab73c3f283f1b087dcd6c25f59c1883ad567e69ba9ef7ee9c26fbac4b4453"
Sep 18 19:31:08.668604 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:31:08.668575    2577 scope.go:117] "RemoveContainer" containerID="f40ab73c3f283f1b087dcd6c25f59c1883ad567e69ba9ef7ee9c26fbac4b4453"
Sep 18 19:31:08.668882 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:31:08.668861    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f40ab73c3f283f1b087dcd6c25f59c1883ad567e69ba9ef7ee9c26fbac4b4453\": container with ID starting with f40ab73c3f283f1b087dcd6c25f59c1883ad567e69ba9ef7ee9c26fbac4b4453 not found: ID does not exist" containerID="f40ab73c3f283f1b087dcd6c25f59c1883ad567e69ba9ef7ee9c26fbac4b4453"
Sep 18 19:31:08.668935 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:31:08.668891    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f40ab73c3f283f1b087dcd6c25f59c1883ad567e69ba9ef7ee9c26fbac4b4453"} err="failed to get container status \"f40ab73c3f283f1b087dcd6c25f59c1883ad567e69ba9ef7ee9c26fbac4b4453\": rpc error: code = NotFound desc = could not find container \"f40ab73c3f283f1b087dcd6c25f59c1883ad567e69ba9ef7ee9c26fbac4b4453\": container with ID starting with f40ab73c3f283f1b087dcd6c25f59c1883ad567e69ba9ef7ee9c26fbac4b4453 not found: ID does not exist"
Sep 18 19:31:08.703228 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:31:08.703169    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dd6ef4a-6e2e-48d6-a401-196df9fac5b4-dshm\") pod \"1dd6ef4a-6e2e-48d6-a401-196df9fac5b4\" (UID: \"1dd6ef4a-6e2e-48d6-a401-196df9fac5b4\") "
Sep 18 19:31:08.703332 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:31:08.703247    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dd6ef4a-6e2e-48d6-a401-196df9fac5b4-tmp-dir\") pod \"1dd6ef4a-6e2e-48d6-a401-196df9fac5b4\" (UID: \"1dd6ef4a-6e2e-48d6-a401-196df9fac5b4\") "
Sep 18 19:31:08.703332 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:31:08.703280    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dd6ef4a-6e2e-48d6-a401-196df9fac5b4-home\") pod \"1dd6ef4a-6e2e-48d6-a401-196df9fac5b4\" (UID: \"1dd6ef4a-6e2e-48d6-a401-196df9fac5b4\") "
Sep 18 19:31:08.703332 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:31:08.703322    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dd6ef4a-6e2e-48d6-a401-196df9fac5b4-model-cache\") pod \"1dd6ef4a-6e2e-48d6-a401-196df9fac5b4\" (UID: \"1dd6ef4a-6e2e-48d6-a401-196df9fac5b4\") "
Sep 18 19:31:08.703494 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:31:08.703345    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dd6ef4a-6e2e-48d6-a401-196df9fac5b4-tls-certs\") pod \"1dd6ef4a-6e2e-48d6-a401-196df9fac5b4\" (UID: \"1dd6ef4a-6e2e-48d6-a401-196df9fac5b4\") "
Sep 18 19:31:08.703494 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:31:08.703361    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrxp2\" (UniqueName: \"kubernetes.io/projected/1dd6ef4a-6e2e-48d6-a401-196df9fac5b4-kube-api-access-hrxp2\") pod \"1dd6ef4a-6e2e-48d6-a401-196df9fac5b4\" (UID: \"1dd6ef4a-6e2e-48d6-a401-196df9fac5b4\") "
Sep 18 19:31:08.703595 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:31:08.703484    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dd6ef4a-6e2e-48d6-a401-196df9fac5b4-home" (OuterVolumeSpecName: "home") pod "1dd6ef4a-6e2e-48d6-a401-196df9fac5b4" (UID: "1dd6ef4a-6e2e-48d6-a401-196df9fac5b4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:31:08.703595 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:31:08.703532    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dd6ef4a-6e2e-48d6-a401-196df9fac5b4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1dd6ef4a-6e2e-48d6-a401-196df9fac5b4" (UID: "1dd6ef4a-6e2e-48d6-a401-196df9fac5b4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:31:08.703595 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:31:08.703575    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dd6ef4a-6e2e-48d6-a401-196df9fac5b4-model-cache" (OuterVolumeSpecName: "model-cache") pod "1dd6ef4a-6e2e-48d6-a401-196df9fac5b4" (UID: "1dd6ef4a-6e2e-48d6-a401-196df9fac5b4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:31:08.705425 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:31:08.705403    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dd6ef4a-6e2e-48d6-a401-196df9fac5b4-dshm" (OuterVolumeSpecName: "dshm") pod "1dd6ef4a-6e2e-48d6-a401-196df9fac5b4" (UID: "1dd6ef4a-6e2e-48d6-a401-196df9fac5b4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:31:08.705534 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:31:08.705478    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dd6ef4a-6e2e-48d6-a401-196df9fac5b4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1dd6ef4a-6e2e-48d6-a401-196df9fac5b4" (UID: "1dd6ef4a-6e2e-48d6-a401-196df9fac5b4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:31:08.705575 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:31:08.705527    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dd6ef4a-6e2e-48d6-a401-196df9fac5b4-kube-api-access-hrxp2" (OuterVolumeSpecName: "kube-api-access-hrxp2") pod "1dd6ef4a-6e2e-48d6-a401-196df9fac5b4" (UID: "1dd6ef4a-6e2e-48d6-a401-196df9fac5b4"). InnerVolumeSpecName "kube-api-access-hrxp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:31:08.804925 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:31:08.804889    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dd6ef4a-6e2e-48d6-a401-196df9fac5b4-dshm\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:31:08.804925 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:31:08.804923    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dd6ef4a-6e2e-48d6-a401-196df9fac5b4-tmp-dir\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:31:08.804925 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:31:08.804932    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dd6ef4a-6e2e-48d6-a401-196df9fac5b4-home\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:31:08.805159 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:31:08.804940    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dd6ef4a-6e2e-48d6-a401-196df9fac5b4-model-cache\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:31:08.805159 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:31:08.804950    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dd6ef4a-6e2e-48d6-a401-196df9fac5b4-tls-certs\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:31:08.805159 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:31:08.804957    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrxp2\" (UniqueName: \"kubernetes.io/projected/1dd6ef4a-6e2e-48d6-a401-196df9fac5b4-kube-api-access-hrxp2\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:31:08.987749 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:31:08.987711    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fvlgm4"]
Sep 18 19:31:08.993142 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:31:08.993113    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fvlgm4"]
Sep 18 19:31:09.438326 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:31:09.438286    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1dd6ef4a-6e2e-48d6-a401-196df9fac5b4" path="/var/lib/kubelet/pods/1dd6ef4a-6e2e-48d6-a401-196df9fac5b4/volumes"
Sep 18 19:34:00.770147 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:34:00.770106    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5fcps"]
Sep 18 19:34:00.770681 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:34:00.770425    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dd6ef4a-6e2e-48d6-a401-196df9fac5b4" containerName="main"
Sep 18 19:34:00.770681 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:34:00.770436    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dd6ef4a-6e2e-48d6-a401-196df9fac5b4" containerName="main"
Sep 18 19:34:00.770681 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:34:00.770497    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dd6ef4a-6e2e-48d6-a401-196df9fac5b4" containerName="main"
Sep 18 19:34:00.773302 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:34:00.773283    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5fcps"
Sep 18 19:34:00.777137 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:34:00.777110    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 18 19:34:00.777137 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:34:00.777125    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 18 19:34:00.777332 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:34:00.777198    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 18 19:34:00.777332 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:34:00.777266    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-6xxsg\""
Sep 18 19:34:00.786092 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:34:00.786065    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5fcps"]
Sep 18 19:34:00.871459 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:34:00.871426    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9497c6e9-4da4-4714-84f9-bfc917ccd75b-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-5fcps\" (UID: \"9497c6e9-4da4-4714-84f9-bfc917ccd75b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5fcps"
Sep 18 19:34:00.871459 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:34:00.871461    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9497c6e9-4da4-4714-84f9-bfc917ccd75b-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-5fcps\" (UID: \"9497c6e9-4da4-4714-84f9-bfc917ccd75b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5fcps"
Sep 18 19:34:00.871667 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:34:00.871494    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzn5b\" (UniqueName: \"kubernetes.io/projected/9497c6e9-4da4-4714-84f9-bfc917ccd75b-kube-api-access-zzn5b\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-5fcps\" (UID: \"9497c6e9-4da4-4714-84f9-bfc917ccd75b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5fcps"
Sep 18 19:34:00.871667 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:34:00.871546    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9497c6e9-4da4-4714-84f9-bfc917ccd75b-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-5fcps\" (UID: \"9497c6e9-4da4-4714-84f9-bfc917ccd75b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5fcps"
Sep 18 19:34:00.871667 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:34:00.871577    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9497c6e9-4da4-4714-84f9-bfc917ccd75b-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-5fcps\" (UID: \"9497c6e9-4da4-4714-84f9-bfc917ccd75b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5fcps"
Sep 18 19:34:00.871667 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:34:00.871620    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9497c6e9-4da4-4714-84f9-bfc917ccd75b-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-5fcps\" (UID: \"9497c6e9-4da4-4714-84f9-bfc917ccd75b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5fcps"
Sep 18 19:34:00.972546 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:34:00.972500    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9497c6e9-4da4-4714-84f9-bfc917ccd75b-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-5fcps\" (UID: \"9497c6e9-4da4-4714-84f9-bfc917ccd75b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5fcps"
Sep 18 19:34:00.972750 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:34:00.972568    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9497c6e9-4da4-4714-84f9-bfc917ccd75b-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-5fcps\" (UID: \"9497c6e9-4da4-4714-84f9-bfc917ccd75b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5fcps"
Sep 18 19:34:00.972750 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:34:00.972604    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9497c6e9-4da4-4714-84f9-bfc917ccd75b-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-5fcps\" (UID: \"9497c6e9-4da4-4714-84f9-bfc917ccd75b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5fcps"
Sep 18 19:34:00.972750 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:34:00.972621    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9497c6e9-4da4-4714-84f9-bfc917ccd75b-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-5fcps\" (UID: \"9497c6e9-4da4-4714-84f9-bfc917ccd75b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5fcps"
Sep 18 19:34:00.972750 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:34:00.972661    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzn5b\" (UniqueName: \"kubernetes.io/projected/9497c6e9-4da4-4714-84f9-bfc917ccd75b-kube-api-access-zzn5b\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-5fcps\" (UID: \"9497c6e9-4da4-4714-84f9-bfc917ccd75b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5fcps"
Sep 18 19:34:00.972750 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:34:00.972683    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9497c6e9-4da4-4714-84f9-bfc917ccd75b-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-5fcps\" (UID: \"9497c6e9-4da4-4714-84f9-bfc917ccd75b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5fcps"
Sep 18 19:34:00.973108 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:34:00.973082    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9497c6e9-4da4-4714-84f9-bfc917ccd75b-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-5fcps\" (UID: \"9497c6e9-4da4-4714-84f9-bfc917ccd75b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5fcps"
Sep 18 19:34:00.973167 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:34:00.973102    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9497c6e9-4da4-4714-84f9-bfc917ccd75b-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-5fcps\" (UID: \"9497c6e9-4da4-4714-84f9-bfc917ccd75b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5fcps"
Sep 18 19:34:00.973167 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:34:00.973160    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9497c6e9-4da4-4714-84f9-bfc917ccd75b-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-5fcps\" (UID: \"9497c6e9-4da4-4714-84f9-bfc917ccd75b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5fcps"
Sep 18 19:34:00.974883 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:34:00.974866    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9497c6e9-4da4-4714-84f9-bfc917ccd75b-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-5fcps\" (UID: \"9497c6e9-4da4-4714-84f9-bfc917ccd75b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5fcps"
Sep 18 19:34:00.975194 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:34:00.975177    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9497c6e9-4da4-4714-84f9-bfc917ccd75b-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-5fcps\" (UID: \"9497c6e9-4da4-4714-84f9-bfc917ccd75b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5fcps"
Sep 18 19:34:00.982024 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:34:00.982000    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzn5b\" (UniqueName: \"kubernetes.io/projected/9497c6e9-4da4-4714-84f9-bfc917ccd75b-kube-api-access-zzn5b\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-5fcps\" (UID: \"9497c6e9-4da4-4714-84f9-bfc917ccd75b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5fcps"
Sep 18 19:34:01.083078 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:34:01.082978    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5fcps"
Sep 18 19:34:01.357853 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:34:01.357790    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5fcps"]
Sep 18 19:34:01.361428 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:34:01.361392    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9497c6e9_4da4_4714_84f9_bfc917ccd75b.slice/crio-884fd253b788c27988500293b609580bb3aa2e4a9b61e4d13ac0978d18b7ad0c WatchSource:0}: Error finding container 884fd253b788c27988500293b609580bb3aa2e4a9b61e4d13ac0978d18b7ad0c: Status 404 returned error can't find the container with id 884fd253b788c27988500293b609580bb3aa2e4a9b61e4d13ac0978d18b7ad0c
Sep 18 19:34:01.374872 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:34:01.374848    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5fcps" event={"ID":"9497c6e9-4da4-4714-84f9-bfc917ccd75b","Type":"ContainerStarted","Data":"884fd253b788c27988500293b609580bb3aa2e4a9b61e4d13ac0978d18b7ad0c"}
Sep 18 19:34:02.380829 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:34:02.380789    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5fcps" event={"ID":"9497c6e9-4da4-4714-84f9-bfc917ccd75b","Type":"ContainerStarted","Data":"3bd1635d185d676d7648a295fa2aa73b41c35cc930206affbea216b88b6fefb5"}
Sep 18 19:34:02.403134 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:34:02.403082    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5fcps" podStartSLOduration=2.403067087 podStartE2EDuration="2.403067087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:34:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:34:02.401227167 +0000 UTC m=+1213.253527538" watchObservedRunningTime="2026-09-18 19:34:02.403067087 +0000 UTC m=+1213.255367457"
Sep 18 19:34:03.083804 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:34:03.083767    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5fcps"
Sep 18 19:34:11.083931 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:34:11.083888    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5fcps"
Sep 18 19:34:11.096771 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:34:11.096746    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5fcps"
Sep 18 19:34:11.438745 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:34:11.438720    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5fcps"
Sep 18 19:35:12.445014 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:35:12.444969    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5fcps"]
Sep 18 19:35:12.453558 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:35:12.453526    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 19:35:12.453558 ip-10-0-129-30 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:35:12.453558 ip-10-0-129-30 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 18 19:35:12.453558 ip-10-0-129-30 kubenswrapper[2577]:  > containerID="3bd1635d185d676d7648a295fa2aa73b41c35cc930206affbea216b88b6fefb5" cmd=["/bin/sleep","15"]
Sep 18 19:35:12.453783 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:35:12.453573    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5fcps" message="" err=<
Sep 18 19:35:12.453783 ip-10-0-129-30 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:35:12.453783 ip-10-0-129-30 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 18 19:35:12.453783 ip-10-0-129-30 kubenswrapper[2577]:  >
Sep 18 19:35:12.453783 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:35:12.453585    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 19:35:12.453783 ip-10-0-129-30 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:35:12.453783 ip-10-0-129-30 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 18 19:35:12.453783 ip-10-0-129-30 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5fcps" podUID="9497c6e9-4da4-4714-84f9-bfc917ccd75b" containerName="main" containerID="cri-o://3bd1635d185d676d7648a295fa2aa73b41c35cc930206affbea216b88b6fefb5"
Sep 18 19:35:12.453783 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:35:12.453618    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5fcps" podUID="9497c6e9-4da4-4714-84f9-bfc917ccd75b" containerName="main" containerID="cri-o://3bd1635d185d676d7648a295fa2aa73b41c35cc930206affbea216b88b6fefb5" gracePeriod=60
Sep 18 19:35:12.687739 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:35:12.687710    2577 generic.go:356] "Generic (PLEG): container finished" podID="9497c6e9-4da4-4714-84f9-bfc917ccd75b" containerID="3bd1635d185d676d7648a295fa2aa73b41c35cc930206affbea216b88b6fefb5" exitCode=0
Sep 18 19:35:12.687887 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:35:12.687764    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5fcps" event={"ID":"9497c6e9-4da4-4714-84f9-bfc917ccd75b","Type":"ContainerDied","Data":"3bd1635d185d676d7648a295fa2aa73b41c35cc930206affbea216b88b6fefb5"}
Sep 18 19:35:12.803972 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:35:12.803940    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5fcps"
Sep 18 19:35:12.907790 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:35:12.907758    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9497c6e9-4da4-4714-84f9-bfc917ccd75b-dshm\") pod \"9497c6e9-4da4-4714-84f9-bfc917ccd75b\" (UID: \"9497c6e9-4da4-4714-84f9-bfc917ccd75b\") "
Sep 18 19:35:12.907790 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:35:12.907793    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9497c6e9-4da4-4714-84f9-bfc917ccd75b-tmp-dir\") pod \"9497c6e9-4da4-4714-84f9-bfc917ccd75b\" (UID: \"9497c6e9-4da4-4714-84f9-bfc917ccd75b\") "
Sep 18 19:35:12.908078 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:35:12.907832    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzn5b\" (UniqueName: \"kubernetes.io/projected/9497c6e9-4da4-4714-84f9-bfc917ccd75b-kube-api-access-zzn5b\") pod \"9497c6e9-4da4-4714-84f9-bfc917ccd75b\" (UID: \"9497c6e9-4da4-4714-84f9-bfc917ccd75b\") "
Sep 18 19:35:12.908078 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:35:12.907866    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9497c6e9-4da4-4714-84f9-bfc917ccd75b-model-cache\") pod \"9497c6e9-4da4-4714-84f9-bfc917ccd75b\" (UID: \"9497c6e9-4da4-4714-84f9-bfc917ccd75b\") "
Sep 18 19:35:12.908078 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:35:12.907888    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9497c6e9-4da4-4714-84f9-bfc917ccd75b-home\") pod \"9497c6e9-4da4-4714-84f9-bfc917ccd75b\" (UID: \"9497c6e9-4da4-4714-84f9-bfc917ccd75b\") "
Sep 18 19:35:12.908078 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:35:12.907945    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9497c6e9-4da4-4714-84f9-bfc917ccd75b-tls-certs\") pod \"9497c6e9-4da4-4714-84f9-bfc917ccd75b\" (UID: \"9497c6e9-4da4-4714-84f9-bfc917ccd75b\") "
Sep 18 19:35:12.908286 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:35:12.908120    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9497c6e9-4da4-4714-84f9-bfc917ccd75b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9497c6e9-4da4-4714-84f9-bfc917ccd75b" (UID: "9497c6e9-4da4-4714-84f9-bfc917ccd75b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:35:12.908286 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:35:12.908167    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9497c6e9-4da4-4714-84f9-bfc917ccd75b-model-cache" (OuterVolumeSpecName: "model-cache") pod "9497c6e9-4da4-4714-84f9-bfc917ccd75b" (UID: "9497c6e9-4da4-4714-84f9-bfc917ccd75b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:35:12.908286 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:35:12.908216    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9497c6e9-4da4-4714-84f9-bfc917ccd75b-home" (OuterVolumeSpecName: "home") pod "9497c6e9-4da4-4714-84f9-bfc917ccd75b" (UID: "9497c6e9-4da4-4714-84f9-bfc917ccd75b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:35:12.908286 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:35:12.908231    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9497c6e9-4da4-4714-84f9-bfc917ccd75b-model-cache\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:35:12.908286 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:35:12.908259    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9497c6e9-4da4-4714-84f9-bfc917ccd75b-tmp-dir\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:35:12.910232 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:35:12.910201    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9497c6e9-4da4-4714-84f9-bfc917ccd75b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9497c6e9-4da4-4714-84f9-bfc917ccd75b" (UID: "9497c6e9-4da4-4714-84f9-bfc917ccd75b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:35:12.910232 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:35:12.910206    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9497c6e9-4da4-4714-84f9-bfc917ccd75b-dshm" (OuterVolumeSpecName: "dshm") pod "9497c6e9-4da4-4714-84f9-bfc917ccd75b" (UID: "9497c6e9-4da4-4714-84f9-bfc917ccd75b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:35:12.910232 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:35:12.910223    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9497c6e9-4da4-4714-84f9-bfc917ccd75b-kube-api-access-zzn5b" (OuterVolumeSpecName: "kube-api-access-zzn5b") pod "9497c6e9-4da4-4714-84f9-bfc917ccd75b" (UID: "9497c6e9-4da4-4714-84f9-bfc917ccd75b"). InnerVolumeSpecName "kube-api-access-zzn5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:35:13.009433 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:35:13.009340    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9497c6e9-4da4-4714-84f9-bfc917ccd75b-dshm\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:35:13.009433 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:35:13.009371    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzn5b\" (UniqueName: \"kubernetes.io/projected/9497c6e9-4da4-4714-84f9-bfc917ccd75b-kube-api-access-zzn5b\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:35:13.009433 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:35:13.009388    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9497c6e9-4da4-4714-84f9-bfc917ccd75b-home\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:35:13.009433 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:35:13.009399    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9497c6e9-4da4-4714-84f9-bfc917ccd75b-tls-certs\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:35:13.693908 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:35:13.693876    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5fcps"
Sep 18 19:35:13.694441 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:35:13.693912    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5fcps" event={"ID":"9497c6e9-4da4-4714-84f9-bfc917ccd75b","Type":"ContainerDied","Data":"884fd253b788c27988500293b609580bb3aa2e4a9b61e4d13ac0978d18b7ad0c"}
Sep 18 19:35:13.694441 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:35:13.693973    2577 scope.go:117] "RemoveContainer" containerID="3bd1635d185d676d7648a295fa2aa73b41c35cc930206affbea216b88b6fefb5"
Sep 18 19:35:13.713631 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:35:13.713607    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5fcps"]
Sep 18 19:35:13.718928 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:35:13.718907    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5fcps"]
Sep 18 19:35:15.438077 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:35:15.438043    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9497c6e9-4da4-4714-84f9-bfc917ccd75b" path="/var/lib/kubelet/pods/9497c6e9-4da4-4714-84f9-bfc917ccd75b/volumes"
Sep 18 19:37:40.431223 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:37:40.431189    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc"]
Sep 18 19:37:40.431752 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:37:40.431658    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9497c6e9-4da4-4714-84f9-bfc917ccd75b" containerName="main"
Sep 18 19:37:40.431752 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:37:40.431676    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9497c6e9-4da4-4714-84f9-bfc917ccd75b" containerName="main"
Sep 18 19:37:40.431875 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:37:40.431758    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9497c6e9-4da4-4714-84f9-bfc917ccd75b" containerName="main"
Sep 18 19:37:40.435027 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:37:40.434983    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc"
Sep 18 19:37:40.447548 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:37:40.447530    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 18 19:37:40.448617 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:37:40.448595    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 18 19:37:40.448795 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:37:40.448777    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 18 19:37:40.449102 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:37:40.449086    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-j2f5n\""
Sep 18 19:37:40.455304 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:37:40.455288    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-m29z2\""
Sep 18 19:37:40.470603 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:37:40.470582    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc"]
Sep 18 19:37:40.600854 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:37:40.600818    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shzqx\" (UniqueName: \"kubernetes.io/projected/27f6ca7d-b589-46d2-95fd-df2790d73fcd-kube-api-access-shzqx\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc\" (UID: \"27f6ca7d-b589-46d2-95fd-df2790d73fcd\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc"
Sep 18 19:37:40.601076 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:37:40.600871    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27f6ca7d-b589-46d2-95fd-df2790d73fcd-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc\" (UID: \"27f6ca7d-b589-46d2-95fd-df2790d73fcd\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc"
Sep 18 19:37:40.601076 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:37:40.601021    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27f6ca7d-b589-46d2-95fd-df2790d73fcd-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc\" (UID: \"27f6ca7d-b589-46d2-95fd-df2790d73fcd\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc"
Sep 18 19:37:40.601076 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:37:40.601059    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27f6ca7d-b589-46d2-95fd-df2790d73fcd-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc\" (UID: \"27f6ca7d-b589-46d2-95fd-df2790d73fcd\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc"
Sep 18 19:37:40.601076 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:37:40.601079    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27f6ca7d-b589-46d2-95fd-df2790d73fcd-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc\" (UID: \"27f6ca7d-b589-46d2-95fd-df2790d73fcd\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc"
Sep 18 19:37:40.601302 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:37:40.601110    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27f6ca7d-b589-46d2-95fd-df2790d73fcd-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc\" (UID: \"27f6ca7d-b589-46d2-95fd-df2790d73fcd\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc"
Sep 18 19:37:40.702393 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:37:40.702311    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shzqx\" (UniqueName: \"kubernetes.io/projected/27f6ca7d-b589-46d2-95fd-df2790d73fcd-kube-api-access-shzqx\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc\" (UID: \"27f6ca7d-b589-46d2-95fd-df2790d73fcd\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc"
Sep 18 19:37:40.702393 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:37:40.702349    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27f6ca7d-b589-46d2-95fd-df2790d73fcd-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc\" (UID: \"27f6ca7d-b589-46d2-95fd-df2790d73fcd\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc"
Sep 18 19:37:40.702585 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:37:40.702441    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27f6ca7d-b589-46d2-95fd-df2790d73fcd-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc\" (UID: \"27f6ca7d-b589-46d2-95fd-df2790d73fcd\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc"
Sep 18 19:37:40.702585 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:37:40.702491    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27f6ca7d-b589-46d2-95fd-df2790d73fcd-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc\" (UID: \"27f6ca7d-b589-46d2-95fd-df2790d73fcd\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc"
Sep 18 19:37:40.702585 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:37:40.702515    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27f6ca7d-b589-46d2-95fd-df2790d73fcd-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc\" (UID: \"27f6ca7d-b589-46d2-95fd-df2790d73fcd\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc"
Sep 18 19:37:40.702585 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:37:40.702554    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27f6ca7d-b589-46d2-95fd-df2790d73fcd-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc\" (UID: \"27f6ca7d-b589-46d2-95fd-df2790d73fcd\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc"
Sep 18 19:37:40.702847 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:37:40.702829    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27f6ca7d-b589-46d2-95fd-df2790d73fcd-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc\" (UID: \"27f6ca7d-b589-46d2-95fd-df2790d73fcd\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc"
Sep 18 19:37:40.702921 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:37:40.702860    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27f6ca7d-b589-46d2-95fd-df2790d73fcd-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc\" (UID: \"27f6ca7d-b589-46d2-95fd-df2790d73fcd\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc"
Sep 18 19:37:40.702976 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:37:40.702914    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27f6ca7d-b589-46d2-95fd-df2790d73fcd-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc\" (UID: \"27f6ca7d-b589-46d2-95fd-df2790d73fcd\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc"
Sep 18 19:37:40.704769 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:37:40.704749    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27f6ca7d-b589-46d2-95fd-df2790d73fcd-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc\" (UID: \"27f6ca7d-b589-46d2-95fd-df2790d73fcd\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc"
Sep 18 19:37:40.704874 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:37:40.704858    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27f6ca7d-b589-46d2-95fd-df2790d73fcd-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc\" (UID: \"27f6ca7d-b589-46d2-95fd-df2790d73fcd\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc"
Sep 18 19:37:40.712072 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:37:40.712052    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shzqx\" (UniqueName: \"kubernetes.io/projected/27f6ca7d-b589-46d2-95fd-df2790d73fcd-kube-api-access-shzqx\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc\" (UID: \"27f6ca7d-b589-46d2-95fd-df2790d73fcd\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc"
Sep 18 19:37:40.744748 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:37:40.744716    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc"
Sep 18 19:37:41.026449 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:37:41.026423    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc"]
Sep 18 19:37:41.029117 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:37:41.029080    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f6ca7d_b589_46d2_95fd_df2790d73fcd.slice/crio-7d0e4b3057a9d6de1cf89330072d02bd84398163705db62935d8162c5672b3a6 WatchSource:0}: Error finding container 7d0e4b3057a9d6de1cf89330072d02bd84398163705db62935d8162c5672b3a6: Status 404 returned error can't find the container with id 7d0e4b3057a9d6de1cf89330072d02bd84398163705db62935d8162c5672b3a6
Sep 18 19:37:41.031229 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:37:41.031211    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:37:41.311966 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:37:41.311885    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc" event={"ID":"27f6ca7d-b589-46d2-95fd-df2790d73fcd","Type":"ContainerStarted","Data":"7d0e4b3057a9d6de1cf89330072d02bd84398163705db62935d8162c5672b3a6"}
Sep 18 19:37:44.327934 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:37:44.327894    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc" event={"ID":"27f6ca7d-b589-46d2-95fd-df2790d73fcd","Type":"ContainerStarted","Data":"8e88edadd620a73814247dba219488181db004e823416ddb3a349d05ce12bb8b"}
Sep 18 19:37:44.328357 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:37:44.328027    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc"
Sep 18 19:37:45.334534 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:37:45.334497    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc" event={"ID":"27f6ca7d-b589-46d2-95fd-df2790d73fcd","Type":"ContainerStarted","Data":"2b0cce2485cd37ace99c994d8d100076efdabb7fa6f8b11bbf268551c89ec0d9"}
Sep 18 19:37:45.744878 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:37:45.744830    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc"
Sep 18 19:37:50.745144 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:37:50.745111    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc"
Sep 18 19:37:50.757444 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:37:50.757418    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc"
Sep 18 19:37:51.115579 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:37:51.115519    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc" podStartSLOduration=8.867657579 podStartE2EDuration="11.115496566s" totalImagesPullingTime="2.247838987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:37:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:37:49.927520756 +0000 UTC m=+1440.779821128" watchObservedRunningTime="2026-09-18 19:37:51.115496566 +0000 UTC m=+1441.967796947"
Sep 18 19:37:51.373156 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:37:51.373077    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc"
Sep 18 19:38:02.378667 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:38:02.378634    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc"
Sep 18 19:39:20.633703 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:39:20.633624    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc"]
Sep 18 19:39:20.642702 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:39:20.642670    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 19:39:20.642702 ip-10-0-129-30 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:39:20.642702 ip-10-0-129-30 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 18 19:39:20.642702 ip-10-0-129-30 kubenswrapper[2577]:  > containerID="2b0cce2485cd37ace99c994d8d100076efdabb7fa6f8b11bbf268551c89ec0d9" cmd=["/bin/sleep","15"]
Sep 18 19:39:20.642965 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:39:20.642720    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc" message="" err=<
Sep 18 19:39:20.642965 ip-10-0-129-30 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:39:20.642965 ip-10-0-129-30 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 18 19:39:20.642965 ip-10-0-129-30 kubenswrapper[2577]:  >
Sep 18 19:39:20.642965 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:39:20.642733    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 19:39:20.642965 ip-10-0-129-30 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:39:20.642965 ip-10-0-129-30 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 18 19:39:20.642965 ip-10-0-129-30 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc" podUID="27f6ca7d-b589-46d2-95fd-df2790d73fcd" containerName="main" containerID="cri-o://2b0cce2485cd37ace99c994d8d100076efdabb7fa6f8b11bbf268551c89ec0d9"
Sep 18 19:39:20.642965 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:39:20.642774    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc" podUID="27f6ca7d-b589-46d2-95fd-df2790d73fcd" containerName="main" containerID="cri-o://2b0cce2485cd37ace99c994d8d100076efdabb7fa6f8b11bbf268551c89ec0d9" gracePeriod=60
Sep 18 19:39:20.858832 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:39:20.858795    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc" podUID="27f6ca7d-b589-46d2-95fd-df2790d73fcd" containerName="llm-d-routing-sidecar" containerID="cri-o://8e88edadd620a73814247dba219488181db004e823416ddb3a349d05ce12bb8b" gracePeriod=60
Sep 18 19:39:21.110456 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:39:21.110421    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc"
Sep 18 19:39:21.295529 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:39:21.295433    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27f6ca7d-b589-46d2-95fd-df2790d73fcd-model-cache\") pod \"27f6ca7d-b589-46d2-95fd-df2790d73fcd\" (UID: \"27f6ca7d-b589-46d2-95fd-df2790d73fcd\") "
Sep 18 19:39:21.295529 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:39:21.295483    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27f6ca7d-b589-46d2-95fd-df2790d73fcd-dshm\") pod \"27f6ca7d-b589-46d2-95fd-df2790d73fcd\" (UID: \"27f6ca7d-b589-46d2-95fd-df2790d73fcd\") "
Sep 18 19:39:21.295769 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:39:21.295535    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shzqx\" (UniqueName: \"kubernetes.io/projected/27f6ca7d-b589-46d2-95fd-df2790d73fcd-kube-api-access-shzqx\") pod \"27f6ca7d-b589-46d2-95fd-df2790d73fcd\" (UID: \"27f6ca7d-b589-46d2-95fd-df2790d73fcd\") "
Sep 18 19:39:21.295769 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:39:21.295590    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27f6ca7d-b589-46d2-95fd-df2790d73fcd-tmp-dir\") pod \"27f6ca7d-b589-46d2-95fd-df2790d73fcd\" (UID: \"27f6ca7d-b589-46d2-95fd-df2790d73fcd\") "
Sep 18 19:39:21.295769 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:39:21.295632    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27f6ca7d-b589-46d2-95fd-df2790d73fcd-tls-certs\") pod \"27f6ca7d-b589-46d2-95fd-df2790d73fcd\" (UID: \"27f6ca7d-b589-46d2-95fd-df2790d73fcd\") "
Sep 18 19:39:21.295769 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:39:21.295685    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27f6ca7d-b589-46d2-95fd-df2790d73fcd-home\") pod \"27f6ca7d-b589-46d2-95fd-df2790d73fcd\" (UID: \"27f6ca7d-b589-46d2-95fd-df2790d73fcd\") "
Sep 18 19:39:21.295964 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:39:21.295762    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f6ca7d-b589-46d2-95fd-df2790d73fcd-model-cache" (OuterVolumeSpecName: "model-cache") pod "27f6ca7d-b589-46d2-95fd-df2790d73fcd" (UID: "27f6ca7d-b589-46d2-95fd-df2790d73fcd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:39:21.295964 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:39:21.295883    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f6ca7d-b589-46d2-95fd-df2790d73fcd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "27f6ca7d-b589-46d2-95fd-df2790d73fcd" (UID: "27f6ca7d-b589-46d2-95fd-df2790d73fcd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:39:21.296098 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:39:21.295972    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f6ca7d-b589-46d2-95fd-df2790d73fcd-home" (OuterVolumeSpecName: "home") pod "27f6ca7d-b589-46d2-95fd-df2790d73fcd" (UID: "27f6ca7d-b589-46d2-95fd-df2790d73fcd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:39:21.296098 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:39:21.296075    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27f6ca7d-b589-46d2-95fd-df2790d73fcd-home\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:39:21.296098 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:39:21.296094    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27f6ca7d-b589-46d2-95fd-df2790d73fcd-model-cache\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:39:21.296253 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:39:21.296132    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27f6ca7d-b589-46d2-95fd-df2790d73fcd-tmp-dir\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:39:21.297866 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:39:21.297841    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f6ca7d-b589-46d2-95fd-df2790d73fcd-dshm" (OuterVolumeSpecName: "dshm") pod "27f6ca7d-b589-46d2-95fd-df2790d73fcd" (UID: "27f6ca7d-b589-46d2-95fd-df2790d73fcd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:39:21.298008 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:39:21.297909    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27f6ca7d-b589-46d2-95fd-df2790d73fcd-kube-api-access-shzqx" (OuterVolumeSpecName: "kube-api-access-shzqx") pod "27f6ca7d-b589-46d2-95fd-df2790d73fcd" (UID: "27f6ca7d-b589-46d2-95fd-df2790d73fcd"). InnerVolumeSpecName "kube-api-access-shzqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:39:21.298008 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:39:21.297947    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27f6ca7d-b589-46d2-95fd-df2790d73fcd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "27f6ca7d-b589-46d2-95fd-df2790d73fcd" (UID: "27f6ca7d-b589-46d2-95fd-df2790d73fcd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:39:21.397379 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:39:21.397342    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shzqx\" (UniqueName: \"kubernetes.io/projected/27f6ca7d-b589-46d2-95fd-df2790d73fcd-kube-api-access-shzqx\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:39:21.397379 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:39:21.397372    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27f6ca7d-b589-46d2-95fd-df2790d73fcd-tls-certs\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:39:21.397379 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:39:21.397383    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27f6ca7d-b589-46d2-95fd-df2790d73fcd-dshm\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:39:21.774161 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:39:21.774130    2577 generic.go:356] "Generic (PLEG): container finished" podID="27f6ca7d-b589-46d2-95fd-df2790d73fcd" containerID="2b0cce2485cd37ace99c994d8d100076efdabb7fa6f8b11bbf268551c89ec0d9" exitCode=0
Sep 18 19:39:21.774161 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:39:21.774156    2577 generic.go:356] "Generic (PLEG): container finished" podID="27f6ca7d-b589-46d2-95fd-df2790d73fcd" containerID="8e88edadd620a73814247dba219488181db004e823416ddb3a349d05ce12bb8b" exitCode=0
Sep 18 19:39:21.774649 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:39:21.774196    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc"
Sep 18 19:39:21.774649 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:39:21.774209    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc" event={"ID":"27f6ca7d-b589-46d2-95fd-df2790d73fcd","Type":"ContainerDied","Data":"2b0cce2485cd37ace99c994d8d100076efdabb7fa6f8b11bbf268551c89ec0d9"}
Sep 18 19:39:21.774649 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:39:21.774289    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc" event={"ID":"27f6ca7d-b589-46d2-95fd-df2790d73fcd","Type":"ContainerDied","Data":"8e88edadd620a73814247dba219488181db004e823416ddb3a349d05ce12bb8b"}
Sep 18 19:39:21.774649 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:39:21.774319    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc" event={"ID":"27f6ca7d-b589-46d2-95fd-df2790d73fcd","Type":"ContainerDied","Data":"7d0e4b3057a9d6de1cf89330072d02bd84398163705db62935d8162c5672b3a6"}
Sep 18 19:39:21.774649 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:39:21.774348    2577 scope.go:117] "RemoveContainer" containerID="2b0cce2485cd37ace99c994d8d100076efdabb7fa6f8b11bbf268551c89ec0d9"
Sep 18 19:39:21.782875 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:39:21.782846    2577 scope.go:117] "RemoveContainer" containerID="8e88edadd620a73814247dba219488181db004e823416ddb3a349d05ce12bb8b"
Sep 18 19:39:21.789770 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:39:21.789750    2577 scope.go:117] "RemoveContainer" containerID="2b0cce2485cd37ace99c994d8d100076efdabb7fa6f8b11bbf268551c89ec0d9"
Sep 18 19:39:21.790042 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:39:21.790024    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b0cce2485cd37ace99c994d8d100076efdabb7fa6f8b11bbf268551c89ec0d9\": container with ID starting with 2b0cce2485cd37ace99c994d8d100076efdabb7fa6f8b11bbf268551c89ec0d9 not found: ID does not exist" containerID="2b0cce2485cd37ace99c994d8d100076efdabb7fa6f8b11bbf268551c89ec0d9"
Sep 18 19:39:21.790120 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:39:21.790047    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b0cce2485cd37ace99c994d8d100076efdabb7fa6f8b11bbf268551c89ec0d9"} err="failed to get container status \"2b0cce2485cd37ace99c994d8d100076efdabb7fa6f8b11bbf268551c89ec0d9\": rpc error: code = NotFound desc = could not find container \"2b0cce2485cd37ace99c994d8d100076efdabb7fa6f8b11bbf268551c89ec0d9\": container with ID starting with 2b0cce2485cd37ace99c994d8d100076efdabb7fa6f8b11bbf268551c89ec0d9 not found: ID does not exist"
Sep 18 19:39:21.790120 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:39:21.790065    2577 scope.go:117] "RemoveContainer" containerID="8e88edadd620a73814247dba219488181db004e823416ddb3a349d05ce12bb8b"
Sep 18 19:39:21.790294 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:39:21.790279    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e88edadd620a73814247dba219488181db004e823416ddb3a349d05ce12bb8b\": container with ID starting with 8e88edadd620a73814247dba219488181db004e823416ddb3a349d05ce12bb8b not found: ID does not exist" containerID="8e88edadd620a73814247dba219488181db004e823416ddb3a349d05ce12bb8b"
Sep 18 19:39:21.790373 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:39:21.790298    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e88edadd620a73814247dba219488181db004e823416ddb3a349d05ce12bb8b"} err="failed to get container status \"8e88edadd620a73814247dba219488181db004e823416ddb3a349d05ce12bb8b\": rpc error: code = NotFound desc = could not find container \"8e88edadd620a73814247dba219488181db004e823416ddb3a349d05ce12bb8b\": container with ID starting with 8e88edadd620a73814247dba219488181db004e823416ddb3a349d05ce12bb8b not found: ID does not exist"
Sep 18 19:39:21.790373 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:39:21.790310    2577 scope.go:117] "RemoveContainer" containerID="2b0cce2485cd37ace99c994d8d100076efdabb7fa6f8b11bbf268551c89ec0d9"
Sep 18 19:39:21.790526 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:39:21.790511    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b0cce2485cd37ace99c994d8d100076efdabb7fa6f8b11bbf268551c89ec0d9"} err="failed to get container status \"2b0cce2485cd37ace99c994d8d100076efdabb7fa6f8b11bbf268551c89ec0d9\": rpc error: code = NotFound desc = could not find container \"2b0cce2485cd37ace99c994d8d100076efdabb7fa6f8b11bbf268551c89ec0d9\": container with ID starting with 2b0cce2485cd37ace99c994d8d100076efdabb7fa6f8b11bbf268551c89ec0d9 not found: ID does not exist"
Sep 18 19:39:21.790567 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:39:21.790526    2577 scope.go:117] "RemoveContainer" containerID="8e88edadd620a73814247dba219488181db004e823416ddb3a349d05ce12bb8b"
Sep 18 19:39:21.790773 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:39:21.790754    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e88edadd620a73814247dba219488181db004e823416ddb3a349d05ce12bb8b"} err="failed to get container status \"8e88edadd620a73814247dba219488181db004e823416ddb3a349d05ce12bb8b\": rpc error: code = NotFound desc = could not find container \"8e88edadd620a73814247dba219488181db004e823416ddb3a349d05ce12bb8b\": container with ID starting with 8e88edadd620a73814247dba219488181db004e823416ddb3a349d05ce12bb8b not found: ID does not exist"
Sep 18 19:39:21.795778 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:39:21.795751    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc"]
Sep 18 19:39:21.800088 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:39:21.800068    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-q2xkc"]
Sep 18 19:39:23.438088 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:39:23.438052    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27f6ca7d-b589-46d2-95fd-df2790d73fcd" path="/var/lib/kubelet/pods/27f6ca7d-b589-46d2-95fd-df2790d73fcd/volumes"
Sep 18 19:40:21.255945 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:21.255907    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7"]
Sep 18 19:40:21.256371 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:21.256274    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27f6ca7d-b589-46d2-95fd-df2790d73fcd" containerName="main"
Sep 18 19:40:21.256371 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:21.256285    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="27f6ca7d-b589-46d2-95fd-df2790d73fcd" containerName="main"
Sep 18 19:40:21.256371 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:21.256295    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27f6ca7d-b589-46d2-95fd-df2790d73fcd" containerName="llm-d-routing-sidecar"
Sep 18 19:40:21.256371 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:21.256301    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="27f6ca7d-b589-46d2-95fd-df2790d73fcd" containerName="llm-d-routing-sidecar"
Sep 18 19:40:21.256371 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:21.256357    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="27f6ca7d-b589-46d2-95fd-df2790d73fcd" containerName="llm-d-routing-sidecar"
Sep 18 19:40:21.256371 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:21.256367    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="27f6ca7d-b589-46d2-95fd-df2790d73fcd" containerName="main"
Sep 18 19:40:21.259405 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:21.259388    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7"
Sep 18 19:40:21.263932 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:21.263907    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 18 19:40:21.264595 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:21.264564    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 18 19:40:21.264595 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:21.264571    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-dockercfg-xf5vh\""
Sep 18 19:40:21.264777 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:21.264575    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-r66hn\""
Sep 18 19:40:21.264777 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:21.264680    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 18 19:40:21.285744 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:21.285719    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7"]
Sep 18 19:40:21.302261 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:21.302236    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8a75bef-1c87-4259-925d-fa2e356b87d0-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7\" (UID: \"e8a75bef-1c87-4259-925d-fa2e356b87d0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7"
Sep 18 19:40:21.302390 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:21.302267    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8a75bef-1c87-4259-925d-fa2e356b87d0-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7\" (UID: \"e8a75bef-1c87-4259-925d-fa2e356b87d0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7"
Sep 18 19:40:21.302390 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:21.302329    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpb2t\" (UniqueName: \"kubernetes.io/projected/e8a75bef-1c87-4259-925d-fa2e356b87d0-kube-api-access-cpb2t\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7\" (UID: \"e8a75bef-1c87-4259-925d-fa2e356b87d0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7"
Sep 18 19:40:21.302390 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:21.302371    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8a75bef-1c87-4259-925d-fa2e356b87d0-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7\" (UID: \"e8a75bef-1c87-4259-925d-fa2e356b87d0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7"
Sep 18 19:40:21.302584 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:21.302417    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8a75bef-1c87-4259-925d-fa2e356b87d0-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7\" (UID: \"e8a75bef-1c87-4259-925d-fa2e356b87d0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7"
Sep 18 19:40:21.302584 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:21.302483    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8a75bef-1c87-4259-925d-fa2e356b87d0-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7\" (UID: \"e8a75bef-1c87-4259-925d-fa2e356b87d0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7"
Sep 18 19:40:21.403230 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:21.403189    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8a75bef-1c87-4259-925d-fa2e356b87d0-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7\" (UID: \"e8a75bef-1c87-4259-925d-fa2e356b87d0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7"
Sep 18 19:40:21.403230 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:21.403235    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8a75bef-1c87-4259-925d-fa2e356b87d0-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7\" (UID: \"e8a75bef-1c87-4259-925d-fa2e356b87d0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7"
Sep 18 19:40:21.403481 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:21.403284    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpb2t\" (UniqueName: \"kubernetes.io/projected/e8a75bef-1c87-4259-925d-fa2e356b87d0-kube-api-access-cpb2t\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7\" (UID: \"e8a75bef-1c87-4259-925d-fa2e356b87d0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7"
Sep 18 19:40:21.403481 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:21.403333    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8a75bef-1c87-4259-925d-fa2e356b87d0-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7\" (UID: \"e8a75bef-1c87-4259-925d-fa2e356b87d0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7"
Sep 18 19:40:21.403481 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:21.403366    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8a75bef-1c87-4259-925d-fa2e356b87d0-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7\" (UID: \"e8a75bef-1c87-4259-925d-fa2e356b87d0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7"
Sep 18 19:40:21.403481 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:21.403393    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8a75bef-1c87-4259-925d-fa2e356b87d0-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7\" (UID: \"e8a75bef-1c87-4259-925d-fa2e356b87d0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7"
Sep 18 19:40:21.403699 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:21.403595    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8a75bef-1c87-4259-925d-fa2e356b87d0-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7\" (UID: \"e8a75bef-1c87-4259-925d-fa2e356b87d0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7"
Sep 18 19:40:21.403699 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:21.403649    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8a75bef-1c87-4259-925d-fa2e356b87d0-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7\" (UID: \"e8a75bef-1c87-4259-925d-fa2e356b87d0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7"
Sep 18 19:40:21.403789 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:21.403715    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8a75bef-1c87-4259-925d-fa2e356b87d0-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7\" (UID: \"e8a75bef-1c87-4259-925d-fa2e356b87d0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7"
Sep 18 19:40:21.405680 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:21.405652    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8a75bef-1c87-4259-925d-fa2e356b87d0-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7\" (UID: \"e8a75bef-1c87-4259-925d-fa2e356b87d0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7"
Sep 18 19:40:21.405874 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:21.405859    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8a75bef-1c87-4259-925d-fa2e356b87d0-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7\" (UID: \"e8a75bef-1c87-4259-925d-fa2e356b87d0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7"
Sep 18 19:40:21.414910 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:21.414880    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpb2t\" (UniqueName: \"kubernetes.io/projected/e8a75bef-1c87-4259-925d-fa2e356b87d0-kube-api-access-cpb2t\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7\" (UID: \"e8a75bef-1c87-4259-925d-fa2e356b87d0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7"
Sep 18 19:40:21.570387 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:21.570303    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7"
Sep 18 19:40:21.845273 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:21.845196    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7"]
Sep 18 19:40:21.848062 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:40:21.848033    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8a75bef_1c87_4259_925d_fa2e356b87d0.slice/crio-bc859fac1980ba9e870c6499fa9e0be8bbb31f7dcbd0c90ec734c23eb3bdcde1 WatchSource:0}: Error finding container bc859fac1980ba9e870c6499fa9e0be8bbb31f7dcbd0c90ec734c23eb3bdcde1: Status 404 returned error can't find the container with id bc859fac1980ba9e870c6499fa9e0be8bbb31f7dcbd0c90ec734c23eb3bdcde1
Sep 18 19:40:22.025077 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:22.025042    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7" event={"ID":"e8a75bef-1c87-4259-925d-fa2e356b87d0","Type":"ContainerStarted","Data":"8f911faf7fbc548f2825f35cca486f97e8e6a61f32e6c2b2a677330245e92f88"}
Sep 18 19:40:22.025260 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:22.025102    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7" event={"ID":"e8a75bef-1c87-4259-925d-fa2e356b87d0","Type":"ContainerStarted","Data":"bc859fac1980ba9e870c6499fa9e0be8bbb31f7dcbd0c90ec734c23eb3bdcde1"}
Sep 18 19:40:22.025260 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:22.025135    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7"
Sep 18 19:40:23.037766 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:23.037724    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7" event={"ID":"e8a75bef-1c87-4259-925d-fa2e356b87d0","Type":"ContainerStarted","Data":"c5e1eda06f75ae56c3089580d5a46d074e1335eff6533dff2bc674a580153615"}
Sep 18 19:40:23.063737 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:23.063687    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7" podStartSLOduration=2.063671939 podStartE2EDuration="2.063671939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:40:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:40:23.061656867 +0000 UTC m=+1593.913957237" watchObservedRunningTime="2026-09-18 19:40:23.063671939 +0000 UTC m=+1593.915972310"
Sep 18 19:40:23.570460 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:23.570430    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7"
Sep 18 19:40:31.571376 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:31.571336    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7"
Sep 18 19:40:31.584146 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:31.584110    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7"
Sep 18 19:40:31.585325 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:31.585301    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7"
Sep 18 19:40:32.088944 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:32.088867    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7"
Sep 18 19:40:32.684755 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:32.684724    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6c5d9d5896-s6dn2"]
Sep 18 19:40:32.688646 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:32.688626    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6c5d9d5896-s6dn2"
Sep 18 19:40:32.692285 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:32.692264    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"router-gateway-1-openshift-default-dockercfg-56h9d\""
Sep 18 19:40:32.692416 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:32.692398    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Sep 18 19:40:32.698721 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:32.698706    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Sep 18 19:40:32.698969 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:32.698956    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-gw-ca-root-cert\""
Sep 18 19:40:32.750104 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:32.750073    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6c5d9d5896-s6dn2"]
Sep 18 19:40:32.807063 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:32.807029    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltcms\" (UniqueName: \"kubernetes.io/projected/11e6710f-408d-4c78-be64-fcf7fae65ffb-kube-api-access-ltcms\") pod \"router-gateway-1-openshift-default-6c5d9d5896-s6dn2\" (UID: \"11e6710f-408d-4c78-be64-fcf7fae65ffb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6c5d9d5896-s6dn2"
Sep 18 19:40:32.807246 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:32.807091    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e6710f-408d-4c78-be64-fcf7fae65ffb-workload-socket\") pod \"router-gateway-1-openshift-default-6c5d9d5896-s6dn2\" (UID: \"11e6710f-408d-4c78-be64-fcf7fae65ffb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6c5d9d5896-s6dn2"
Sep 18 19:40:32.807246 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:32.807134    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e6710f-408d-4c78-be64-fcf7fae65ffb-credential-socket\") pod \"router-gateway-1-openshift-default-6c5d9d5896-s6dn2\" (UID: \"11e6710f-408d-4c78-be64-fcf7fae65ffb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6c5d9d5896-s6dn2"
Sep 18 19:40:32.807246 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:32.807160    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11e6710f-408d-4c78-be64-fcf7fae65ffb-istio-podinfo\") pod \"router-gateway-1-openshift-default-6c5d9d5896-s6dn2\" (UID: \"11e6710f-408d-4c78-be64-fcf7fae65ffb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6c5d9d5896-s6dn2"
Sep 18 19:40:32.807246 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:32.807227    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11e6710f-408d-4c78-be64-fcf7fae65ffb-istio-token\") pod \"router-gateway-1-openshift-default-6c5d9d5896-s6dn2\" (UID: \"11e6710f-408d-4c78-be64-fcf7fae65ffb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6c5d9d5896-s6dn2"
Sep 18 19:40:32.807390 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:32.807271    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11e6710f-408d-4c78-be64-fcf7fae65ffb-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6c5d9d5896-s6dn2\" (UID: \"11e6710f-408d-4c78-be64-fcf7fae65ffb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6c5d9d5896-s6dn2"
Sep 18 19:40:32.807390 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:32.807315    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11e6710f-408d-4c78-be64-fcf7fae65ffb-istio-envoy\") pod \"router-gateway-1-openshift-default-6c5d9d5896-s6dn2\" (UID: \"11e6710f-408d-4c78-be64-fcf7fae65ffb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6c5d9d5896-s6dn2"
Sep 18 19:40:32.807390 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:32.807342    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11e6710f-408d-4c78-be64-fcf7fae65ffb-workload-certs\") pod \"router-gateway-1-openshift-default-6c5d9d5896-s6dn2\" (UID: \"11e6710f-408d-4c78-be64-fcf7fae65ffb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6c5d9d5896-s6dn2"
Sep 18 19:40:32.807390 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:32.807363    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11e6710f-408d-4c78-be64-fcf7fae65ffb-istio-data\") pod \"router-gateway-1-openshift-default-6c5d9d5896-s6dn2\" (UID: \"11e6710f-408d-4c78-be64-fcf7fae65ffb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6c5d9d5896-s6dn2"
Sep 18 19:40:32.908275 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:32.908241    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11e6710f-408d-4c78-be64-fcf7fae65ffb-istio-envoy\") pod \"router-gateway-1-openshift-default-6c5d9d5896-s6dn2\" (UID: \"11e6710f-408d-4c78-be64-fcf7fae65ffb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6c5d9d5896-s6dn2"
Sep 18 19:40:32.908513 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:32.908286    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11e6710f-408d-4c78-be64-fcf7fae65ffb-workload-certs\") pod \"router-gateway-1-openshift-default-6c5d9d5896-s6dn2\" (UID: \"11e6710f-408d-4c78-be64-fcf7fae65ffb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6c5d9d5896-s6dn2"
Sep 18 19:40:32.908513 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:32.908311    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11e6710f-408d-4c78-be64-fcf7fae65ffb-istio-data\") pod \"router-gateway-1-openshift-default-6c5d9d5896-s6dn2\" (UID: \"11e6710f-408d-4c78-be64-fcf7fae65ffb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6c5d9d5896-s6dn2"
Sep 18 19:40:32.908513 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:32.908331    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltcms\" (UniqueName: \"kubernetes.io/projected/11e6710f-408d-4c78-be64-fcf7fae65ffb-kube-api-access-ltcms\") pod \"router-gateway-1-openshift-default-6c5d9d5896-s6dn2\" (UID: \"11e6710f-408d-4c78-be64-fcf7fae65ffb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6c5d9d5896-s6dn2"
Sep 18 19:40:32.908513 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:32.908370    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e6710f-408d-4c78-be64-fcf7fae65ffb-workload-socket\") pod \"router-gateway-1-openshift-default-6c5d9d5896-s6dn2\" (UID: \"11e6710f-408d-4c78-be64-fcf7fae65ffb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6c5d9d5896-s6dn2"
Sep 18 19:40:32.908513 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:32.908393    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e6710f-408d-4c78-be64-fcf7fae65ffb-credential-socket\") pod \"router-gateway-1-openshift-default-6c5d9d5896-s6dn2\" (UID: \"11e6710f-408d-4c78-be64-fcf7fae65ffb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6c5d9d5896-s6dn2"
Sep 18 19:40:32.908513 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:32.908418    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11e6710f-408d-4c78-be64-fcf7fae65ffb-istio-podinfo\") pod \"router-gateway-1-openshift-default-6c5d9d5896-s6dn2\" (UID: \"11e6710f-408d-4c78-be64-fcf7fae65ffb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6c5d9d5896-s6dn2"
Sep 18 19:40:32.908513 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:32.908456    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11e6710f-408d-4c78-be64-fcf7fae65ffb-istio-token\") pod \"router-gateway-1-openshift-default-6c5d9d5896-s6dn2\" (UID: \"11e6710f-408d-4c78-be64-fcf7fae65ffb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6c5d9d5896-s6dn2"
Sep 18 19:40:32.908513 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:32.908485    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11e6710f-408d-4c78-be64-fcf7fae65ffb-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6c5d9d5896-s6dn2\" (UID: \"11e6710f-408d-4c78-be64-fcf7fae65ffb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6c5d9d5896-s6dn2"
Sep 18 19:40:32.909017 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:32.908730    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11e6710f-408d-4c78-be64-fcf7fae65ffb-workload-certs\") pod \"router-gateway-1-openshift-default-6c5d9d5896-s6dn2\" (UID: \"11e6710f-408d-4c78-be64-fcf7fae65ffb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6c5d9d5896-s6dn2"
Sep 18 19:40:32.909017 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:32.908766    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11e6710f-408d-4c78-be64-fcf7fae65ffb-istio-data\") pod \"router-gateway-1-openshift-default-6c5d9d5896-s6dn2\" (UID: \"11e6710f-408d-4c78-be64-fcf7fae65ffb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6c5d9d5896-s6dn2"
Sep 18 19:40:32.909017 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:32.908831    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e6710f-408d-4c78-be64-fcf7fae65ffb-workload-socket\") pod \"router-gateway-1-openshift-default-6c5d9d5896-s6dn2\" (UID: \"11e6710f-408d-4c78-be64-fcf7fae65ffb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6c5d9d5896-s6dn2"
Sep 18 19:40:32.909197 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:32.909029    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e6710f-408d-4c78-be64-fcf7fae65ffb-credential-socket\") pod \"router-gateway-1-openshift-default-6c5d9d5896-s6dn2\" (UID: \"11e6710f-408d-4c78-be64-fcf7fae65ffb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6c5d9d5896-s6dn2"
Sep 18 19:40:32.909255 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:32.909213    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11e6710f-408d-4c78-be64-fcf7fae65ffb-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6c5d9d5896-s6dn2\" (UID: \"11e6710f-408d-4c78-be64-fcf7fae65ffb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6c5d9d5896-s6dn2"
Sep 18 19:40:32.910743 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:32.910721    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11e6710f-408d-4c78-be64-fcf7fae65ffb-istio-envoy\") pod \"router-gateway-1-openshift-default-6c5d9d5896-s6dn2\" (UID: \"11e6710f-408d-4c78-be64-fcf7fae65ffb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6c5d9d5896-s6dn2"
Sep 18 19:40:32.910882 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:32.910863    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11e6710f-408d-4c78-be64-fcf7fae65ffb-istio-podinfo\") pod \"router-gateway-1-openshift-default-6c5d9d5896-s6dn2\" (UID: \"11e6710f-408d-4c78-be64-fcf7fae65ffb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6c5d9d5896-s6dn2"
Sep 18 19:40:32.923399 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:32.923339    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltcms\" (UniqueName: \"kubernetes.io/projected/11e6710f-408d-4c78-be64-fcf7fae65ffb-kube-api-access-ltcms\") pod \"router-gateway-1-openshift-default-6c5d9d5896-s6dn2\" (UID: \"11e6710f-408d-4c78-be64-fcf7fae65ffb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6c5d9d5896-s6dn2"
Sep 18 19:40:32.923790 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:32.923768    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11e6710f-408d-4c78-be64-fcf7fae65ffb-istio-token\") pod \"router-gateway-1-openshift-default-6c5d9d5896-s6dn2\" (UID: \"11e6710f-408d-4c78-be64-fcf7fae65ffb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6c5d9d5896-s6dn2"
Sep 18 19:40:32.999103 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:32.999070    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6c5d9d5896-s6dn2"
Sep 18 19:40:33.301913 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:33.301830    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6c5d9d5896-s6dn2"]
Sep 18 19:40:33.305011 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:40:33.304970    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11e6710f_408d_4c78_be64_fcf7fae65ffb.slice/crio-e501a95e286f391ccbca160824317eaffa8d132aaa43b05505cf1d80a9d19888 WatchSource:0}: Error finding container e501a95e286f391ccbca160824317eaffa8d132aaa43b05505cf1d80a9d19888: Status 404 returned error can't find the container with id e501a95e286f391ccbca160824317eaffa8d132aaa43b05505cf1d80a9d19888
Sep 18 19:40:33.307077 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:33.307048    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 19:40:33.307162 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:33.307116    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 19:40:33.307162 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:33.307143    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 19:40:34.090683 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:34.090645    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6c5d9d5896-s6dn2" event={"ID":"11e6710f-408d-4c78-be64-fcf7fae65ffb","Type":"ContainerStarted","Data":"7096b644c50afbc530ef8f3e632651b0cb2792a7aa488cd3ac728d04470906c1"}
Sep 18 19:40:34.091082 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:34.090710    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6c5d9d5896-s6dn2" event={"ID":"11e6710f-408d-4c78-be64-fcf7fae65ffb","Type":"ContainerStarted","Data":"e501a95e286f391ccbca160824317eaffa8d132aaa43b05505cf1d80a9d19888"}
Sep 18 19:40:35.000058 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:35.000018    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6c5d9d5896-s6dn2"
Sep 18 19:40:35.005215 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:35.005187    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6c5d9d5896-s6dn2"
Sep 18 19:40:35.031186 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:35.031138    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6c5d9d5896-s6dn2" podStartSLOduration=3.031121991 podStartE2EDuration="3.031121991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:40:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:40:34.12431193 +0000 UTC m=+1604.976612299" watchObservedRunningTime="2026-09-18 19:40:35.031121991 +0000 UTC m=+1605.883422402"
Sep 18 19:40:35.095494 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:35.095467    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6c5d9d5896-s6dn2"
Sep 18 19:40:35.096522 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:40:35.096502    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6c5d9d5896-s6dn2"
Sep 18 19:41:18.522268 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:18.522228    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-5c6944466d-ft8vb"]
Sep 18 19:41:18.525533 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:18.525508    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-5c6944466d-ft8vb"
Sep 18 19:41:18.542857 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:18.541974    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"router-gateway-1-openshift-default-dockercfg-f6bf4\""
Sep 18 19:41:18.542857 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:18.542295    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-gw-ca-root-cert\""
Sep 18 19:41:18.542857 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:18.542481    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Sep 18 19:41:18.542857 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:18.542697    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Sep 18 19:41:18.555450 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:18.555422    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-5c6944466d-ft8vb"]
Sep 18 19:41:18.620276 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:18.620238    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e8ca221-7c88-4fa6-82ae-601906c8d602-istio-podinfo\") pod \"router-gateway-1-openshift-default-5c6944466d-ft8vb\" (UID: \"3e8ca221-7c88-4fa6-82ae-601906c8d602\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-5c6944466d-ft8vb"
Sep 18 19:41:18.620276 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:18.620277    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e8ca221-7c88-4fa6-82ae-601906c8d602-istio-envoy\") pod \"router-gateway-1-openshift-default-5c6944466d-ft8vb\" (UID: \"3e8ca221-7c88-4fa6-82ae-601906c8d602\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-5c6944466d-ft8vb"
Sep 18 19:41:18.620482 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:18.620333    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxh7c\" (UniqueName: \"kubernetes.io/projected/3e8ca221-7c88-4fa6-82ae-601906c8d602-kube-api-access-hxh7c\") pod \"router-gateway-1-openshift-default-5c6944466d-ft8vb\" (UID: \"3e8ca221-7c88-4fa6-82ae-601906c8d602\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-5c6944466d-ft8vb"
Sep 18 19:41:18.620482 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:18.620378    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e8ca221-7c88-4fa6-82ae-601906c8d602-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5c6944466d-ft8vb\" (UID: \"3e8ca221-7c88-4fa6-82ae-601906c8d602\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-5c6944466d-ft8vb"
Sep 18 19:41:18.620482 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:18.620426    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8ca221-7c88-4fa6-82ae-601906c8d602-credential-socket\") pod \"router-gateway-1-openshift-default-5c6944466d-ft8vb\" (UID: \"3e8ca221-7c88-4fa6-82ae-601906c8d602\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-5c6944466d-ft8vb"
Sep 18 19:41:18.620482 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:18.620457    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e8ca221-7c88-4fa6-82ae-601906c8d602-workload-certs\") pod \"router-gateway-1-openshift-default-5c6944466d-ft8vb\" (UID: \"3e8ca221-7c88-4fa6-82ae-601906c8d602\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-5c6944466d-ft8vb"
Sep 18 19:41:18.620482 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:18.620479    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e8ca221-7c88-4fa6-82ae-601906c8d602-istio-data\") pod \"router-gateway-1-openshift-default-5c6944466d-ft8vb\" (UID: \"3e8ca221-7c88-4fa6-82ae-601906c8d602\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-5c6944466d-ft8vb"
Sep 18 19:41:18.620646 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:18.620499    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e8ca221-7c88-4fa6-82ae-601906c8d602-istio-token\") pod \"router-gateway-1-openshift-default-5c6944466d-ft8vb\" (UID: \"3e8ca221-7c88-4fa6-82ae-601906c8d602\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-5c6944466d-ft8vb"
Sep 18 19:41:18.620646 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:18.620561    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8ca221-7c88-4fa6-82ae-601906c8d602-workload-socket\") pod \"router-gateway-1-openshift-default-5c6944466d-ft8vb\" (UID: \"3e8ca221-7c88-4fa6-82ae-601906c8d602\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-5c6944466d-ft8vb"
Sep 18 19:41:18.721239 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:18.721156    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8ca221-7c88-4fa6-82ae-601906c8d602-credential-socket\") pod \"router-gateway-1-openshift-default-5c6944466d-ft8vb\" (UID: \"3e8ca221-7c88-4fa6-82ae-601906c8d602\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-5c6944466d-ft8vb"
Sep 18 19:41:18.721239 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:18.721197    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e8ca221-7c88-4fa6-82ae-601906c8d602-workload-certs\") pod \"router-gateway-1-openshift-default-5c6944466d-ft8vb\" (UID: \"3e8ca221-7c88-4fa6-82ae-601906c8d602\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-5c6944466d-ft8vb"
Sep 18 19:41:18.721239 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:18.721219    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e8ca221-7c88-4fa6-82ae-601906c8d602-istio-data\") pod \"router-gateway-1-openshift-default-5c6944466d-ft8vb\" (UID: \"3e8ca221-7c88-4fa6-82ae-601906c8d602\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-5c6944466d-ft8vb"
Sep 18 19:41:18.721484 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:18.721326    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e8ca221-7c88-4fa6-82ae-601906c8d602-istio-token\") pod \"router-gateway-1-openshift-default-5c6944466d-ft8vb\" (UID: \"3e8ca221-7c88-4fa6-82ae-601906c8d602\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-5c6944466d-ft8vb"
Sep 18 19:41:18.721484 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:18.721373    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8ca221-7c88-4fa6-82ae-601906c8d602-workload-socket\") pod \"router-gateway-1-openshift-default-5c6944466d-ft8vb\" (UID: \"3e8ca221-7c88-4fa6-82ae-601906c8d602\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-5c6944466d-ft8vb"
Sep 18 19:41:18.721484 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:18.721461    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e8ca221-7c88-4fa6-82ae-601906c8d602-istio-podinfo\") pod \"router-gateway-1-openshift-default-5c6944466d-ft8vb\" (UID: \"3e8ca221-7c88-4fa6-82ae-601906c8d602\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-5c6944466d-ft8vb"
Sep 18 19:41:18.721652 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:18.721493    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e8ca221-7c88-4fa6-82ae-601906c8d602-istio-envoy\") pod \"router-gateway-1-openshift-default-5c6944466d-ft8vb\" (UID: \"3e8ca221-7c88-4fa6-82ae-601906c8d602\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-5c6944466d-ft8vb"
Sep 18 19:41:18.721652 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:18.721549    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxh7c\" (UniqueName: \"kubernetes.io/projected/3e8ca221-7c88-4fa6-82ae-601906c8d602-kube-api-access-hxh7c\") pod \"router-gateway-1-openshift-default-5c6944466d-ft8vb\" (UID: \"3e8ca221-7c88-4fa6-82ae-601906c8d602\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-5c6944466d-ft8vb"
Sep 18 19:41:18.721652 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:18.721569    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e8ca221-7c88-4fa6-82ae-601906c8d602-istio-data\") pod \"router-gateway-1-openshift-default-5c6944466d-ft8vb\" (UID: \"3e8ca221-7c88-4fa6-82ae-601906c8d602\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-5c6944466d-ft8vb"
Sep 18 19:41:18.721652 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:18.721585    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e8ca221-7c88-4fa6-82ae-601906c8d602-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5c6944466d-ft8vb\" (UID: \"3e8ca221-7c88-4fa6-82ae-601906c8d602\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-5c6944466d-ft8vb"
Sep 18 19:41:18.721652 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:18.721634    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e8ca221-7c88-4fa6-82ae-601906c8d602-workload-certs\") pod \"router-gateway-1-openshift-default-5c6944466d-ft8vb\" (UID: \"3e8ca221-7c88-4fa6-82ae-601906c8d602\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-5c6944466d-ft8vb"
Sep 18 19:41:18.721855 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:18.721681    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8ca221-7c88-4fa6-82ae-601906c8d602-credential-socket\") pod \"router-gateway-1-openshift-default-5c6944466d-ft8vb\" (UID: \"3e8ca221-7c88-4fa6-82ae-601906c8d602\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-5c6944466d-ft8vb"
Sep 18 19:41:18.721855 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:18.721722    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8ca221-7c88-4fa6-82ae-601906c8d602-workload-socket\") pod \"router-gateway-1-openshift-default-5c6944466d-ft8vb\" (UID: \"3e8ca221-7c88-4fa6-82ae-601906c8d602\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-5c6944466d-ft8vb"
Sep 18 19:41:18.722330 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:18.722299    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e8ca221-7c88-4fa6-82ae-601906c8d602-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5c6944466d-ft8vb\" (UID: \"3e8ca221-7c88-4fa6-82ae-601906c8d602\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-5c6944466d-ft8vb"
Sep 18 19:41:18.723814 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:18.723789    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e8ca221-7c88-4fa6-82ae-601906c8d602-istio-envoy\") pod \"router-gateway-1-openshift-default-5c6944466d-ft8vb\" (UID: \"3e8ca221-7c88-4fa6-82ae-601906c8d602\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-5c6944466d-ft8vb"
Sep 18 19:41:18.723916 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:18.723856    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e8ca221-7c88-4fa6-82ae-601906c8d602-istio-podinfo\") pod \"router-gateway-1-openshift-default-5c6944466d-ft8vb\" (UID: \"3e8ca221-7c88-4fa6-82ae-601906c8d602\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-5c6944466d-ft8vb"
Sep 18 19:41:18.740458 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:18.740425    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxh7c\" (UniqueName: \"kubernetes.io/projected/3e8ca221-7c88-4fa6-82ae-601906c8d602-kube-api-access-hxh7c\") pod \"router-gateway-1-openshift-default-5c6944466d-ft8vb\" (UID: \"3e8ca221-7c88-4fa6-82ae-601906c8d602\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-5c6944466d-ft8vb"
Sep 18 19:41:18.742817 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:18.742785    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e8ca221-7c88-4fa6-82ae-601906c8d602-istio-token\") pod \"router-gateway-1-openshift-default-5c6944466d-ft8vb\" (UID: \"3e8ca221-7c88-4fa6-82ae-601906c8d602\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-5c6944466d-ft8vb"
Sep 18 19:41:18.852698 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:18.852643    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-5c6944466d-ft8vb"
Sep 18 19:41:19.137009 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:19.136954    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-5c6944466d-ft8vb"]
Sep 18 19:41:19.160965 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:19.160929    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 19:41:19.161108 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:19.161076    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 19:41:19.161178 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:19.161129    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 19:41:19.291036 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:19.291001    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-5c6944466d-ft8vb" event={"ID":"3e8ca221-7c88-4fa6-82ae-601906c8d602","Type":"ContainerStarted","Data":"965152e7ae1b71314ee3e74698093f22bcbb4e69d27345ff9cedb89e95275e23"}
Sep 18 19:41:19.291207 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:19.291052    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-5c6944466d-ft8vb" event={"ID":"3e8ca221-7c88-4fa6-82ae-601906c8d602","Type":"ContainerStarted","Data":"bee8a767d1898b2b0448dc0f0839ef261a042f3404eb340ec35c7f5477592f4a"}
Sep 18 19:41:19.315350 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:19.315289    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-5c6944466d-ft8vb" podStartSLOduration=1.315270444 podStartE2EDuration="1.315270444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:41:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:41:19.314122568 +0000 UTC m=+1650.166422938" watchObservedRunningTime="2026-09-18 19:41:19.315270444 +0000 UTC m=+1650.167570814"
Sep 18 19:41:19.853895 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:19.853850    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-5c6944466d-ft8vb"
Sep 18 19:41:20.858522 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:20.858490    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-5c6944466d-ft8vb"
Sep 18 19:41:21.299704 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:21.299674    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-5c6944466d-ft8vb"
Sep 18 19:41:21.300667 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:21.300644    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-5c6944466d-ft8vb"
Sep 18 19:41:22.847202 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:22.847114    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6c5d9d5896-s6dn2"]
Sep 18 19:41:22.847848 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:22.847342    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6c5d9d5896-s6dn2" podUID="11e6710f-408d-4c78-be64-fcf7fae65ffb" containerName="istio-proxy" containerID="cri-o://7096b644c50afbc530ef8f3e632651b0cb2792a7aa488cd3ac728d04470906c1" gracePeriod=30
Sep 18 19:41:25.487508 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:25.487480    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6c5d9d5896-s6dn2"
Sep 18 19:41:25.599676 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:25.599643    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11e6710f-408d-4c78-be64-fcf7fae65ffb-istio-token\") pod \"11e6710f-408d-4c78-be64-fcf7fae65ffb\" (UID: \"11e6710f-408d-4c78-be64-fcf7fae65ffb\") "
Sep 18 19:41:25.599847 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:25.599685    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11e6710f-408d-4c78-be64-fcf7fae65ffb-istio-podinfo\") pod \"11e6710f-408d-4c78-be64-fcf7fae65ffb\" (UID: \"11e6710f-408d-4c78-be64-fcf7fae65ffb\") "
Sep 18 19:41:25.599847 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:25.599711    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e6710f-408d-4c78-be64-fcf7fae65ffb-workload-socket\") pod \"11e6710f-408d-4c78-be64-fcf7fae65ffb\" (UID: \"11e6710f-408d-4c78-be64-fcf7fae65ffb\") "
Sep 18 19:41:25.599847 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:25.599739    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11e6710f-408d-4c78-be64-fcf7fae65ffb-istiod-ca-cert\") pod \"11e6710f-408d-4c78-be64-fcf7fae65ffb\" (UID: \"11e6710f-408d-4c78-be64-fcf7fae65ffb\") "
Sep 18 19:41:25.599847 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:25.599775    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ltcms\" (UniqueName: \"kubernetes.io/projected/11e6710f-408d-4c78-be64-fcf7fae65ffb-kube-api-access-ltcms\") pod \"11e6710f-408d-4c78-be64-fcf7fae65ffb\" (UID: \"11e6710f-408d-4c78-be64-fcf7fae65ffb\") "
Sep 18 19:41:25.599847 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:25.599811    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11e6710f-408d-4c78-be64-fcf7fae65ffb-workload-certs\") pod \"11e6710f-408d-4c78-be64-fcf7fae65ffb\" (UID: \"11e6710f-408d-4c78-be64-fcf7fae65ffb\") "
Sep 18 19:41:25.599847 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:25.599837    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e6710f-408d-4c78-be64-fcf7fae65ffb-credential-socket\") pod \"11e6710f-408d-4c78-be64-fcf7fae65ffb\" (UID: \"11e6710f-408d-4c78-be64-fcf7fae65ffb\") "
Sep 18 19:41:25.600202 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:25.599860    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11e6710f-408d-4c78-be64-fcf7fae65ffb-istio-envoy\") pod \"11e6710f-408d-4c78-be64-fcf7fae65ffb\" (UID: \"11e6710f-408d-4c78-be64-fcf7fae65ffb\") "
Sep 18 19:41:25.600202 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:25.599920    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11e6710f-408d-4c78-be64-fcf7fae65ffb-istio-data\") pod \"11e6710f-408d-4c78-be64-fcf7fae65ffb\" (UID: \"11e6710f-408d-4c78-be64-fcf7fae65ffb\") "
Sep 18 19:41:25.600202 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:25.600154    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11e6710f-408d-4c78-be64-fcf7fae65ffb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "11e6710f-408d-4c78-be64-fcf7fae65ffb" (UID: "11e6710f-408d-4c78-be64-fcf7fae65ffb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:41:25.600376 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:25.600288    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11e6710f-408d-4c78-be64-fcf7fae65ffb-workload-certs\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:41:25.600376 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:25.600294    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11e6710f-408d-4c78-be64-fcf7fae65ffb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "11e6710f-408d-4c78-be64-fcf7fae65ffb" (UID: "11e6710f-408d-4c78-be64-fcf7fae65ffb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:41:25.600376 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:25.600327    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11e6710f-408d-4c78-be64-fcf7fae65ffb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "11e6710f-408d-4c78-be64-fcf7fae65ffb" (UID: "11e6710f-408d-4c78-be64-fcf7fae65ffb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:41:25.600376 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:25.600350    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11e6710f-408d-4c78-be64-fcf7fae65ffb-istio-data" (OuterVolumeSpecName: "istio-data") pod "11e6710f-408d-4c78-be64-fcf7fae65ffb" (UID: "11e6710f-408d-4c78-be64-fcf7fae65ffb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:41:25.600587 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:25.600379    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11e6710f-408d-4c78-be64-fcf7fae65ffb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "11e6710f-408d-4c78-be64-fcf7fae65ffb" (UID: "11e6710f-408d-4c78-be64-fcf7fae65ffb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:41:25.602108 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:25.602071    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11e6710f-408d-4c78-be64-fcf7fae65ffb-istio-token" (OuterVolumeSpecName: "istio-token") pod "11e6710f-408d-4c78-be64-fcf7fae65ffb" (UID: "11e6710f-408d-4c78-be64-fcf7fae65ffb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:41:25.602219 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:25.602150    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11e6710f-408d-4c78-be64-fcf7fae65ffb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "11e6710f-408d-4c78-be64-fcf7fae65ffb" (UID: "11e6710f-408d-4c78-be64-fcf7fae65ffb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 19:41:25.602219 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:25.602159    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11e6710f-408d-4c78-be64-fcf7fae65ffb-kube-api-access-ltcms" (OuterVolumeSpecName: "kube-api-access-ltcms") pod "11e6710f-408d-4c78-be64-fcf7fae65ffb" (UID: "11e6710f-408d-4c78-be64-fcf7fae65ffb"). InnerVolumeSpecName "kube-api-access-ltcms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:41:25.602369 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:25.602354    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11e6710f-408d-4c78-be64-fcf7fae65ffb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "11e6710f-408d-4c78-be64-fcf7fae65ffb" (UID: "11e6710f-408d-4c78-be64-fcf7fae65ffb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:41:25.701757 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:25.701719    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11e6710f-408d-4c78-be64-fcf7fae65ffb-istio-data\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:41:25.701757 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:25.701750    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11e6710f-408d-4c78-be64-fcf7fae65ffb-istio-token\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:41:25.701757 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:25.701760    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11e6710f-408d-4c78-be64-fcf7fae65ffb-istio-podinfo\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:41:25.701757 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:25.701768    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e6710f-408d-4c78-be64-fcf7fae65ffb-workload-socket\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:41:25.702060 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:25.701776    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11e6710f-408d-4c78-be64-fcf7fae65ffb-istiod-ca-cert\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:41:25.702060 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:25.701784    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltcms\" (UniqueName: \"kubernetes.io/projected/11e6710f-408d-4c78-be64-fcf7fae65ffb-kube-api-access-ltcms\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:41:25.702060 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:25.701795    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e6710f-408d-4c78-be64-fcf7fae65ffb-credential-socket\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:41:25.702060 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:25.701803    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11e6710f-408d-4c78-be64-fcf7fae65ffb-istio-envoy\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:41:26.326615 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:26.326582    2577 generic.go:356] "Generic (PLEG): container finished" podID="11e6710f-408d-4c78-be64-fcf7fae65ffb" containerID="7096b644c50afbc530ef8f3e632651b0cb2792a7aa488cd3ac728d04470906c1" exitCode=0
Sep 18 19:41:26.326797 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:26.326629    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6c5d9d5896-s6dn2"
Sep 18 19:41:26.326797 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:26.326661    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6c5d9d5896-s6dn2" event={"ID":"11e6710f-408d-4c78-be64-fcf7fae65ffb","Type":"ContainerDied","Data":"7096b644c50afbc530ef8f3e632651b0cb2792a7aa488cd3ac728d04470906c1"}
Sep 18 19:41:26.326797 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:26.326716    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6c5d9d5896-s6dn2" event={"ID":"11e6710f-408d-4c78-be64-fcf7fae65ffb","Type":"ContainerDied","Data":"e501a95e286f391ccbca160824317eaffa8d132aaa43b05505cf1d80a9d19888"}
Sep 18 19:41:26.326797 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:26.326747    2577 scope.go:117] "RemoveContainer" containerID="7096b644c50afbc530ef8f3e632651b0cb2792a7aa488cd3ac728d04470906c1"
Sep 18 19:41:26.335326 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:26.335309    2577 scope.go:117] "RemoveContainer" containerID="7096b644c50afbc530ef8f3e632651b0cb2792a7aa488cd3ac728d04470906c1"
Sep 18 19:41:26.335573 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:41:26.335552    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7096b644c50afbc530ef8f3e632651b0cb2792a7aa488cd3ac728d04470906c1\": container with ID starting with 7096b644c50afbc530ef8f3e632651b0cb2792a7aa488cd3ac728d04470906c1 not found: ID does not exist" containerID="7096b644c50afbc530ef8f3e632651b0cb2792a7aa488cd3ac728d04470906c1"
Sep 18 19:41:26.335664 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:26.335579    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7096b644c50afbc530ef8f3e632651b0cb2792a7aa488cd3ac728d04470906c1"} err="failed to get container status \"7096b644c50afbc530ef8f3e632651b0cb2792a7aa488cd3ac728d04470906c1\": rpc error: code = NotFound desc = could not find container \"7096b644c50afbc530ef8f3e632651b0cb2792a7aa488cd3ac728d04470906c1\": container with ID starting with 7096b644c50afbc530ef8f3e632651b0cb2792a7aa488cd3ac728d04470906c1 not found: ID does not exist"
Sep 18 19:41:26.358861 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:26.358832    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6c5d9d5896-s6dn2"]
Sep 18 19:41:26.369633 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:26.369602    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6c5d9d5896-s6dn2"]
Sep 18 19:41:27.439062 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:27.439027    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11e6710f-408d-4c78-be64-fcf7fae65ffb" path="/var/lib/kubelet/pods/11e6710f-408d-4c78-be64-fcf7fae65ffb/volumes"
Sep 18 19:41:52.203036 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:52.203000    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7"]
Sep 18 19:41:52.211875 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:41:52.211831    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 19:41:52.211875 ip-10-0-129-30 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:41:52.211875 ip-10-0-129-30 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 18 19:41:52.211875 ip-10-0-129-30 kubenswrapper[2577]:  > containerID="c5e1eda06f75ae56c3089580d5a46d074e1335eff6533dff2bc674a580153615" cmd=["/bin/sleep","15"]
Sep 18 19:41:52.212188 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:52.211894    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7" message="" err=<
Sep 18 19:41:52.212188 ip-10-0-129-30 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:41:52.212188 ip-10-0-129-30 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 18 19:41:52.212188 ip-10-0-129-30 kubenswrapper[2577]:  >
Sep 18 19:41:52.212188 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:41:52.211911    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 19:41:52.212188 ip-10-0-129-30 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:41:52.212188 ip-10-0-129-30 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 18 19:41:52.212188 ip-10-0-129-30 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7" podUID="e8a75bef-1c87-4259-925d-fa2e356b87d0" containerName="main" containerID="cri-o://c5e1eda06f75ae56c3089580d5a46d074e1335eff6533dff2bc674a580153615"
Sep 18 19:41:52.212188 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:52.211954    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7" podUID="e8a75bef-1c87-4259-925d-fa2e356b87d0" containerName="main" containerID="cri-o://c5e1eda06f75ae56c3089580d5a46d074e1335eff6533dff2bc674a580153615" gracePeriod=60
Sep 18 19:41:52.427834 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:52.427796    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7" podUID="e8a75bef-1c87-4259-925d-fa2e356b87d0" containerName="llm-d-routing-sidecar" containerID="cri-o://8f911faf7fbc548f2825f35cca486f97e8e6a61f32e6c2b2a677330245e92f88" gracePeriod=60
Sep 18 19:41:52.461017 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:52.460970    2577 generic.go:356] "Generic (PLEG): container finished" podID="e8a75bef-1c87-4259-925d-fa2e356b87d0" containerID="c5e1eda06f75ae56c3089580d5a46d074e1335eff6533dff2bc674a580153615" exitCode=0
Sep 18 19:41:52.461134 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:52.461019    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7" event={"ID":"e8a75bef-1c87-4259-925d-fa2e356b87d0","Type":"ContainerDied","Data":"c5e1eda06f75ae56c3089580d5a46d074e1335eff6533dff2bc674a580153615"}
Sep 18 19:41:52.718358 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:52.718295    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7"
Sep 18 19:41:52.761456 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:52.761415    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8a75bef-1c87-4259-925d-fa2e356b87d0-model-cache\") pod \"e8a75bef-1c87-4259-925d-fa2e356b87d0\" (UID: \"e8a75bef-1c87-4259-925d-fa2e356b87d0\") "
Sep 18 19:41:52.761456 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:52.761461    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8a75bef-1c87-4259-925d-fa2e356b87d0-tls-certs\") pod \"e8a75bef-1c87-4259-925d-fa2e356b87d0\" (UID: \"e8a75bef-1c87-4259-925d-fa2e356b87d0\") "
Sep 18 19:41:52.761708 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:52.761496    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8a75bef-1c87-4259-925d-fa2e356b87d0-dshm\") pod \"e8a75bef-1c87-4259-925d-fa2e356b87d0\" (UID: \"e8a75bef-1c87-4259-925d-fa2e356b87d0\") "
Sep 18 19:41:52.761708 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:52.761519    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cpb2t\" (UniqueName: \"kubernetes.io/projected/e8a75bef-1c87-4259-925d-fa2e356b87d0-kube-api-access-cpb2t\") pod \"e8a75bef-1c87-4259-925d-fa2e356b87d0\" (UID: \"e8a75bef-1c87-4259-925d-fa2e356b87d0\") "
Sep 18 19:41:52.761708 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:52.761564    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8a75bef-1c87-4259-925d-fa2e356b87d0-home\") pod \"e8a75bef-1c87-4259-925d-fa2e356b87d0\" (UID: \"e8a75bef-1c87-4259-925d-fa2e356b87d0\") "
Sep 18 19:41:52.761708 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:52.761591    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8a75bef-1c87-4259-925d-fa2e356b87d0-tmp-dir\") pod \"e8a75bef-1c87-4259-925d-fa2e356b87d0\" (UID: \"e8a75bef-1c87-4259-925d-fa2e356b87d0\") "
Sep 18 19:41:52.761923 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:52.761721    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a75bef-1c87-4259-925d-fa2e356b87d0-model-cache" (OuterVolumeSpecName: "model-cache") pod "e8a75bef-1c87-4259-925d-fa2e356b87d0" (UID: "e8a75bef-1c87-4259-925d-fa2e356b87d0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:41:52.761923 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:52.761825    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a75bef-1c87-4259-925d-fa2e356b87d0-home" (OuterVolumeSpecName: "home") pod "e8a75bef-1c87-4259-925d-fa2e356b87d0" (UID: "e8a75bef-1c87-4259-925d-fa2e356b87d0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:41:52.761923 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:52.761898    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8a75bef-1c87-4259-925d-fa2e356b87d0-model-cache\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:41:52.761923 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:52.761919    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8a75bef-1c87-4259-925d-fa2e356b87d0-home\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:41:52.762144 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:52.761939    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a75bef-1c87-4259-925d-fa2e356b87d0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e8a75bef-1c87-4259-925d-fa2e356b87d0" (UID: "e8a75bef-1c87-4259-925d-fa2e356b87d0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:41:52.763977 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:52.763917    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a75bef-1c87-4259-925d-fa2e356b87d0-dshm" (OuterVolumeSpecName: "dshm") pod "e8a75bef-1c87-4259-925d-fa2e356b87d0" (UID: "e8a75bef-1c87-4259-925d-fa2e356b87d0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:41:52.763977 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:52.763939    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8a75bef-1c87-4259-925d-fa2e356b87d0-kube-api-access-cpb2t" (OuterVolumeSpecName: "kube-api-access-cpb2t") pod "e8a75bef-1c87-4259-925d-fa2e356b87d0" (UID: "e8a75bef-1c87-4259-925d-fa2e356b87d0"). InnerVolumeSpecName "kube-api-access-cpb2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:41:52.763977 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:52.763959    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8a75bef-1c87-4259-925d-fa2e356b87d0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e8a75bef-1c87-4259-925d-fa2e356b87d0" (UID: "e8a75bef-1c87-4259-925d-fa2e356b87d0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:41:52.862607 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:52.862575    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8a75bef-1c87-4259-925d-fa2e356b87d0-tls-certs\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:41:52.862607 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:52.862604    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8a75bef-1c87-4259-925d-fa2e356b87d0-dshm\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:41:52.862607 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:52.862615    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpb2t\" (UniqueName: \"kubernetes.io/projected/e8a75bef-1c87-4259-925d-fa2e356b87d0-kube-api-access-cpb2t\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:41:52.862859 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:52.862624    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8a75bef-1c87-4259-925d-fa2e356b87d0-tmp-dir\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:41:53.467182 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:53.467154    2577 generic.go:356] "Generic (PLEG): container finished" podID="e8a75bef-1c87-4259-925d-fa2e356b87d0" containerID="8f911faf7fbc548f2825f35cca486f97e8e6a61f32e6c2b2a677330245e92f88" exitCode=0
Sep 18 19:41:53.467542 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:53.467199    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7"
Sep 18 19:41:53.467542 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:53.467239    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7" event={"ID":"e8a75bef-1c87-4259-925d-fa2e356b87d0","Type":"ContainerDied","Data":"8f911faf7fbc548f2825f35cca486f97e8e6a61f32e6c2b2a677330245e92f88"}
Sep 18 19:41:53.467542 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:53.467291    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7" event={"ID":"e8a75bef-1c87-4259-925d-fa2e356b87d0","Type":"ContainerDied","Data":"bc859fac1980ba9e870c6499fa9e0be8bbb31f7dcbd0c90ec734c23eb3bdcde1"}
Sep 18 19:41:53.467542 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:53.467316    2577 scope.go:117] "RemoveContainer" containerID="c5e1eda06f75ae56c3089580d5a46d074e1335eff6533dff2bc674a580153615"
Sep 18 19:41:53.476408 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:53.476389    2577 scope.go:117] "RemoveContainer" containerID="8f911faf7fbc548f2825f35cca486f97e8e6a61f32e6c2b2a677330245e92f88"
Sep 18 19:41:53.483871 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:53.483847    2577 scope.go:117] "RemoveContainer" containerID="c5e1eda06f75ae56c3089580d5a46d074e1335eff6533dff2bc674a580153615"
Sep 18 19:41:53.484174 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:41:53.484153    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5e1eda06f75ae56c3089580d5a46d074e1335eff6533dff2bc674a580153615\": container with ID starting with c5e1eda06f75ae56c3089580d5a46d074e1335eff6533dff2bc674a580153615 not found: ID does not exist" containerID="c5e1eda06f75ae56c3089580d5a46d074e1335eff6533dff2bc674a580153615"
Sep 18 19:41:53.484251 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:53.484183    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5e1eda06f75ae56c3089580d5a46d074e1335eff6533dff2bc674a580153615"} err="failed to get container status \"c5e1eda06f75ae56c3089580d5a46d074e1335eff6533dff2bc674a580153615\": rpc error: code = NotFound desc = could not find container \"c5e1eda06f75ae56c3089580d5a46d074e1335eff6533dff2bc674a580153615\": container with ID starting with c5e1eda06f75ae56c3089580d5a46d074e1335eff6533dff2bc674a580153615 not found: ID does not exist"
Sep 18 19:41:53.484251 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:53.484199    2577 scope.go:117] "RemoveContainer" containerID="8f911faf7fbc548f2825f35cca486f97e8e6a61f32e6c2b2a677330245e92f88"
Sep 18 19:41:53.484458 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:41:53.484438    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f911faf7fbc548f2825f35cca486f97e8e6a61f32e6c2b2a677330245e92f88\": container with ID starting with 8f911faf7fbc548f2825f35cca486f97e8e6a61f32e6c2b2a677330245e92f88 not found: ID does not exist" containerID="8f911faf7fbc548f2825f35cca486f97e8e6a61f32e6c2b2a677330245e92f88"
Sep 18 19:41:53.484501 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:53.484464    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f911faf7fbc548f2825f35cca486f97e8e6a61f32e6c2b2a677330245e92f88"} err="failed to get container status \"8f911faf7fbc548f2825f35cca486f97e8e6a61f32e6c2b2a677330245e92f88\": rpc error: code = NotFound desc = could not find container \"8f911faf7fbc548f2825f35cca486f97e8e6a61f32e6c2b2a677330245e92f88\": container with ID starting with 8f911faf7fbc548f2825f35cca486f97e8e6a61f32e6c2b2a677330245e92f88 not found: ID does not exist"
Sep 18 19:41:53.553889 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:53.553847    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7"]
Sep 18 19:41:53.563061 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:53.563029    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-626c7"]
Sep 18 19:41:55.442388 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:55.442346    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8a75bef-1c87-4259-925d-fa2e356b87d0" path="/var/lib/kubelet/pods/e8a75bef-1c87-4259-925d-fa2e356b87d0/volumes"
Sep 18 19:41:56.706974 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:56.706930    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-5c6944466d-ft8vb"]
Sep 18 19:41:56.707454 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:56.707199    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-5c6944466d-ft8vb" podUID="3e8ca221-7c88-4fa6-82ae-601906c8d602" containerName="istio-proxy" containerID="cri-o://965152e7ae1b71314ee3e74698093f22bcbb4e69d27345ff9cedb89e95275e23" gracePeriod=30
Sep 18 19:41:59.657763 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:59.657737    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-5c6944466d-ft8vb"
Sep 18 19:41:59.728124 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:59.728086    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8ca221-7c88-4fa6-82ae-601906c8d602-workload-socket\") pod \"3e8ca221-7c88-4fa6-82ae-601906c8d602\" (UID: \"3e8ca221-7c88-4fa6-82ae-601906c8d602\") "
Sep 18 19:41:59.728124 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:59.728134    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e8ca221-7c88-4fa6-82ae-601906c8d602-istiod-ca-cert\") pod \"3e8ca221-7c88-4fa6-82ae-601906c8d602\" (UID: \"3e8ca221-7c88-4fa6-82ae-601906c8d602\") "
Sep 18 19:41:59.728414 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:59.728180    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e8ca221-7c88-4fa6-82ae-601906c8d602-istio-data\") pod \"3e8ca221-7c88-4fa6-82ae-601906c8d602\" (UID: \"3e8ca221-7c88-4fa6-82ae-601906c8d602\") "
Sep 18 19:41:59.728414 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:59.728230    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8ca221-7c88-4fa6-82ae-601906c8d602-credential-socket\") pod \"3e8ca221-7c88-4fa6-82ae-601906c8d602\" (UID: \"3e8ca221-7c88-4fa6-82ae-601906c8d602\") "
Sep 18 19:41:59.728414 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:59.728266    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e8ca221-7c88-4fa6-82ae-601906c8d602-istio-token\") pod \"3e8ca221-7c88-4fa6-82ae-601906c8d602\" (UID: \"3e8ca221-7c88-4fa6-82ae-601906c8d602\") "
Sep 18 19:41:59.728414 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:59.728323    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e8ca221-7c88-4fa6-82ae-601906c8d602-istio-envoy\") pod \"3e8ca221-7c88-4fa6-82ae-601906c8d602\" (UID: \"3e8ca221-7c88-4fa6-82ae-601906c8d602\") "
Sep 18 19:41:59.728414 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:59.728366    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hxh7c\" (UniqueName: \"kubernetes.io/projected/3e8ca221-7c88-4fa6-82ae-601906c8d602-kube-api-access-hxh7c\") pod \"3e8ca221-7c88-4fa6-82ae-601906c8d602\" (UID: \"3e8ca221-7c88-4fa6-82ae-601906c8d602\") "
Sep 18 19:41:59.728414 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:59.728391    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e8ca221-7c88-4fa6-82ae-601906c8d602-workload-certs\") pod \"3e8ca221-7c88-4fa6-82ae-601906c8d602\" (UID: \"3e8ca221-7c88-4fa6-82ae-601906c8d602\") "
Sep 18 19:41:59.728414 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:59.728416    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e8ca221-7c88-4fa6-82ae-601906c8d602-istio-podinfo\") pod \"3e8ca221-7c88-4fa6-82ae-601906c8d602\" (UID: \"3e8ca221-7c88-4fa6-82ae-601906c8d602\") "
Sep 18 19:41:59.728789 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:59.728413    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8ca221-7c88-4fa6-82ae-601906c8d602-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3e8ca221-7c88-4fa6-82ae-601906c8d602" (UID: "3e8ca221-7c88-4fa6-82ae-601906c8d602"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:41:59.728789 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:59.728604    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e8ca221-7c88-4fa6-82ae-601906c8d602-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3e8ca221-7c88-4fa6-82ae-601906c8d602" (UID: "3e8ca221-7c88-4fa6-82ae-601906c8d602"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:41:59.728789 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:59.728775    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8ca221-7c88-4fa6-82ae-601906c8d602-workload-socket\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:41:59.728947 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:59.728792    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e8ca221-7c88-4fa6-82ae-601906c8d602-istiod-ca-cert\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:41:59.729020 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:59.728956    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8ca221-7c88-4fa6-82ae-601906c8d602-istio-data" (OuterVolumeSpecName: "istio-data") pod "3e8ca221-7c88-4fa6-82ae-601906c8d602" (UID: "3e8ca221-7c88-4fa6-82ae-601906c8d602"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:41:59.729126 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:59.729106    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8ca221-7c88-4fa6-82ae-601906c8d602-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3e8ca221-7c88-4fa6-82ae-601906c8d602" (UID: "3e8ca221-7c88-4fa6-82ae-601906c8d602"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:41:59.729640 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:59.729589    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8ca221-7c88-4fa6-82ae-601906c8d602-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3e8ca221-7c88-4fa6-82ae-601906c8d602" (UID: "3e8ca221-7c88-4fa6-82ae-601906c8d602"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:41:59.731451 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:59.731412    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e8ca221-7c88-4fa6-82ae-601906c8d602-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3e8ca221-7c88-4fa6-82ae-601906c8d602" (UID: "3e8ca221-7c88-4fa6-82ae-601906c8d602"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 19:41:59.731774 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:59.731738    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e8ca221-7c88-4fa6-82ae-601906c8d602-istio-token" (OuterVolumeSpecName: "istio-token") pod "3e8ca221-7c88-4fa6-82ae-601906c8d602" (UID: "3e8ca221-7c88-4fa6-82ae-601906c8d602"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:41:59.731949 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:59.731924    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e8ca221-7c88-4fa6-82ae-601906c8d602-kube-api-access-hxh7c" (OuterVolumeSpecName: "kube-api-access-hxh7c") pod "3e8ca221-7c88-4fa6-82ae-601906c8d602" (UID: "3e8ca221-7c88-4fa6-82ae-601906c8d602"). InnerVolumeSpecName "kube-api-access-hxh7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:41:59.732329 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:59.732304    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8ca221-7c88-4fa6-82ae-601906c8d602-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3e8ca221-7c88-4fa6-82ae-601906c8d602" (UID: "3e8ca221-7c88-4fa6-82ae-601906c8d602"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:41:59.829858 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:59.829825    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e8ca221-7c88-4fa6-82ae-601906c8d602-istio-data\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:41:59.829858 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:59.829854    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8ca221-7c88-4fa6-82ae-601906c8d602-credential-socket\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:41:59.829858 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:59.829863    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e8ca221-7c88-4fa6-82ae-601906c8d602-istio-token\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:41:59.830134 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:59.829871    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e8ca221-7c88-4fa6-82ae-601906c8d602-istio-envoy\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:41:59.830134 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:59.829879    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxh7c\" (UniqueName: \"kubernetes.io/projected/3e8ca221-7c88-4fa6-82ae-601906c8d602-kube-api-access-hxh7c\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:41:59.830134 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:59.829889    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e8ca221-7c88-4fa6-82ae-601906c8d602-workload-certs\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:41:59.830134 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:41:59.829897    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e8ca221-7c88-4fa6-82ae-601906c8d602-istio-podinfo\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:42:00.502922 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:42:00.502885    2577 generic.go:356] "Generic (PLEG): container finished" podID="3e8ca221-7c88-4fa6-82ae-601906c8d602" containerID="965152e7ae1b71314ee3e74698093f22bcbb4e69d27345ff9cedb89e95275e23" exitCode=0
Sep 18 19:42:00.503127 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:42:00.502939    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-5c6944466d-ft8vb"
Sep 18 19:42:00.503127 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:42:00.502967    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-5c6944466d-ft8vb" event={"ID":"3e8ca221-7c88-4fa6-82ae-601906c8d602","Type":"ContainerDied","Data":"965152e7ae1b71314ee3e74698093f22bcbb4e69d27345ff9cedb89e95275e23"}
Sep 18 19:42:00.503127 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:42:00.503039    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-5c6944466d-ft8vb" event={"ID":"3e8ca221-7c88-4fa6-82ae-601906c8d602","Type":"ContainerDied","Data":"bee8a767d1898b2b0448dc0f0839ef261a042f3404eb340ec35c7f5477592f4a"}
Sep 18 19:42:00.503127 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:42:00.503055    2577 scope.go:117] "RemoveContainer" containerID="965152e7ae1b71314ee3e74698093f22bcbb4e69d27345ff9cedb89e95275e23"
Sep 18 19:42:00.511922 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:42:00.511903    2577 scope.go:117] "RemoveContainer" containerID="965152e7ae1b71314ee3e74698093f22bcbb4e69d27345ff9cedb89e95275e23"
Sep 18 19:42:00.512183 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:42:00.512165    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"965152e7ae1b71314ee3e74698093f22bcbb4e69d27345ff9cedb89e95275e23\": container with ID starting with 965152e7ae1b71314ee3e74698093f22bcbb4e69d27345ff9cedb89e95275e23 not found: ID does not exist" containerID="965152e7ae1b71314ee3e74698093f22bcbb4e69d27345ff9cedb89e95275e23"
Sep 18 19:42:00.512246 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:42:00.512192    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"965152e7ae1b71314ee3e74698093f22bcbb4e69d27345ff9cedb89e95275e23"} err="failed to get container status \"965152e7ae1b71314ee3e74698093f22bcbb4e69d27345ff9cedb89e95275e23\": rpc error: code = NotFound desc = could not find container \"965152e7ae1b71314ee3e74698093f22bcbb4e69d27345ff9cedb89e95275e23\": container with ID starting with 965152e7ae1b71314ee3e74698093f22bcbb4e69d27345ff9cedb89e95275e23 not found: ID does not exist"
Sep 18 19:42:00.526023 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:42:00.526001    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-5c6944466d-ft8vb"]
Sep 18 19:42:00.531243 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:42:00.531221    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-5c6944466d-ft8vb"]
Sep 18 19:42:01.438547 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:42:01.438521    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e8ca221-7c88-4fa6-82ae-601906c8d602" path="/var/lib/kubelet/pods/3e8ca221-7c88-4fa6-82ae-601906c8d602/volumes"
Sep 18 19:43:15.378138 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:15.378106    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-qwp6t"]
Sep 18 19:43:15.378612 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:15.378430    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8a75bef-1c87-4259-925d-fa2e356b87d0" containerName="llm-d-routing-sidecar"
Sep 18 19:43:15.378612 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:15.378442    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8a75bef-1c87-4259-925d-fa2e356b87d0" containerName="llm-d-routing-sidecar"
Sep 18 19:43:15.378612 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:15.378453    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8a75bef-1c87-4259-925d-fa2e356b87d0" containerName="main"
Sep 18 19:43:15.378612 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:15.378458    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8a75bef-1c87-4259-925d-fa2e356b87d0" containerName="main"
Sep 18 19:43:15.378612 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:15.378464    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11e6710f-408d-4c78-be64-fcf7fae65ffb" containerName="istio-proxy"
Sep 18 19:43:15.378612 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:15.378470    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="11e6710f-408d-4c78-be64-fcf7fae65ffb" containerName="istio-proxy"
Sep 18 19:43:15.378612 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:15.378483    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e8ca221-7c88-4fa6-82ae-601906c8d602" containerName="istio-proxy"
Sep 18 19:43:15.378612 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:15.378487    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e8ca221-7c88-4fa6-82ae-601906c8d602" containerName="istio-proxy"
Sep 18 19:43:15.378612 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:15.378537    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e8ca221-7c88-4fa6-82ae-601906c8d602" containerName="istio-proxy"
Sep 18 19:43:15.378612 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:15.378547    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8a75bef-1c87-4259-925d-fa2e356b87d0" containerName="llm-d-routing-sidecar"
Sep 18 19:43:15.378612 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:15.378552    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8a75bef-1c87-4259-925d-fa2e356b87d0" containerName="main"
Sep 18 19:43:15.378612 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:15.378557    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="11e6710f-408d-4c78-be64-fcf7fae65ffb" containerName="istio-proxy"
Sep 18 19:43:15.380587 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:15.380571    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-qwp6t"
Sep 18 19:43:15.383916 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:15.383889    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 19:43:15.383916 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:15.383911    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-xlhct\""
Sep 18 19:43:15.384165 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:15.383918    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 18 19:43:15.385036 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:15.385016    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 18 19:43:15.385221 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:15.385174    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 18 19:43:15.385467 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:15.385453    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 18 19:43:15.400031 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:15.400004    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-qwp6t"]
Sep 18 19:43:15.522269 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:15.522238    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cdc252d-5f28-4636-8da8-7f71b3157d20-home\") pod \"tokenizer-clean-path-test-kserve-5c88c58655-qwp6t\" (UID: \"6cdc252d-5f28-4636-8da8-7f71b3157d20\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-qwp6t"
Sep 18 19:43:15.522462 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:15.522287    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cdc252d-5f28-4636-8da8-7f71b3157d20-tls-certs\") pod \"tokenizer-clean-path-test-kserve-5c88c58655-qwp6t\" (UID: \"6cdc252d-5f28-4636-8da8-7f71b3157d20\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-qwp6t"
Sep 18 19:43:15.522462 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:15.522309    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cdc252d-5f28-4636-8da8-7f71b3157d20-dshm\") pod \"tokenizer-clean-path-test-kserve-5c88c58655-qwp6t\" (UID: \"6cdc252d-5f28-4636-8da8-7f71b3157d20\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-qwp6t"
Sep 18 19:43:15.522462 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:15.522352    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6cdc252d-5f28-4636-8da8-7f71b3157d20-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-5c88c58655-qwp6t\" (UID: \"6cdc252d-5f28-4636-8da8-7f71b3157d20\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-qwp6t"
Sep 18 19:43:15.522462 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:15.522388    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm8cs\" (UniqueName: \"kubernetes.io/projected/6cdc252d-5f28-4636-8da8-7f71b3157d20-kube-api-access-lm8cs\") pod \"tokenizer-clean-path-test-kserve-5c88c58655-qwp6t\" (UID: \"6cdc252d-5f28-4636-8da8-7f71b3157d20\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-qwp6t"
Sep 18 19:43:15.522462 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:15.522451    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cdc252d-5f28-4636-8da8-7f71b3157d20-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-5c88c58655-qwp6t\" (UID: \"6cdc252d-5f28-4636-8da8-7f71b3157d20\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-qwp6t"
Sep 18 19:43:15.522712 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:15.522488    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6cdc252d-5f28-4636-8da8-7f71b3157d20-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-5c88c58655-qwp6t\" (UID: \"6cdc252d-5f28-4636-8da8-7f71b3157d20\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-qwp6t"
Sep 18 19:43:15.522712 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:15.522555    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cdc252d-5f28-4636-8da8-7f71b3157d20-model-cache\") pod \"tokenizer-clean-path-test-kserve-5c88c58655-qwp6t\" (UID: \"6cdc252d-5f28-4636-8da8-7f71b3157d20\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-qwp6t"
Sep 18 19:43:15.623466 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:15.623433    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cdc252d-5f28-4636-8da8-7f71b3157d20-model-cache\") pod \"tokenizer-clean-path-test-kserve-5c88c58655-qwp6t\" (UID: \"6cdc252d-5f28-4636-8da8-7f71b3157d20\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-qwp6t"
Sep 18 19:43:15.623669 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:15.623500    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cdc252d-5f28-4636-8da8-7f71b3157d20-home\") pod \"tokenizer-clean-path-test-kserve-5c88c58655-qwp6t\" (UID: \"6cdc252d-5f28-4636-8da8-7f71b3157d20\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-qwp6t"
Sep 18 19:43:15.623669 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:15.623538    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cdc252d-5f28-4636-8da8-7f71b3157d20-tls-certs\") pod \"tokenizer-clean-path-test-kserve-5c88c58655-qwp6t\" (UID: \"6cdc252d-5f28-4636-8da8-7f71b3157d20\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-qwp6t"
Sep 18 19:43:15.623669 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:15.623567    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cdc252d-5f28-4636-8da8-7f71b3157d20-dshm\") pod \"tokenizer-clean-path-test-kserve-5c88c58655-qwp6t\" (UID: \"6cdc252d-5f28-4636-8da8-7f71b3157d20\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-qwp6t"
Sep 18 19:43:15.623669 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:15.623591    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6cdc252d-5f28-4636-8da8-7f71b3157d20-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-5c88c58655-qwp6t\" (UID: \"6cdc252d-5f28-4636-8da8-7f71b3157d20\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-qwp6t"
Sep 18 19:43:15.623669 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:15.623621    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm8cs\" (UniqueName: \"kubernetes.io/projected/6cdc252d-5f28-4636-8da8-7f71b3157d20-kube-api-access-lm8cs\") pod \"tokenizer-clean-path-test-kserve-5c88c58655-qwp6t\" (UID: \"6cdc252d-5f28-4636-8da8-7f71b3157d20\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-qwp6t"
Sep 18 19:43:15.623669 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:15.623646    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cdc252d-5f28-4636-8da8-7f71b3157d20-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-5c88c58655-qwp6t\" (UID: \"6cdc252d-5f28-4636-8da8-7f71b3157d20\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-qwp6t"
Sep 18 19:43:15.624019 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:15.623670    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6cdc252d-5f28-4636-8da8-7f71b3157d20-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-5c88c58655-qwp6t\" (UID: \"6cdc252d-5f28-4636-8da8-7f71b3157d20\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-qwp6t"
Sep 18 19:43:15.624019 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:15.623895    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cdc252d-5f28-4636-8da8-7f71b3157d20-model-cache\") pod \"tokenizer-clean-path-test-kserve-5c88c58655-qwp6t\" (UID: \"6cdc252d-5f28-4636-8da8-7f71b3157d20\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-qwp6t"
Sep 18 19:43:15.624019 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:15.623948    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cdc252d-5f28-4636-8da8-7f71b3157d20-home\") pod \"tokenizer-clean-path-test-kserve-5c88c58655-qwp6t\" (UID: \"6cdc252d-5f28-4636-8da8-7f71b3157d20\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-qwp6t"
Sep 18 19:43:15.624255 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:15.624231    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cdc252d-5f28-4636-8da8-7f71b3157d20-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-5c88c58655-qwp6t\" (UID: \"6cdc252d-5f28-4636-8da8-7f71b3157d20\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-qwp6t"
Sep 18 19:43:15.624375 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:15.624301    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6cdc252d-5f28-4636-8da8-7f71b3157d20-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-5c88c58655-qwp6t\" (UID: \"6cdc252d-5f28-4636-8da8-7f71b3157d20\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-qwp6t"
Sep 18 19:43:15.624423 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:15.624368    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6cdc252d-5f28-4636-8da8-7f71b3157d20-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-5c88c58655-qwp6t\" (UID: \"6cdc252d-5f28-4636-8da8-7f71b3157d20\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-qwp6t"
Sep 18 19:43:15.626135 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:15.626113    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cdc252d-5f28-4636-8da8-7f71b3157d20-dshm\") pod \"tokenizer-clean-path-test-kserve-5c88c58655-qwp6t\" (UID: \"6cdc252d-5f28-4636-8da8-7f71b3157d20\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-qwp6t"
Sep 18 19:43:15.626135 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:15.626126    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cdc252d-5f28-4636-8da8-7f71b3157d20-tls-certs\") pod \"tokenizer-clean-path-test-kserve-5c88c58655-qwp6t\" (UID: \"6cdc252d-5f28-4636-8da8-7f71b3157d20\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-qwp6t"
Sep 18 19:43:15.645875 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:15.645807    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm8cs\" (UniqueName: \"kubernetes.io/projected/6cdc252d-5f28-4636-8da8-7f71b3157d20-kube-api-access-lm8cs\") pod \"tokenizer-clean-path-test-kserve-5c88c58655-qwp6t\" (UID: \"6cdc252d-5f28-4636-8da8-7f71b3157d20\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-qwp6t"
Sep 18 19:43:15.691301 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:15.691266    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-qwp6t"
Sep 18 19:43:15.967638 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:15.967613    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-qwp6t"]
Sep 18 19:43:15.969455 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:43:15.969432    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cdc252d_5f28_4636_8da8_7f71b3157d20.slice/crio-5b84252871effdfd37b14035c8cdcf50f40208b9ece4b6e83d70fef8e48deb03 WatchSource:0}: Error finding container 5b84252871effdfd37b14035c8cdcf50f40208b9ece4b6e83d70fef8e48deb03: Status 404 returned error can't find the container with id 5b84252871effdfd37b14035c8cdcf50f40208b9ece4b6e83d70fef8e48deb03
Sep 18 19:43:15.971381 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:15.971364    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:43:16.840956 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:16.840875    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-qwp6t" event={"ID":"6cdc252d-5f28-4636-8da8-7f71b3157d20","Type":"ContainerStarted","Data":"5b84252871effdfd37b14035c8cdcf50f40208b9ece4b6e83d70fef8e48deb03"}
Sep 18 19:43:19.860370 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:19.860286    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-qwp6t" event={"ID":"6cdc252d-5f28-4636-8da8-7f71b3157d20","Type":"ContainerStarted","Data":"3a53f34dcc50ccd99cdfbdfe79873e06c121e4241aa8ca6ea173e9c545f5dbf1"}
Sep 18 19:43:21.871584 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:21.871551    2577 generic.go:356] "Generic (PLEG): container finished" podID="6cdc252d-5f28-4636-8da8-7f71b3157d20" containerID="3a53f34dcc50ccd99cdfbdfe79873e06c121e4241aa8ca6ea173e9c545f5dbf1" exitCode=0
Sep 18 19:43:21.872126 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:21.871619    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-qwp6t" event={"ID":"6cdc252d-5f28-4636-8da8-7f71b3157d20","Type":"ContainerDied","Data":"3a53f34dcc50ccd99cdfbdfe79873e06c121e4241aa8ca6ea173e9c545f5dbf1"}
Sep 18 19:43:22.877595 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:22.877559    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-qwp6t" event={"ID":"6cdc252d-5f28-4636-8da8-7f71b3157d20","Type":"ContainerStarted","Data":"9db11c2ef9f966d55b3901b0fae5a40654198776f95312d17ce867090cb378e4"}
Sep 18 19:43:22.930141 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:22.930091    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-qwp6t" podStartSLOduration=2.32075355 podStartE2EDuration="7.930073848s" totalImagesPullingTime="3.552019824s" totalInitContainerRuntime="2.057300474s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:43:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:43:22.929938273 +0000 UTC m=+1773.782238644" watchObservedRunningTime="2026-09-18 19:43:22.930073848 +0000 UTC m=+1773.782374219"
Sep 18 19:43:23.691736 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:23.691699    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-qwp6t"
Sep 18 19:43:25.692428 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:25.692387    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-qwp6t"
Sep 18 19:43:25.704773 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:25.704747    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-qwp6t"
Sep 18 19:43:25.900821 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:25.900790    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-qwp6t"
Sep 18 19:43:52.769671 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:52.769632    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-qwp6t"]
Sep 18 19:43:52.778706 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:43:52.778665    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 19:43:52.778706 ip-10-0-129-30 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:43:52.778706 ip-10-0-129-30 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 18 19:43:52.778706 ip-10-0-129-30 kubenswrapper[2577]:  > containerID="9db11c2ef9f966d55b3901b0fae5a40654198776f95312d17ce867090cb378e4" cmd=["/bin/sleep","15"]
Sep 18 19:43:52.778946 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:52.778719    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-qwp6t" message="" err=<
Sep 18 19:43:52.778946 ip-10-0-129-30 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:43:52.778946 ip-10-0-129-30 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 18 19:43:52.778946 ip-10-0-129-30 kubenswrapper[2577]:  >
Sep 18 19:43:52.778946 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:43:52.778730    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 19:43:52.778946 ip-10-0-129-30 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:43:52.778946 ip-10-0-129-30 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 18 19:43:52.778946 ip-10-0-129-30 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-qwp6t" podUID="6cdc252d-5f28-4636-8da8-7f71b3157d20" containerName="main" containerID="cri-o://9db11c2ef9f966d55b3901b0fae5a40654198776f95312d17ce867090cb378e4"
Sep 18 19:43:52.778946 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:52.778778    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-qwp6t" podUID="6cdc252d-5f28-4636-8da8-7f71b3157d20" containerName="main" containerID="cri-o://9db11c2ef9f966d55b3901b0fae5a40654198776f95312d17ce867090cb378e4" gracePeriod=60
Sep 18 19:43:52.890690 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:52.890649    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-qwp6t" podUID="6cdc252d-5f28-4636-8da8-7f71b3157d20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/health\": dial tcp 10.132.0.33:8000: connect: connection refused"
Sep 18 19:43:53.007506 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:53.007476    2577 generic.go:356] "Generic (PLEG): container finished" podID="6cdc252d-5f28-4636-8da8-7f71b3157d20" containerID="9db11c2ef9f966d55b3901b0fae5a40654198776f95312d17ce867090cb378e4" exitCode=0
Sep 18 19:43:53.007654 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:53.007559    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-qwp6t" event={"ID":"6cdc252d-5f28-4636-8da8-7f71b3157d20","Type":"ContainerDied","Data":"9db11c2ef9f966d55b3901b0fae5a40654198776f95312d17ce867090cb378e4"}
Sep 18 19:43:53.124390 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:53.124366    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-qwp6t"
Sep 18 19:43:53.166523 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:53.166492    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lm8cs\" (UniqueName: \"kubernetes.io/projected/6cdc252d-5f28-4636-8da8-7f71b3157d20-kube-api-access-lm8cs\") pod \"6cdc252d-5f28-4636-8da8-7f71b3157d20\" (UID: \"6cdc252d-5f28-4636-8da8-7f71b3157d20\") "
Sep 18 19:43:53.166671 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:53.166540    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cdc252d-5f28-4636-8da8-7f71b3157d20-model-cache\") pod \"6cdc252d-5f28-4636-8da8-7f71b3157d20\" (UID: \"6cdc252d-5f28-4636-8da8-7f71b3157d20\") "
Sep 18 19:43:53.166671 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:53.166577    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cdc252d-5f28-4636-8da8-7f71b3157d20-tmp-dir\") pod \"6cdc252d-5f28-4636-8da8-7f71b3157d20\" (UID: \"6cdc252d-5f28-4636-8da8-7f71b3157d20\") "
Sep 18 19:43:53.166671 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:53.166648    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6cdc252d-5f28-4636-8da8-7f71b3157d20-cabundle-cert\") pod \"6cdc252d-5f28-4636-8da8-7f71b3157d20\" (UID: \"6cdc252d-5f28-4636-8da8-7f71b3157d20\") "
Sep 18 19:43:53.166831 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:53.166695    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cdc252d-5f28-4636-8da8-7f71b3157d20-tls-certs\") pod \"6cdc252d-5f28-4636-8da8-7f71b3157d20\" (UID: \"6cdc252d-5f28-4636-8da8-7f71b3157d20\") "
Sep 18 19:43:53.166831 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:53.166729    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cdc252d-5f28-4636-8da8-7f71b3157d20-dshm\") pod \"6cdc252d-5f28-4636-8da8-7f71b3157d20\" (UID: \"6cdc252d-5f28-4636-8da8-7f71b3157d20\") "
Sep 18 19:43:53.166950 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:53.166856    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6cdc252d-5f28-4636-8da8-7f71b3157d20-kserve-provision-location\") pod \"6cdc252d-5f28-4636-8da8-7f71b3157d20\" (UID: \"6cdc252d-5f28-4636-8da8-7f71b3157d20\") "
Sep 18 19:43:53.166950 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:53.166915    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cdc252d-5f28-4636-8da8-7f71b3157d20-home\") pod \"6cdc252d-5f28-4636-8da8-7f71b3157d20\" (UID: \"6cdc252d-5f28-4636-8da8-7f71b3157d20\") "
Sep 18 19:43:53.167098 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:53.166832    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cdc252d-5f28-4636-8da8-7f71b3157d20-model-cache" (OuterVolumeSpecName: "model-cache") pod "6cdc252d-5f28-4636-8da8-7f71b3157d20" (UID: "6cdc252d-5f28-4636-8da8-7f71b3157d20"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:43:53.167098 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:53.166911    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cdc252d-5f28-4636-8da8-7f71b3157d20-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6cdc252d-5f28-4636-8da8-7f71b3157d20" (UID: "6cdc252d-5f28-4636-8da8-7f71b3157d20"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:43:53.167205 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:53.167093    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cdc252d-5f28-4636-8da8-7f71b3157d20-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6cdc252d-5f28-4636-8da8-7f71b3157d20" (UID: "6cdc252d-5f28-4636-8da8-7f71b3157d20"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:43:53.167205 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:53.167176    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cdc252d-5f28-4636-8da8-7f71b3157d20-home" (OuterVolumeSpecName: "home") pod "6cdc252d-5f28-4636-8da8-7f71b3157d20" (UID: "6cdc252d-5f28-4636-8da8-7f71b3157d20"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:43:53.167308 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:53.167246    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cdc252d-5f28-4636-8da8-7f71b3157d20-model-cache\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:43:53.167308 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:53.167267    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cdc252d-5f28-4636-8da8-7f71b3157d20-tmp-dir\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:43:53.169473 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:53.169451    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cdc252d-5f28-4636-8da8-7f71b3157d20-dshm" (OuterVolumeSpecName: "dshm") pod "6cdc252d-5f28-4636-8da8-7f71b3157d20" (UID: "6cdc252d-5f28-4636-8da8-7f71b3157d20"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:43:53.169473 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:53.169460    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cdc252d-5f28-4636-8da8-7f71b3157d20-kube-api-access-lm8cs" (OuterVolumeSpecName: "kube-api-access-lm8cs") pod "6cdc252d-5f28-4636-8da8-7f71b3157d20" (UID: "6cdc252d-5f28-4636-8da8-7f71b3157d20"). InnerVolumeSpecName "kube-api-access-lm8cs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:43:53.169639 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:53.169472    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cdc252d-5f28-4636-8da8-7f71b3157d20-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6cdc252d-5f28-4636-8da8-7f71b3157d20" (UID: "6cdc252d-5f28-4636-8da8-7f71b3157d20"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:43:53.187002 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:53.186961    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cdc252d-5f28-4636-8da8-7f71b3157d20-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6cdc252d-5f28-4636-8da8-7f71b3157d20" (UID: "6cdc252d-5f28-4636-8da8-7f71b3157d20"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:43:53.268657 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:53.268616    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6cdc252d-5f28-4636-8da8-7f71b3157d20-cabundle-cert\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:43:53.268657 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:53.268650    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cdc252d-5f28-4636-8da8-7f71b3157d20-tls-certs\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:43:53.268657 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:53.268659    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cdc252d-5f28-4636-8da8-7f71b3157d20-dshm\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:43:53.268898 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:53.268668    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6cdc252d-5f28-4636-8da8-7f71b3157d20-kserve-provision-location\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:43:53.268898 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:53.268683    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cdc252d-5f28-4636-8da8-7f71b3157d20-home\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:43:53.268898 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:53.268692    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm8cs\" (UniqueName: \"kubernetes.io/projected/6cdc252d-5f28-4636-8da8-7f71b3157d20-kube-api-access-lm8cs\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:43:54.013453 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:54.013421    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-qwp6t"
Sep 18 19:43:54.013935 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:54.013424    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-qwp6t" event={"ID":"6cdc252d-5f28-4636-8da8-7f71b3157d20","Type":"ContainerDied","Data":"5b84252871effdfd37b14035c8cdcf50f40208b9ece4b6e83d70fef8e48deb03"}
Sep 18 19:43:54.013935 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:54.013562    2577 scope.go:117] "RemoveContainer" containerID="9db11c2ef9f966d55b3901b0fae5a40654198776f95312d17ce867090cb378e4"
Sep 18 19:43:54.022305 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:54.022286    2577 scope.go:117] "RemoveContainer" containerID="3a53f34dcc50ccd99cdfbdfe79873e06c121e4241aa8ca6ea173e9c545f5dbf1"
Sep 18 19:43:54.036580 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:54.036558    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-qwp6t"]
Sep 18 19:43:54.041839 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:54.041816    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-qwp6t"]
Sep 18 19:43:55.438810 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:43:55.438781    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cdc252d-5f28-4636-8da8-7f71b3157d20" path="/var/lib/kubelet/pods/6cdc252d-5f28-4636-8da8-7f71b3157d20/volumes"
Sep 18 19:44:45.305421 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:44:45.305381    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-v52jz"]
Sep 18 19:44:45.305969 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:44:45.305878    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cdc252d-5f28-4636-8da8-7f71b3157d20" containerName="main"
Sep 18 19:44:45.305969 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:44:45.305899    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cdc252d-5f28-4636-8da8-7f71b3157d20" containerName="main"
Sep 18 19:44:45.305969 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:44:45.305921    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cdc252d-5f28-4636-8da8-7f71b3157d20" containerName="storage-initializer"
Sep 18 19:44:45.305969 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:44:45.305931    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cdc252d-5f28-4636-8da8-7f71b3157d20" containerName="storage-initializer"
Sep 18 19:44:45.306230 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:44:45.306035    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cdc252d-5f28-4636-8da8-7f71b3157d20" containerName="main"
Sep 18 19:44:45.309079 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:44:45.309058    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-v52jz"
Sep 18 19:44:45.311747 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:44:45.311724    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-sxvdz\""
Sep 18 19:44:45.313165 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:44:45.313131    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 19:44:45.313165 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:44:45.313152    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 18 19:44:45.313349 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:44:45.313136    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 18 19:44:45.313421 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:44:45.313407    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 18 19:44:45.313775 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:44:45.313760    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 18 19:44:45.323427 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:44:45.323407    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-v52jz"]
Sep 18 19:44:45.426311 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:44:45.426277    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a34ec3d5-57e7-4c5b-ab7b-041c503e451f-home\") pod \"tokenizer-migration-path-test-kserve-c6f9494cd-v52jz\" (UID: \"a34ec3d5-57e7-4c5b-ab7b-041c503e451f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-v52jz"
Sep 18 19:44:45.426487 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:44:45.426325    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a34ec3d5-57e7-4c5b-ab7b-041c503e451f-dshm\") pod \"tokenizer-migration-path-test-kserve-c6f9494cd-v52jz\" (UID: \"a34ec3d5-57e7-4c5b-ab7b-041c503e451f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-v52jz"
Sep 18 19:44:45.426487 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:44:45.426395    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a34ec3d5-57e7-4c5b-ab7b-041c503e451f-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-c6f9494cd-v52jz\" (UID: \"a34ec3d5-57e7-4c5b-ab7b-041c503e451f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-v52jz"
Sep 18 19:44:45.426487 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:44:45.426415    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jpjv\" (UniqueName: \"kubernetes.io/projected/a34ec3d5-57e7-4c5b-ab7b-041c503e451f-kube-api-access-2jpjv\") pod \"tokenizer-migration-path-test-kserve-c6f9494cd-v52jz\" (UID: \"a34ec3d5-57e7-4c5b-ab7b-041c503e451f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-v52jz"
Sep 18 19:44:45.426487 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:44:45.426474    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a34ec3d5-57e7-4c5b-ab7b-041c503e451f-model-cache\") pod \"tokenizer-migration-path-test-kserve-c6f9494cd-v52jz\" (UID: \"a34ec3d5-57e7-4c5b-ab7b-041c503e451f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-v52jz"
Sep 18 19:44:45.426633 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:44:45.426490    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a34ec3d5-57e7-4c5b-ab7b-041c503e451f-tls-certs\") pod \"tokenizer-migration-path-test-kserve-c6f9494cd-v52jz\" (UID: \"a34ec3d5-57e7-4c5b-ab7b-041c503e451f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-v52jz"
Sep 18 19:44:45.426633 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:44:45.426520    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a34ec3d5-57e7-4c5b-ab7b-041c503e451f-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-c6f9494cd-v52jz\" (UID: \"a34ec3d5-57e7-4c5b-ab7b-041c503e451f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-v52jz"
Sep 18 19:44:45.426633 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:44:45.426558    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a34ec3d5-57e7-4c5b-ab7b-041c503e451f-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-c6f9494cd-v52jz\" (UID: \"a34ec3d5-57e7-4c5b-ab7b-041c503e451f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-v52jz"
Sep 18 19:44:45.527693 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:44:45.527656    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a34ec3d5-57e7-4c5b-ab7b-041c503e451f-model-cache\") pod \"tokenizer-migration-path-test-kserve-c6f9494cd-v52jz\" (UID: \"a34ec3d5-57e7-4c5b-ab7b-041c503e451f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-v52jz"
Sep 18 19:44:45.527693 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:44:45.527694    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a34ec3d5-57e7-4c5b-ab7b-041c503e451f-tls-certs\") pod \"tokenizer-migration-path-test-kserve-c6f9494cd-v52jz\" (UID: \"a34ec3d5-57e7-4c5b-ab7b-041c503e451f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-v52jz"
Sep 18 19:44:45.527937 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:44:45.527713    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a34ec3d5-57e7-4c5b-ab7b-041c503e451f-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-c6f9494cd-v52jz\" (UID: \"a34ec3d5-57e7-4c5b-ab7b-041c503e451f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-v52jz"
Sep 18 19:44:45.527937 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:44:45.527736    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a34ec3d5-57e7-4c5b-ab7b-041c503e451f-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-c6f9494cd-v52jz\" (UID: \"a34ec3d5-57e7-4c5b-ab7b-041c503e451f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-v52jz"
Sep 18 19:44:45.527937 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:44:45.527781    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a34ec3d5-57e7-4c5b-ab7b-041c503e451f-home\") pod \"tokenizer-migration-path-test-kserve-c6f9494cd-v52jz\" (UID: \"a34ec3d5-57e7-4c5b-ab7b-041c503e451f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-v52jz"
Sep 18 19:44:45.527937 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:44:45.527822    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a34ec3d5-57e7-4c5b-ab7b-041c503e451f-dshm\") pod \"tokenizer-migration-path-test-kserve-c6f9494cd-v52jz\" (UID: \"a34ec3d5-57e7-4c5b-ab7b-041c503e451f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-v52jz"
Sep 18 19:44:45.527937 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:44:45.527877    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a34ec3d5-57e7-4c5b-ab7b-041c503e451f-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-c6f9494cd-v52jz\" (UID: \"a34ec3d5-57e7-4c5b-ab7b-041c503e451f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-v52jz"
Sep 18 19:44:45.527937 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:44:45.527902    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jpjv\" (UniqueName: \"kubernetes.io/projected/a34ec3d5-57e7-4c5b-ab7b-041c503e451f-kube-api-access-2jpjv\") pod \"tokenizer-migration-path-test-kserve-c6f9494cd-v52jz\" (UID: \"a34ec3d5-57e7-4c5b-ab7b-041c503e451f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-v52jz"
Sep 18 19:44:45.528289 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:44:45.528147    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a34ec3d5-57e7-4c5b-ab7b-041c503e451f-model-cache\") pod \"tokenizer-migration-path-test-kserve-c6f9494cd-v52jz\" (UID: \"a34ec3d5-57e7-4c5b-ab7b-041c503e451f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-v52jz"
Sep 18 19:44:45.528397 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:44:45.528376    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a34ec3d5-57e7-4c5b-ab7b-041c503e451f-home\") pod \"tokenizer-migration-path-test-kserve-c6f9494cd-v52jz\" (UID: \"a34ec3d5-57e7-4c5b-ab7b-041c503e451f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-v52jz"
Sep 18 19:44:45.528461 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:44:45.528402    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a34ec3d5-57e7-4c5b-ab7b-041c503e451f-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-c6f9494cd-v52jz\" (UID: \"a34ec3d5-57e7-4c5b-ab7b-041c503e451f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-v52jz"
Sep 18 19:44:45.528500 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:44:45.528453    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a34ec3d5-57e7-4c5b-ab7b-041c503e451f-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-c6f9494cd-v52jz\" (UID: \"a34ec3d5-57e7-4c5b-ab7b-041c503e451f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-v52jz"
Sep 18 19:44:45.528736 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:44:45.528717    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a34ec3d5-57e7-4c5b-ab7b-041c503e451f-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-c6f9494cd-v52jz\" (UID: \"a34ec3d5-57e7-4c5b-ab7b-041c503e451f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-v52jz"
Sep 18 19:44:45.530102 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:44:45.530084    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a34ec3d5-57e7-4c5b-ab7b-041c503e451f-dshm\") pod \"tokenizer-migration-path-test-kserve-c6f9494cd-v52jz\" (UID: \"a34ec3d5-57e7-4c5b-ab7b-041c503e451f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-v52jz"
Sep 18 19:44:45.530490 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:44:45.530471    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a34ec3d5-57e7-4c5b-ab7b-041c503e451f-tls-certs\") pod \"tokenizer-migration-path-test-kserve-c6f9494cd-v52jz\" (UID: \"a34ec3d5-57e7-4c5b-ab7b-041c503e451f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-v52jz"
Sep 18 19:44:45.548862 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:44:45.548834    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jpjv\" (UniqueName: \"kubernetes.io/projected/a34ec3d5-57e7-4c5b-ab7b-041c503e451f-kube-api-access-2jpjv\") pod \"tokenizer-migration-path-test-kserve-c6f9494cd-v52jz\" (UID: \"a34ec3d5-57e7-4c5b-ab7b-041c503e451f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-v52jz"
Sep 18 19:44:45.619159 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:44:45.619130    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-v52jz"
Sep 18 19:44:45.915360 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:44:45.915335    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-v52jz"]
Sep 18 19:44:45.918215 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:44:45.918189    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda34ec3d5_57e7_4c5b_ab7b_041c503e451f.slice/crio-fcb5e9150fc5d1fc1e328c1855c694020b0a34ea585e609ce7eb375439a96962 WatchSource:0}: Error finding container fcb5e9150fc5d1fc1e328c1855c694020b0a34ea585e609ce7eb375439a96962: Status 404 returned error can't find the container with id fcb5e9150fc5d1fc1e328c1855c694020b0a34ea585e609ce7eb375439a96962
Sep 18 19:44:46.243146 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:44:46.243055    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-v52jz" event={"ID":"a34ec3d5-57e7-4c5b-ab7b-041c503e451f","Type":"ContainerStarted","Data":"2fc144705a0324de6c7757c433172a4e462e80a14a4c83a4e54b9dd3ba4600c2"}
Sep 18 19:44:46.243293 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:44:46.243150    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-v52jz" event={"ID":"a34ec3d5-57e7-4c5b-ab7b-041c503e451f","Type":"ContainerStarted","Data":"fcb5e9150fc5d1fc1e328c1855c694020b0a34ea585e609ce7eb375439a96962"}
Sep 18 19:44:48.254003 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:44:48.253958    2577 generic.go:356] "Generic (PLEG): container finished" podID="a34ec3d5-57e7-4c5b-ab7b-041c503e451f" containerID="2fc144705a0324de6c7757c433172a4e462e80a14a4c83a4e54b9dd3ba4600c2" exitCode=0
Sep 18 19:44:48.254375 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:44:48.254024    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-v52jz" event={"ID":"a34ec3d5-57e7-4c5b-ab7b-041c503e451f","Type":"ContainerDied","Data":"2fc144705a0324de6c7757c433172a4e462e80a14a4c83a4e54b9dd3ba4600c2"}
Sep 18 19:44:49.259945 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:44:49.259913    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-v52jz" event={"ID":"a34ec3d5-57e7-4c5b-ab7b-041c503e451f","Type":"ContainerStarted","Data":"38b213fc437cc3f97881e9be16060419534c4c17d13cc7da2b24a5a5eae34a25"}
Sep 18 19:44:49.296697 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:44:49.296631    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-v52jz" podStartSLOduration=2.120995976 podStartE2EDuration="4.296614337s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.175618361s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:44:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:44:49.294813517 +0000 UTC m=+1860.147113886" watchObservedRunningTime="2026-09-18 19:44:49.296614337 +0000 UTC m=+1860.148914707"
Sep 18 19:44:49.619875 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:44:49.619836    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-v52jz"
Sep 18 19:44:55.619940 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:44:55.619904    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-v52jz"
Sep 18 19:44:55.632428 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:44:55.632401    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-v52jz"
Sep 18 19:44:56.301704 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:44:56.301675    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-v52jz"
Sep 18 19:45:15.806407 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:15.806370    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d5c7c8987-mxkw7"]
Sep 18 19:45:15.809642 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:15.809624    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d5c7c8987-mxkw7"
Sep 18 19:45:15.812713 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:15.812688    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Sep 18 19:45:15.813502 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:15.813471    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Sep 18 19:45:15.813630 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:15.813534    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 19:45:15.813630 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:15.813536    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-nlb9d\""
Sep 18 19:45:15.813630 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:15.813482    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"seaweedfs-s3-creds\""
Sep 18 19:45:15.813630 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:15.813577    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Sep 18 19:45:15.827046 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:15.827019    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d5c7c8987-mxkw7"]
Sep 18 19:45:15.902776 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:15.902746    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/569bc59d-5833-4190-a316-242303c52bd5-dshm\") pod \"auth-invalid-token-test-kserve-5d5c7c8987-mxkw7\" (UID: \"569bc59d-5833-4190-a316-242303c52bd5\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d5c7c8987-mxkw7"
Sep 18 19:45:15.902776 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:15.902776    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/569bc59d-5833-4190-a316-242303c52bd5-model-cache\") pod \"auth-invalid-token-test-kserve-5d5c7c8987-mxkw7\" (UID: \"569bc59d-5833-4190-a316-242303c52bd5\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d5c7c8987-mxkw7"
Sep 18 19:45:15.902967 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:15.902814    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/569bc59d-5833-4190-a316-242303c52bd5-tmp-dir\") pod \"auth-invalid-token-test-kserve-5d5c7c8987-mxkw7\" (UID: \"569bc59d-5833-4190-a316-242303c52bd5\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d5c7c8987-mxkw7"
Sep 18 19:45:15.902967 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:15.902831    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/569bc59d-5833-4190-a316-242303c52bd5-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-5d5c7c8987-mxkw7\" (UID: \"569bc59d-5833-4190-a316-242303c52bd5\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d5c7c8987-mxkw7"
Sep 18 19:45:15.902967 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:15.902898    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/569bc59d-5833-4190-a316-242303c52bd5-cabundle-cert\") pod \"auth-invalid-token-test-kserve-5d5c7c8987-mxkw7\" (UID: \"569bc59d-5833-4190-a316-242303c52bd5\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d5c7c8987-mxkw7"
Sep 18 19:45:15.902967 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:15.902940    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjl5x\" (UniqueName: \"kubernetes.io/projected/569bc59d-5833-4190-a316-242303c52bd5-kube-api-access-pjl5x\") pod \"auth-invalid-token-test-kserve-5d5c7c8987-mxkw7\" (UID: \"569bc59d-5833-4190-a316-242303c52bd5\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d5c7c8987-mxkw7"
Sep 18 19:45:15.903125 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:15.902983    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/569bc59d-5833-4190-a316-242303c52bd5-tls-certs\") pod \"auth-invalid-token-test-kserve-5d5c7c8987-mxkw7\" (UID: \"569bc59d-5833-4190-a316-242303c52bd5\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d5c7c8987-mxkw7"
Sep 18 19:45:15.903125 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:15.903044    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/569bc59d-5833-4190-a316-242303c52bd5-home\") pod \"auth-invalid-token-test-kserve-5d5c7c8987-mxkw7\" (UID: \"569bc59d-5833-4190-a316-242303c52bd5\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d5c7c8987-mxkw7"
Sep 18 19:45:16.003551 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:16.003516    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/569bc59d-5833-4190-a316-242303c52bd5-tls-certs\") pod \"auth-invalid-token-test-kserve-5d5c7c8987-mxkw7\" (UID: \"569bc59d-5833-4190-a316-242303c52bd5\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d5c7c8987-mxkw7"
Sep 18 19:45:16.003551 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:16.003556    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/569bc59d-5833-4190-a316-242303c52bd5-home\") pod \"auth-invalid-token-test-kserve-5d5c7c8987-mxkw7\" (UID: \"569bc59d-5833-4190-a316-242303c52bd5\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d5c7c8987-mxkw7"
Sep 18 19:45:16.003802 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:16.003587    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/569bc59d-5833-4190-a316-242303c52bd5-dshm\") pod \"auth-invalid-token-test-kserve-5d5c7c8987-mxkw7\" (UID: \"569bc59d-5833-4190-a316-242303c52bd5\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d5c7c8987-mxkw7"
Sep 18 19:45:16.003802 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:16.003603    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/569bc59d-5833-4190-a316-242303c52bd5-model-cache\") pod \"auth-invalid-token-test-kserve-5d5c7c8987-mxkw7\" (UID: \"569bc59d-5833-4190-a316-242303c52bd5\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d5c7c8987-mxkw7"
Sep 18 19:45:16.003802 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:16.003641    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/569bc59d-5833-4190-a316-242303c52bd5-tmp-dir\") pod \"auth-invalid-token-test-kserve-5d5c7c8987-mxkw7\" (UID: \"569bc59d-5833-4190-a316-242303c52bd5\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d5c7c8987-mxkw7"
Sep 18 19:45:16.003802 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:16.003657    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/569bc59d-5833-4190-a316-242303c52bd5-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-5d5c7c8987-mxkw7\" (UID: \"569bc59d-5833-4190-a316-242303c52bd5\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d5c7c8987-mxkw7"
Sep 18 19:45:16.003802 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:16.003676    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/569bc59d-5833-4190-a316-242303c52bd5-cabundle-cert\") pod \"auth-invalid-token-test-kserve-5d5c7c8987-mxkw7\" (UID: \"569bc59d-5833-4190-a316-242303c52bd5\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d5c7c8987-mxkw7"
Sep 18 19:45:16.003802 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:16.003705    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjl5x\" (UniqueName: \"kubernetes.io/projected/569bc59d-5833-4190-a316-242303c52bd5-kube-api-access-pjl5x\") pod \"auth-invalid-token-test-kserve-5d5c7c8987-mxkw7\" (UID: \"569bc59d-5833-4190-a316-242303c52bd5\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d5c7c8987-mxkw7"
Sep 18 19:45:16.004181 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:16.003966    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/569bc59d-5833-4190-a316-242303c52bd5-home\") pod \"auth-invalid-token-test-kserve-5d5c7c8987-mxkw7\" (UID: \"569bc59d-5833-4190-a316-242303c52bd5\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d5c7c8987-mxkw7"
Sep 18 19:45:16.004181 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:16.004119    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/569bc59d-5833-4190-a316-242303c52bd5-model-cache\") pod \"auth-invalid-token-test-kserve-5d5c7c8987-mxkw7\" (UID: \"569bc59d-5833-4190-a316-242303c52bd5\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d5c7c8987-mxkw7"
Sep 18 19:45:16.004288 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:16.004231    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/569bc59d-5833-4190-a316-242303c52bd5-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-5d5c7c8987-mxkw7\" (UID: \"569bc59d-5833-4190-a316-242303c52bd5\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d5c7c8987-mxkw7"
Sep 18 19:45:16.004288 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:16.004232    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/569bc59d-5833-4190-a316-242303c52bd5-tmp-dir\") pod \"auth-invalid-token-test-kserve-5d5c7c8987-mxkw7\" (UID: \"569bc59d-5833-4190-a316-242303c52bd5\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d5c7c8987-mxkw7"
Sep 18 19:45:16.004426 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:16.004408    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/569bc59d-5833-4190-a316-242303c52bd5-cabundle-cert\") pod \"auth-invalid-token-test-kserve-5d5c7c8987-mxkw7\" (UID: \"569bc59d-5833-4190-a316-242303c52bd5\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d5c7c8987-mxkw7"
Sep 18 19:45:16.006517 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:16.006488    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/569bc59d-5833-4190-a316-242303c52bd5-dshm\") pod \"auth-invalid-token-test-kserve-5d5c7c8987-mxkw7\" (UID: \"569bc59d-5833-4190-a316-242303c52bd5\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d5c7c8987-mxkw7"
Sep 18 19:45:16.006665 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:16.006647    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/569bc59d-5833-4190-a316-242303c52bd5-tls-certs\") pod \"auth-invalid-token-test-kserve-5d5c7c8987-mxkw7\" (UID: \"569bc59d-5833-4190-a316-242303c52bd5\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d5c7c8987-mxkw7"
Sep 18 19:45:16.012867 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:16.012847    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjl5x\" (UniqueName: \"kubernetes.io/projected/569bc59d-5833-4190-a316-242303c52bd5-kube-api-access-pjl5x\") pod \"auth-invalid-token-test-kserve-5d5c7c8987-mxkw7\" (UID: \"569bc59d-5833-4190-a316-242303c52bd5\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d5c7c8987-mxkw7"
Sep 18 19:45:16.119713 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:16.119664    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d5c7c8987-mxkw7"
Sep 18 19:45:16.404384 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:16.404304    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d5c7c8987-mxkw7"]
Sep 18 19:45:16.407854 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:45:16.407820    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod569bc59d_5833_4190_a316_242303c52bd5.slice/crio-5a5d6608519053e7e241cb756b613cb0c13439fd95c3eec826862b42140a4097 WatchSource:0}: Error finding container 5a5d6608519053e7e241cb756b613cb0c13439fd95c3eec826862b42140a4097: Status 404 returned error can't find the container with id 5a5d6608519053e7e241cb756b613cb0c13439fd95c3eec826862b42140a4097
Sep 18 19:45:17.386377 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:17.386341    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d5c7c8987-mxkw7" event={"ID":"569bc59d-5833-4190-a316-242303c52bd5","Type":"ContainerStarted","Data":"031e0e1561b928730630f8debf9b5e3d016b5e668bdd5a77e487155bec92f8a7"}
Sep 18 19:45:17.386770 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:17.386395    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d5c7c8987-mxkw7" event={"ID":"569bc59d-5833-4190-a316-242303c52bd5","Type":"ContainerStarted","Data":"5a5d6608519053e7e241cb756b613cb0c13439fd95c3eec826862b42140a4097"}
Sep 18 19:45:19.396277 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:19.396244    2577 generic.go:356] "Generic (PLEG): container finished" podID="569bc59d-5833-4190-a316-242303c52bd5" containerID="031e0e1561b928730630f8debf9b5e3d016b5e668bdd5a77e487155bec92f8a7" exitCode=0
Sep 18 19:45:19.396688 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:19.396298    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d5c7c8987-mxkw7" event={"ID":"569bc59d-5833-4190-a316-242303c52bd5","Type":"ContainerDied","Data":"031e0e1561b928730630f8debf9b5e3d016b5e668bdd5a77e487155bec92f8a7"}
Sep 18 19:45:44.570139 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:44.570101    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-v52jz"]
Sep 18 19:45:44.581266 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:45:44.581221    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 19:45:44.581266 ip-10-0-129-30 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:45:44.581266 ip-10-0-129-30 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 18 19:45:44.581266 ip-10-0-129-30 kubenswrapper[2577]:  > containerID="38b213fc437cc3f97881e9be16060419534c4c17d13cc7da2b24a5a5eae34a25" cmd=["/bin/sleep","15"]
Sep 18 19:45:44.581594 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:44.581288    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-v52jz" message="" err=<
Sep 18 19:45:44.581594 ip-10-0-129-30 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:45:44.581594 ip-10-0-129-30 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 18 19:45:44.581594 ip-10-0-129-30 kubenswrapper[2577]:  >
Sep 18 19:45:44.581594 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:45:44.581300    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 19:45:44.581594 ip-10-0-129-30 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:45:44.581594 ip-10-0-129-30 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 18 19:45:44.581594 ip-10-0-129-30 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-v52jz" podUID="a34ec3d5-57e7-4c5b-ab7b-041c503e451f" containerName="main" containerID="cri-o://38b213fc437cc3f97881e9be16060419534c4c17d13cc7da2b24a5a5eae34a25"
Sep 18 19:45:44.581594 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:44.581344    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-v52jz" podUID="a34ec3d5-57e7-4c5b-ab7b-041c503e451f" containerName="main" containerID="cri-o://38b213fc437cc3f97881e9be16060419534c4c17d13cc7da2b24a5a5eae34a25" gracePeriod=60
Sep 18 19:45:45.291449 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:45.291394    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-v52jz" podUID="a34ec3d5-57e7-4c5b-ab7b-041c503e451f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8000/health\": dial tcp 10.132.0.34:8000: connect: connection refused"
Sep 18 19:45:46.291664 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:46.291619    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-v52jz" podUID="a34ec3d5-57e7-4c5b-ab7b-041c503e451f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8000/health\": dial tcp 10.132.0.34:8000: connect: connection refused"
Sep 18 19:45:46.292126 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:46.291743    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-v52jz"
Sep 18 19:45:46.553971 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:46.553891    2577 generic.go:356] "Generic (PLEG): container finished" podID="a34ec3d5-57e7-4c5b-ab7b-041c503e451f" containerID="38b213fc437cc3f97881e9be16060419534c4c17d13cc7da2b24a5a5eae34a25" exitCode=0
Sep 18 19:45:46.553971 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:46.553934    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-v52jz" event={"ID":"a34ec3d5-57e7-4c5b-ab7b-041c503e451f","Type":"ContainerDied","Data":"38b213fc437cc3f97881e9be16060419534c4c17d13cc7da2b24a5a5eae34a25"}
Sep 18 19:45:47.916114 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:47.916086    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-v52jz"
Sep 18 19:45:47.921930 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:47.921907    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a34ec3d5-57e7-4c5b-ab7b-041c503e451f-cabundle-cert\") pod \"a34ec3d5-57e7-4c5b-ab7b-041c503e451f\" (UID: \"a34ec3d5-57e7-4c5b-ab7b-041c503e451f\") "
Sep 18 19:45:47.922059 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:47.921937    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2jpjv\" (UniqueName: \"kubernetes.io/projected/a34ec3d5-57e7-4c5b-ab7b-041c503e451f-kube-api-access-2jpjv\") pod \"a34ec3d5-57e7-4c5b-ab7b-041c503e451f\" (UID: \"a34ec3d5-57e7-4c5b-ab7b-041c503e451f\") "
Sep 18 19:45:47.922059 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:47.921983    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a34ec3d5-57e7-4c5b-ab7b-041c503e451f-home\") pod \"a34ec3d5-57e7-4c5b-ab7b-041c503e451f\" (UID: \"a34ec3d5-57e7-4c5b-ab7b-041c503e451f\") "
Sep 18 19:45:47.922147 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:47.922057    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a34ec3d5-57e7-4c5b-ab7b-041c503e451f-tls-certs\") pod \"a34ec3d5-57e7-4c5b-ab7b-041c503e451f\" (UID: \"a34ec3d5-57e7-4c5b-ab7b-041c503e451f\") "
Sep 18 19:45:47.922147 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:47.922138    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a34ec3d5-57e7-4c5b-ab7b-041c503e451f-dshm\") pod \"a34ec3d5-57e7-4c5b-ab7b-041c503e451f\" (UID: \"a34ec3d5-57e7-4c5b-ab7b-041c503e451f\") "
Sep 18 19:45:47.922234 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:47.922175    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a34ec3d5-57e7-4c5b-ab7b-041c503e451f-tmp-dir\") pod \"a34ec3d5-57e7-4c5b-ab7b-041c503e451f\" (UID: \"a34ec3d5-57e7-4c5b-ab7b-041c503e451f\") "
Sep 18 19:45:47.922234 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:47.922209    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a34ec3d5-57e7-4c5b-ab7b-041c503e451f-home" (OuterVolumeSpecName: "home") pod "a34ec3d5-57e7-4c5b-ab7b-041c503e451f" (UID: "a34ec3d5-57e7-4c5b-ab7b-041c503e451f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:45:47.922234 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:47.922216    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a34ec3d5-57e7-4c5b-ab7b-041c503e451f-kserve-provision-location\") pod \"a34ec3d5-57e7-4c5b-ab7b-041c503e451f\" (UID: \"a34ec3d5-57e7-4c5b-ab7b-041c503e451f\") "
Sep 18 19:45:47.922391 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:47.922244    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a34ec3d5-57e7-4c5b-ab7b-041c503e451f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a34ec3d5-57e7-4c5b-ab7b-041c503e451f" (UID: "a34ec3d5-57e7-4c5b-ab7b-041c503e451f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:45:47.922391 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:47.922276    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a34ec3d5-57e7-4c5b-ab7b-041c503e451f-model-cache\") pod \"a34ec3d5-57e7-4c5b-ab7b-041c503e451f\" (UID: \"a34ec3d5-57e7-4c5b-ab7b-041c503e451f\") "
Sep 18 19:45:47.922523 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:47.922474    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a34ec3d5-57e7-4c5b-ab7b-041c503e451f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a34ec3d5-57e7-4c5b-ab7b-041c503e451f" (UID: "a34ec3d5-57e7-4c5b-ab7b-041c503e451f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:45:47.922613 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:47.922590    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a34ec3d5-57e7-4c5b-ab7b-041c503e451f-model-cache" (OuterVolumeSpecName: "model-cache") pod "a34ec3d5-57e7-4c5b-ab7b-041c503e451f" (UID: "a34ec3d5-57e7-4c5b-ab7b-041c503e451f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:45:47.922676 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:47.922647    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a34ec3d5-57e7-4c5b-ab7b-041c503e451f-cabundle-cert\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:45:47.922676 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:47.922661    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a34ec3d5-57e7-4c5b-ab7b-041c503e451f-home\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:45:47.922676 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:47.922669    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a34ec3d5-57e7-4c5b-ab7b-041c503e451f-tmp-dir\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:45:47.924310 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:47.924274    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a34ec3d5-57e7-4c5b-ab7b-041c503e451f-dshm" (OuterVolumeSpecName: "dshm") pod "a34ec3d5-57e7-4c5b-ab7b-041c503e451f" (UID: "a34ec3d5-57e7-4c5b-ab7b-041c503e451f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:45:47.924412 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:47.924325    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a34ec3d5-57e7-4c5b-ab7b-041c503e451f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a34ec3d5-57e7-4c5b-ab7b-041c503e451f" (UID: "a34ec3d5-57e7-4c5b-ab7b-041c503e451f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:45:47.924559 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:47.924537    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a34ec3d5-57e7-4c5b-ab7b-041c503e451f-kube-api-access-2jpjv" (OuterVolumeSpecName: "kube-api-access-2jpjv") pod "a34ec3d5-57e7-4c5b-ab7b-041c503e451f" (UID: "a34ec3d5-57e7-4c5b-ab7b-041c503e451f"). InnerVolumeSpecName "kube-api-access-2jpjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:45:47.943094 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:47.943060    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a34ec3d5-57e7-4c5b-ab7b-041c503e451f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a34ec3d5-57e7-4c5b-ab7b-041c503e451f" (UID: "a34ec3d5-57e7-4c5b-ab7b-041c503e451f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:45:48.026163 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:48.025118    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jpjv\" (UniqueName: \"kubernetes.io/projected/a34ec3d5-57e7-4c5b-ab7b-041c503e451f-kube-api-access-2jpjv\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:45:48.026163 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:48.025160    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a34ec3d5-57e7-4c5b-ab7b-041c503e451f-tls-certs\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:45:48.026163 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:48.025176    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a34ec3d5-57e7-4c5b-ab7b-041c503e451f-dshm\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:45:48.026163 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:48.025189    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a34ec3d5-57e7-4c5b-ab7b-041c503e451f-kserve-provision-location\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:45:48.026163 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:48.025202    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a34ec3d5-57e7-4c5b-ab7b-041c503e451f-model-cache\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:45:48.292100 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:48.292067    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-v52jz" podUID="a34ec3d5-57e7-4c5b-ab7b-041c503e451f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8000/health\": context deadline exceeded"
Sep 18 19:45:48.565116 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:48.565027    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-v52jz"
Sep 18 19:45:48.565116 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:48.565029    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-v52jz" event={"ID":"a34ec3d5-57e7-4c5b-ab7b-041c503e451f","Type":"ContainerDied","Data":"fcb5e9150fc5d1fc1e328c1855c694020b0a34ea585e609ce7eb375439a96962"}
Sep 18 19:45:48.565116 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:48.565114    2577 scope.go:117] "RemoveContainer" containerID="38b213fc437cc3f97881e9be16060419534c4c17d13cc7da2b24a5a5eae34a25"
Sep 18 19:45:48.567335 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:48.567304    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d5c7c8987-mxkw7" event={"ID":"569bc59d-5833-4190-a316-242303c52bd5","Type":"ContainerStarted","Data":"b671a5c88c9d9f041c52c74c2ddda848b54ab181070ffd9b1003d8c409c94a16"}
Sep 18 19:45:48.575726 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:48.575704    2577 scope.go:117] "RemoveContainer" containerID="2fc144705a0324de6c7757c433172a4e462e80a14a4c83a4e54b9dd3ba4600c2"
Sep 18 19:45:48.599702 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:48.599644    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d5c7c8987-mxkw7" podStartSLOduration=2.668087333 podStartE2EDuration="33.599622153s" totalImagesPullingTime="28.798156966s" totalInitContainerRuntime="2.133377854s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:45:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:45:48.597434136 +0000 UTC m=+1919.449734507" watchObservedRunningTime="2026-09-18 19:45:48.599622153 +0000 UTC m=+1919.451922526"
Sep 18 19:45:48.612211 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:48.612180    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-v52jz"]
Sep 18 19:45:48.617276 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:48.617249    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-v52jz"]
Sep 18 19:45:49.119802 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:49.119767    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d5c7c8987-mxkw7"
Sep 18 19:45:49.440718 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:49.440637    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a34ec3d5-57e7-4c5b-ab7b-041c503e451f" path="/var/lib/kubelet/pods/a34ec3d5-57e7-4c5b-ab7b-041c503e451f/volumes"
Sep 18 19:45:56.120518 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:56.120483    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d5c7c8987-mxkw7"
Sep 18 19:45:56.122270 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:45:56.122241    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d5c7c8987-mxkw7" podUID="569bc59d-5833-4190-a316-242303c52bd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Sep 18 19:46:06.121005 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:46:06.120955    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d5c7c8987-mxkw7" podUID="569bc59d-5833-4190-a316-242303c52bd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Sep 18 19:46:16.120865 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:46:16.120822    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d5c7c8987-mxkw7" podUID="569bc59d-5833-4190-a316-242303c52bd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Sep 18 19:46:26.120887 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:46:26.120841    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d5c7c8987-mxkw7" podUID="569bc59d-5833-4190-a316-242303c52bd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Sep 18 19:46:36.120967 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:46:36.120924    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d5c7c8987-mxkw7" podUID="569bc59d-5833-4190-a316-242303c52bd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Sep 18 19:46:44.756070 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:46:44.756031    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-sj97q"]
Sep 18 19:46:44.756453 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:46:44.756359    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a34ec3d5-57e7-4c5b-ab7b-041c503e451f" containerName="main"
Sep 18 19:46:44.756453 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:46:44.756370    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a34ec3d5-57e7-4c5b-ab7b-041c503e451f" containerName="main"
Sep 18 19:46:44.756453 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:46:44.756379    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a34ec3d5-57e7-4c5b-ab7b-041c503e451f" containerName="storage-initializer"
Sep 18 19:46:44.756453 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:46:44.756385    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a34ec3d5-57e7-4c5b-ab7b-041c503e451f" containerName="storage-initializer"
Sep 18 19:46:44.756453 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:46:44.756439    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a34ec3d5-57e7-4c5b-ab7b-041c503e451f" containerName="main"
Sep 18 19:46:44.758484 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:46:44.758467    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-sj97q"
Sep 18 19:46:44.761731 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:46:44.761704    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 18 19:46:44.761863 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:46:44.761704    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 18 19:46:44.762624 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:46:44.762602    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 18 19:46:44.762741 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:46:44.762613    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 19:46:44.762741 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:46:44.762735    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-fbm2d\""
Sep 18 19:46:44.762843 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:46:44.762779    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 18 19:46:44.773000 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:46:44.772965    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-sj97q"]
Sep 18 19:46:44.819592 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:46:44.819553    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5802c8c-73b3-4062-8e3d-037325e1c354-model-cache\") pod \"tokenizer-template-args-test-kserve-6475d67bc4-sj97q\" (UID: \"f5802c8c-73b3-4062-8e3d-037325e1c354\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-sj97q"
Sep 18 19:46:44.819592 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:46:44.819594    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5802c8c-73b3-4062-8e3d-037325e1c354-tls-certs\") pod \"tokenizer-template-args-test-kserve-6475d67bc4-sj97q\" (UID: \"f5802c8c-73b3-4062-8e3d-037325e1c354\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-sj97q"
Sep 18 19:46:44.819809 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:46:44.819618    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5802c8c-73b3-4062-8e3d-037325e1c354-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-6475d67bc4-sj97q\" (UID: \"f5802c8c-73b3-4062-8e3d-037325e1c354\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-sj97q"
Sep 18 19:46:44.819809 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:46:44.819649    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5802c8c-73b3-4062-8e3d-037325e1c354-dshm\") pod \"tokenizer-template-args-test-kserve-6475d67bc4-sj97q\" (UID: \"f5802c8c-73b3-4062-8e3d-037325e1c354\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-sj97q"
Sep 18 19:46:44.819809 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:46:44.819682    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5802c8c-73b3-4062-8e3d-037325e1c354-home\") pod \"tokenizer-template-args-test-kserve-6475d67bc4-sj97q\" (UID: \"f5802c8c-73b3-4062-8e3d-037325e1c354\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-sj97q"
Sep 18 19:46:44.819809 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:46:44.819760    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jp5g\" (UniqueName: \"kubernetes.io/projected/f5802c8c-73b3-4062-8e3d-037325e1c354-kube-api-access-5jp5g\") pod \"tokenizer-template-args-test-kserve-6475d67bc4-sj97q\" (UID: \"f5802c8c-73b3-4062-8e3d-037325e1c354\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-sj97q"
Sep 18 19:46:44.819809 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:46:44.819799    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5802c8c-73b3-4062-8e3d-037325e1c354-tmp-dir\") pod \"tokenizer-template-args-test-kserve-6475d67bc4-sj97q\" (UID: \"f5802c8c-73b3-4062-8e3d-037325e1c354\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-sj97q"
Sep 18 19:46:44.820024 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:46:44.819869    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5802c8c-73b3-4062-8e3d-037325e1c354-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-6475d67bc4-sj97q\" (UID: \"f5802c8c-73b3-4062-8e3d-037325e1c354\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-sj97q"
Sep 18 19:46:44.921228 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:46:44.921195    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jp5g\" (UniqueName: \"kubernetes.io/projected/f5802c8c-73b3-4062-8e3d-037325e1c354-kube-api-access-5jp5g\") pod \"tokenizer-template-args-test-kserve-6475d67bc4-sj97q\" (UID: \"f5802c8c-73b3-4062-8e3d-037325e1c354\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-sj97q"
Sep 18 19:46:44.921422 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:46:44.921237    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5802c8c-73b3-4062-8e3d-037325e1c354-tmp-dir\") pod \"tokenizer-template-args-test-kserve-6475d67bc4-sj97q\" (UID: \"f5802c8c-73b3-4062-8e3d-037325e1c354\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-sj97q"
Sep 18 19:46:44.921422 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:46:44.921372    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5802c8c-73b3-4062-8e3d-037325e1c354-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-6475d67bc4-sj97q\" (UID: \"f5802c8c-73b3-4062-8e3d-037325e1c354\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-sj97q"
Sep 18 19:46:44.921540 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:46:44.921424    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5802c8c-73b3-4062-8e3d-037325e1c354-model-cache\") pod \"tokenizer-template-args-test-kserve-6475d67bc4-sj97q\" (UID: \"f5802c8c-73b3-4062-8e3d-037325e1c354\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-sj97q"
Sep 18 19:46:44.921600 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:46:44.921525    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5802c8c-73b3-4062-8e3d-037325e1c354-tls-certs\") pod \"tokenizer-template-args-test-kserve-6475d67bc4-sj97q\" (UID: \"f5802c8c-73b3-4062-8e3d-037325e1c354\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-sj97q"
Sep 18 19:46:44.921600 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:46:44.921571    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5802c8c-73b3-4062-8e3d-037325e1c354-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-6475d67bc4-sj97q\" (UID: \"f5802c8c-73b3-4062-8e3d-037325e1c354\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-sj97q"
Sep 18 19:46:44.921709 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:46:44.921612    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5802c8c-73b3-4062-8e3d-037325e1c354-dshm\") pod \"tokenizer-template-args-test-kserve-6475d67bc4-sj97q\" (UID: \"f5802c8c-73b3-4062-8e3d-037325e1c354\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-sj97q"
Sep 18 19:46:44.921709 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:46:44.921671    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5802c8c-73b3-4062-8e3d-037325e1c354-home\") pod \"tokenizer-template-args-test-kserve-6475d67bc4-sj97q\" (UID: \"f5802c8c-73b3-4062-8e3d-037325e1c354\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-sj97q"
Sep 18 19:46:44.921813 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:46:44.921748    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5802c8c-73b3-4062-8e3d-037325e1c354-tmp-dir\") pod \"tokenizer-template-args-test-kserve-6475d67bc4-sj97q\" (UID: \"f5802c8c-73b3-4062-8e3d-037325e1c354\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-sj97q"
Sep 18 19:46:44.921864 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:46:44.921809    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5802c8c-73b3-4062-8e3d-037325e1c354-model-cache\") pod \"tokenizer-template-args-test-kserve-6475d67bc4-sj97q\" (UID: \"f5802c8c-73b3-4062-8e3d-037325e1c354\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-sj97q"
Sep 18 19:46:44.921984 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:46:44.921958    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5802c8c-73b3-4062-8e3d-037325e1c354-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-6475d67bc4-sj97q\" (UID: \"f5802c8c-73b3-4062-8e3d-037325e1c354\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-sj97q"
Sep 18 19:46:44.922167 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:46:44.922140    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5802c8c-73b3-4062-8e3d-037325e1c354-home\") pod \"tokenizer-template-args-test-kserve-6475d67bc4-sj97q\" (UID: \"f5802c8c-73b3-4062-8e3d-037325e1c354\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-sj97q"
Sep 18 19:46:44.922319 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:46:44.922301    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5802c8c-73b3-4062-8e3d-037325e1c354-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-6475d67bc4-sj97q\" (UID: \"f5802c8c-73b3-4062-8e3d-037325e1c354\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-sj97q"
Sep 18 19:46:44.923908 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:46:44.923885    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5802c8c-73b3-4062-8e3d-037325e1c354-dshm\") pod \"tokenizer-template-args-test-kserve-6475d67bc4-sj97q\" (UID: \"f5802c8c-73b3-4062-8e3d-037325e1c354\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-sj97q"
Sep 18 19:46:44.924279 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:46:44.924261    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5802c8c-73b3-4062-8e3d-037325e1c354-tls-certs\") pod \"tokenizer-template-args-test-kserve-6475d67bc4-sj97q\" (UID: \"f5802c8c-73b3-4062-8e3d-037325e1c354\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-sj97q"
Sep 18 19:46:44.930898 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:46:44.930864    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jp5g\" (UniqueName: \"kubernetes.io/projected/f5802c8c-73b3-4062-8e3d-037325e1c354-kube-api-access-5jp5g\") pod \"tokenizer-template-args-test-kserve-6475d67bc4-sj97q\" (UID: \"f5802c8c-73b3-4062-8e3d-037325e1c354\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-sj97q"
Sep 18 19:46:45.070945 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:46:45.070845    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-sj97q"
Sep 18 19:46:45.357461 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:46:45.357428    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-sj97q"]
Sep 18 19:46:45.360205 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:46:45.360169    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5802c8c_73b3_4062_8e3d_037325e1c354.slice/crio-56820fe9fe3f6f08def87ad7d3546864701f26a8a0ac287f38b204b03749cef8 WatchSource:0}: Error finding container 56820fe9fe3f6f08def87ad7d3546864701f26a8a0ac287f38b204b03749cef8: Status 404 returned error can't find the container with id 56820fe9fe3f6f08def87ad7d3546864701f26a8a0ac287f38b204b03749cef8
Sep 18 19:46:45.837921 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:46:45.837883    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-sj97q" event={"ID":"f5802c8c-73b3-4062-8e3d-037325e1c354","Type":"ContainerStarted","Data":"81359a49ac699831c0b8f811be0703a233bcf53673115f46e0dd70aec07aaaea"}
Sep 18 19:46:45.838427 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:46:45.837944    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-sj97q" event={"ID":"f5802c8c-73b3-4062-8e3d-037325e1c354","Type":"ContainerStarted","Data":"56820fe9fe3f6f08def87ad7d3546864701f26a8a0ac287f38b204b03749cef8"}
Sep 18 19:46:46.120912 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:46:46.120869    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d5c7c8987-mxkw7" podUID="569bc59d-5833-4190-a316-242303c52bd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Sep 18 19:46:47.851204 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:46:47.851166    2577 generic.go:356] "Generic (PLEG): container finished" podID="f5802c8c-73b3-4062-8e3d-037325e1c354" containerID="81359a49ac699831c0b8f811be0703a233bcf53673115f46e0dd70aec07aaaea" exitCode=0
Sep 18 19:46:47.851621 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:46:47.851247    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-sj97q" event={"ID":"f5802c8c-73b3-4062-8e3d-037325e1c354","Type":"ContainerDied","Data":"81359a49ac699831c0b8f811be0703a233bcf53673115f46e0dd70aec07aaaea"}
Sep 18 19:46:48.858176 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:46:48.858141    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-sj97q" event={"ID":"f5802c8c-73b3-4062-8e3d-037325e1c354","Type":"ContainerStarted","Data":"c24dd72b453cb2e8957b2d49c2ee8a652707499d2a8527d9b5c05f99fd17c1df"}
Sep 18 19:46:49.071420 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:46:49.071385    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-sj97q"
Sep 18 19:46:55.071602 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:46:55.071558    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-sj97q"
Sep 18 19:46:55.084740 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:46:55.084717    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-sj97q"
Sep 18 19:46:55.111820 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:46:55.111762    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-sj97q" podStartSLOduration=8.876498256 podStartE2EDuration="11.111747514s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.235249258s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:46:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:46:48.886969928 +0000 UTC m=+1979.739270299" watchObservedRunningTime="2026-09-18 19:46:55.111747514 +0000 UTC m=+1985.964047883"
Sep 18 19:46:55.906201 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:46:55.906170    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-sj97q"
Sep 18 19:46:56.120346 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:46:56.120303    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d5c7c8987-mxkw7" podUID="569bc59d-5833-4190-a316-242303c52bd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Sep 18 19:47:06.120248 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:47:06.120201    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d5c7c8987-mxkw7" podUID="569bc59d-5833-4190-a316-242303c52bd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Sep 18 19:47:16.121109 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:47:16.121065    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d5c7c8987-mxkw7" podUID="569bc59d-5833-4190-a316-242303c52bd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Sep 18 19:47:22.858935 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:47:22.858895    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-sj97q"]
Sep 18 19:47:22.868287 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:47:22.868251    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 19:47:22.868287 ip-10-0-129-30 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:47:22.868287 ip-10-0-129-30 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 18 19:47:22.868287 ip-10-0-129-30 kubenswrapper[2577]:  > containerID="c24dd72b453cb2e8957b2d49c2ee8a652707499d2a8527d9b5c05f99fd17c1df" cmd=["/bin/sleep","15"]
Sep 18 19:47:22.868645 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:47:22.868309    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-sj97q" message="" err=<
Sep 18 19:47:22.868645 ip-10-0-129-30 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:47:22.868645 ip-10-0-129-30 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 18 19:47:22.868645 ip-10-0-129-30 kubenswrapper[2577]:  >
Sep 18 19:47:22.868645 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:47:22.868322    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 19:47:22.868645 ip-10-0-129-30 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:47:22.868645 ip-10-0-129-30 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 18 19:47:22.868645 ip-10-0-129-30 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-sj97q" podUID="f5802c8c-73b3-4062-8e3d-037325e1c354" containerName="main" containerID="cri-o://c24dd72b453cb2e8957b2d49c2ee8a652707499d2a8527d9b5c05f99fd17c1df"
Sep 18 19:47:22.868645 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:47:22.868360    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-sj97q" podUID="f5802c8c-73b3-4062-8e3d-037325e1c354" containerName="main" containerID="cri-o://c24dd72b453cb2e8957b2d49c2ee8a652707499d2a8527d9b5c05f99fd17c1df" gracePeriod=60
Sep 18 19:47:22.896286 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:47:22.896249    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-sj97q" podUID="f5802c8c-73b3-4062-8e3d-037325e1c354" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Sep 18 19:47:23.218777 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:47:23.218750    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-sj97q"
Sep 18 19:47:23.273422 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:47:23.273386    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5802c8c-73b3-4062-8e3d-037325e1c354-dshm\") pod \"f5802c8c-73b3-4062-8e3d-037325e1c354\" (UID: \"f5802c8c-73b3-4062-8e3d-037325e1c354\") "
Sep 18 19:47:23.273422 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:47:23.273427    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5802c8c-73b3-4062-8e3d-037325e1c354-cabundle-cert\") pod \"f5802c8c-73b3-4062-8e3d-037325e1c354\" (UID: \"f5802c8c-73b3-4062-8e3d-037325e1c354\") "
Sep 18 19:47:23.273661 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:47:23.273470    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5802c8c-73b3-4062-8e3d-037325e1c354-tmp-dir\") pod \"f5802c8c-73b3-4062-8e3d-037325e1c354\" (UID: \"f5802c8c-73b3-4062-8e3d-037325e1c354\") "
Sep 18 19:47:23.273661 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:47:23.273509    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jp5g\" (UniqueName: \"kubernetes.io/projected/f5802c8c-73b3-4062-8e3d-037325e1c354-kube-api-access-5jp5g\") pod \"f5802c8c-73b3-4062-8e3d-037325e1c354\" (UID: \"f5802c8c-73b3-4062-8e3d-037325e1c354\") "
Sep 18 19:47:23.273661 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:47:23.273575    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5802c8c-73b3-4062-8e3d-037325e1c354-home\") pod \"f5802c8c-73b3-4062-8e3d-037325e1c354\" (UID: \"f5802c8c-73b3-4062-8e3d-037325e1c354\") "
Sep 18 19:47:23.273661 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:47:23.273601    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5802c8c-73b3-4062-8e3d-037325e1c354-model-cache\") pod \"f5802c8c-73b3-4062-8e3d-037325e1c354\" (UID: \"f5802c8c-73b3-4062-8e3d-037325e1c354\") "
Sep 18 19:47:23.273661 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:47:23.273626    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5802c8c-73b3-4062-8e3d-037325e1c354-tls-certs\") pod \"f5802c8c-73b3-4062-8e3d-037325e1c354\" (UID: \"f5802c8c-73b3-4062-8e3d-037325e1c354\") "
Sep 18 19:47:23.273928 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:47:23.273679    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5802c8c-73b3-4062-8e3d-037325e1c354-kserve-provision-location\") pod \"f5802c8c-73b3-4062-8e3d-037325e1c354\" (UID: \"f5802c8c-73b3-4062-8e3d-037325e1c354\") "
Sep 18 19:47:23.273928 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:47:23.273785    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5802c8c-73b3-4062-8e3d-037325e1c354-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f5802c8c-73b3-4062-8e3d-037325e1c354" (UID: "f5802c8c-73b3-4062-8e3d-037325e1c354"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:47:23.273928 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:47:23.273801    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5802c8c-73b3-4062-8e3d-037325e1c354-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f5802c8c-73b3-4062-8e3d-037325e1c354" (UID: "f5802c8c-73b3-4062-8e3d-037325e1c354"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:47:23.274109 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:47:23.274012    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5802c8c-73b3-4062-8e3d-037325e1c354-cabundle-cert\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:47:23.274109 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:47:23.274033    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5802c8c-73b3-4062-8e3d-037325e1c354-tmp-dir\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:47:23.274283 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:47:23.274262    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5802c8c-73b3-4062-8e3d-037325e1c354-home" (OuterVolumeSpecName: "home") pod "f5802c8c-73b3-4062-8e3d-037325e1c354" (UID: "f5802c8c-73b3-4062-8e3d-037325e1c354"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:47:23.274545 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:47:23.274518    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5802c8c-73b3-4062-8e3d-037325e1c354-model-cache" (OuterVolumeSpecName: "model-cache") pod "f5802c8c-73b3-4062-8e3d-037325e1c354" (UID: "f5802c8c-73b3-4062-8e3d-037325e1c354"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:47:23.276213 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:47:23.276190    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5802c8c-73b3-4062-8e3d-037325e1c354-dshm" (OuterVolumeSpecName: "dshm") pod "f5802c8c-73b3-4062-8e3d-037325e1c354" (UID: "f5802c8c-73b3-4062-8e3d-037325e1c354"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:47:23.276334 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:47:23.276258    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5802c8c-73b3-4062-8e3d-037325e1c354-kube-api-access-5jp5g" (OuterVolumeSpecName: "kube-api-access-5jp5g") pod "f5802c8c-73b3-4062-8e3d-037325e1c354" (UID: "f5802c8c-73b3-4062-8e3d-037325e1c354"). InnerVolumeSpecName "kube-api-access-5jp5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:47:23.276547 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:47:23.276521    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5802c8c-73b3-4062-8e3d-037325e1c354-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f5802c8c-73b3-4062-8e3d-037325e1c354" (UID: "f5802c8c-73b3-4062-8e3d-037325e1c354"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:47:23.293928 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:47:23.293890    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5802c8c-73b3-4062-8e3d-037325e1c354-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f5802c8c-73b3-4062-8e3d-037325e1c354" (UID: "f5802c8c-73b3-4062-8e3d-037325e1c354"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:47:23.375424 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:47:23.375388    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jp5g\" (UniqueName: \"kubernetes.io/projected/f5802c8c-73b3-4062-8e3d-037325e1c354-kube-api-access-5jp5g\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:47:23.375424 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:47:23.375421    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5802c8c-73b3-4062-8e3d-037325e1c354-home\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:47:23.375638 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:47:23.375435    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5802c8c-73b3-4062-8e3d-037325e1c354-model-cache\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:47:23.375638 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:47:23.375447    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5802c8c-73b3-4062-8e3d-037325e1c354-tls-certs\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:47:23.375638 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:47:23.375458    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5802c8c-73b3-4062-8e3d-037325e1c354-kserve-provision-location\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:47:23.375638 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:47:23.375468    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5802c8c-73b3-4062-8e3d-037325e1c354-dshm\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:47:24.024157 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:47:24.024114    2577 generic.go:356] "Generic (PLEG): container finished" podID="f5802c8c-73b3-4062-8e3d-037325e1c354" containerID="c24dd72b453cb2e8957b2d49c2ee8a652707499d2a8527d9b5c05f99fd17c1df" exitCode=0
Sep 18 19:47:24.024666 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:47:24.024171    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-sj97q"
Sep 18 19:47:24.024666 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:47:24.024190    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-sj97q" event={"ID":"f5802c8c-73b3-4062-8e3d-037325e1c354","Type":"ContainerDied","Data":"c24dd72b453cb2e8957b2d49c2ee8a652707499d2a8527d9b5c05f99fd17c1df"}
Sep 18 19:47:24.024666 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:47:24.024245    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-sj97q" event={"ID":"f5802c8c-73b3-4062-8e3d-037325e1c354","Type":"ContainerDied","Data":"56820fe9fe3f6f08def87ad7d3546864701f26a8a0ac287f38b204b03749cef8"}
Sep 18 19:47:24.024666 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:47:24.024271    2577 scope.go:117] "RemoveContainer" containerID="c24dd72b453cb2e8957b2d49c2ee8a652707499d2a8527d9b5c05f99fd17c1df"
Sep 18 19:47:24.033355 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:47:24.033321    2577 scope.go:117] "RemoveContainer" containerID="81359a49ac699831c0b8f811be0703a233bcf53673115f46e0dd70aec07aaaea"
Sep 18 19:47:24.041243 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:47:24.041224    2577 scope.go:117] "RemoveContainer" containerID="c24dd72b453cb2e8957b2d49c2ee8a652707499d2a8527d9b5c05f99fd17c1df"
Sep 18 19:47:24.041509 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:47:24.041487    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c24dd72b453cb2e8957b2d49c2ee8a652707499d2a8527d9b5c05f99fd17c1df\": container with ID starting with c24dd72b453cb2e8957b2d49c2ee8a652707499d2a8527d9b5c05f99fd17c1df not found: ID does not exist" containerID="c24dd72b453cb2e8957b2d49c2ee8a652707499d2a8527d9b5c05f99fd17c1df"
Sep 18 19:47:24.041582 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:47:24.041517    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c24dd72b453cb2e8957b2d49c2ee8a652707499d2a8527d9b5c05f99fd17c1df"} err="failed to get container status \"c24dd72b453cb2e8957b2d49c2ee8a652707499d2a8527d9b5c05f99fd17c1df\": rpc error: code = NotFound desc = could not find container \"c24dd72b453cb2e8957b2d49c2ee8a652707499d2a8527d9b5c05f99fd17c1df\": container with ID starting with c24dd72b453cb2e8957b2d49c2ee8a652707499d2a8527d9b5c05f99fd17c1df not found: ID does not exist"
Sep 18 19:47:24.041582 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:47:24.041532    2577 scope.go:117] "RemoveContainer" containerID="81359a49ac699831c0b8f811be0703a233bcf53673115f46e0dd70aec07aaaea"
Sep 18 19:47:24.041777 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:47:24.041758    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81359a49ac699831c0b8f811be0703a233bcf53673115f46e0dd70aec07aaaea\": container with ID starting with 81359a49ac699831c0b8f811be0703a233bcf53673115f46e0dd70aec07aaaea not found: ID does not exist" containerID="81359a49ac699831c0b8f811be0703a233bcf53673115f46e0dd70aec07aaaea"
Sep 18 19:47:24.041827 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:47:24.041785    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81359a49ac699831c0b8f811be0703a233bcf53673115f46e0dd70aec07aaaea"} err="failed to get container status \"81359a49ac699831c0b8f811be0703a233bcf53673115f46e0dd70aec07aaaea\": rpc error: code = NotFound desc = could not find container \"81359a49ac699831c0b8f811be0703a233bcf53673115f46e0dd70aec07aaaea\": container with ID starting with 81359a49ac699831c0b8f811be0703a233bcf53673115f46e0dd70aec07aaaea not found: ID does not exist"
Sep 18 19:47:24.045100 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:47:24.045079    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-sj97q"]
Sep 18 19:47:24.051961 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:47:24.051939    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-sj97q"]
Sep 18 19:47:25.438948 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:47:25.438915    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5802c8c-73b3-4062-8e3d-037325e1c354" path="/var/lib/kubelet/pods/f5802c8c-73b3-4062-8e3d-037325e1c354/volumes"
Sep 18 19:47:26.120322 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:47:26.120275    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d5c7c8987-mxkw7" podUID="569bc59d-5833-4190-a316-242303c52bd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Sep 18 19:47:36.120420 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:47:36.120374    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d5c7c8987-mxkw7" podUID="569bc59d-5833-4190-a316-242303c52bd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Sep 18 19:47:46.132080 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:47:46.132046    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d5c7c8987-mxkw7"
Sep 18 19:47:46.140554 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:47:46.140523    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d5c7c8987-mxkw7"
Sep 18 19:47:49.766085 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:47:49.766053    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d5c7c8987-mxkw7"]
Sep 18 19:48:04.788316 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:04.788202    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d5c7c8987-mxkw7" podUID="569bc59d-5833-4190-a316-242303c52bd5" containerName="main" containerID="cri-o://b671a5c88c9d9f041c52c74c2ddda848b54ab181070ffd9b1003d8c409c94a16" gracePeriod=45
Sep 18 19:48:05.132712 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:05.132661    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d5c7c8987-mxkw7" podUID="569bc59d-5833-4190-a316-242303c52bd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Sep 18 19:48:05.539068 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:05.539045    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d5c7c8987-mxkw7"
Sep 18 19:48:05.550711 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:05.550674    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/569bc59d-5833-4190-a316-242303c52bd5-kserve-provision-location\") pod \"569bc59d-5833-4190-a316-242303c52bd5\" (UID: \"569bc59d-5833-4190-a316-242303c52bd5\") "
Sep 18 19:48:05.550851 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:05.550720    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/569bc59d-5833-4190-a316-242303c52bd5-tmp-dir\") pod \"569bc59d-5833-4190-a316-242303c52bd5\" (UID: \"569bc59d-5833-4190-a316-242303c52bd5\") "
Sep 18 19:48:05.550851 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:05.550774    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/569bc59d-5833-4190-a316-242303c52bd5-dshm\") pod \"569bc59d-5833-4190-a316-242303c52bd5\" (UID: \"569bc59d-5833-4190-a316-242303c52bd5\") "
Sep 18 19:48:05.550851 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:05.550835    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjl5x\" (UniqueName: \"kubernetes.io/projected/569bc59d-5833-4190-a316-242303c52bd5-kube-api-access-pjl5x\") pod \"569bc59d-5833-4190-a316-242303c52bd5\" (UID: \"569bc59d-5833-4190-a316-242303c52bd5\") "
Sep 18 19:48:05.551047 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:05.550858    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/569bc59d-5833-4190-a316-242303c52bd5-tls-certs\") pod \"569bc59d-5833-4190-a316-242303c52bd5\" (UID: \"569bc59d-5833-4190-a316-242303c52bd5\") "
Sep 18 19:48:05.551047 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:05.550893    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/569bc59d-5833-4190-a316-242303c52bd5-cabundle-cert\") pod \"569bc59d-5833-4190-a316-242303c52bd5\" (UID: \"569bc59d-5833-4190-a316-242303c52bd5\") "
Sep 18 19:48:05.551047 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:05.550921    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/569bc59d-5833-4190-a316-242303c52bd5-model-cache\") pod \"569bc59d-5833-4190-a316-242303c52bd5\" (UID: \"569bc59d-5833-4190-a316-242303c52bd5\") "
Sep 18 19:48:05.551047 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:05.550946    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/569bc59d-5833-4190-a316-242303c52bd5-home\") pod \"569bc59d-5833-4190-a316-242303c52bd5\" (UID: \"569bc59d-5833-4190-a316-242303c52bd5\") "
Sep 18 19:48:05.551494 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:05.551365    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/569bc59d-5833-4190-a316-242303c52bd5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "569bc59d-5833-4190-a316-242303c52bd5" (UID: "569bc59d-5833-4190-a316-242303c52bd5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:48:05.551615 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:05.551572    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/569bc59d-5833-4190-a316-242303c52bd5-model-cache" (OuterVolumeSpecName: "model-cache") pod "569bc59d-5833-4190-a316-242303c52bd5" (UID: "569bc59d-5833-4190-a316-242303c52bd5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:48:05.551877 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:05.551852    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/569bc59d-5833-4190-a316-242303c52bd5-home" (OuterVolumeSpecName: "home") pod "569bc59d-5833-4190-a316-242303c52bd5" (UID: "569bc59d-5833-4190-a316-242303c52bd5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:48:05.553287 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:05.553263    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/569bc59d-5833-4190-a316-242303c52bd5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "569bc59d-5833-4190-a316-242303c52bd5" (UID: "569bc59d-5833-4190-a316-242303c52bd5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:48:05.553532 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:05.553504    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/569bc59d-5833-4190-a316-242303c52bd5-dshm" (OuterVolumeSpecName: "dshm") pod "569bc59d-5833-4190-a316-242303c52bd5" (UID: "569bc59d-5833-4190-a316-242303c52bd5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:48:05.553532 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:05.553517    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/569bc59d-5833-4190-a316-242303c52bd5-kube-api-access-pjl5x" (OuterVolumeSpecName: "kube-api-access-pjl5x") pod "569bc59d-5833-4190-a316-242303c52bd5" (UID: "569bc59d-5833-4190-a316-242303c52bd5"). InnerVolumeSpecName "kube-api-access-pjl5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:48:05.569549 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:05.569508    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/569bc59d-5833-4190-a316-242303c52bd5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "569bc59d-5833-4190-a316-242303c52bd5" (UID: "569bc59d-5833-4190-a316-242303c52bd5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:48:05.582660 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:05.582621    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/569bc59d-5833-4190-a316-242303c52bd5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "569bc59d-5833-4190-a316-242303c52bd5" (UID: "569bc59d-5833-4190-a316-242303c52bd5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:48:05.651904 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:05.651867    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjl5x\" (UniqueName: \"kubernetes.io/projected/569bc59d-5833-4190-a316-242303c52bd5-kube-api-access-pjl5x\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:48:05.651904 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:05.651900    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/569bc59d-5833-4190-a316-242303c52bd5-tls-certs\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:48:05.651904 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:05.651911    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/569bc59d-5833-4190-a316-242303c52bd5-cabundle-cert\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:48:05.652153 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:05.651919    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/569bc59d-5833-4190-a316-242303c52bd5-model-cache\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:48:05.652153 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:05.651927    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/569bc59d-5833-4190-a316-242303c52bd5-home\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:48:05.652153 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:05.651935    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/569bc59d-5833-4190-a316-242303c52bd5-kserve-provision-location\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:48:05.652153 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:05.651944    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/569bc59d-5833-4190-a316-242303c52bd5-tmp-dir\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:48:05.652153 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:05.651951    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/569bc59d-5833-4190-a316-242303c52bd5-dshm\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:48:06.215397 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:06.215359    2577 generic.go:356] "Generic (PLEG): container finished" podID="569bc59d-5833-4190-a316-242303c52bd5" containerID="b671a5c88c9d9f041c52c74c2ddda848b54ab181070ffd9b1003d8c409c94a16" exitCode=0
Sep 18 19:48:06.215858 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:06.215420    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d5c7c8987-mxkw7"
Sep 18 19:48:06.215858 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:06.215442    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d5c7c8987-mxkw7" event={"ID":"569bc59d-5833-4190-a316-242303c52bd5","Type":"ContainerDied","Data":"b671a5c88c9d9f041c52c74c2ddda848b54ab181070ffd9b1003d8c409c94a16"}
Sep 18 19:48:06.215858 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:06.215520    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d5c7c8987-mxkw7" event={"ID":"569bc59d-5833-4190-a316-242303c52bd5","Type":"ContainerDied","Data":"5a5d6608519053e7e241cb756b613cb0c13439fd95c3eec826862b42140a4097"}
Sep 18 19:48:06.215858 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:06.215542    2577 scope.go:117] "RemoveContainer" containerID="b671a5c88c9d9f041c52c74c2ddda848b54ab181070ffd9b1003d8c409c94a16"
Sep 18 19:48:06.224313 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:06.224295    2577 scope.go:117] "RemoveContainer" containerID="031e0e1561b928730630f8debf9b5e3d016b5e668bdd5a77e487155bec92f8a7"
Sep 18 19:48:06.231716 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:06.231701    2577 scope.go:117] "RemoveContainer" containerID="b671a5c88c9d9f041c52c74c2ddda848b54ab181070ffd9b1003d8c409c94a16"
Sep 18 19:48:06.231962 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:48:06.231944    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b671a5c88c9d9f041c52c74c2ddda848b54ab181070ffd9b1003d8c409c94a16\": container with ID starting with b671a5c88c9d9f041c52c74c2ddda848b54ab181070ffd9b1003d8c409c94a16 not found: ID does not exist" containerID="b671a5c88c9d9f041c52c74c2ddda848b54ab181070ffd9b1003d8c409c94a16"
Sep 18 19:48:06.232033 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:06.231969    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b671a5c88c9d9f041c52c74c2ddda848b54ab181070ffd9b1003d8c409c94a16"} err="failed to get container status \"b671a5c88c9d9f041c52c74c2ddda848b54ab181070ffd9b1003d8c409c94a16\": rpc error: code = NotFound desc = could not find container \"b671a5c88c9d9f041c52c74c2ddda848b54ab181070ffd9b1003d8c409c94a16\": container with ID starting with b671a5c88c9d9f041c52c74c2ddda848b54ab181070ffd9b1003d8c409c94a16 not found: ID does not exist"
Sep 18 19:48:06.232033 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:06.231984    2577 scope.go:117] "RemoveContainer" containerID="031e0e1561b928730630f8debf9b5e3d016b5e668bdd5a77e487155bec92f8a7"
Sep 18 19:48:06.232230 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:48:06.232210    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"031e0e1561b928730630f8debf9b5e3d016b5e668bdd5a77e487155bec92f8a7\": container with ID starting with 031e0e1561b928730630f8debf9b5e3d016b5e668bdd5a77e487155bec92f8a7 not found: ID does not exist" containerID="031e0e1561b928730630f8debf9b5e3d016b5e668bdd5a77e487155bec92f8a7"
Sep 18 19:48:06.232290 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:06.232240    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"031e0e1561b928730630f8debf9b5e3d016b5e668bdd5a77e487155bec92f8a7"} err="failed to get container status \"031e0e1561b928730630f8debf9b5e3d016b5e668bdd5a77e487155bec92f8a7\": rpc error: code = NotFound desc = could not find container \"031e0e1561b928730630f8debf9b5e3d016b5e668bdd5a77e487155bec92f8a7\": container with ID starting with 031e0e1561b928730630f8debf9b5e3d016b5e668bdd5a77e487155bec92f8a7 not found: ID does not exist"
Sep 18 19:48:06.240815 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:06.240792    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d5c7c8987-mxkw7"]
Sep 18 19:48:06.244642 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:06.244622    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5d5c7c8987-mxkw7"]
Sep 18 19:48:07.438295 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:07.438262    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="569bc59d-5833-4190-a316-242303c52bd5" path="/var/lib/kubelet/pods/569bc59d-5833-4190-a316-242303c52bd5/volumes"
Sep 18 19:48:47.391267 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:47.391228    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-85487667b4-g8cg4"]
Sep 18 19:48:47.391893 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:47.391720    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5802c8c-73b3-4062-8e3d-037325e1c354" containerName="storage-initializer"
Sep 18 19:48:47.391893 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:47.391739    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5802c8c-73b3-4062-8e3d-037325e1c354" containerName="storage-initializer"
Sep 18 19:48:47.391893 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:47.391761    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5802c8c-73b3-4062-8e3d-037325e1c354" containerName="main"
Sep 18 19:48:47.391893 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:47.391770    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5802c8c-73b3-4062-8e3d-037325e1c354" containerName="main"
Sep 18 19:48:47.391893 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:47.391787    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="569bc59d-5833-4190-a316-242303c52bd5" containerName="storage-initializer"
Sep 18 19:48:47.391893 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:47.391797    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="569bc59d-5833-4190-a316-242303c52bd5" containerName="storage-initializer"
Sep 18 19:48:47.391893 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:47.391808    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="569bc59d-5833-4190-a316-242303c52bd5" containerName="main"
Sep 18 19:48:47.391893 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:47.391818    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="569bc59d-5833-4190-a316-242303c52bd5" containerName="main"
Sep 18 19:48:47.392339 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:47.391900    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="569bc59d-5833-4190-a316-242303c52bd5" containerName="main"
Sep 18 19:48:47.392339 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:47.391916    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5802c8c-73b3-4062-8e3d-037325e1c354" containerName="main"
Sep 18 19:48:47.394884 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:47.394864    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-85487667b4-g8cg4"
Sep 18 19:48:47.399536 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:47.399510    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Sep 18 19:48:47.399727 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:47.399713    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"seaweedfs-s3-creds\""
Sep 18 19:48:47.399795 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:47.399751    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 19:48:47.399852 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:47.399821    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-lh7td\""
Sep 18 19:48:47.400232 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:47.400213    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Sep 18 19:48:47.400741 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:47.400721    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Sep 18 19:48:47.417857 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:47.417834    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-85487667b4-g8cg4"]
Sep 18 19:48:47.508675 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:47.508644    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99ec07f1-50eb-499f-a7c8-740bc7c195fc-tmp-dir\") pod \"auth-disabled-test-kserve-85487667b4-g8cg4\" (UID: \"99ec07f1-50eb-499f-a7c8-740bc7c195fc\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-85487667b4-g8cg4"
Sep 18 19:48:47.508856 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:47.508684    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99ec07f1-50eb-499f-a7c8-740bc7c195fc-model-cache\") pod \"auth-disabled-test-kserve-85487667b4-g8cg4\" (UID: \"99ec07f1-50eb-499f-a7c8-740bc7c195fc\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-85487667b4-g8cg4"
Sep 18 19:48:47.508856 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:47.508712    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99ec07f1-50eb-499f-a7c8-740bc7c195fc-kserve-provision-location\") pod \"auth-disabled-test-kserve-85487667b4-g8cg4\" (UID: \"99ec07f1-50eb-499f-a7c8-740bc7c195fc\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-85487667b4-g8cg4"
Sep 18 19:48:47.508856 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:47.508786    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99ec07f1-50eb-499f-a7c8-740bc7c195fc-cabundle-cert\") pod \"auth-disabled-test-kserve-85487667b4-g8cg4\" (UID: \"99ec07f1-50eb-499f-a7c8-740bc7c195fc\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-85487667b4-g8cg4"
Sep 18 19:48:47.508984 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:47.508879    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99ec07f1-50eb-499f-a7c8-740bc7c195fc-home\") pod \"auth-disabled-test-kserve-85487667b4-g8cg4\" (UID: \"99ec07f1-50eb-499f-a7c8-740bc7c195fc\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-85487667b4-g8cg4"
Sep 18 19:48:47.508984 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:47.508910    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99ec07f1-50eb-499f-a7c8-740bc7c195fc-tls-certs\") pod \"auth-disabled-test-kserve-85487667b4-g8cg4\" (UID: \"99ec07f1-50eb-499f-a7c8-740bc7c195fc\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-85487667b4-g8cg4"
Sep 18 19:48:47.508984 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:47.508931    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqrx5\" (UniqueName: \"kubernetes.io/projected/99ec07f1-50eb-499f-a7c8-740bc7c195fc-kube-api-access-wqrx5\") pod \"auth-disabled-test-kserve-85487667b4-g8cg4\" (UID: \"99ec07f1-50eb-499f-a7c8-740bc7c195fc\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-85487667b4-g8cg4"
Sep 18 19:48:47.509113 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:47.509022    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99ec07f1-50eb-499f-a7c8-740bc7c195fc-dshm\") pod \"auth-disabled-test-kserve-85487667b4-g8cg4\" (UID: \"99ec07f1-50eb-499f-a7c8-740bc7c195fc\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-85487667b4-g8cg4"
Sep 18 19:48:47.609692 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:47.609660    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99ec07f1-50eb-499f-a7c8-740bc7c195fc-home\") pod \"auth-disabled-test-kserve-85487667b4-g8cg4\" (UID: \"99ec07f1-50eb-499f-a7c8-740bc7c195fc\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-85487667b4-g8cg4"
Sep 18 19:48:47.609692 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:47.609695    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99ec07f1-50eb-499f-a7c8-740bc7c195fc-tls-certs\") pod \"auth-disabled-test-kserve-85487667b4-g8cg4\" (UID: \"99ec07f1-50eb-499f-a7c8-740bc7c195fc\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-85487667b4-g8cg4"
Sep 18 19:48:47.609967 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:47.609712    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqrx5\" (UniqueName: \"kubernetes.io/projected/99ec07f1-50eb-499f-a7c8-740bc7c195fc-kube-api-access-wqrx5\") pod \"auth-disabled-test-kserve-85487667b4-g8cg4\" (UID: \"99ec07f1-50eb-499f-a7c8-740bc7c195fc\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-85487667b4-g8cg4"
Sep 18 19:48:47.609967 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:47.609842    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99ec07f1-50eb-499f-a7c8-740bc7c195fc-dshm\") pod \"auth-disabled-test-kserve-85487667b4-g8cg4\" (UID: \"99ec07f1-50eb-499f-a7c8-740bc7c195fc\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-85487667b4-g8cg4"
Sep 18 19:48:47.609967 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:47.609898    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99ec07f1-50eb-499f-a7c8-740bc7c195fc-tmp-dir\") pod \"auth-disabled-test-kserve-85487667b4-g8cg4\" (UID: \"99ec07f1-50eb-499f-a7c8-740bc7c195fc\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-85487667b4-g8cg4"
Sep 18 19:48:47.609967 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:47.609930    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99ec07f1-50eb-499f-a7c8-740bc7c195fc-model-cache\") pod \"auth-disabled-test-kserve-85487667b4-g8cg4\" (UID: \"99ec07f1-50eb-499f-a7c8-740bc7c195fc\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-85487667b4-g8cg4"
Sep 18 19:48:47.609967 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:47.609948    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99ec07f1-50eb-499f-a7c8-740bc7c195fc-kserve-provision-location\") pod \"auth-disabled-test-kserve-85487667b4-g8cg4\" (UID: \"99ec07f1-50eb-499f-a7c8-740bc7c195fc\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-85487667b4-g8cg4"
Sep 18 19:48:47.610279 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:47.609999    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99ec07f1-50eb-499f-a7c8-740bc7c195fc-cabundle-cert\") pod \"auth-disabled-test-kserve-85487667b4-g8cg4\" (UID: \"99ec07f1-50eb-499f-a7c8-740bc7c195fc\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-85487667b4-g8cg4"
Sep 18 19:48:47.610339 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:47.610288    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99ec07f1-50eb-499f-a7c8-740bc7c195fc-model-cache\") pod \"auth-disabled-test-kserve-85487667b4-g8cg4\" (UID: \"99ec07f1-50eb-499f-a7c8-740bc7c195fc\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-85487667b4-g8cg4"
Sep 18 19:48:47.610339 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:47.610311    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99ec07f1-50eb-499f-a7c8-740bc7c195fc-tmp-dir\") pod \"auth-disabled-test-kserve-85487667b4-g8cg4\" (UID: \"99ec07f1-50eb-499f-a7c8-740bc7c195fc\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-85487667b4-g8cg4"
Sep 18 19:48:47.610453 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:47.610358    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99ec07f1-50eb-499f-a7c8-740bc7c195fc-kserve-provision-location\") pod \"auth-disabled-test-kserve-85487667b4-g8cg4\" (UID: \"99ec07f1-50eb-499f-a7c8-740bc7c195fc\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-85487667b4-g8cg4"
Sep 18 19:48:47.610453 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:47.610417    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99ec07f1-50eb-499f-a7c8-740bc7c195fc-home\") pod \"auth-disabled-test-kserve-85487667b4-g8cg4\" (UID: \"99ec07f1-50eb-499f-a7c8-740bc7c195fc\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-85487667b4-g8cg4"
Sep 18 19:48:47.610687 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:47.610670    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99ec07f1-50eb-499f-a7c8-740bc7c195fc-cabundle-cert\") pod \"auth-disabled-test-kserve-85487667b4-g8cg4\" (UID: \"99ec07f1-50eb-499f-a7c8-740bc7c195fc\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-85487667b4-g8cg4"
Sep 18 19:48:47.612090 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:47.612067    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99ec07f1-50eb-499f-a7c8-740bc7c195fc-dshm\") pod \"auth-disabled-test-kserve-85487667b4-g8cg4\" (UID: \"99ec07f1-50eb-499f-a7c8-740bc7c195fc\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-85487667b4-g8cg4"
Sep 18 19:48:47.612597 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:47.612575    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99ec07f1-50eb-499f-a7c8-740bc7c195fc-tls-certs\") pod \"auth-disabled-test-kserve-85487667b4-g8cg4\" (UID: \"99ec07f1-50eb-499f-a7c8-740bc7c195fc\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-85487667b4-g8cg4"
Sep 18 19:48:47.621333 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:47.621307    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqrx5\" (UniqueName: \"kubernetes.io/projected/99ec07f1-50eb-499f-a7c8-740bc7c195fc-kube-api-access-wqrx5\") pod \"auth-disabled-test-kserve-85487667b4-g8cg4\" (UID: \"99ec07f1-50eb-499f-a7c8-740bc7c195fc\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-85487667b4-g8cg4"
Sep 18 19:48:47.704809 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:47.704733    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-85487667b4-g8cg4"
Sep 18 19:48:47.997352 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:47.997324    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-85487667b4-g8cg4"]
Sep 18 19:48:47.999226 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:48:47.999202    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99ec07f1_50eb_499f_a7c8_740bc7c195fc.slice/crio-0d8f33d4e0689acc85d42f32fa26c242a21fff7e884c3f68051fdd88d2b272ca WatchSource:0}: Error finding container 0d8f33d4e0689acc85d42f32fa26c242a21fff7e884c3f68051fdd88d2b272ca: Status 404 returned error can't find the container with id 0d8f33d4e0689acc85d42f32fa26c242a21fff7e884c3f68051fdd88d2b272ca
Sep 18 19:48:48.404518 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:48.404483    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-85487667b4-g8cg4" event={"ID":"99ec07f1-50eb-499f-a7c8-740bc7c195fc","Type":"ContainerStarted","Data":"e213439f09ad6370d91abe843b095cbaae88a498213b1ee8a58fbc692cbea50d"}
Sep 18 19:48:48.404956 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:48.404534    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-85487667b4-g8cg4" event={"ID":"99ec07f1-50eb-499f-a7c8-740bc7c195fc","Type":"ContainerStarted","Data":"0d8f33d4e0689acc85d42f32fa26c242a21fff7e884c3f68051fdd88d2b272ca"}
Sep 18 19:48:50.415171 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:50.415136    2577 generic.go:356] "Generic (PLEG): container finished" podID="99ec07f1-50eb-499f-a7c8-740bc7c195fc" containerID="e213439f09ad6370d91abe843b095cbaae88a498213b1ee8a58fbc692cbea50d" exitCode=0
Sep 18 19:48:50.415548 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:50.415209    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-85487667b4-g8cg4" event={"ID":"99ec07f1-50eb-499f-a7c8-740bc7c195fc","Type":"ContainerDied","Data":"e213439f09ad6370d91abe843b095cbaae88a498213b1ee8a58fbc692cbea50d"}
Sep 18 19:48:51.422769 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:51.422737    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-85487667b4-g8cg4" event={"ID":"99ec07f1-50eb-499f-a7c8-740bc7c195fc","Type":"ContainerStarted","Data":"3cbb49666206020cafdc7e7d16f29aa516ebc2495df7e6f56da530092984560d"}
Sep 18 19:48:51.458073 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:51.458011    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-85487667b4-g8cg4" podStartSLOduration=2.317848385 podStartE2EDuration="4.457971692s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.140123307s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:48:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:48:51.454530268 +0000 UTC m=+2102.306830636" watchObservedRunningTime="2026-09-18 19:48:51.457971692 +0000 UTC m=+2102.310272063"
Sep 18 19:48:51.705615 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:51.705531    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-85487667b4-g8cg4"
Sep 18 19:48:57.705877 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:57.705840    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-85487667b4-g8cg4"
Sep 18 19:48:57.707443 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:48:57.707413    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-85487667b4-g8cg4" podUID="99ec07f1-50eb-499f-a7c8-740bc7c195fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
Sep 18 19:49:07.706101 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:49:07.706048    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-85487667b4-g8cg4" podUID="99ec07f1-50eb-499f-a7c8-740bc7c195fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
Sep 18 19:49:17.705886 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:49:17.705823    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-85487667b4-g8cg4" podUID="99ec07f1-50eb-499f-a7c8-740bc7c195fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
Sep 18 19:49:27.705282 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:49:27.705240    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-85487667b4-g8cg4" podUID="99ec07f1-50eb-499f-a7c8-740bc7c195fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
Sep 18 19:49:37.705564 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:49:37.705520    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-85487667b4-g8cg4" podUID="99ec07f1-50eb-499f-a7c8-740bc7c195fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
Sep 18 19:49:47.705728 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:49:47.705680    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-85487667b4-g8cg4" podUID="99ec07f1-50eb-499f-a7c8-740bc7c195fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
Sep 18 19:49:57.705952 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:49:57.705909    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-85487667b4-g8cg4" podUID="99ec07f1-50eb-499f-a7c8-740bc7c195fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
Sep 18 19:50:07.705720 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:50:07.705668    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-85487667b4-g8cg4" podUID="99ec07f1-50eb-499f-a7c8-740bc7c195fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
Sep 18 19:50:17.706141 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:50:17.706097    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-85487667b4-g8cg4" podUID="99ec07f1-50eb-499f-a7c8-740bc7c195fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
Sep 18 19:50:27.705388 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:50:27.705347    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-85487667b4-g8cg4" podUID="99ec07f1-50eb-499f-a7c8-740bc7c195fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
Sep 18 19:50:37.705904 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:50:37.705855    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-85487667b4-g8cg4" podUID="99ec07f1-50eb-499f-a7c8-740bc7c195fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
Sep 18 19:50:47.715381 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:50:47.715341    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-85487667b4-g8cg4"
Sep 18 19:50:47.723246 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:50:47.723220    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-85487667b4-g8cg4"
Sep 18 19:51:10.331393 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:51:10.331305    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-85487667b4-g8cg4"]
Sep 18 19:51:25.353411 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:51:25.353355    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-85487667b4-g8cg4" podUID="99ec07f1-50eb-499f-a7c8-740bc7c195fc" containerName="main" containerID="cri-o://3cbb49666206020cafdc7e7d16f29aa516ebc2495df7e6f56da530092984560d" gracePeriod=45
Sep 18 19:51:25.716721 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:51:25.716675    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-85487667b4-g8cg4" podUID="99ec07f1-50eb-499f-a7c8-740bc7c195fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
Sep 18 19:51:26.031530 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:51:26.031507    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-85487667b4-g8cg4"
Sep 18 19:51:26.137882 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:51:26.137850    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99ec07f1-50eb-499f-a7c8-740bc7c195fc-tls-certs\") pod \"99ec07f1-50eb-499f-a7c8-740bc7c195fc\" (UID: \"99ec07f1-50eb-499f-a7c8-740bc7c195fc\") "
Sep 18 19:51:26.138110 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:51:26.137890    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99ec07f1-50eb-499f-a7c8-740bc7c195fc-tmp-dir\") pod \"99ec07f1-50eb-499f-a7c8-740bc7c195fc\" (UID: \"99ec07f1-50eb-499f-a7c8-740bc7c195fc\") "
Sep 18 19:51:26.138110 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:51:26.137917    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99ec07f1-50eb-499f-a7c8-740bc7c195fc-kserve-provision-location\") pod \"99ec07f1-50eb-499f-a7c8-740bc7c195fc\" (UID: \"99ec07f1-50eb-499f-a7c8-740bc7c195fc\") "
Sep 18 19:51:26.138110 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:51:26.137936    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wqrx5\" (UniqueName: \"kubernetes.io/projected/99ec07f1-50eb-499f-a7c8-740bc7c195fc-kube-api-access-wqrx5\") pod \"99ec07f1-50eb-499f-a7c8-740bc7c195fc\" (UID: \"99ec07f1-50eb-499f-a7c8-740bc7c195fc\") "
Sep 18 19:51:26.138110 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:51:26.137956    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99ec07f1-50eb-499f-a7c8-740bc7c195fc-model-cache\") pod \"99ec07f1-50eb-499f-a7c8-740bc7c195fc\" (UID: \"99ec07f1-50eb-499f-a7c8-740bc7c195fc\") "
Sep 18 19:51:26.138110 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:51:26.137974    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99ec07f1-50eb-499f-a7c8-740bc7c195fc-cabundle-cert\") pod \"99ec07f1-50eb-499f-a7c8-740bc7c195fc\" (UID: \"99ec07f1-50eb-499f-a7c8-740bc7c195fc\") "
Sep 18 19:51:26.138110 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:51:26.138023    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99ec07f1-50eb-499f-a7c8-740bc7c195fc-home\") pod \"99ec07f1-50eb-499f-a7c8-740bc7c195fc\" (UID: \"99ec07f1-50eb-499f-a7c8-740bc7c195fc\") "
Sep 18 19:51:26.138110 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:51:26.138095    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99ec07f1-50eb-499f-a7c8-740bc7c195fc-dshm\") pod \"99ec07f1-50eb-499f-a7c8-740bc7c195fc\" (UID: \"99ec07f1-50eb-499f-a7c8-740bc7c195fc\") "
Sep 18 19:51:26.138504 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:51:26.138328    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99ec07f1-50eb-499f-a7c8-740bc7c195fc-model-cache" (OuterVolumeSpecName: "model-cache") pod "99ec07f1-50eb-499f-a7c8-740bc7c195fc" (UID: "99ec07f1-50eb-499f-a7c8-740bc7c195fc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:51:26.138504 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:51:26.138421    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99ec07f1-50eb-499f-a7c8-740bc7c195fc-model-cache\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:51:26.138614 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:51:26.138539    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99ec07f1-50eb-499f-a7c8-740bc7c195fc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "99ec07f1-50eb-499f-a7c8-740bc7c195fc" (UID: "99ec07f1-50eb-499f-a7c8-740bc7c195fc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:51:26.138788 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:51:26.138765    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99ec07f1-50eb-499f-a7c8-740bc7c195fc-home" (OuterVolumeSpecName: "home") pod "99ec07f1-50eb-499f-a7c8-740bc7c195fc" (UID: "99ec07f1-50eb-499f-a7c8-740bc7c195fc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:51:26.140747 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:51:26.140694    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99ec07f1-50eb-499f-a7c8-740bc7c195fc-dshm" (OuterVolumeSpecName: "dshm") pod "99ec07f1-50eb-499f-a7c8-740bc7c195fc" (UID: "99ec07f1-50eb-499f-a7c8-740bc7c195fc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:51:26.140960 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:51:26.140926    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99ec07f1-50eb-499f-a7c8-740bc7c195fc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "99ec07f1-50eb-499f-a7c8-740bc7c195fc" (UID: "99ec07f1-50eb-499f-a7c8-740bc7c195fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:51:26.141103 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:51:26.141089    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99ec07f1-50eb-499f-a7c8-740bc7c195fc-kube-api-access-wqrx5" (OuterVolumeSpecName: "kube-api-access-wqrx5") pod "99ec07f1-50eb-499f-a7c8-740bc7c195fc" (UID: "99ec07f1-50eb-499f-a7c8-740bc7c195fc"). InnerVolumeSpecName "kube-api-access-wqrx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:51:26.144533 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:51:26.144508    2577 generic.go:356] "Generic (PLEG): container finished" podID="99ec07f1-50eb-499f-a7c8-740bc7c195fc" containerID="3cbb49666206020cafdc7e7d16f29aa516ebc2495df7e6f56da530092984560d" exitCode=0
Sep 18 19:51:26.144654 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:51:26.144567    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-85487667b4-g8cg4"
Sep 18 19:51:26.144654 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:51:26.144564    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-85487667b4-g8cg4" event={"ID":"99ec07f1-50eb-499f-a7c8-740bc7c195fc","Type":"ContainerDied","Data":"3cbb49666206020cafdc7e7d16f29aa516ebc2495df7e6f56da530092984560d"}
Sep 18 19:51:26.144654 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:51:26.144631    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-85487667b4-g8cg4" event={"ID":"99ec07f1-50eb-499f-a7c8-740bc7c195fc","Type":"ContainerDied","Data":"0d8f33d4e0689acc85d42f32fa26c242a21fff7e884c3f68051fdd88d2b272ca"}
Sep 18 19:51:26.144818 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:51:26.144664    2577 scope.go:117] "RemoveContainer" containerID="3cbb49666206020cafdc7e7d16f29aa516ebc2495df7e6f56da530092984560d"
Sep 18 19:51:26.157443 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:51:26.157415    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99ec07f1-50eb-499f-a7c8-740bc7c195fc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "99ec07f1-50eb-499f-a7c8-740bc7c195fc" (UID: "99ec07f1-50eb-499f-a7c8-740bc7c195fc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:51:26.158744 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:51:26.158727    2577 scope.go:117] "RemoveContainer" containerID="e213439f09ad6370d91abe843b095cbaae88a498213b1ee8a58fbc692cbea50d"
Sep 18 19:51:26.166131 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:51:26.166115    2577 scope.go:117] "RemoveContainer" containerID="3cbb49666206020cafdc7e7d16f29aa516ebc2495df7e6f56da530092984560d"
Sep 18 19:51:26.166409 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:51:26.166380    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cbb49666206020cafdc7e7d16f29aa516ebc2495df7e6f56da530092984560d\": container with ID starting with 3cbb49666206020cafdc7e7d16f29aa516ebc2495df7e6f56da530092984560d not found: ID does not exist" containerID="3cbb49666206020cafdc7e7d16f29aa516ebc2495df7e6f56da530092984560d"
Sep 18 19:51:26.166484 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:51:26.166417    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cbb49666206020cafdc7e7d16f29aa516ebc2495df7e6f56da530092984560d"} err="failed to get container status \"3cbb49666206020cafdc7e7d16f29aa516ebc2495df7e6f56da530092984560d\": rpc error: code = NotFound desc = could not find container \"3cbb49666206020cafdc7e7d16f29aa516ebc2495df7e6f56da530092984560d\": container with ID starting with 3cbb49666206020cafdc7e7d16f29aa516ebc2495df7e6f56da530092984560d not found: ID does not exist"
Sep 18 19:51:26.166484 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:51:26.166441    2577 scope.go:117] "RemoveContainer" containerID="e213439f09ad6370d91abe843b095cbaae88a498213b1ee8a58fbc692cbea50d"
Sep 18 19:51:26.166715 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:51:26.166698    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e213439f09ad6370d91abe843b095cbaae88a498213b1ee8a58fbc692cbea50d\": container with ID starting with e213439f09ad6370d91abe843b095cbaae88a498213b1ee8a58fbc692cbea50d not found: ID does not exist" containerID="e213439f09ad6370d91abe843b095cbaae88a498213b1ee8a58fbc692cbea50d"
Sep 18 19:51:26.166762 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:51:26.166721    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e213439f09ad6370d91abe843b095cbaae88a498213b1ee8a58fbc692cbea50d"} err="failed to get container status \"e213439f09ad6370d91abe843b095cbaae88a498213b1ee8a58fbc692cbea50d\": rpc error: code = NotFound desc = could not find container \"e213439f09ad6370d91abe843b095cbaae88a498213b1ee8a58fbc692cbea50d\": container with ID starting with e213439f09ad6370d91abe843b095cbaae88a498213b1ee8a58fbc692cbea50d not found: ID does not exist"
Sep 18 19:51:26.170191 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:51:26.170170    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99ec07f1-50eb-499f-a7c8-740bc7c195fc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "99ec07f1-50eb-499f-a7c8-740bc7c195fc" (UID: "99ec07f1-50eb-499f-a7c8-740bc7c195fc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:51:26.238960 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:51:26.238926    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99ec07f1-50eb-499f-a7c8-740bc7c195fc-dshm\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:51:26.238960 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:51:26.238954    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99ec07f1-50eb-499f-a7c8-740bc7c195fc-tls-certs\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:51:26.238960 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:51:26.238964    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99ec07f1-50eb-499f-a7c8-740bc7c195fc-tmp-dir\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:51:26.239195 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:51:26.238973    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99ec07f1-50eb-499f-a7c8-740bc7c195fc-kserve-provision-location\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:51:26.239195 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:51:26.238982    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqrx5\" (UniqueName: \"kubernetes.io/projected/99ec07f1-50eb-499f-a7c8-740bc7c195fc-kube-api-access-wqrx5\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:51:26.239195 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:51:26.239012    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99ec07f1-50eb-499f-a7c8-740bc7c195fc-cabundle-cert\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:51:26.239195 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:51:26.239021    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99ec07f1-50eb-499f-a7c8-740bc7c195fc-home\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:51:26.471350 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:51:26.471321    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-85487667b4-g8cg4"]
Sep 18 19:51:26.477614 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:51:26.477587    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-85487667b4-g8cg4"]
Sep 18 19:51:27.437827 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:51:27.437791    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99ec07f1-50eb-499f-a7c8-740bc7c195fc" path="/var/lib/kubelet/pods/99ec07f1-50eb-499f-a7c8-740bc7c195fc/volumes"
Sep 18 19:52:00.582133 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:00.582091    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-k6jrj"]
Sep 18 19:52:00.582721 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:00.582434    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99ec07f1-50eb-499f-a7c8-740bc7c195fc" containerName="storage-initializer"
Sep 18 19:52:00.582721 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:00.582446    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="99ec07f1-50eb-499f-a7c8-740bc7c195fc" containerName="storage-initializer"
Sep 18 19:52:00.582721 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:00.582460    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99ec07f1-50eb-499f-a7c8-740bc7c195fc" containerName="main"
Sep 18 19:52:00.582721 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:00.582465    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="99ec07f1-50eb-499f-a7c8-740bc7c195fc" containerName="main"
Sep 18 19:52:00.582721 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:00.582529    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="99ec07f1-50eb-499f-a7c8-740bc7c195fc" containerName="main"
Sep 18 19:52:00.586825 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:00.586808    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-k6jrj"
Sep 18 19:52:00.592567 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:00.592538    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Sep 18 19:52:00.592714 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:00.592602    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-gw-ca-root-cert\""
Sep 18 19:52:00.592929 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:00.592819    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"router-gateway-1-openshift-default-dockercfg-99jdq\""
Sep 18 19:52:00.593222 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:00.593204    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Sep 18 19:52:00.613015 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:00.612970    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-k6jrj"]
Sep 18 19:52:00.624250 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:00.624219    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c4a937f-37ce-4f81-b64e-ab8375a91312-istio-token\") pod \"router-gateway-1-openshift-default-6f9866b8dc-k6jrj\" (UID: \"9c4a937f-37ce-4f81-b64e-ab8375a91312\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-k6jrj"
Sep 18 19:52:00.624406 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:00.624255    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv68p\" (UniqueName: \"kubernetes.io/projected/9c4a937f-37ce-4f81-b64e-ab8375a91312-kube-api-access-qv68p\") pod \"router-gateway-1-openshift-default-6f9866b8dc-k6jrj\" (UID: \"9c4a937f-37ce-4f81-b64e-ab8375a91312\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-k6jrj"
Sep 18 19:52:00.624406 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:00.624281    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c4a937f-37ce-4f81-b64e-ab8375a91312-workload-socket\") pod \"router-gateway-1-openshift-default-6f9866b8dc-k6jrj\" (UID: \"9c4a937f-37ce-4f81-b64e-ab8375a91312\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-k6jrj"
Sep 18 19:52:00.624406 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:00.624352    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c4a937f-37ce-4f81-b64e-ab8375a91312-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6f9866b8dc-k6jrj\" (UID: \"9c4a937f-37ce-4f81-b64e-ab8375a91312\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-k6jrj"
Sep 18 19:52:00.624532 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:00.624428    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c4a937f-37ce-4f81-b64e-ab8375a91312-credential-socket\") pod \"router-gateway-1-openshift-default-6f9866b8dc-k6jrj\" (UID: \"9c4a937f-37ce-4f81-b64e-ab8375a91312\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-k6jrj"
Sep 18 19:52:00.624532 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:00.624474    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c4a937f-37ce-4f81-b64e-ab8375a91312-istio-podinfo\") pod \"router-gateway-1-openshift-default-6f9866b8dc-k6jrj\" (UID: \"9c4a937f-37ce-4f81-b64e-ab8375a91312\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-k6jrj"
Sep 18 19:52:00.624532 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:00.624513    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c4a937f-37ce-4f81-b64e-ab8375a91312-workload-certs\") pod \"router-gateway-1-openshift-default-6f9866b8dc-k6jrj\" (UID: \"9c4a937f-37ce-4f81-b64e-ab8375a91312\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-k6jrj"
Sep 18 19:52:00.624640 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:00.624561    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c4a937f-37ce-4f81-b64e-ab8375a91312-istio-envoy\") pod \"router-gateway-1-openshift-default-6f9866b8dc-k6jrj\" (UID: \"9c4a937f-37ce-4f81-b64e-ab8375a91312\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-k6jrj"
Sep 18 19:52:00.624640 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:00.624594    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c4a937f-37ce-4f81-b64e-ab8375a91312-istio-data\") pod \"router-gateway-1-openshift-default-6f9866b8dc-k6jrj\" (UID: \"9c4a937f-37ce-4f81-b64e-ab8375a91312\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-k6jrj"
Sep 18 19:52:00.725815 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:00.725778    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c4a937f-37ce-4f81-b64e-ab8375a91312-istio-token\") pod \"router-gateway-1-openshift-default-6f9866b8dc-k6jrj\" (UID: \"9c4a937f-37ce-4f81-b64e-ab8375a91312\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-k6jrj"
Sep 18 19:52:00.725815 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:00.725819    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv68p\" (UniqueName: \"kubernetes.io/projected/9c4a937f-37ce-4f81-b64e-ab8375a91312-kube-api-access-qv68p\") pod \"router-gateway-1-openshift-default-6f9866b8dc-k6jrj\" (UID: \"9c4a937f-37ce-4f81-b64e-ab8375a91312\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-k6jrj"
Sep 18 19:52:00.726079 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:00.725840    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c4a937f-37ce-4f81-b64e-ab8375a91312-workload-socket\") pod \"router-gateway-1-openshift-default-6f9866b8dc-k6jrj\" (UID: \"9c4a937f-37ce-4f81-b64e-ab8375a91312\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-k6jrj"
Sep 18 19:52:00.726079 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:00.725973    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c4a937f-37ce-4f81-b64e-ab8375a91312-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6f9866b8dc-k6jrj\" (UID: \"9c4a937f-37ce-4f81-b64e-ab8375a91312\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-k6jrj"
Sep 18 19:52:00.726178 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:00.726075    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c4a937f-37ce-4f81-b64e-ab8375a91312-credential-socket\") pod \"router-gateway-1-openshift-default-6f9866b8dc-k6jrj\" (UID: \"9c4a937f-37ce-4f81-b64e-ab8375a91312\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-k6jrj"
Sep 18 19:52:00.726178 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:00.726125    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c4a937f-37ce-4f81-b64e-ab8375a91312-istio-podinfo\") pod \"router-gateway-1-openshift-default-6f9866b8dc-k6jrj\" (UID: \"9c4a937f-37ce-4f81-b64e-ab8375a91312\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-k6jrj"
Sep 18 19:52:00.726178 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:00.726157    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c4a937f-37ce-4f81-b64e-ab8375a91312-workload-certs\") pod \"router-gateway-1-openshift-default-6f9866b8dc-k6jrj\" (UID: \"9c4a937f-37ce-4f81-b64e-ab8375a91312\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-k6jrj"
Sep 18 19:52:00.726312 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:00.726198    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c4a937f-37ce-4f81-b64e-ab8375a91312-istio-envoy\") pod \"router-gateway-1-openshift-default-6f9866b8dc-k6jrj\" (UID: \"9c4a937f-37ce-4f81-b64e-ab8375a91312\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-k6jrj"
Sep 18 19:52:00.726312 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:00.726218    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c4a937f-37ce-4f81-b64e-ab8375a91312-workload-socket\") pod \"router-gateway-1-openshift-default-6f9866b8dc-k6jrj\" (UID: \"9c4a937f-37ce-4f81-b64e-ab8375a91312\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-k6jrj"
Sep 18 19:52:00.726312 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:00.726232    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c4a937f-37ce-4f81-b64e-ab8375a91312-istio-data\") pod \"router-gateway-1-openshift-default-6f9866b8dc-k6jrj\" (UID: \"9c4a937f-37ce-4f81-b64e-ab8375a91312\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-k6jrj"
Sep 18 19:52:00.726442 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:00.726399    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c4a937f-37ce-4f81-b64e-ab8375a91312-credential-socket\") pod \"router-gateway-1-openshift-default-6f9866b8dc-k6jrj\" (UID: \"9c4a937f-37ce-4f81-b64e-ab8375a91312\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-k6jrj"
Sep 18 19:52:00.726563 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:00.726535    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c4a937f-37ce-4f81-b64e-ab8375a91312-workload-certs\") pod \"router-gateway-1-openshift-default-6f9866b8dc-k6jrj\" (UID: \"9c4a937f-37ce-4f81-b64e-ab8375a91312\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-k6jrj"
Sep 18 19:52:00.726703 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:00.726590    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c4a937f-37ce-4f81-b64e-ab8375a91312-istio-data\") pod \"router-gateway-1-openshift-default-6f9866b8dc-k6jrj\" (UID: \"9c4a937f-37ce-4f81-b64e-ab8375a91312\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-k6jrj"
Sep 18 19:52:00.726780 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:00.726738    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c4a937f-37ce-4f81-b64e-ab8375a91312-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6f9866b8dc-k6jrj\" (UID: \"9c4a937f-37ce-4f81-b64e-ab8375a91312\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-k6jrj"
Sep 18 19:52:00.728407 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:00.728379    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c4a937f-37ce-4f81-b64e-ab8375a91312-istio-envoy\") pod \"router-gateway-1-openshift-default-6f9866b8dc-k6jrj\" (UID: \"9c4a937f-37ce-4f81-b64e-ab8375a91312\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-k6jrj"
Sep 18 19:52:00.728670 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:00.728650    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c4a937f-37ce-4f81-b64e-ab8375a91312-istio-podinfo\") pod \"router-gateway-1-openshift-default-6f9866b8dc-k6jrj\" (UID: \"9c4a937f-37ce-4f81-b64e-ab8375a91312\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-k6jrj"
Sep 18 19:52:00.745278 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:00.745255    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv68p\" (UniqueName: \"kubernetes.io/projected/9c4a937f-37ce-4f81-b64e-ab8375a91312-kube-api-access-qv68p\") pod \"router-gateway-1-openshift-default-6f9866b8dc-k6jrj\" (UID: \"9c4a937f-37ce-4f81-b64e-ab8375a91312\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-k6jrj"
Sep 18 19:52:00.745538 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:00.745515    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c4a937f-37ce-4f81-b64e-ab8375a91312-istio-token\") pod \"router-gateway-1-openshift-default-6f9866b8dc-k6jrj\" (UID: \"9c4a937f-37ce-4f81-b64e-ab8375a91312\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-k6jrj"
Sep 18 19:52:00.898023 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:00.897969    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-k6jrj"
Sep 18 19:52:01.184289 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:01.184208    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-k6jrj"]
Sep 18 19:52:01.191224 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:01.191192    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 19:52:01.191313 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:01.191269    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 19:52:01.191313 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:01.191299    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 19:52:01.296222 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:01.296187    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-k6jrj" event={"ID":"9c4a937f-37ce-4f81-b64e-ab8375a91312","Type":"ContainerStarted","Data":"3c2c877abfede1fb4fec4d29f05990e83fc6019797360cfc22cccc907b47ab06"}
Sep 18 19:52:01.296431 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:01.296256    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-k6jrj" event={"ID":"9c4a937f-37ce-4f81-b64e-ab8375a91312","Type":"ContainerStarted","Data":"41756e99fcbc539c96c0979a5b3be3ba70d16975557fa682ceb225c47895bbdd"}
Sep 18 19:52:01.898682 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:01.898639    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-k6jrj"
Sep 18 19:52:02.903820 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:02.903788    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-k6jrj"
Sep 18 19:52:02.932505 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:02.932452    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-k6jrj" podStartSLOduration=2.932435794 podStartE2EDuration="2.932435794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:52:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:52:01.324497121 +0000 UTC m=+2292.176797506" watchObservedRunningTime="2026-09-18 19:52:02.932435794 +0000 UTC m=+2293.784736164"
Sep 18 19:52:03.305051 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:03.304931    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-k6jrj"
Sep 18 19:52:03.306103 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:03.306078    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-k6jrj"
Sep 18 19:52:05.980356 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:05.980319    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972kxsl"]
Sep 18 19:52:05.984487 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:05.984465    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972kxsl"
Sep 18 19:52:05.989070 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:05.989049    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"default-dockercfg-wcf7r\""
Sep 18 19:52:05.989218 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:05.989077    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs\""
Sep 18 19:52:06.005220 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:06.005194    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972kxsl"]
Sep 18 19:52:06.071351 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:06.071314    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6afa7184-6099-41e8-8fb7-d1c8b890e315-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972kxsl\" (UID: \"6afa7184-6099-41e8-8fb7-d1c8b890e315\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972kxsl"
Sep 18 19:52:06.071552 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:06.071361    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6afa7184-6099-41e8-8fb7-d1c8b890e315-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972kxsl\" (UID: \"6afa7184-6099-41e8-8fb7-d1c8b890e315\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972kxsl"
Sep 18 19:52:06.071552 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:06.071463    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55kx8\" (UniqueName: \"kubernetes.io/projected/6afa7184-6099-41e8-8fb7-d1c8b890e315-kube-api-access-55kx8\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972kxsl\" (UID: \"6afa7184-6099-41e8-8fb7-d1c8b890e315\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972kxsl"
Sep 18 19:52:06.071552 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:06.071513    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6afa7184-6099-41e8-8fb7-d1c8b890e315-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972kxsl\" (UID: \"6afa7184-6099-41e8-8fb7-d1c8b890e315\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972kxsl"
Sep 18 19:52:06.071715 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:06.071559    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6afa7184-6099-41e8-8fb7-d1c8b890e315-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972kxsl\" (UID: \"6afa7184-6099-41e8-8fb7-d1c8b890e315\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972kxsl"
Sep 18 19:52:06.071715 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:06.071588    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6afa7184-6099-41e8-8fb7-d1c8b890e315-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972kxsl\" (UID: \"6afa7184-6099-41e8-8fb7-d1c8b890e315\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972kxsl"
Sep 18 19:52:06.172772 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:06.172642    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6afa7184-6099-41e8-8fb7-d1c8b890e315-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972kxsl\" (UID: \"6afa7184-6099-41e8-8fb7-d1c8b890e315\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972kxsl"
Sep 18 19:52:06.172772 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:06.172704    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6afa7184-6099-41e8-8fb7-d1c8b890e315-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972kxsl\" (UID: \"6afa7184-6099-41e8-8fb7-d1c8b890e315\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972kxsl"
Sep 18 19:52:06.172772 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:06.172764    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55kx8\" (UniqueName: \"kubernetes.io/projected/6afa7184-6099-41e8-8fb7-d1c8b890e315-kube-api-access-55kx8\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972kxsl\" (UID: \"6afa7184-6099-41e8-8fb7-d1c8b890e315\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972kxsl"
Sep 18 19:52:06.185853 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:06.172811    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6afa7184-6099-41e8-8fb7-d1c8b890e315-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972kxsl\" (UID: \"6afa7184-6099-41e8-8fb7-d1c8b890e315\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972kxsl"
Sep 18 19:52:06.185853 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:06.172855    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6afa7184-6099-41e8-8fb7-d1c8b890e315-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972kxsl\" (UID: \"6afa7184-6099-41e8-8fb7-d1c8b890e315\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972kxsl"
Sep 18 19:52:06.185853 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:06.172882    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6afa7184-6099-41e8-8fb7-d1c8b890e315-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972kxsl\" (UID: \"6afa7184-6099-41e8-8fb7-d1c8b890e315\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972kxsl"
Sep 18 19:52:06.185853 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:06.173119    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6afa7184-6099-41e8-8fb7-d1c8b890e315-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972kxsl\" (UID: \"6afa7184-6099-41e8-8fb7-d1c8b890e315\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972kxsl"
Sep 18 19:52:06.185853 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:06.173160    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6afa7184-6099-41e8-8fb7-d1c8b890e315-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972kxsl\" (UID: \"6afa7184-6099-41e8-8fb7-d1c8b890e315\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972kxsl"
Sep 18 19:52:06.185853 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:06.173187    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6afa7184-6099-41e8-8fb7-d1c8b890e315-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972kxsl\" (UID: \"6afa7184-6099-41e8-8fb7-d1c8b890e315\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972kxsl"
Sep 18 19:52:06.185853 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:06.175116    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6afa7184-6099-41e8-8fb7-d1c8b890e315-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972kxsl\" (UID: \"6afa7184-6099-41e8-8fb7-d1c8b890e315\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972kxsl"
Sep 18 19:52:06.185853 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:06.175273    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6afa7184-6099-41e8-8fb7-d1c8b890e315-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972kxsl\" (UID: \"6afa7184-6099-41e8-8fb7-d1c8b890e315\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972kxsl"
Sep 18 19:52:06.185853 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:06.182510    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55kx8\" (UniqueName: \"kubernetes.io/projected/6afa7184-6099-41e8-8fb7-d1c8b890e315-kube-api-access-55kx8\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972kxsl\" (UID: \"6afa7184-6099-41e8-8fb7-d1c8b890e315\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972kxsl"
Sep 18 19:52:06.296953 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:06.296874    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972kxsl"
Sep 18 19:52:06.578360 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:06.578277    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972kxsl"]
Sep 18 19:52:06.581600 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:52:06.581569    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6afa7184_6099_41e8_8fb7_d1c8b890e315.slice/crio-50bebf01d9139e1cc0293d5be8d20750a442d741612d232ccb5afa912b0ae3e0 WatchSource:0}: Error finding container 50bebf01d9139e1cc0293d5be8d20750a442d741612d232ccb5afa912b0ae3e0: Status 404 returned error can't find the container with id 50bebf01d9139e1cc0293d5be8d20750a442d741612d232ccb5afa912b0ae3e0
Sep 18 19:52:07.324841 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:07.324797    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972kxsl" event={"ID":"6afa7184-6099-41e8-8fb7-d1c8b890e315","Type":"ContainerStarted","Data":"9bc6b4ea89c289f5c37430d6fa29aa71046b8797c7fd059a3bba6841ad8b5f02"}
Sep 18 19:52:07.325280 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:07.324913    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972kxsl" event={"ID":"6afa7184-6099-41e8-8fb7-d1c8b890e315","Type":"ContainerStarted","Data":"50bebf01d9139e1cc0293d5be8d20750a442d741612d232ccb5afa912b0ae3e0"}
Sep 18 19:52:08.297742 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:08.297704    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972kxsl"
Sep 18 19:52:15.370348 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:15.370313    2577 generic.go:356] "Generic (PLEG): container finished" podID="9c4a937f-37ce-4f81-b64e-ab8375a91312" containerID="3c2c877abfede1fb4fec4d29f05990e83fc6019797360cfc22cccc907b47ab06" exitCode=0
Sep 18 19:52:15.370870 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:15.370392    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-k6jrj" event={"ID":"9c4a937f-37ce-4f81-b64e-ab8375a91312","Type":"ContainerDied","Data":"3c2c877abfede1fb4fec4d29f05990e83fc6019797360cfc22cccc907b47ab06"}
Sep 18 19:52:15.370942 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:15.370926    2577 scope.go:117] "RemoveContainer" containerID="3c2c877abfede1fb4fec4d29f05990e83fc6019797360cfc22cccc907b47ab06"
Sep 18 19:52:15.371919 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:15.371892    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 19:52:15.372036 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:15.371961    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 19:52:15.372036 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:15.372011    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 19:52:15.411921 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:15.411877    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972kxsl" podStartSLOduration=10.41186014 podStartE2EDuration="10.41186014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:52:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:52:07.355150466 +0000 UTC m=+2298.207450837" watchObservedRunningTime="2026-09-18 19:52:15.41186014 +0000 UTC m=+2306.264160517"
Sep 18 19:52:16.297111 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:16.297055    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972kxsl"
Sep 18 19:52:16.309655 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:16.309625    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972kxsl"
Sep 18 19:52:16.377597 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:16.377563    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-k6jrj" event={"ID":"9c4a937f-37ce-4f81-b64e-ab8375a91312","Type":"ContainerStarted","Data":"39a9c033da9383e8fd51fcfd1a98e62da2fd36a362b0845c8440540dc0325bbb"}
Sep 18 19:52:16.388736 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:16.388706    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972kxsl"
Sep 18 19:52:16.898318 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:16.898284    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-k6jrj"
Sep 18 19:52:16.899456 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:16.899421    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-k6jrj" podUID="9c4a937f-37ce-4f81-b64e-ab8375a91312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Sep 18 19:52:17.898895 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:17.898850    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-k6jrj" podUID="9c4a937f-37ce-4f81-b64e-ab8375a91312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Sep 18 19:52:18.305313 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:18.305232    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-k6jrj"
Sep 18 19:52:18.898545 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:18.898499    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-k6jrj" podUID="9c4a937f-37ce-4f81-b64e-ab8375a91312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Sep 18 19:52:19.898739 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:19.898687    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-k6jrj" podUID="9c4a937f-37ce-4f81-b64e-ab8375a91312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Sep 18 19:52:20.898912 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:20.898873    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-k6jrj" podUID="9c4a937f-37ce-4f81-b64e-ab8375a91312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Sep 18 19:52:21.902805 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:21.902775    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-k6jrj"
Sep 18 19:52:21.903757 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:52:21.903739    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-k6jrj"
Sep 18 19:53:21.882691 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:21.882656    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972kxsl"]
Sep 18 19:53:21.891405 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:53:21.891363    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 19:53:21.891405 ip-10-0-129-30 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:53:21.891405 ip-10-0-129-30 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 18 19:53:21.891405 ip-10-0-129-30 kubenswrapper[2577]:  > containerID="9bc6b4ea89c289f5c37430d6fa29aa71046b8797c7fd059a3bba6841ad8b5f02" cmd=["/bin/sleep","15"]
Sep 18 19:53:21.891629 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:21.891421    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972kxsl" message="" err=<
Sep 18 19:53:21.891629 ip-10-0-129-30 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:53:21.891629 ip-10-0-129-30 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 18 19:53:21.891629 ip-10-0-129-30 kubenswrapper[2577]:  >
Sep 18 19:53:21.891629 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:53:21.891439    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 19:53:21.891629 ip-10-0-129-30 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:53:21.891629 ip-10-0-129-30 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 18 19:53:21.891629 ip-10-0-129-30 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972kxsl" podUID="6afa7184-6099-41e8-8fb7-d1c8b890e315" containerName="main" containerID="cri-o://9bc6b4ea89c289f5c37430d6fa29aa71046b8797c7fd059a3bba6841ad8b5f02"
Sep 18 19:53:21.891629 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:21.891488    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972kxsl" podUID="6afa7184-6099-41e8-8fb7-d1c8b890e315" containerName="main" containerID="cri-o://9bc6b4ea89c289f5c37430d6fa29aa71046b8797c7fd059a3bba6841ad8b5f02" gracePeriod=60
Sep 18 19:53:22.230385 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:22.230353    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972kxsl"
Sep 18 19:53:22.350426 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:22.350371    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6afa7184-6099-41e8-8fb7-d1c8b890e315-tls-certs\") pod \"6afa7184-6099-41e8-8fb7-d1c8b890e315\" (UID: \"6afa7184-6099-41e8-8fb7-d1c8b890e315\") "
Sep 18 19:53:22.350426 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:22.350436    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6afa7184-6099-41e8-8fb7-d1c8b890e315-home\") pod \"6afa7184-6099-41e8-8fb7-d1c8b890e315\" (UID: \"6afa7184-6099-41e8-8fb7-d1c8b890e315\") "
Sep 18 19:53:22.350652 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:22.350469    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6afa7184-6099-41e8-8fb7-d1c8b890e315-model-cache\") pod \"6afa7184-6099-41e8-8fb7-d1c8b890e315\" (UID: \"6afa7184-6099-41e8-8fb7-d1c8b890e315\") "
Sep 18 19:53:22.350652 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:22.350559    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6afa7184-6099-41e8-8fb7-d1c8b890e315-dshm\") pod \"6afa7184-6099-41e8-8fb7-d1c8b890e315\" (UID: \"6afa7184-6099-41e8-8fb7-d1c8b890e315\") "
Sep 18 19:53:22.350652 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:22.350597    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55kx8\" (UniqueName: \"kubernetes.io/projected/6afa7184-6099-41e8-8fb7-d1c8b890e315-kube-api-access-55kx8\") pod \"6afa7184-6099-41e8-8fb7-d1c8b890e315\" (UID: \"6afa7184-6099-41e8-8fb7-d1c8b890e315\") "
Sep 18 19:53:22.350652 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:22.350632    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6afa7184-6099-41e8-8fb7-d1c8b890e315-tmp-dir\") pod \"6afa7184-6099-41e8-8fb7-d1c8b890e315\" (UID: \"6afa7184-6099-41e8-8fb7-d1c8b890e315\") "
Sep 18 19:53:22.350858 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:22.350728    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6afa7184-6099-41e8-8fb7-d1c8b890e315-home" (OuterVolumeSpecName: "home") pod "6afa7184-6099-41e8-8fb7-d1c8b890e315" (UID: "6afa7184-6099-41e8-8fb7-d1c8b890e315"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:53:22.350858 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:22.350737    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6afa7184-6099-41e8-8fb7-d1c8b890e315-model-cache" (OuterVolumeSpecName: "model-cache") pod "6afa7184-6099-41e8-8fb7-d1c8b890e315" (UID: "6afa7184-6099-41e8-8fb7-d1c8b890e315"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:53:22.350972 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:22.350916    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6afa7184-6099-41e8-8fb7-d1c8b890e315-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6afa7184-6099-41e8-8fb7-d1c8b890e315" (UID: "6afa7184-6099-41e8-8fb7-d1c8b890e315"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:53:22.350972 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:22.350940    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6afa7184-6099-41e8-8fb7-d1c8b890e315-home\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:53:22.350972 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:22.350957    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6afa7184-6099-41e8-8fb7-d1c8b890e315-model-cache\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:53:22.352832 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:22.352810    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6afa7184-6099-41e8-8fb7-d1c8b890e315-dshm" (OuterVolumeSpecName: "dshm") pod "6afa7184-6099-41e8-8fb7-d1c8b890e315" (UID: "6afa7184-6099-41e8-8fb7-d1c8b890e315"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:53:22.352832 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:22.352822    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6afa7184-6099-41e8-8fb7-d1c8b890e315-kube-api-access-55kx8" (OuterVolumeSpecName: "kube-api-access-55kx8") pod "6afa7184-6099-41e8-8fb7-d1c8b890e315" (UID: "6afa7184-6099-41e8-8fb7-d1c8b890e315"). InnerVolumeSpecName "kube-api-access-55kx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:53:22.352968 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:22.352894    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6afa7184-6099-41e8-8fb7-d1c8b890e315-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6afa7184-6099-41e8-8fb7-d1c8b890e315" (UID: "6afa7184-6099-41e8-8fb7-d1c8b890e315"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:53:22.451631 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:22.451528    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6afa7184-6099-41e8-8fb7-d1c8b890e315-dshm\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:53:22.451631 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:22.451575    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55kx8\" (UniqueName: \"kubernetes.io/projected/6afa7184-6099-41e8-8fb7-d1c8b890e315-kube-api-access-55kx8\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:53:22.451631 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:22.451605    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6afa7184-6099-41e8-8fb7-d1c8b890e315-tmp-dir\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:53:22.451631 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:22.451615    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6afa7184-6099-41e8-8fb7-d1c8b890e315-tls-certs\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:53:22.666795 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:22.666763    2577 generic.go:356] "Generic (PLEG): container finished" podID="6afa7184-6099-41e8-8fb7-d1c8b890e315" containerID="9bc6b4ea89c289f5c37430d6fa29aa71046b8797c7fd059a3bba6841ad8b5f02" exitCode=0
Sep 18 19:53:22.666976 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:22.666809    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972kxsl"
Sep 18 19:53:22.666976 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:22.666827    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972kxsl" event={"ID":"6afa7184-6099-41e8-8fb7-d1c8b890e315","Type":"ContainerDied","Data":"9bc6b4ea89c289f5c37430d6fa29aa71046b8797c7fd059a3bba6841ad8b5f02"}
Sep 18 19:53:22.666976 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:22.666871    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972kxsl" event={"ID":"6afa7184-6099-41e8-8fb7-d1c8b890e315","Type":"ContainerDied","Data":"50bebf01d9139e1cc0293d5be8d20750a442d741612d232ccb5afa912b0ae3e0"}
Sep 18 19:53:22.666976 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:22.666891    2577 scope.go:117] "RemoveContainer" containerID="9bc6b4ea89c289f5c37430d6fa29aa71046b8797c7fd059a3bba6841ad8b5f02"
Sep 18 19:53:22.676217 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:22.676201    2577 scope.go:117] "RemoveContainer" containerID="9bc6b4ea89c289f5c37430d6fa29aa71046b8797c7fd059a3bba6841ad8b5f02"
Sep 18 19:53:22.676481 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:53:22.676463    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bc6b4ea89c289f5c37430d6fa29aa71046b8797c7fd059a3bba6841ad8b5f02\": container with ID starting with 9bc6b4ea89c289f5c37430d6fa29aa71046b8797c7fd059a3bba6841ad8b5f02 not found: ID does not exist" containerID="9bc6b4ea89c289f5c37430d6fa29aa71046b8797c7fd059a3bba6841ad8b5f02"
Sep 18 19:53:22.676529 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:22.676488    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bc6b4ea89c289f5c37430d6fa29aa71046b8797c7fd059a3bba6841ad8b5f02"} err="failed to get container status \"9bc6b4ea89c289f5c37430d6fa29aa71046b8797c7fd059a3bba6841ad8b5f02\": rpc error: code = NotFound desc = could not find container \"9bc6b4ea89c289f5c37430d6fa29aa71046b8797c7fd059a3bba6841ad8b5f02\": container with ID starting with 9bc6b4ea89c289f5c37430d6fa29aa71046b8797c7fd059a3bba6841ad8b5f02 not found: ID does not exist"
Sep 18 19:53:22.702418 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:22.702337    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972kxsl"]
Sep 18 19:53:22.707270 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:22.707244    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972kxsl"]
Sep 18 19:53:23.438092 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:23.438054    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6afa7184-6099-41e8-8fb7-d1c8b890e315" path="/var/lib/kubelet/pods/6afa7184-6099-41e8-8fb7-d1c8b890e315/volumes"
Sep 18 19:53:26.861945 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:26.861912    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6"]
Sep 18 19:53:26.862352 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:26.862283    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6afa7184-6099-41e8-8fb7-d1c8b890e315" containerName="main"
Sep 18 19:53:26.862352 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:26.862295    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6afa7184-6099-41e8-8fb7-d1c8b890e315" containerName="main"
Sep 18 19:53:26.862426 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:26.862372    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6afa7184-6099-41e8-8fb7-d1c8b890e315" containerName="main"
Sep 18 19:53:26.866613 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:26.866593    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6"
Sep 18 19:53:26.869693 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:26.869672    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 19:53:26.870819 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:26.870802    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Sep 18 19:53:26.870903 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:26.870821    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Sep 18 19:53:26.870949 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:26.870921    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-697gw\""
Sep 18 19:53:26.871086 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:26.871070    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"seaweedfs-s3-creds\""
Sep 18 19:53:26.871368 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:26.871352    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Sep 18 19:53:26.887423 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:26.887386    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6"]
Sep 18 19:53:26.994428 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:26.994390    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28e50adb-d3f3-4d10-ab28-ff9f09bf40da-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6\" (UID: \"28e50adb-d3f3-4d10-ab28-ff9f09bf40da\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6"
Sep 18 19:53:26.994626 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:26.994443    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28e50adb-d3f3-4d10-ab28-ff9f09bf40da-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6\" (UID: \"28e50adb-d3f3-4d10-ab28-ff9f09bf40da\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6"
Sep 18 19:53:26.994626 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:26.994467    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28e50adb-d3f3-4d10-ab28-ff9f09bf40da-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6\" (UID: \"28e50adb-d3f3-4d10-ab28-ff9f09bf40da\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6"
Sep 18 19:53:26.994626 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:26.994515    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/28e50adb-d3f3-4d10-ab28-ff9f09bf40da-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6\" (UID: \"28e50adb-d3f3-4d10-ab28-ff9f09bf40da\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6"
Sep 18 19:53:26.994626 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:26.994559    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28e50adb-d3f3-4d10-ab28-ff9f09bf40da-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6\" (UID: \"28e50adb-d3f3-4d10-ab28-ff9f09bf40da\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6"
Sep 18 19:53:26.994626 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:26.994596    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct9sx\" (UniqueName: \"kubernetes.io/projected/28e50adb-d3f3-4d10-ab28-ff9f09bf40da-kube-api-access-ct9sx\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6\" (UID: \"28e50adb-d3f3-4d10-ab28-ff9f09bf40da\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6"
Sep 18 19:53:26.994626 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:26.994614    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28e50adb-d3f3-4d10-ab28-ff9f09bf40da-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6\" (UID: \"28e50adb-d3f3-4d10-ab28-ff9f09bf40da\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6"
Sep 18 19:53:26.994855 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:26.994641    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28e50adb-d3f3-4d10-ab28-ff9f09bf40da-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6\" (UID: \"28e50adb-d3f3-4d10-ab28-ff9f09bf40da\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6"
Sep 18 19:53:27.095546 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:27.095505    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28e50adb-d3f3-4d10-ab28-ff9f09bf40da-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6\" (UID: \"28e50adb-d3f3-4d10-ab28-ff9f09bf40da\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6"
Sep 18 19:53:27.095741 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:27.095666    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28e50adb-d3f3-4d10-ab28-ff9f09bf40da-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6\" (UID: \"28e50adb-d3f3-4d10-ab28-ff9f09bf40da\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6"
Sep 18 19:53:27.095741 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:27.095721    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28e50adb-d3f3-4d10-ab28-ff9f09bf40da-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6\" (UID: \"28e50adb-d3f3-4d10-ab28-ff9f09bf40da\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6"
Sep 18 19:53:27.095879 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:27.095750    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28e50adb-d3f3-4d10-ab28-ff9f09bf40da-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6\" (UID: \"28e50adb-d3f3-4d10-ab28-ff9f09bf40da\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6"
Sep 18 19:53:27.095879 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:27.095779    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/28e50adb-d3f3-4d10-ab28-ff9f09bf40da-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6\" (UID: \"28e50adb-d3f3-4d10-ab28-ff9f09bf40da\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6"
Sep 18 19:53:27.095879 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:27.095816    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28e50adb-d3f3-4d10-ab28-ff9f09bf40da-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6\" (UID: \"28e50adb-d3f3-4d10-ab28-ff9f09bf40da\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6"
Sep 18 19:53:27.095879 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:27.095843    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct9sx\" (UniqueName: \"kubernetes.io/projected/28e50adb-d3f3-4d10-ab28-ff9f09bf40da-kube-api-access-ct9sx\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6\" (UID: \"28e50adb-d3f3-4d10-ab28-ff9f09bf40da\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6"
Sep 18 19:53:27.095879 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:27.095870    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28e50adb-d3f3-4d10-ab28-ff9f09bf40da-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6\" (UID: \"28e50adb-d3f3-4d10-ab28-ff9f09bf40da\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6"
Sep 18 19:53:27.096193 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:27.096121    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28e50adb-d3f3-4d10-ab28-ff9f09bf40da-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6\" (UID: \"28e50adb-d3f3-4d10-ab28-ff9f09bf40da\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6"
Sep 18 19:53:27.096243 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:27.096213    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28e50adb-d3f3-4d10-ab28-ff9f09bf40da-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6\" (UID: \"28e50adb-d3f3-4d10-ab28-ff9f09bf40da\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6"
Sep 18 19:53:27.096307 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:27.096280    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28e50adb-d3f3-4d10-ab28-ff9f09bf40da-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6\" (UID: \"28e50adb-d3f3-4d10-ab28-ff9f09bf40da\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6"
Sep 18 19:53:27.096474 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:27.096452    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28e50adb-d3f3-4d10-ab28-ff9f09bf40da-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6\" (UID: \"28e50adb-d3f3-4d10-ab28-ff9f09bf40da\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6"
Sep 18 19:53:27.096578 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:27.096561    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/28e50adb-d3f3-4d10-ab28-ff9f09bf40da-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6\" (UID: \"28e50adb-d3f3-4d10-ab28-ff9f09bf40da\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6"
Sep 18 19:53:27.097937 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:27.097917    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28e50adb-d3f3-4d10-ab28-ff9f09bf40da-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6\" (UID: \"28e50adb-d3f3-4d10-ab28-ff9f09bf40da\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6"
Sep 18 19:53:27.098391 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:27.098371    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28e50adb-d3f3-4d10-ab28-ff9f09bf40da-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6\" (UID: \"28e50adb-d3f3-4d10-ab28-ff9f09bf40da\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6"
Sep 18 19:53:27.107720 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:27.107695    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct9sx\" (UniqueName: \"kubernetes.io/projected/28e50adb-d3f3-4d10-ab28-ff9f09bf40da-kube-api-access-ct9sx\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6\" (UID: \"28e50adb-d3f3-4d10-ab28-ff9f09bf40da\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6"
Sep 18 19:53:27.177200 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:27.177120    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6"
Sep 18 19:53:27.467926 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:27.467893    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6"]
Sep 18 19:53:27.469059 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:53:27.469036    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28e50adb_d3f3_4d10_ab28_ff9f09bf40da.slice/crio-d0783876e6e68d34243129d07b0d279ebd24b37d35b83716472f05b2bc7d483a WatchSource:0}: Error finding container d0783876e6e68d34243129d07b0d279ebd24b37d35b83716472f05b2bc7d483a: Status 404 returned error can't find the container with id d0783876e6e68d34243129d07b0d279ebd24b37d35b83716472f05b2bc7d483a
Sep 18 19:53:27.690926 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:27.690890    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6" event={"ID":"28e50adb-d3f3-4d10-ab28-ff9f09bf40da","Type":"ContainerStarted","Data":"6ca7d5c7f32c5a0ba32750ae21e68159991b670fd25899d38dfdc9156de7ea4f"}
Sep 18 19:53:27.691117 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:27.690948    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6" event={"ID":"28e50adb-d3f3-4d10-ab28-ff9f09bf40da","Type":"ContainerStarted","Data":"d0783876e6e68d34243129d07b0d279ebd24b37d35b83716472f05b2bc7d483a"}
Sep 18 19:53:28.250485 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:28.250449    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-k6jrj"]
Sep 18 19:53:28.250965 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:28.250706    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-k6jrj" podUID="9c4a937f-37ce-4f81-b64e-ab8375a91312" containerName="istio-proxy" containerID="cri-o://39a9c033da9383e8fd51fcfd1a98e62da2fd36a362b0845c8440540dc0325bbb" gracePeriod=30
Sep 18 19:53:29.702622 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:29.702587    2577 generic.go:356] "Generic (PLEG): container finished" podID="28e50adb-d3f3-4d10-ab28-ff9f09bf40da" containerID="6ca7d5c7f32c5a0ba32750ae21e68159991b670fd25899d38dfdc9156de7ea4f" exitCode=0
Sep 18 19:53:29.703046 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:29.702670    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6" event={"ID":"28e50adb-d3f3-4d10-ab28-ff9f09bf40da","Type":"ContainerDied","Data":"6ca7d5c7f32c5a0ba32750ae21e68159991b670fd25899d38dfdc9156de7ea4f"}
Sep 18 19:53:30.708818 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:30.708780    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6" event={"ID":"28e50adb-d3f3-4d10-ab28-ff9f09bf40da","Type":"ContainerStarted","Data":"86d66ffb6f63f63465ac66d2f1ea50c6e3ed0fdfb44295e5b50eb5821215cce4"}
Sep 18 19:53:30.747203 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:30.747142    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6" podStartSLOduration=2.601516333 podStartE2EDuration="4.747123555s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.145607222s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:53:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:53:30.745177463 +0000 UTC m=+2381.597477833" watchObservedRunningTime="2026-09-18 19:53:30.747123555 +0000 UTC m=+2381.599423926"
Sep 18 19:53:31.178091 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:31.178053    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6"
Sep 18 19:53:33.493261 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:33.493236    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-k6jrj"
Sep 18 19:53:33.655806 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:33.655770    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c4a937f-37ce-4f81-b64e-ab8375a91312-istio-data\") pod \"9c4a937f-37ce-4f81-b64e-ab8375a91312\" (UID: \"9c4a937f-37ce-4f81-b64e-ab8375a91312\") "
Sep 18 19:53:33.656031 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:33.655843    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c4a937f-37ce-4f81-b64e-ab8375a91312-credential-socket\") pod \"9c4a937f-37ce-4f81-b64e-ab8375a91312\" (UID: \"9c4a937f-37ce-4f81-b64e-ab8375a91312\") "
Sep 18 19:53:33.656031 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:33.655879    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c4a937f-37ce-4f81-b64e-ab8375a91312-istio-token\") pod \"9c4a937f-37ce-4f81-b64e-ab8375a91312\" (UID: \"9c4a937f-37ce-4f81-b64e-ab8375a91312\") "
Sep 18 19:53:33.656031 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:33.655917    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c4a937f-37ce-4f81-b64e-ab8375a91312-istiod-ca-cert\") pod \"9c4a937f-37ce-4f81-b64e-ab8375a91312\" (UID: \"9c4a937f-37ce-4f81-b64e-ab8375a91312\") "
Sep 18 19:53:33.656031 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:33.655938    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c4a937f-37ce-4f81-b64e-ab8375a91312-workload-socket\") pod \"9c4a937f-37ce-4f81-b64e-ab8375a91312\" (UID: \"9c4a937f-37ce-4f81-b64e-ab8375a91312\") "
Sep 18 19:53:33.656031 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:33.655964    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qv68p\" (UniqueName: \"kubernetes.io/projected/9c4a937f-37ce-4f81-b64e-ab8375a91312-kube-api-access-qv68p\") pod \"9c4a937f-37ce-4f81-b64e-ab8375a91312\" (UID: \"9c4a937f-37ce-4f81-b64e-ab8375a91312\") "
Sep 18 19:53:33.656031 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:33.656017    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c4a937f-37ce-4f81-b64e-ab8375a91312-istio-podinfo\") pod \"9c4a937f-37ce-4f81-b64e-ab8375a91312\" (UID: \"9c4a937f-37ce-4f81-b64e-ab8375a91312\") "
Sep 18 19:53:33.656273 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:33.656041    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c4a937f-37ce-4f81-b64e-ab8375a91312-istio-envoy\") pod \"9c4a937f-37ce-4f81-b64e-ab8375a91312\" (UID: \"9c4a937f-37ce-4f81-b64e-ab8375a91312\") "
Sep 18 19:53:33.656273 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:33.656095    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c4a937f-37ce-4f81-b64e-ab8375a91312-workload-certs\") pod \"9c4a937f-37ce-4f81-b64e-ab8375a91312\" (UID: \"9c4a937f-37ce-4f81-b64e-ab8375a91312\") "
Sep 18 19:53:33.656273 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:33.656164    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c4a937f-37ce-4f81-b64e-ab8375a91312-istio-data" (OuterVolumeSpecName: "istio-data") pod "9c4a937f-37ce-4f81-b64e-ab8375a91312" (UID: "9c4a937f-37ce-4f81-b64e-ab8375a91312"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:53:33.656438 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:33.656340    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c4a937f-37ce-4f81-b64e-ab8375a91312-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9c4a937f-37ce-4f81-b64e-ab8375a91312" (UID: "9c4a937f-37ce-4f81-b64e-ab8375a91312"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:53:33.656438 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:33.656381    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c4a937f-37ce-4f81-b64e-ab8375a91312-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9c4a937f-37ce-4f81-b64e-ab8375a91312" (UID: "9c4a937f-37ce-4f81-b64e-ab8375a91312"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:53:33.656537 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:33.656452    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c4a937f-37ce-4f81-b64e-ab8375a91312-workload-certs\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:53:33.656537 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:33.656470    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c4a937f-37ce-4f81-b64e-ab8375a91312-istio-data\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:53:33.656537 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:33.656482    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c4a937f-37ce-4f81-b64e-ab8375a91312-workload-socket\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:53:33.656686 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:33.656657    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c4a937f-37ce-4f81-b64e-ab8375a91312-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9c4a937f-37ce-4f81-b64e-ab8375a91312" (UID: "9c4a937f-37ce-4f81-b64e-ab8375a91312"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:53:33.656838 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:33.656801    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c4a937f-37ce-4f81-b64e-ab8375a91312-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9c4a937f-37ce-4f81-b64e-ab8375a91312" (UID: "9c4a937f-37ce-4f81-b64e-ab8375a91312"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:53:33.658366 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:33.658340    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c4a937f-37ce-4f81-b64e-ab8375a91312-istio-token" (OuterVolumeSpecName: "istio-token") pod "9c4a937f-37ce-4f81-b64e-ab8375a91312" (UID: "9c4a937f-37ce-4f81-b64e-ab8375a91312"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:53:33.658473 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:33.658442    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c4a937f-37ce-4f81-b64e-ab8375a91312-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9c4a937f-37ce-4f81-b64e-ab8375a91312" (UID: "9c4a937f-37ce-4f81-b64e-ab8375a91312"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:53:33.658654 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:33.658631    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c4a937f-37ce-4f81-b64e-ab8375a91312-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9c4a937f-37ce-4f81-b64e-ab8375a91312" (UID: "9c4a937f-37ce-4f81-b64e-ab8375a91312"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 19:53:33.658832 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:33.658812    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c4a937f-37ce-4f81-b64e-ab8375a91312-kube-api-access-qv68p" (OuterVolumeSpecName: "kube-api-access-qv68p") pod "9c4a937f-37ce-4f81-b64e-ab8375a91312" (UID: "9c4a937f-37ce-4f81-b64e-ab8375a91312"). InnerVolumeSpecName "kube-api-access-qv68p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:53:33.725881 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:33.725844    2577 generic.go:356] "Generic (PLEG): container finished" podID="9c4a937f-37ce-4f81-b64e-ab8375a91312" containerID="39a9c033da9383e8fd51fcfd1a98e62da2fd36a362b0845c8440540dc0325bbb" exitCode=0
Sep 18 19:53:33.726093 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:33.725888    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-k6jrj"
Sep 18 19:53:33.726093 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:33.725902    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-k6jrj" event={"ID":"9c4a937f-37ce-4f81-b64e-ab8375a91312","Type":"ContainerDied","Data":"39a9c033da9383e8fd51fcfd1a98e62da2fd36a362b0845c8440540dc0325bbb"}
Sep 18 19:53:33.726093 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:33.725954    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-k6jrj" event={"ID":"9c4a937f-37ce-4f81-b64e-ab8375a91312","Type":"ContainerDied","Data":"41756e99fcbc539c96c0979a5b3be3ba70d16975557fa682ceb225c47895bbdd"}
Sep 18 19:53:33.726093 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:33.725984    2577 scope.go:117] "RemoveContainer" containerID="39a9c033da9383e8fd51fcfd1a98e62da2fd36a362b0845c8440540dc0325bbb"
Sep 18 19:53:33.738694 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:33.738674    2577 scope.go:117] "RemoveContainer" containerID="3c2c877abfede1fb4fec4d29f05990e83fc6019797360cfc22cccc907b47ab06"
Sep 18 19:53:33.746584 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:33.746566    2577 scope.go:117] "RemoveContainer" containerID="39a9c033da9383e8fd51fcfd1a98e62da2fd36a362b0845c8440540dc0325bbb"
Sep 18 19:53:33.746876 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:53:33.746852    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39a9c033da9383e8fd51fcfd1a98e62da2fd36a362b0845c8440540dc0325bbb\": container with ID starting with 39a9c033da9383e8fd51fcfd1a98e62da2fd36a362b0845c8440540dc0325bbb not found: ID does not exist" containerID="39a9c033da9383e8fd51fcfd1a98e62da2fd36a362b0845c8440540dc0325bbb"
Sep 18 19:53:33.746952 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:33.746883    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39a9c033da9383e8fd51fcfd1a98e62da2fd36a362b0845c8440540dc0325bbb"} err="failed to get container status \"39a9c033da9383e8fd51fcfd1a98e62da2fd36a362b0845c8440540dc0325bbb\": rpc error: code = NotFound desc = could not find container \"39a9c033da9383e8fd51fcfd1a98e62da2fd36a362b0845c8440540dc0325bbb\": container with ID starting with 39a9c033da9383e8fd51fcfd1a98e62da2fd36a362b0845c8440540dc0325bbb not found: ID does not exist"
Sep 18 19:53:33.746952 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:33.746899    2577 scope.go:117] "RemoveContainer" containerID="3c2c877abfede1fb4fec4d29f05990e83fc6019797360cfc22cccc907b47ab06"
Sep 18 19:53:33.747166 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:53:33.747151    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c2c877abfede1fb4fec4d29f05990e83fc6019797360cfc22cccc907b47ab06\": container with ID starting with 3c2c877abfede1fb4fec4d29f05990e83fc6019797360cfc22cccc907b47ab06 not found: ID does not exist" containerID="3c2c877abfede1fb4fec4d29f05990e83fc6019797360cfc22cccc907b47ab06"
Sep 18 19:53:33.747228 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:33.747167    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c2c877abfede1fb4fec4d29f05990e83fc6019797360cfc22cccc907b47ab06"} err="failed to get container status \"3c2c877abfede1fb4fec4d29f05990e83fc6019797360cfc22cccc907b47ab06\": rpc error: code = NotFound desc = could not find container \"3c2c877abfede1fb4fec4d29f05990e83fc6019797360cfc22cccc907b47ab06\": container with ID starting with 3c2c877abfede1fb4fec4d29f05990e83fc6019797360cfc22cccc907b47ab06 not found: ID does not exist"
Sep 18 19:53:33.757335 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:33.757302    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c4a937f-37ce-4f81-b64e-ab8375a91312-credential-socket\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:53:33.757335 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:33.757326    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c4a937f-37ce-4f81-b64e-ab8375a91312-istio-token\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:53:33.757335 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:33.757336    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c4a937f-37ce-4f81-b64e-ab8375a91312-istiod-ca-cert\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:53:33.757576 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:33.757344    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv68p\" (UniqueName: \"kubernetes.io/projected/9c4a937f-37ce-4f81-b64e-ab8375a91312-kube-api-access-qv68p\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:53:33.757576 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:33.757352    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c4a937f-37ce-4f81-b64e-ab8375a91312-istio-podinfo\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:53:33.757576 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:33.757360    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c4a937f-37ce-4f81-b64e-ab8375a91312-istio-envoy\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:53:33.757794 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:33.757773    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-k6jrj"]
Sep 18 19:53:33.776211 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:33.776174    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-k6jrj"]
Sep 18 19:53:35.439420 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:35.439378    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c4a937f-37ce-4f81-b64e-ab8375a91312" path="/var/lib/kubelet/pods/9c4a937f-37ce-4f81-b64e-ab8375a91312/volumes"
Sep 18 19:53:37.177559 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:37.177522    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6"
Sep 18 19:53:37.178869 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:37.178840    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6" podUID="28e50adb-d3f3-4d10-ab28-ff9f09bf40da" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
Sep 18 19:53:47.178267 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:47.178222    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6" podUID="28e50adb-d3f3-4d10-ab28-ff9f09bf40da" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
Sep 18 19:53:57.177933 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:53:57.177888    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6" podUID="28e50adb-d3f3-4d10-ab28-ff9f09bf40da" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
Sep 18 19:54:02.107137 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:02.107059    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fj29c"]
Sep 18 19:54:02.107488 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:02.107394    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c4a937f-37ce-4f81-b64e-ab8375a91312" containerName="istio-proxy"
Sep 18 19:54:02.107488 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:02.107406    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c4a937f-37ce-4f81-b64e-ab8375a91312" containerName="istio-proxy"
Sep 18 19:54:02.107488 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:02.107422    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c4a937f-37ce-4f81-b64e-ab8375a91312" containerName="istio-proxy"
Sep 18 19:54:02.107488 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:02.107428    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c4a937f-37ce-4f81-b64e-ab8375a91312" containerName="istio-proxy"
Sep 18 19:54:02.107488 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:02.107478    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c4a937f-37ce-4f81-b64e-ab8375a91312" containerName="istio-proxy"
Sep 18 19:54:02.111739 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:02.111716    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fj29c"
Sep 18 19:54:02.118601 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:02.118571    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 18 19:54:02.119228 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:02.119206    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-ltr9b\""
Sep 18 19:54:02.119341 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:02.119227    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 18 19:54:02.119341 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:02.119246    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Sep 18 19:54:02.128353 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:02.128331    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fj29c"]
Sep 18 19:54:02.216489 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:02.216457    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb52b9a9-f757-4676-86d5-c9f0f51133ec-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fj29c\" (UID: \"cb52b9a9-f757-4676-86d5-c9f0f51133ec\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fj29c"
Sep 18 19:54:02.216489 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:02.216493    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q8zh\" (UniqueName: \"kubernetes.io/projected/cb52b9a9-f757-4676-86d5-c9f0f51133ec-kube-api-access-4q8zh\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fj29c\" (UID: \"cb52b9a9-f757-4676-86d5-c9f0f51133ec\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fj29c"
Sep 18 19:54:02.216741 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:02.216519    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb52b9a9-f757-4676-86d5-c9f0f51133ec-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fj29c\" (UID: \"cb52b9a9-f757-4676-86d5-c9f0f51133ec\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fj29c"
Sep 18 19:54:02.216741 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:02.216625    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb52b9a9-f757-4676-86d5-c9f0f51133ec-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fj29c\" (UID: \"cb52b9a9-f757-4676-86d5-c9f0f51133ec\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fj29c"
Sep 18 19:54:02.216826 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:02.216746    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb52b9a9-f757-4676-86d5-c9f0f51133ec-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fj29c\" (UID: \"cb52b9a9-f757-4676-86d5-c9f0f51133ec\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fj29c"
Sep 18 19:54:02.216826 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:02.216782    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb52b9a9-f757-4676-86d5-c9f0f51133ec-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fj29c\" (UID: \"cb52b9a9-f757-4676-86d5-c9f0f51133ec\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fj29c"
Sep 18 19:54:02.318165 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:02.318125    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb52b9a9-f757-4676-86d5-c9f0f51133ec-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fj29c\" (UID: \"cb52b9a9-f757-4676-86d5-c9f0f51133ec\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fj29c"
Sep 18 19:54:02.318373 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:02.318170    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q8zh\" (UniqueName: \"kubernetes.io/projected/cb52b9a9-f757-4676-86d5-c9f0f51133ec-kube-api-access-4q8zh\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fj29c\" (UID: \"cb52b9a9-f757-4676-86d5-c9f0f51133ec\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fj29c"
Sep 18 19:54:02.318373 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:02.318212    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb52b9a9-f757-4676-86d5-c9f0f51133ec-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fj29c\" (UID: \"cb52b9a9-f757-4676-86d5-c9f0f51133ec\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fj29c"
Sep 18 19:54:02.318373 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:02.318264    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb52b9a9-f757-4676-86d5-c9f0f51133ec-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fj29c\" (UID: \"cb52b9a9-f757-4676-86d5-c9f0f51133ec\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fj29c"
Sep 18 19:54:02.318373 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:02.318336    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb52b9a9-f757-4676-86d5-c9f0f51133ec-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fj29c\" (UID: \"cb52b9a9-f757-4676-86d5-c9f0f51133ec\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fj29c"
Sep 18 19:54:02.318373 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:02.318362    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb52b9a9-f757-4676-86d5-c9f0f51133ec-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fj29c\" (UID: \"cb52b9a9-f757-4676-86d5-c9f0f51133ec\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fj29c"
Sep 18 19:54:02.318804 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:02.318773    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb52b9a9-f757-4676-86d5-c9f0f51133ec-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fj29c\" (UID: \"cb52b9a9-f757-4676-86d5-c9f0f51133ec\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fj29c"
Sep 18 19:54:02.318916 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:02.318803    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb52b9a9-f757-4676-86d5-c9f0f51133ec-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fj29c\" (UID: \"cb52b9a9-f757-4676-86d5-c9f0f51133ec\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fj29c"
Sep 18 19:54:02.318916 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:02.318834    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb52b9a9-f757-4676-86d5-c9f0f51133ec-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fj29c\" (UID: \"cb52b9a9-f757-4676-86d5-c9f0f51133ec\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fj29c"
Sep 18 19:54:02.320576 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:02.320555    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb52b9a9-f757-4676-86d5-c9f0f51133ec-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fj29c\" (UID: \"cb52b9a9-f757-4676-86d5-c9f0f51133ec\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fj29c"
Sep 18 19:54:02.320847 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:02.320828    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb52b9a9-f757-4676-86d5-c9f0f51133ec-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fj29c\" (UID: \"cb52b9a9-f757-4676-86d5-c9f0f51133ec\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fj29c"
Sep 18 19:54:02.333165 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:02.333133    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q8zh\" (UniqueName: \"kubernetes.io/projected/cb52b9a9-f757-4676-86d5-c9f0f51133ec-kube-api-access-4q8zh\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fj29c\" (UID: \"cb52b9a9-f757-4676-86d5-c9f0f51133ec\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fj29c"
Sep 18 19:54:02.422065 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:02.422021    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fj29c"
Sep 18 19:54:02.705723 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:54:02.705693    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb52b9a9_f757_4676_86d5_c9f0f51133ec.slice/crio-3adbe0703877eecae9d8f36ec85aa84d4621d48512b5c2d40ab8da0c47e493e8 WatchSource:0}: Error finding container 3adbe0703877eecae9d8f36ec85aa84d4621d48512b5c2d40ab8da0c47e493e8: Status 404 returned error can't find the container with id 3adbe0703877eecae9d8f36ec85aa84d4621d48512b5c2d40ab8da0c47e493e8
Sep 18 19:54:02.713564 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:02.713537    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fj29c"]
Sep 18 19:54:02.863563 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:02.863516    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fj29c" event={"ID":"cb52b9a9-f757-4676-86d5-c9f0f51133ec","Type":"ContainerStarted","Data":"a5b01d2870620a7ca8b84a8daf6c2fcd0fddc9e6c32299144c9837dc66f516f7"}
Sep 18 19:54:02.863762 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:02.863582    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fj29c" event={"ID":"cb52b9a9-f757-4676-86d5-c9f0f51133ec","Type":"ContainerStarted","Data":"3adbe0703877eecae9d8f36ec85aa84d4621d48512b5c2d40ab8da0c47e493e8"}
Sep 18 19:54:02.893043 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:02.892968    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fj29c" podStartSLOduration=0.892950248 podStartE2EDuration="892.950248ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:54:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:54:02.889687483 +0000 UTC m=+2413.741987854" watchObservedRunningTime="2026-09-18 19:54:02.892950248 +0000 UTC m=+2413.745250617"
Sep 18 19:54:03.422412 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:03.422374    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fj29c"
Sep 18 19:54:07.177646 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:07.177592    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6" podUID="28e50adb-d3f3-4d10-ab28-ff9f09bf40da" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
Sep 18 19:54:12.423028 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:12.422972    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fj29c"
Sep 18 19:54:12.435653 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:12.435627    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fj29c"
Sep 18 19:54:12.926766 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:12.926732    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fj29c"
Sep 18 19:54:17.178325 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:17.178284    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6" podUID="28e50adb-d3f3-4d10-ab28-ff9f09bf40da" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
Sep 18 19:54:27.177892 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:27.177843    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6" podUID="28e50adb-d3f3-4d10-ab28-ff9f09bf40da" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
Sep 18 19:54:36.467424 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:36.467391    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fj29c"]
Sep 18 19:54:36.477581 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:54:36.477541    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 19:54:36.477581 ip-10-0-129-30 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:54:36.477581 ip-10-0-129-30 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 18 19:54:36.477581 ip-10-0-129-30 kubenswrapper[2577]:  > containerID="a5b01d2870620a7ca8b84a8daf6c2fcd0fddc9e6c32299144c9837dc66f516f7" cmd=["/bin/sleep","15"]
Sep 18 19:54:36.477871 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:36.477602    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fj29c" message="" err=<
Sep 18 19:54:36.477871 ip-10-0-129-30 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:54:36.477871 ip-10-0-129-30 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 18 19:54:36.477871 ip-10-0-129-30 kubenswrapper[2577]:  >
Sep 18 19:54:36.477871 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:54:36.477614    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 19:54:36.477871 ip-10-0-129-30 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:54:36.477871 ip-10-0-129-30 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 18 19:54:36.477871 ip-10-0-129-30 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fj29c" podUID="cb52b9a9-f757-4676-86d5-c9f0f51133ec" containerName="main" containerID="cri-o://a5b01d2870620a7ca8b84a8daf6c2fcd0fddc9e6c32299144c9837dc66f516f7"
Sep 18 19:54:36.477871 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:36.477648    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fj29c" podUID="cb52b9a9-f757-4676-86d5-c9f0f51133ec" containerName="main" containerID="cri-o://a5b01d2870620a7ca8b84a8daf6c2fcd0fddc9e6c32299144c9837dc66f516f7" gracePeriod=60
Sep 18 19:54:36.825196 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:36.825172    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fj29c"
Sep 18 19:54:36.952847 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:36.952817    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb52b9a9-f757-4676-86d5-c9f0f51133ec-home\") pod \"cb52b9a9-f757-4676-86d5-c9f0f51133ec\" (UID: \"cb52b9a9-f757-4676-86d5-c9f0f51133ec\") "
Sep 18 19:54:36.953049 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:36.952876    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb52b9a9-f757-4676-86d5-c9f0f51133ec-tls-certs\") pod \"cb52b9a9-f757-4676-86d5-c9f0f51133ec\" (UID: \"cb52b9a9-f757-4676-86d5-c9f0f51133ec\") "
Sep 18 19:54:36.953049 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:36.952912    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb52b9a9-f757-4676-86d5-c9f0f51133ec-dshm\") pod \"cb52b9a9-f757-4676-86d5-c9f0f51133ec\" (UID: \"cb52b9a9-f757-4676-86d5-c9f0f51133ec\") "
Sep 18 19:54:36.953049 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:36.952980    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb52b9a9-f757-4676-86d5-c9f0f51133ec-model-cache\") pod \"cb52b9a9-f757-4676-86d5-c9f0f51133ec\" (UID: \"cb52b9a9-f757-4676-86d5-c9f0f51133ec\") "
Sep 18 19:54:36.953313 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:36.953057    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb52b9a9-f757-4676-86d5-c9f0f51133ec-tmp-dir\") pod \"cb52b9a9-f757-4676-86d5-c9f0f51133ec\" (UID: \"cb52b9a9-f757-4676-86d5-c9f0f51133ec\") "
Sep 18 19:54:36.953313 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:36.953089    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4q8zh\" (UniqueName: \"kubernetes.io/projected/cb52b9a9-f757-4676-86d5-c9f0f51133ec-kube-api-access-4q8zh\") pod \"cb52b9a9-f757-4676-86d5-c9f0f51133ec\" (UID: \"cb52b9a9-f757-4676-86d5-c9f0f51133ec\") "
Sep 18 19:54:36.953313 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:36.953109    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb52b9a9-f757-4676-86d5-c9f0f51133ec-home" (OuterVolumeSpecName: "home") pod "cb52b9a9-f757-4676-86d5-c9f0f51133ec" (UID: "cb52b9a9-f757-4676-86d5-c9f0f51133ec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:54:36.953470 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:36.953316    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb52b9a9-f757-4676-86d5-c9f0f51133ec-model-cache" (OuterVolumeSpecName: "model-cache") pod "cb52b9a9-f757-4676-86d5-c9f0f51133ec" (UID: "cb52b9a9-f757-4676-86d5-c9f0f51133ec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:54:36.953470 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:36.953413    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb52b9a9-f757-4676-86d5-c9f0f51133ec-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cb52b9a9-f757-4676-86d5-c9f0f51133ec" (UID: "cb52b9a9-f757-4676-86d5-c9f0f51133ec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:54:36.953578 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:36.953471    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb52b9a9-f757-4676-86d5-c9f0f51133ec-home\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:54:36.953578 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:36.953490    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb52b9a9-f757-4676-86d5-c9f0f51133ec-model-cache\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:54:36.955362 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:36.955327    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb52b9a9-f757-4676-86d5-c9f0f51133ec-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cb52b9a9-f757-4676-86d5-c9f0f51133ec" (UID: "cb52b9a9-f757-4676-86d5-c9f0f51133ec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:54:36.955498 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:36.955363    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb52b9a9-f757-4676-86d5-c9f0f51133ec-kube-api-access-4q8zh" (OuterVolumeSpecName: "kube-api-access-4q8zh") pod "cb52b9a9-f757-4676-86d5-c9f0f51133ec" (UID: "cb52b9a9-f757-4676-86d5-c9f0f51133ec"). InnerVolumeSpecName "kube-api-access-4q8zh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:54:36.955498 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:36.955477    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb52b9a9-f757-4676-86d5-c9f0f51133ec-dshm" (OuterVolumeSpecName: "dshm") pod "cb52b9a9-f757-4676-86d5-c9f0f51133ec" (UID: "cb52b9a9-f757-4676-86d5-c9f0f51133ec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:54:37.046135 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:37.046045    2577 generic.go:356] "Generic (PLEG): container finished" podID="cb52b9a9-f757-4676-86d5-c9f0f51133ec" containerID="a5b01d2870620a7ca8b84a8daf6c2fcd0fddc9e6c32299144c9837dc66f516f7" exitCode=0
Sep 18 19:54:37.046135 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:37.046090    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fj29c"
Sep 18 19:54:37.046321 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:37.046130    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fj29c" event={"ID":"cb52b9a9-f757-4676-86d5-c9f0f51133ec","Type":"ContainerDied","Data":"a5b01d2870620a7ca8b84a8daf6c2fcd0fddc9e6c32299144c9837dc66f516f7"}
Sep 18 19:54:37.046321 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:37.046188    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fj29c" event={"ID":"cb52b9a9-f757-4676-86d5-c9f0f51133ec","Type":"ContainerDied","Data":"3adbe0703877eecae9d8f36ec85aa84d4621d48512b5c2d40ab8da0c47e493e8"}
Sep 18 19:54:37.046321 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:37.046217    2577 scope.go:117] "RemoveContainer" containerID="a5b01d2870620a7ca8b84a8daf6c2fcd0fddc9e6c32299144c9837dc66f516f7"
Sep 18 19:54:37.054513 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:37.054491    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb52b9a9-f757-4676-86d5-c9f0f51133ec-tmp-dir\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:54:37.054513 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:37.054515    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4q8zh\" (UniqueName: \"kubernetes.io/projected/cb52b9a9-f757-4676-86d5-c9f0f51133ec-kube-api-access-4q8zh\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:54:37.054687 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:37.054524    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb52b9a9-f757-4676-86d5-c9f0f51133ec-tls-certs\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:54:37.054687 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:37.054533    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb52b9a9-f757-4676-86d5-c9f0f51133ec-dshm\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:54:37.057168 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:37.057151    2577 scope.go:117] "RemoveContainer" containerID="a5b01d2870620a7ca8b84a8daf6c2fcd0fddc9e6c32299144c9837dc66f516f7"
Sep 18 19:54:37.057474 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:54:37.057452    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5b01d2870620a7ca8b84a8daf6c2fcd0fddc9e6c32299144c9837dc66f516f7\": container with ID starting with a5b01d2870620a7ca8b84a8daf6c2fcd0fddc9e6c32299144c9837dc66f516f7 not found: ID does not exist" containerID="a5b01d2870620a7ca8b84a8daf6c2fcd0fddc9e6c32299144c9837dc66f516f7"
Sep 18 19:54:37.057571 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:37.057481    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5b01d2870620a7ca8b84a8daf6c2fcd0fddc9e6c32299144c9837dc66f516f7"} err="failed to get container status \"a5b01d2870620a7ca8b84a8daf6c2fcd0fddc9e6c32299144c9837dc66f516f7\": rpc error: code = NotFound desc = could not find container \"a5b01d2870620a7ca8b84a8daf6c2fcd0fddc9e6c32299144c9837dc66f516f7\": container with ID starting with a5b01d2870620a7ca8b84a8daf6c2fcd0fddc9e6c32299144c9837dc66f516f7 not found: ID does not exist"
Sep 18 19:54:37.070121 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:37.070095    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fj29c"]
Sep 18 19:54:37.080478 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:37.080456    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fj29c"]
Sep 18 19:54:37.177813 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:37.177766    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6" podUID="28e50adb-d3f3-4d10-ab28-ff9f09bf40da" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
Sep 18 19:54:37.439516 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:37.439482    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb52b9a9-f757-4676-86d5-c9f0f51133ec" path="/var/lib/kubelet/pods/cb52b9a9-f757-4676-86d5-c9f0f51133ec/volumes"
Sep 18 19:54:47.178437 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:47.178380    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6" podUID="28e50adb-d3f3-4d10-ab28-ff9f09bf40da" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
Sep 18 19:54:57.177931 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:54:57.177884    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6" podUID="28e50adb-d3f3-4d10-ab28-ff9f09bf40da" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
Sep 18 19:55:07.177574 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:55:07.177521    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6" podUID="28e50adb-d3f3-4d10-ab28-ff9f09bf40da" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
Sep 18 19:55:17.178212 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:55:17.178160    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6" podUID="28e50adb-d3f3-4d10-ab28-ff9f09bf40da" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
Sep 18 19:55:27.194678 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:55:27.194637    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6"
Sep 18 19:55:27.206142 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:55:27.206109    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6"
Sep 18 19:55:33.037385 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:55:33.037302    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6"]
Sep 18 19:55:48.059653 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:55:48.059604    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6" podUID="28e50adb-d3f3-4d10-ab28-ff9f09bf40da" containerName="main" containerID="cri-o://86d66ffb6f63f63465ac66d2f1ea50c6e3ed0fdfb44295e5b50eb5821215cce4" gracePeriod=45
Sep 18 19:55:48.195722 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:55:48.195676    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6" podUID="28e50adb-d3f3-4d10-ab28-ff9f09bf40da" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
Sep 18 19:55:48.808433 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:55:48.808395    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6"
Sep 18 19:55:48.915928 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:55:48.915892    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28e50adb-d3f3-4d10-ab28-ff9f09bf40da-model-cache\") pod \"28e50adb-d3f3-4d10-ab28-ff9f09bf40da\" (UID: \"28e50adb-d3f3-4d10-ab28-ff9f09bf40da\") "
Sep 18 19:55:48.915928 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:55:48.915934    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28e50adb-d3f3-4d10-ab28-ff9f09bf40da-home\") pod \"28e50adb-d3f3-4d10-ab28-ff9f09bf40da\" (UID: \"28e50adb-d3f3-4d10-ab28-ff9f09bf40da\") "
Sep 18 19:55:48.916229 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:55:48.915969    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28e50adb-d3f3-4d10-ab28-ff9f09bf40da-tls-certs\") pod \"28e50adb-d3f3-4d10-ab28-ff9f09bf40da\" (UID: \"28e50adb-d3f3-4d10-ab28-ff9f09bf40da\") "
Sep 18 19:55:48.916229 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:55:48.916019    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/28e50adb-d3f3-4d10-ab28-ff9f09bf40da-cabundle-cert\") pod \"28e50adb-d3f3-4d10-ab28-ff9f09bf40da\" (UID: \"28e50adb-d3f3-4d10-ab28-ff9f09bf40da\") "
Sep 18 19:55:48.916229 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:55:48.916076    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28e50adb-d3f3-4d10-ab28-ff9f09bf40da-kserve-provision-location\") pod \"28e50adb-d3f3-4d10-ab28-ff9f09bf40da\" (UID: \"28e50adb-d3f3-4d10-ab28-ff9f09bf40da\") "
Sep 18 19:55:48.916229 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:55:48.916095    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ct9sx\" (UniqueName: \"kubernetes.io/projected/28e50adb-d3f3-4d10-ab28-ff9f09bf40da-kube-api-access-ct9sx\") pod \"28e50adb-d3f3-4d10-ab28-ff9f09bf40da\" (UID: \"28e50adb-d3f3-4d10-ab28-ff9f09bf40da\") "
Sep 18 19:55:48.916229 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:55:48.916133    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28e50adb-d3f3-4d10-ab28-ff9f09bf40da-dshm\") pod \"28e50adb-d3f3-4d10-ab28-ff9f09bf40da\" (UID: \"28e50adb-d3f3-4d10-ab28-ff9f09bf40da\") "
Sep 18 19:55:48.916229 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:55:48.916164    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28e50adb-d3f3-4d10-ab28-ff9f09bf40da-tmp-dir\") pod \"28e50adb-d3f3-4d10-ab28-ff9f09bf40da\" (UID: \"28e50adb-d3f3-4d10-ab28-ff9f09bf40da\") "
Sep 18 19:55:48.916551 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:55:48.916451    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28e50adb-d3f3-4d10-ab28-ff9f09bf40da-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "28e50adb-d3f3-4d10-ab28-ff9f09bf40da" (UID: "28e50adb-d3f3-4d10-ab28-ff9f09bf40da"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:55:48.916666 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:55:48.916620    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28e50adb-d3f3-4d10-ab28-ff9f09bf40da-model-cache" (OuterVolumeSpecName: "model-cache") pod "28e50adb-d3f3-4d10-ab28-ff9f09bf40da" (UID: "28e50adb-d3f3-4d10-ab28-ff9f09bf40da"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:55:48.916800 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:55:48.916702    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28e50adb-d3f3-4d10-ab28-ff9f09bf40da-home" (OuterVolumeSpecName: "home") pod "28e50adb-d3f3-4d10-ab28-ff9f09bf40da" (UID: "28e50adb-d3f3-4d10-ab28-ff9f09bf40da"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:55:48.918734 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:55:48.918616    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28e50adb-d3f3-4d10-ab28-ff9f09bf40da-dshm" (OuterVolumeSpecName: "dshm") pod "28e50adb-d3f3-4d10-ab28-ff9f09bf40da" (UID: "28e50adb-d3f3-4d10-ab28-ff9f09bf40da"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:55:48.918831 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:55:48.918696    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28e50adb-d3f3-4d10-ab28-ff9f09bf40da-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "28e50adb-d3f3-4d10-ab28-ff9f09bf40da" (UID: "28e50adb-d3f3-4d10-ab28-ff9f09bf40da"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:55:48.918831 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:55:48.918698    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28e50adb-d3f3-4d10-ab28-ff9f09bf40da-kube-api-access-ct9sx" (OuterVolumeSpecName: "kube-api-access-ct9sx") pod "28e50adb-d3f3-4d10-ab28-ff9f09bf40da" (UID: "28e50adb-d3f3-4d10-ab28-ff9f09bf40da"). InnerVolumeSpecName "kube-api-access-ct9sx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:55:48.934695 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:55:48.934662    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28e50adb-d3f3-4d10-ab28-ff9f09bf40da-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "28e50adb-d3f3-4d10-ab28-ff9f09bf40da" (UID: "28e50adb-d3f3-4d10-ab28-ff9f09bf40da"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:55:48.944794 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:55:48.944761    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28e50adb-d3f3-4d10-ab28-ff9f09bf40da-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "28e50adb-d3f3-4d10-ab28-ff9f09bf40da" (UID: "28e50adb-d3f3-4d10-ab28-ff9f09bf40da"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:55:49.016908 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:55:49.016875    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28e50adb-d3f3-4d10-ab28-ff9f09bf40da-kserve-provision-location\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:55:49.016908 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:55:49.016906    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ct9sx\" (UniqueName: \"kubernetes.io/projected/28e50adb-d3f3-4d10-ab28-ff9f09bf40da-kube-api-access-ct9sx\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:55:49.016908 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:55:49.016915    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28e50adb-d3f3-4d10-ab28-ff9f09bf40da-dshm\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:55:49.017166 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:55:49.016926    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28e50adb-d3f3-4d10-ab28-ff9f09bf40da-tmp-dir\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:55:49.017166 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:55:49.016935    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28e50adb-d3f3-4d10-ab28-ff9f09bf40da-model-cache\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:55:49.017166 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:55:49.016943    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28e50adb-d3f3-4d10-ab28-ff9f09bf40da-home\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:55:49.017166 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:55:49.016950    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28e50adb-d3f3-4d10-ab28-ff9f09bf40da-tls-certs\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:55:49.017166 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:55:49.016961    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/28e50adb-d3f3-4d10-ab28-ff9f09bf40da-cabundle-cert\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:55:49.379766 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:55:49.379728    2577 generic.go:356] "Generic (PLEG): container finished" podID="28e50adb-d3f3-4d10-ab28-ff9f09bf40da" containerID="86d66ffb6f63f63465ac66d2f1ea50c6e3ed0fdfb44295e5b50eb5821215cce4" exitCode=0
Sep 18 19:55:49.380342 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:55:49.379789    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6" event={"ID":"28e50adb-d3f3-4d10-ab28-ff9f09bf40da","Type":"ContainerDied","Data":"86d66ffb6f63f63465ac66d2f1ea50c6e3ed0fdfb44295e5b50eb5821215cce4"}
Sep 18 19:55:49.380342 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:55:49.379803    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6"
Sep 18 19:55:49.380342 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:55:49.379872    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6" event={"ID":"28e50adb-d3f3-4d10-ab28-ff9f09bf40da","Type":"ContainerDied","Data":"d0783876e6e68d34243129d07b0d279ebd24b37d35b83716472f05b2bc7d483a"}
Sep 18 19:55:49.380342 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:55:49.379906    2577 scope.go:117] "RemoveContainer" containerID="86d66ffb6f63f63465ac66d2f1ea50c6e3ed0fdfb44295e5b50eb5821215cce4"
Sep 18 19:55:49.389170 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:55:49.389149    2577 scope.go:117] "RemoveContainer" containerID="6ca7d5c7f32c5a0ba32750ae21e68159991b670fd25899d38dfdc9156de7ea4f"
Sep 18 19:55:49.396605 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:55:49.396585    2577 scope.go:117] "RemoveContainer" containerID="86d66ffb6f63f63465ac66d2f1ea50c6e3ed0fdfb44295e5b50eb5821215cce4"
Sep 18 19:55:49.396877 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:55:49.396857    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86d66ffb6f63f63465ac66d2f1ea50c6e3ed0fdfb44295e5b50eb5821215cce4\": container with ID starting with 86d66ffb6f63f63465ac66d2f1ea50c6e3ed0fdfb44295e5b50eb5821215cce4 not found: ID does not exist" containerID="86d66ffb6f63f63465ac66d2f1ea50c6e3ed0fdfb44295e5b50eb5821215cce4"
Sep 18 19:55:49.396927 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:55:49.396887    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86d66ffb6f63f63465ac66d2f1ea50c6e3ed0fdfb44295e5b50eb5821215cce4"} err="failed to get container status \"86d66ffb6f63f63465ac66d2f1ea50c6e3ed0fdfb44295e5b50eb5821215cce4\": rpc error: code = NotFound desc = could not find container \"86d66ffb6f63f63465ac66d2f1ea50c6e3ed0fdfb44295e5b50eb5821215cce4\": container with ID starting with 86d66ffb6f63f63465ac66d2f1ea50c6e3ed0fdfb44295e5b50eb5821215cce4 not found: ID does not exist"
Sep 18 19:55:49.396927 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:55:49.396904    2577 scope.go:117] "RemoveContainer" containerID="6ca7d5c7f32c5a0ba32750ae21e68159991b670fd25899d38dfdc9156de7ea4f"
Sep 18 19:55:49.397170 ip-10-0-129-30 kubenswrapper[2577]: E0918 19:55:49.397151    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ca7d5c7f32c5a0ba32750ae21e68159991b670fd25899d38dfdc9156de7ea4f\": container with ID starting with 6ca7d5c7f32c5a0ba32750ae21e68159991b670fd25899d38dfdc9156de7ea4f not found: ID does not exist" containerID="6ca7d5c7f32c5a0ba32750ae21e68159991b670fd25899d38dfdc9156de7ea4f"
Sep 18 19:55:49.397214 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:55:49.397176    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ca7d5c7f32c5a0ba32750ae21e68159991b670fd25899d38dfdc9156de7ea4f"} err="failed to get container status \"6ca7d5c7f32c5a0ba32750ae21e68159991b670fd25899d38dfdc9156de7ea4f\": rpc error: code = NotFound desc = could not find container \"6ca7d5c7f32c5a0ba32750ae21e68159991b670fd25899d38dfdc9156de7ea4f\": container with ID starting with 6ca7d5c7f32c5a0ba32750ae21e68159991b670fd25899d38dfdc9156de7ea4f not found: ID does not exist"
Sep 18 19:55:49.403934 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:55:49.403910    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6"]
Sep 18 19:55:49.408073 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:55:49.408051    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d899cf9d9sd6j6"]
Sep 18 19:55:49.438585 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:55:49.438554    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28e50adb-d3f3-4d10-ab28-ff9f09bf40da" path="/var/lib/kubelet/pods/28e50adb-d3f3-4d10-ab28-ff9f09bf40da/volumes"
Sep 18 19:56:45.408225 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:56:45.408189    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk"]
Sep 18 19:56:45.408705 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:56:45.408526    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28e50adb-d3f3-4d10-ab28-ff9f09bf40da" containerName="main"
Sep 18 19:56:45.408705 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:56:45.408536    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="28e50adb-d3f3-4d10-ab28-ff9f09bf40da" containerName="main"
Sep 18 19:56:45.408705 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:56:45.408543    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb52b9a9-f757-4676-86d5-c9f0f51133ec" containerName="main"
Sep 18 19:56:45.408705 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:56:45.408548    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb52b9a9-f757-4676-86d5-c9f0f51133ec" containerName="main"
Sep 18 19:56:45.408705 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:56:45.408558    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28e50adb-d3f3-4d10-ab28-ff9f09bf40da" containerName="storage-initializer"
Sep 18 19:56:45.408705 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:56:45.408564    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="28e50adb-d3f3-4d10-ab28-ff9f09bf40da" containerName="storage-initializer"
Sep 18 19:56:45.408705 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:56:45.408620    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c4a937f-37ce-4f81-b64e-ab8375a91312" containerName="istio-proxy"
Sep 18 19:56:45.408705 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:56:45.408627    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb52b9a9-f757-4676-86d5-c9f0f51133ec" containerName="main"
Sep 18 19:56:45.408705 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:56:45.408633    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="28e50adb-d3f3-4d10-ab28-ff9f09bf40da" containerName="main"
Sep 18 19:56:45.412078 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:56:45.412059    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk"
Sep 18 19:56:45.414963 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:56:45.414943    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Sep 18 19:56:45.415107 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:56:45.414953    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-r5kzs\""
Sep 18 19:56:45.415107 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:56:45.415089    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Sep 18 19:56:45.415374 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:56:45.415356    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Sep 18 19:56:45.426821 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:56:45.426800    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk"]
Sep 18 19:56:45.521869 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:56:45.521836    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wh8b\" (UniqueName: \"kubernetes.io/projected/59cebc93-92ca-400d-9343-a505176cdc79-kube-api-access-4wh8b\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk\" (UID: \"59cebc93-92ca-400d-9343-a505176cdc79\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk"
Sep 18 19:56:45.522063 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:56:45.521883    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59cebc93-92ca-400d-9343-a505176cdc79-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk\" (UID: \"59cebc93-92ca-400d-9343-a505176cdc79\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk"
Sep 18 19:56:45.522063 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:56:45.521932    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59cebc93-92ca-400d-9343-a505176cdc79-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk\" (UID: \"59cebc93-92ca-400d-9343-a505176cdc79\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk"
Sep 18 19:56:45.522063 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:56:45.522025    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59cebc93-92ca-400d-9343-a505176cdc79-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk\" (UID: \"59cebc93-92ca-400d-9343-a505176cdc79\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk"
Sep 18 19:56:45.522192 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:56:45.522100    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59cebc93-92ca-400d-9343-a505176cdc79-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk\" (UID: \"59cebc93-92ca-400d-9343-a505176cdc79\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk"
Sep 18 19:56:45.522192 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:56:45.522132    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59cebc93-92ca-400d-9343-a505176cdc79-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk\" (UID: \"59cebc93-92ca-400d-9343-a505176cdc79\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk"
Sep 18 19:56:45.522192 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:56:45.522150    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59cebc93-92ca-400d-9343-a505176cdc79-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk\" (UID: \"59cebc93-92ca-400d-9343-a505176cdc79\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk"
Sep 18 19:56:45.623328 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:56:45.623294    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59cebc93-92ca-400d-9343-a505176cdc79-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk\" (UID: \"59cebc93-92ca-400d-9343-a505176cdc79\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk"
Sep 18 19:56:45.623328 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:56:45.623330    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59cebc93-92ca-400d-9343-a505176cdc79-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk\" (UID: \"59cebc93-92ca-400d-9343-a505176cdc79\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk"
Sep 18 19:56:45.623601 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:56:45.623360    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wh8b\" (UniqueName: \"kubernetes.io/projected/59cebc93-92ca-400d-9343-a505176cdc79-kube-api-access-4wh8b\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk\" (UID: \"59cebc93-92ca-400d-9343-a505176cdc79\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk"
Sep 18 19:56:45.623601 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:56:45.623392    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59cebc93-92ca-400d-9343-a505176cdc79-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk\" (UID: \"59cebc93-92ca-400d-9343-a505176cdc79\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk"
Sep 18 19:56:45.623601 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:56:45.623416    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59cebc93-92ca-400d-9343-a505176cdc79-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk\" (UID: \"59cebc93-92ca-400d-9343-a505176cdc79\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk"
Sep 18 19:56:45.623601 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:56:45.623434    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59cebc93-92ca-400d-9343-a505176cdc79-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk\" (UID: \"59cebc93-92ca-400d-9343-a505176cdc79\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk"
Sep 18 19:56:45.623601 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:56:45.623483    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59cebc93-92ca-400d-9343-a505176cdc79-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk\" (UID: \"59cebc93-92ca-400d-9343-a505176cdc79\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk"
Sep 18 19:56:45.623868 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:56:45.623762    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59cebc93-92ca-400d-9343-a505176cdc79-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk\" (UID: \"59cebc93-92ca-400d-9343-a505176cdc79\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk"
Sep 18 19:56:45.623868 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:56:45.623803    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59cebc93-92ca-400d-9343-a505176cdc79-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk\" (UID: \"59cebc93-92ca-400d-9343-a505176cdc79\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk"
Sep 18 19:56:45.623868 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:56:45.623834    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59cebc93-92ca-400d-9343-a505176cdc79-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk\" (UID: \"59cebc93-92ca-400d-9343-a505176cdc79\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk"
Sep 18 19:56:45.623979 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:56:45.623941    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59cebc93-92ca-400d-9343-a505176cdc79-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk\" (UID: \"59cebc93-92ca-400d-9343-a505176cdc79\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk"
Sep 18 19:56:45.625744 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:56:45.625715    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59cebc93-92ca-400d-9343-a505176cdc79-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk\" (UID: \"59cebc93-92ca-400d-9343-a505176cdc79\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk"
Sep 18 19:56:45.626141 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:56:45.626121    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59cebc93-92ca-400d-9343-a505176cdc79-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk\" (UID: \"59cebc93-92ca-400d-9343-a505176cdc79\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk"
Sep 18 19:56:45.633238 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:56:45.633219    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wh8b\" (UniqueName: \"kubernetes.io/projected/59cebc93-92ca-400d-9343-a505176cdc79-kube-api-access-4wh8b\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk\" (UID: \"59cebc93-92ca-400d-9343-a505176cdc79\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk"
Sep 18 19:56:45.721656 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:56:45.721564    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk"
Sep 18 19:56:46.001620 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:56:46.001594    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk"]
Sep 18 19:56:46.044716 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:56:46.044697    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:56:46.625783 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:56:46.625738    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk" event={"ID":"59cebc93-92ca-400d-9343-a505176cdc79","Type":"ContainerStarted","Data":"c3c8fce7a8ec9fa3b3d54c1503b6becf5ba38cf1ce307b27c8031583d8c177f9"}
Sep 18 19:56:52.661220 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:56:52.661181    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk" event={"ID":"59cebc93-92ca-400d-9343-a505176cdc79","Type":"ContainerStarted","Data":"28f1707b8a65fc8f6197ef620edd0fbf598a948577c508b9ef169e9ab490ff4f"}
Sep 18 19:56:53.667687 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:56:53.667654    2577 generic.go:356] "Generic (PLEG): container finished" podID="59cebc93-92ca-400d-9343-a505176cdc79" containerID="28f1707b8a65fc8f6197ef620edd0fbf598a948577c508b9ef169e9ab490ff4f" exitCode=0
Sep 18 19:56:53.668159 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:56:53.667726    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk" event={"ID":"59cebc93-92ca-400d-9343-a505176cdc79","Type":"ContainerDied","Data":"28f1707b8a65fc8f6197ef620edd0fbf598a948577c508b9ef169e9ab490ff4f"}
Sep 18 19:56:54.674388 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:56:54.674353    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk" event={"ID":"59cebc93-92ca-400d-9343-a505176cdc79","Type":"ContainerStarted","Data":"4db2a2200457d3887656f278572c9e9ea0b371d044d462a8fc23de67c908f1af"}
Sep 18 19:56:54.674783 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:56:54.674412    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk" event={"ID":"59cebc93-92ca-400d-9343-a505176cdc79","Type":"ContainerStarted","Data":"71a94759ad1e4fc006cd2f755ac31d49d7420085e92138070ba5d15a100e9b2a"}
Sep 18 19:56:54.705010 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:56:54.704936    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk" podStartSLOduration=2.405953812 podStartE2EDuration="9.704916433s" totalImagesPullingTime="6.112792797s" totalInitContainerRuntime="1.186169824s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:56:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:56:54.703643262 +0000 UTC m=+2585.555943632" watchObservedRunningTime="2026-09-18 19:56:54.704916433 +0000 UTC m=+2585.557216803"
Sep 18 19:56:54.721794 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:56:54.721749    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk"
Sep 18 19:56:55.721819 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:56:55.721766    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk"
Sep 18 19:56:55.723259 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:56:55.723226    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk" podUID="59cebc93-92ca-400d-9343-a505176cdc79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Sep 18 19:57:05.722908 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:57:05.722800    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk" podUID="59cebc93-92ca-400d-9343-a505176cdc79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Sep 18 19:57:15.722553 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:57:15.722489    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk" podUID="59cebc93-92ca-400d-9343-a505176cdc79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Sep 18 19:57:25.722517 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:57:25.722473    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk" podUID="59cebc93-92ca-400d-9343-a505176cdc79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Sep 18 19:57:35.722200 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:57:35.722154    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk" podUID="59cebc93-92ca-400d-9343-a505176cdc79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Sep 18 19:57:45.722715 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:57:45.722665    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk" podUID="59cebc93-92ca-400d-9343-a505176cdc79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Sep 18 19:57:55.722463 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:57:55.722404    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk" podUID="59cebc93-92ca-400d-9343-a505176cdc79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Sep 18 19:58:05.722506 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:58:05.722459    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk" podUID="59cebc93-92ca-400d-9343-a505176cdc79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Sep 18 19:58:15.722568 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:58:15.722520    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk" podUID="59cebc93-92ca-400d-9343-a505176cdc79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Sep 18 19:58:25.722379 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:58:25.722334    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk" podUID="59cebc93-92ca-400d-9343-a505176cdc79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Sep 18 19:58:35.723009 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:58:35.722934    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk" podUID="59cebc93-92ca-400d-9343-a505176cdc79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Sep 18 19:58:45.732460 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:58:45.732428    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk"
Sep 18 19:58:45.740407 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:58:45.740384    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk"
Sep 18 19:58:50.113949 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:58:50.113899    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk"]
Sep 18 19:58:50.114406 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:58:50.114181    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk" podUID="59cebc93-92ca-400d-9343-a505176cdc79" containerName="modelcar" containerID="cri-o://4db2a2200457d3887656f278572c9e9ea0b371d044d462a8fc23de67c908f1af" gracePeriod=60
Sep 18 19:58:51.217300 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:58:51.217273    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk_59cebc93-92ca-400d-9343-a505176cdc79/modelcar/0.log"
Sep 18 19:58:51.217907 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:58:51.217887    2577 generic.go:356] "Generic (PLEG): container finished" podID="59cebc93-92ca-400d-9343-a505176cdc79" containerID="4db2a2200457d3887656f278572c9e9ea0b371d044d462a8fc23de67c908f1af" exitCode=143
Sep 18 19:58:51.218017 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:58:51.217966    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk" event={"ID":"59cebc93-92ca-400d-9343-a505176cdc79","Type":"ContainerDied","Data":"4db2a2200457d3887656f278572c9e9ea0b371d044d462a8fc23de67c908f1af"}
Sep 18 19:58:51.557459 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:58:51.557380    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk_59cebc93-92ca-400d-9343-a505176cdc79/modelcar/0.log"
Sep 18 19:58:51.566560 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:58:51.566533    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk_59cebc93-92ca-400d-9343-a505176cdc79/modelcar/0.log"
Sep 18 19:59:05.136629 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:05.136562    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk" podUID="59cebc93-92ca-400d-9343-a505176cdc79" containerName="main" containerID="cri-o://71a94759ad1e4fc006cd2f755ac31d49d7420085e92138070ba5d15a100e9b2a" gracePeriod=45
Sep 18 19:59:05.733302 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:05.733261    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk" podUID="59cebc93-92ca-400d-9343-a505176cdc79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Sep 18 19:59:06.289688 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:06.289659    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk_59cebc93-92ca-400d-9343-a505176cdc79/modelcar/0.log"
Sep 18 19:59:06.290434 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:06.290409    2577 generic.go:356] "Generic (PLEG): container finished" podID="59cebc93-92ca-400d-9343-a505176cdc79" containerID="71a94759ad1e4fc006cd2f755ac31d49d7420085e92138070ba5d15a100e9b2a" exitCode=0
Sep 18 19:59:06.290530 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:06.290462    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk" event={"ID":"59cebc93-92ca-400d-9343-a505176cdc79","Type":"ContainerDied","Data":"71a94759ad1e4fc006cd2f755ac31d49d7420085e92138070ba5d15a100e9b2a"}
Sep 18 19:59:06.290530 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:06.290507    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk" event={"ID":"59cebc93-92ca-400d-9343-a505176cdc79","Type":"ContainerDied","Data":"c3c8fce7a8ec9fa3b3d54c1503b6becf5ba38cf1ce307b27c8031583d8c177f9"}
Sep 18 19:59:06.290530 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:06.290529    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3c8fce7a8ec9fa3b3d54c1503b6becf5ba38cf1ce307b27c8031583d8c177f9"
Sep 18 19:59:06.382839 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:06.382818    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk_59cebc93-92ca-400d-9343-a505176cdc79/modelcar/0.log"
Sep 18 19:59:06.385506 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:06.385488    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk"
Sep 18 19:59:06.415515 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:06.415417    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59cebc93-92ca-400d-9343-a505176cdc79-home\") pod \"59cebc93-92ca-400d-9343-a505176cdc79\" (UID: \"59cebc93-92ca-400d-9343-a505176cdc79\") "
Sep 18 19:59:06.415515 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:06.415458    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59cebc93-92ca-400d-9343-a505176cdc79-dshm\") pod \"59cebc93-92ca-400d-9343-a505176cdc79\" (UID: \"59cebc93-92ca-400d-9343-a505176cdc79\") "
Sep 18 19:59:06.415515 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:06.415510    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59cebc93-92ca-400d-9343-a505176cdc79-model-cache\") pod \"59cebc93-92ca-400d-9343-a505176cdc79\" (UID: \"59cebc93-92ca-400d-9343-a505176cdc79\") "
Sep 18 19:59:06.415740 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:06.415549    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59cebc93-92ca-400d-9343-a505176cdc79-tmp-dir\") pod \"59cebc93-92ca-400d-9343-a505176cdc79\" (UID: \"59cebc93-92ca-400d-9343-a505176cdc79\") "
Sep 18 19:59:06.415740 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:06.415584    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wh8b\" (UniqueName: \"kubernetes.io/projected/59cebc93-92ca-400d-9343-a505176cdc79-kube-api-access-4wh8b\") pod \"59cebc93-92ca-400d-9343-a505176cdc79\" (UID: \"59cebc93-92ca-400d-9343-a505176cdc79\") "
Sep 18 19:59:06.415740 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:06.415627    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59cebc93-92ca-400d-9343-a505176cdc79-kserve-provision-location\") pod \"59cebc93-92ca-400d-9343-a505176cdc79\" (UID: \"59cebc93-92ca-400d-9343-a505176cdc79\") "
Sep 18 19:59:06.415740 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:06.415651    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59cebc93-92ca-400d-9343-a505176cdc79-tls-certs\") pod \"59cebc93-92ca-400d-9343-a505176cdc79\" (UID: \"59cebc93-92ca-400d-9343-a505176cdc79\") "
Sep 18 19:59:06.416205 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:06.416165    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59cebc93-92ca-400d-9343-a505176cdc79-model-cache" (OuterVolumeSpecName: "model-cache") pod "59cebc93-92ca-400d-9343-a505176cdc79" (UID: "59cebc93-92ca-400d-9343-a505176cdc79"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:59:06.416324 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:06.416240    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59cebc93-92ca-400d-9343-a505176cdc79-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "59cebc93-92ca-400d-9343-a505176cdc79" (UID: "59cebc93-92ca-400d-9343-a505176cdc79"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:59:06.416324 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:06.416251    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59cebc93-92ca-400d-9343-a505176cdc79-home" (OuterVolumeSpecName: "home") pod "59cebc93-92ca-400d-9343-a505176cdc79" (UID: "59cebc93-92ca-400d-9343-a505176cdc79"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:59:06.417891 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:06.417870    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59cebc93-92ca-400d-9343-a505176cdc79-dshm" (OuterVolumeSpecName: "dshm") pod "59cebc93-92ca-400d-9343-a505176cdc79" (UID: "59cebc93-92ca-400d-9343-a505176cdc79"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:59:06.418224 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:06.418205    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59cebc93-92ca-400d-9343-a505176cdc79-kube-api-access-4wh8b" (OuterVolumeSpecName: "kube-api-access-4wh8b") pod "59cebc93-92ca-400d-9343-a505176cdc79" (UID: "59cebc93-92ca-400d-9343-a505176cdc79"). InnerVolumeSpecName "kube-api-access-4wh8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:59:06.418431 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:06.418413    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59cebc93-92ca-400d-9343-a505176cdc79-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "59cebc93-92ca-400d-9343-a505176cdc79" (UID: "59cebc93-92ca-400d-9343-a505176cdc79"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:59:06.428863 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:06.428839    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59cebc93-92ca-400d-9343-a505176cdc79-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "59cebc93-92ca-400d-9343-a505176cdc79" (UID: "59cebc93-92ca-400d-9343-a505176cdc79"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:59:06.516362 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:06.516329    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59cebc93-92ca-400d-9343-a505176cdc79-home\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:59:06.516362 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:06.516356    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59cebc93-92ca-400d-9343-a505176cdc79-dshm\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:59:06.516362 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:06.516367    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59cebc93-92ca-400d-9343-a505176cdc79-model-cache\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:59:06.516589 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:06.516376    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59cebc93-92ca-400d-9343-a505176cdc79-tmp-dir\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:59:06.516589 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:06.516385    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wh8b\" (UniqueName: \"kubernetes.io/projected/59cebc93-92ca-400d-9343-a505176cdc79-kube-api-access-4wh8b\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:59:06.516589 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:06.516393    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59cebc93-92ca-400d-9343-a505176cdc79-kserve-provision-location\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:59:06.516589 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:06.516401    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59cebc93-92ca-400d-9343-a505176cdc79-tls-certs\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 19:59:07.295584 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:07.295549    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk"
Sep 18 19:59:07.324003 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:07.323953    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk"]
Sep 18 19:59:07.331907 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:07.331877    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dcc6mk"]
Sep 18 19:59:07.439303 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:07.439272    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59cebc93-92ca-400d-9343-a505176cdc79" path="/var/lib/kubelet/pods/59cebc93-92ca-400d-9343-a505176cdc79/volumes"
Sep 18 19:59:13.547239 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:13.547200    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr"]
Sep 18 19:59:13.547630 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:13.547528    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59cebc93-92ca-400d-9343-a505176cdc79" containerName="modelcar-init"
Sep 18 19:59:13.547630 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:13.547538    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="59cebc93-92ca-400d-9343-a505176cdc79" containerName="modelcar-init"
Sep 18 19:59:13.547630 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:13.547545    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59cebc93-92ca-400d-9343-a505176cdc79" containerName="main"
Sep 18 19:59:13.547630 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:13.547551    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="59cebc93-92ca-400d-9343-a505176cdc79" containerName="main"
Sep 18 19:59:13.547630 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:13.547557    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59cebc93-92ca-400d-9343-a505176cdc79" containerName="modelcar"
Sep 18 19:59:13.547630 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:13.547563    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="59cebc93-92ca-400d-9343-a505176cdc79" containerName="modelcar"
Sep 18 19:59:13.547630 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:13.547615    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="59cebc93-92ca-400d-9343-a505176cdc79" containerName="modelcar"
Sep 18 19:59:13.547630 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:13.547624    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="59cebc93-92ca-400d-9343-a505176cdc79" containerName="main"
Sep 18 19:59:13.552291 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:13.552270    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr"
Sep 18 19:59:13.554968 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:13.554947    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"seaweedfs-s3-creds\""
Sep 18 19:59:13.555258 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:13.555230    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Sep 18 19:59:13.555473 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:13.555257    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 19:59:13.555473 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:13.555289    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Sep 18 19:59:13.555473 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:13.555299    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Sep 18 19:59:13.555704 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:13.555515    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-f2j2n\""
Sep 18 19:59:13.570174 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:13.570150    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr"]
Sep 18 19:59:13.577936 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:13.577914    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d1ae7a9-a940-4d00-8174-8432bbb4538c-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr\" (UID: \"5d1ae7a9-a940-4d00-8174-8432bbb4538c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr"
Sep 18 19:59:13.578064 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:13.577943    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d1ae7a9-a940-4d00-8174-8432bbb4538c-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr\" (UID: \"5d1ae7a9-a940-4d00-8174-8432bbb4538c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr"
Sep 18 19:59:13.578064 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:13.577967    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d1ae7a9-a940-4d00-8174-8432bbb4538c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr\" (UID: \"5d1ae7a9-a940-4d00-8174-8432bbb4538c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr"
Sep 18 19:59:13.578153 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:13.578061    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcbqx\" (UniqueName: \"kubernetes.io/projected/5d1ae7a9-a940-4d00-8174-8432bbb4538c-kube-api-access-tcbqx\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr\" (UID: \"5d1ae7a9-a940-4d00-8174-8432bbb4538c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr"
Sep 18 19:59:13.578153 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:13.578093    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d1ae7a9-a940-4d00-8174-8432bbb4538c-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr\" (UID: \"5d1ae7a9-a940-4d00-8174-8432bbb4538c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr"
Sep 18 19:59:13.578153 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:13.578141    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d1ae7a9-a940-4d00-8174-8432bbb4538c-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr\" (UID: \"5d1ae7a9-a940-4d00-8174-8432bbb4538c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr"
Sep 18 19:59:13.578260 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:13.578159    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d1ae7a9-a940-4d00-8174-8432bbb4538c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr\" (UID: \"5d1ae7a9-a940-4d00-8174-8432bbb4538c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr"
Sep 18 19:59:13.578260 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:13.578180    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d1ae7a9-a940-4d00-8174-8432bbb4538c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr\" (UID: \"5d1ae7a9-a940-4d00-8174-8432bbb4538c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr"
Sep 18 19:59:13.679526 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:13.679493    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d1ae7a9-a940-4d00-8174-8432bbb4538c-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr\" (UID: \"5d1ae7a9-a940-4d00-8174-8432bbb4538c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr"
Sep 18 19:59:13.679526 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:13.679529    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d1ae7a9-a940-4d00-8174-8432bbb4538c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr\" (UID: \"5d1ae7a9-a940-4d00-8174-8432bbb4538c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr"
Sep 18 19:59:13.679526 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:13.679557    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d1ae7a9-a940-4d00-8174-8432bbb4538c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr\" (UID: \"5d1ae7a9-a940-4d00-8174-8432bbb4538c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr"
Sep 18 19:59:13.679943 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:13.679632    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d1ae7a9-a940-4d00-8174-8432bbb4538c-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr\" (UID: \"5d1ae7a9-a940-4d00-8174-8432bbb4538c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr"
Sep 18 19:59:13.679943 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:13.679660    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d1ae7a9-a940-4d00-8174-8432bbb4538c-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr\" (UID: \"5d1ae7a9-a940-4d00-8174-8432bbb4538c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr"
Sep 18 19:59:13.679943 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:13.679683    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d1ae7a9-a940-4d00-8174-8432bbb4538c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr\" (UID: \"5d1ae7a9-a940-4d00-8174-8432bbb4538c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr"
Sep 18 19:59:13.679943 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:13.679711    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcbqx\" (UniqueName: \"kubernetes.io/projected/5d1ae7a9-a940-4d00-8174-8432bbb4538c-kube-api-access-tcbqx\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr\" (UID: \"5d1ae7a9-a940-4d00-8174-8432bbb4538c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr"
Sep 18 19:59:13.679943 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:13.679742    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d1ae7a9-a940-4d00-8174-8432bbb4538c-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr\" (UID: \"5d1ae7a9-a940-4d00-8174-8432bbb4538c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr"
Sep 18 19:59:13.680325 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:13.679998    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d1ae7a9-a940-4d00-8174-8432bbb4538c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr\" (UID: \"5d1ae7a9-a940-4d00-8174-8432bbb4538c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr"
Sep 18 19:59:13.680325 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:13.680259    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d1ae7a9-a940-4d00-8174-8432bbb4538c-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr\" (UID: \"5d1ae7a9-a940-4d00-8174-8432bbb4538c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr"
Sep 18 19:59:13.680436 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:13.680326    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d1ae7a9-a940-4d00-8174-8432bbb4538c-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr\" (UID: \"5d1ae7a9-a940-4d00-8174-8432bbb4538c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr"
Sep 18 19:59:13.680436 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:13.680425    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d1ae7a9-a940-4d00-8174-8432bbb4538c-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr\" (UID: \"5d1ae7a9-a940-4d00-8174-8432bbb4538c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr"
Sep 18 19:59:13.680798 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:13.680781    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d1ae7a9-a940-4d00-8174-8432bbb4538c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr\" (UID: \"5d1ae7a9-a940-4d00-8174-8432bbb4538c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr"
Sep 18 19:59:13.682182 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:13.682150    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d1ae7a9-a940-4d00-8174-8432bbb4538c-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr\" (UID: \"5d1ae7a9-a940-4d00-8174-8432bbb4538c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr"
Sep 18 19:59:13.682483 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:13.682464    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d1ae7a9-a940-4d00-8174-8432bbb4538c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr\" (UID: \"5d1ae7a9-a940-4d00-8174-8432bbb4538c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr"
Sep 18 19:59:13.688320 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:13.688304    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcbqx\" (UniqueName: \"kubernetes.io/projected/5d1ae7a9-a940-4d00-8174-8432bbb4538c-kube-api-access-tcbqx\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr\" (UID: \"5d1ae7a9-a940-4d00-8174-8432bbb4538c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr"
Sep 18 19:59:13.862732 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:13.862708    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr"
Sep 18 19:59:14.154253 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:14.154165    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr"]
Sep 18 19:59:14.157947 ip-10-0-129-30 kubenswrapper[2577]: W0918 19:59:14.157919    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d1ae7a9_a940_4d00_8174_8432bbb4538c.slice/crio-d722947e0782b8ef192ed5b2dd59aae76797be3c03fbebc2a4b059a6cf672a86 WatchSource:0}: Error finding container d722947e0782b8ef192ed5b2dd59aae76797be3c03fbebc2a4b059a6cf672a86: Status 404 returned error can't find the container with id d722947e0782b8ef192ed5b2dd59aae76797be3c03fbebc2a4b059a6cf672a86
Sep 18 19:59:14.332417 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:14.332366    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr" event={"ID":"5d1ae7a9-a940-4d00-8174-8432bbb4538c","Type":"ContainerStarted","Data":"b268ce7cf395a6ffed9f15f10d357b1d8ee7de287089ffb8e1c505fa42f09c1e"}
Sep 18 19:59:14.332570 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:14.332446    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr" event={"ID":"5d1ae7a9-a940-4d00-8174-8432bbb4538c","Type":"ContainerStarted","Data":"d722947e0782b8ef192ed5b2dd59aae76797be3c03fbebc2a4b059a6cf672a86"}
Sep 18 19:59:17.350298 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:17.350207    2577 generic.go:356] "Generic (PLEG): container finished" podID="5d1ae7a9-a940-4d00-8174-8432bbb4538c" containerID="b268ce7cf395a6ffed9f15f10d357b1d8ee7de287089ffb8e1c505fa42f09c1e" exitCode=0
Sep 18 19:59:17.350298 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:17.350277    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr" event={"ID":"5d1ae7a9-a940-4d00-8174-8432bbb4538c","Type":"ContainerDied","Data":"b268ce7cf395a6ffed9f15f10d357b1d8ee7de287089ffb8e1c505fa42f09c1e"}
Sep 18 19:59:18.357053 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:18.357018    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr" event={"ID":"5d1ae7a9-a940-4d00-8174-8432bbb4538c","Type":"ContainerStarted","Data":"875428e5d3da53e227c76ee817c627725c8d2259f01443760da4ca83c9137529"}
Sep 18 19:59:18.383939 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:18.383880    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr" podStartSLOduration=2.725377145 podStartE2EDuration="5.383851931s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.658474786s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:59:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:59:18.382243539 +0000 UTC m=+2729.234543911" watchObservedRunningTime="2026-09-18 19:59:18.383851931 +0000 UTC m=+2729.236152303"
Sep 18 19:59:18.863518 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:18.863479    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr"
Sep 18 19:59:23.863870 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:23.863825    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr"
Sep 18 19:59:23.865456 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:23.865428    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr" podUID="5d1ae7a9-a940-4d00-8174-8432bbb4538c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Sep 18 19:59:33.863487 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:33.863439    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr" podUID="5d1ae7a9-a940-4d00-8174-8432bbb4538c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Sep 18 19:59:43.864116 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:43.864068    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr" podUID="5d1ae7a9-a940-4d00-8174-8432bbb4538c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Sep 18 19:59:53.863755 ip-10-0-129-30 kubenswrapper[2577]: I0918 19:59:53.863709    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr" podUID="5d1ae7a9-a940-4d00-8174-8432bbb4538c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Sep 18 20:00:03.863187 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:00:03.863102    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr" podUID="5d1ae7a9-a940-4d00-8174-8432bbb4538c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Sep 18 20:00:13.863453 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:00:13.863406    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr" podUID="5d1ae7a9-a940-4d00-8174-8432bbb4538c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Sep 18 20:00:23.863518 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:00:23.863458    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr" podUID="5d1ae7a9-a940-4d00-8174-8432bbb4538c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Sep 18 20:00:33.873425 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:00:33.873396    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr"
Sep 18 20:00:33.881084 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:00:33.881059    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr"
Sep 18 20:00:35.301358 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:00:35.301318    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr"]
Sep 18 20:00:50.753511 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:00:50.753447    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr" podUID="5d1ae7a9-a940-4d00-8174-8432bbb4538c" containerName="main" containerID="cri-o://875428e5d3da53e227c76ee817c627725c8d2259f01443760da4ca83c9137529" gracePeriod=45
Sep 18 20:00:50.874542 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:00:50.874491    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr" podUID="5d1ae7a9-a940-4d00-8174-8432bbb4538c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Sep 18 20:00:51.300310 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:00:51.300284    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr"
Sep 18 20:00:51.367693 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:00:51.367612    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d1ae7a9-a940-4d00-8174-8432bbb4538c-home\") pod \"5d1ae7a9-a940-4d00-8174-8432bbb4538c\" (UID: \"5d1ae7a9-a940-4d00-8174-8432bbb4538c\") "
Sep 18 20:00:51.367693 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:00:51.367643    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d1ae7a9-a940-4d00-8174-8432bbb4538c-dshm\") pod \"5d1ae7a9-a940-4d00-8174-8432bbb4538c\" (UID: \"5d1ae7a9-a940-4d00-8174-8432bbb4538c\") "
Sep 18 20:00:51.367927 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:00:51.367698    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tcbqx\" (UniqueName: \"kubernetes.io/projected/5d1ae7a9-a940-4d00-8174-8432bbb4538c-kube-api-access-tcbqx\") pod \"5d1ae7a9-a940-4d00-8174-8432bbb4538c\" (UID: \"5d1ae7a9-a940-4d00-8174-8432bbb4538c\") "
Sep 18 20:00:51.367927 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:00:51.367713    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d1ae7a9-a940-4d00-8174-8432bbb4538c-model-cache\") pod \"5d1ae7a9-a940-4d00-8174-8432bbb4538c\" (UID: \"5d1ae7a9-a940-4d00-8174-8432bbb4538c\") "
Sep 18 20:00:51.367927 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:00:51.367734    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d1ae7a9-a940-4d00-8174-8432bbb4538c-cabundle-cert\") pod \"5d1ae7a9-a940-4d00-8174-8432bbb4538c\" (UID: \"5d1ae7a9-a940-4d00-8174-8432bbb4538c\") "
Sep 18 20:00:51.367927 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:00:51.367761    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d1ae7a9-a940-4d00-8174-8432bbb4538c-tmp-dir\") pod \"5d1ae7a9-a940-4d00-8174-8432bbb4538c\" (UID: \"5d1ae7a9-a940-4d00-8174-8432bbb4538c\") "
Sep 18 20:00:51.367927 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:00:51.367779    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d1ae7a9-a940-4d00-8174-8432bbb4538c-tls-certs\") pod \"5d1ae7a9-a940-4d00-8174-8432bbb4538c\" (UID: \"5d1ae7a9-a940-4d00-8174-8432bbb4538c\") "
Sep 18 20:00:51.367927 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:00:51.367808    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d1ae7a9-a940-4d00-8174-8432bbb4538c-kserve-provision-location\") pod \"5d1ae7a9-a940-4d00-8174-8432bbb4538c\" (UID: \"5d1ae7a9-a940-4d00-8174-8432bbb4538c\") "
Sep 18 20:00:51.368276 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:00:51.368052    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d1ae7a9-a940-4d00-8174-8432bbb4538c-model-cache" (OuterVolumeSpecName: "model-cache") pod "5d1ae7a9-a940-4d00-8174-8432bbb4538c" (UID: "5d1ae7a9-a940-4d00-8174-8432bbb4538c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:00:51.368276 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:00:51.368066    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d1ae7a9-a940-4d00-8174-8432bbb4538c-home" (OuterVolumeSpecName: "home") pod "5d1ae7a9-a940-4d00-8174-8432bbb4538c" (UID: "5d1ae7a9-a940-4d00-8174-8432bbb4538c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:00:51.368276 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:00:51.368151    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d1ae7a9-a940-4d00-8174-8432bbb4538c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5d1ae7a9-a940-4d00-8174-8432bbb4538c" (UID: "5d1ae7a9-a940-4d00-8174-8432bbb4538c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:00:51.368276 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:00:51.368171    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d1ae7a9-a940-4d00-8174-8432bbb4538c-home\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:00:51.368276 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:00:51.368192    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d1ae7a9-a940-4d00-8174-8432bbb4538c-model-cache\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:00:51.368695 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:00:51.368665    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d1ae7a9-a940-4d00-8174-8432bbb4538c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5d1ae7a9-a940-4d00-8174-8432bbb4538c" (UID: "5d1ae7a9-a940-4d00-8174-8432bbb4538c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:00:51.370187 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:00:51.370133    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d1ae7a9-a940-4d00-8174-8432bbb4538c-kube-api-access-tcbqx" (OuterVolumeSpecName: "kube-api-access-tcbqx") pod "5d1ae7a9-a940-4d00-8174-8432bbb4538c" (UID: "5d1ae7a9-a940-4d00-8174-8432bbb4538c"). InnerVolumeSpecName "kube-api-access-tcbqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:00:51.370187 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:00:51.370137    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d1ae7a9-a940-4d00-8174-8432bbb4538c-dshm" (OuterVolumeSpecName: "dshm") pod "5d1ae7a9-a940-4d00-8174-8432bbb4538c" (UID: "5d1ae7a9-a940-4d00-8174-8432bbb4538c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:00:51.370326 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:00:51.370237    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d1ae7a9-a940-4d00-8174-8432bbb4538c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5d1ae7a9-a940-4d00-8174-8432bbb4538c" (UID: "5d1ae7a9-a940-4d00-8174-8432bbb4538c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:00:51.391730 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:00:51.391705    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d1ae7a9-a940-4d00-8174-8432bbb4538c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5d1ae7a9-a940-4d00-8174-8432bbb4538c" (UID: "5d1ae7a9-a940-4d00-8174-8432bbb4538c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:00:51.469410 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:00:51.469383    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d1ae7a9-a940-4d00-8174-8432bbb4538c-dshm\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:00:51.469410 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:00:51.469409    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcbqx\" (UniqueName: \"kubernetes.io/projected/5d1ae7a9-a940-4d00-8174-8432bbb4538c-kube-api-access-tcbqx\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:00:51.469582 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:00:51.469419    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d1ae7a9-a940-4d00-8174-8432bbb4538c-cabundle-cert\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:00:51.469582 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:00:51.469427    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d1ae7a9-a940-4d00-8174-8432bbb4538c-tmp-dir\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:00:51.469582 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:00:51.469435    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d1ae7a9-a940-4d00-8174-8432bbb4538c-tls-certs\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:00:51.469582 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:00:51.469442    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d1ae7a9-a940-4d00-8174-8432bbb4538c-kserve-provision-location\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:00:51.809792 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:00:51.809708    2577 generic.go:356] "Generic (PLEG): container finished" podID="5d1ae7a9-a940-4d00-8174-8432bbb4538c" containerID="875428e5d3da53e227c76ee817c627725c8d2259f01443760da4ca83c9137529" exitCode=0
Sep 18 20:00:51.809792 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:00:51.809761    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr"
Sep 18 20:00:51.810343 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:00:51.809760    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr" event={"ID":"5d1ae7a9-a940-4d00-8174-8432bbb4538c","Type":"ContainerDied","Data":"875428e5d3da53e227c76ee817c627725c8d2259f01443760da4ca83c9137529"}
Sep 18 20:00:51.810343 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:00:51.809879    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr" event={"ID":"5d1ae7a9-a940-4d00-8174-8432bbb4538c","Type":"ContainerDied","Data":"d722947e0782b8ef192ed5b2dd59aae76797be3c03fbebc2a4b059a6cf672a86"}
Sep 18 20:00:51.810343 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:00:51.809902    2577 scope.go:117] "RemoveContainer" containerID="875428e5d3da53e227c76ee817c627725c8d2259f01443760da4ca83c9137529"
Sep 18 20:00:51.818419 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:00:51.818398    2577 scope.go:117] "RemoveContainer" containerID="b268ce7cf395a6ffed9f15f10d357b1d8ee7de287089ffb8e1c505fa42f09c1e"
Sep 18 20:00:51.826300 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:00:51.826284    2577 scope.go:117] "RemoveContainer" containerID="875428e5d3da53e227c76ee817c627725c8d2259f01443760da4ca83c9137529"
Sep 18 20:00:51.826552 ip-10-0-129-30 kubenswrapper[2577]: E0918 20:00:51.826532    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"875428e5d3da53e227c76ee817c627725c8d2259f01443760da4ca83c9137529\": container with ID starting with 875428e5d3da53e227c76ee817c627725c8d2259f01443760da4ca83c9137529 not found: ID does not exist" containerID="875428e5d3da53e227c76ee817c627725c8d2259f01443760da4ca83c9137529"
Sep 18 20:00:51.826627 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:00:51.826565    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"875428e5d3da53e227c76ee817c627725c8d2259f01443760da4ca83c9137529"} err="failed to get container status \"875428e5d3da53e227c76ee817c627725c8d2259f01443760da4ca83c9137529\": rpc error: code = NotFound desc = could not find container \"875428e5d3da53e227c76ee817c627725c8d2259f01443760da4ca83c9137529\": container with ID starting with 875428e5d3da53e227c76ee817c627725c8d2259f01443760da4ca83c9137529 not found: ID does not exist"
Sep 18 20:00:51.826627 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:00:51.826589    2577 scope.go:117] "RemoveContainer" containerID="b268ce7cf395a6ffed9f15f10d357b1d8ee7de287089ffb8e1c505fa42f09c1e"
Sep 18 20:00:51.826852 ip-10-0-129-30 kubenswrapper[2577]: E0918 20:00:51.826833    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b268ce7cf395a6ffed9f15f10d357b1d8ee7de287089ffb8e1c505fa42f09c1e\": container with ID starting with b268ce7cf395a6ffed9f15f10d357b1d8ee7de287089ffb8e1c505fa42f09c1e not found: ID does not exist" containerID="b268ce7cf395a6ffed9f15f10d357b1d8ee7de287089ffb8e1c505fa42f09c1e"
Sep 18 20:00:51.826893 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:00:51.826859    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b268ce7cf395a6ffed9f15f10d357b1d8ee7de287089ffb8e1c505fa42f09c1e"} err="failed to get container status \"b268ce7cf395a6ffed9f15f10d357b1d8ee7de287089ffb8e1c505fa42f09c1e\": rpc error: code = NotFound desc = could not find container \"b268ce7cf395a6ffed9f15f10d357b1d8ee7de287089ffb8e1c505fa42f09c1e\": container with ID starting with b268ce7cf395a6ffed9f15f10d357b1d8ee7de287089ffb8e1c505fa42f09c1e not found: ID does not exist"
Sep 18 20:00:51.832388 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:00:51.832368    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr"]
Sep 18 20:00:51.841355 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:00:51.841334    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5fd49885b8-bkwwr"]
Sep 18 20:00:53.438495 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:00:53.438453    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d1ae7a9-a940-4d00-8174-8432bbb4538c" path="/var/lib/kubelet/pods/5d1ae7a9-a940-4d00-8174-8432bbb4538c/volumes"
Sep 18 20:01:32.221035 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:01:32.220932    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-4bs5t"]
Sep 18 20:01:32.221504 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:01:32.221268    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d1ae7a9-a940-4d00-8174-8432bbb4538c" containerName="storage-initializer"
Sep 18 20:01:32.221504 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:01:32.221281    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d1ae7a9-a940-4d00-8174-8432bbb4538c" containerName="storage-initializer"
Sep 18 20:01:32.221504 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:01:32.221303    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d1ae7a9-a940-4d00-8174-8432bbb4538c" containerName="main"
Sep 18 20:01:32.221504 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:01:32.221309    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d1ae7a9-a940-4d00-8174-8432bbb4538c" containerName="main"
Sep 18 20:01:32.221504 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:01:32.221365    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d1ae7a9-a940-4d00-8174-8432bbb4538c" containerName="main"
Sep 18 20:01:32.224228 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:01:32.224212    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-4bs5t"
Sep 18 20:01:32.226858 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:01:32.226833    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:01:32.227170 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:01:32.227153    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Sep 18 20:01:32.227710 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:01:32.227693    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 18 20:01:32.227915 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:01:32.227889    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-dc84j\""
Sep 18 20:01:32.228027 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:01:32.227945    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 18 20:01:32.228101 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:01:32.228061    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 18 20:01:32.242770 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:01:32.242744    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-4bs5t"]
Sep 18 20:01:32.324881 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:01:32.324847    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f31729a-6bdf-40a6-b4c1-1e05d0348ff8-model-cache\") pod \"stop-feature-test-kserve-58d6cbd986-4bs5t\" (UID: \"4f31729a-6bdf-40a6-b4c1-1e05d0348ff8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-4bs5t"
Sep 18 20:01:32.325079 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:01:32.324903    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f31729a-6bdf-40a6-b4c1-1e05d0348ff8-dshm\") pod \"stop-feature-test-kserve-58d6cbd986-4bs5t\" (UID: \"4f31729a-6bdf-40a6-b4c1-1e05d0348ff8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-4bs5t"
Sep 18 20:01:32.325079 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:01:32.324923    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f31729a-6bdf-40a6-b4c1-1e05d0348ff8-kserve-provision-location\") pod \"stop-feature-test-kserve-58d6cbd986-4bs5t\" (UID: \"4f31729a-6bdf-40a6-b4c1-1e05d0348ff8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-4bs5t"
Sep 18 20:01:32.325079 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:01:32.325006    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f31729a-6bdf-40a6-b4c1-1e05d0348ff8-tls-certs\") pod \"stop-feature-test-kserve-58d6cbd986-4bs5t\" (UID: \"4f31729a-6bdf-40a6-b4c1-1e05d0348ff8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-4bs5t"
Sep 18 20:01:32.325079 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:01:32.325049    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f31729a-6bdf-40a6-b4c1-1e05d0348ff8-cabundle-cert\") pod \"stop-feature-test-kserve-58d6cbd986-4bs5t\" (UID: \"4f31729a-6bdf-40a6-b4c1-1e05d0348ff8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-4bs5t"
Sep 18 20:01:32.325226 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:01:32.325077    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlfhd\" (UniqueName: \"kubernetes.io/projected/4f31729a-6bdf-40a6-b4c1-1e05d0348ff8-kube-api-access-xlfhd\") pod \"stop-feature-test-kserve-58d6cbd986-4bs5t\" (UID: \"4f31729a-6bdf-40a6-b4c1-1e05d0348ff8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-4bs5t"
Sep 18 20:01:32.325226 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:01:32.325101    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f31729a-6bdf-40a6-b4c1-1e05d0348ff8-home\") pod \"stop-feature-test-kserve-58d6cbd986-4bs5t\" (UID: \"4f31729a-6bdf-40a6-b4c1-1e05d0348ff8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-4bs5t"
Sep 18 20:01:32.325226 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:01:32.325116    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f31729a-6bdf-40a6-b4c1-1e05d0348ff8-tmp-dir\") pod \"stop-feature-test-kserve-58d6cbd986-4bs5t\" (UID: \"4f31729a-6bdf-40a6-b4c1-1e05d0348ff8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-4bs5t"
Sep 18 20:01:32.426492 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:01:32.426449    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f31729a-6bdf-40a6-b4c1-1e05d0348ff8-dshm\") pod \"stop-feature-test-kserve-58d6cbd986-4bs5t\" (UID: \"4f31729a-6bdf-40a6-b4c1-1e05d0348ff8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-4bs5t"
Sep 18 20:01:32.426492 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:01:32.426494    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f31729a-6bdf-40a6-b4c1-1e05d0348ff8-kserve-provision-location\") pod \"stop-feature-test-kserve-58d6cbd986-4bs5t\" (UID: \"4f31729a-6bdf-40a6-b4c1-1e05d0348ff8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-4bs5t"
Sep 18 20:01:32.426744 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:01:32.426521    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f31729a-6bdf-40a6-b4c1-1e05d0348ff8-tls-certs\") pod \"stop-feature-test-kserve-58d6cbd986-4bs5t\" (UID: \"4f31729a-6bdf-40a6-b4c1-1e05d0348ff8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-4bs5t"
Sep 18 20:01:32.426744 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:01:32.426547    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f31729a-6bdf-40a6-b4c1-1e05d0348ff8-cabundle-cert\") pod \"stop-feature-test-kserve-58d6cbd986-4bs5t\" (UID: \"4f31729a-6bdf-40a6-b4c1-1e05d0348ff8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-4bs5t"
Sep 18 20:01:32.426744 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:01:32.426564    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlfhd\" (UniqueName: \"kubernetes.io/projected/4f31729a-6bdf-40a6-b4c1-1e05d0348ff8-kube-api-access-xlfhd\") pod \"stop-feature-test-kserve-58d6cbd986-4bs5t\" (UID: \"4f31729a-6bdf-40a6-b4c1-1e05d0348ff8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-4bs5t"
Sep 18 20:01:32.426744 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:01:32.426580    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f31729a-6bdf-40a6-b4c1-1e05d0348ff8-home\") pod \"stop-feature-test-kserve-58d6cbd986-4bs5t\" (UID: \"4f31729a-6bdf-40a6-b4c1-1e05d0348ff8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-4bs5t"
Sep 18 20:01:32.426744 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:01:32.426594    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f31729a-6bdf-40a6-b4c1-1e05d0348ff8-tmp-dir\") pod \"stop-feature-test-kserve-58d6cbd986-4bs5t\" (UID: \"4f31729a-6bdf-40a6-b4c1-1e05d0348ff8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-4bs5t"
Sep 18 20:01:32.427064 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:01:32.426752    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f31729a-6bdf-40a6-b4c1-1e05d0348ff8-model-cache\") pod \"stop-feature-test-kserve-58d6cbd986-4bs5t\" (UID: \"4f31729a-6bdf-40a6-b4c1-1e05d0348ff8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-4bs5t"
Sep 18 20:01:32.427064 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:01:32.426941    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f31729a-6bdf-40a6-b4c1-1e05d0348ff8-kserve-provision-location\") pod \"stop-feature-test-kserve-58d6cbd986-4bs5t\" (UID: \"4f31729a-6bdf-40a6-b4c1-1e05d0348ff8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-4bs5t"
Sep 18 20:01:32.427064 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:01:32.426985    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f31729a-6bdf-40a6-b4c1-1e05d0348ff8-tmp-dir\") pod \"stop-feature-test-kserve-58d6cbd986-4bs5t\" (UID: \"4f31729a-6bdf-40a6-b4c1-1e05d0348ff8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-4bs5t"
Sep 18 20:01:32.427239 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:01:32.427195    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f31729a-6bdf-40a6-b4c1-1e05d0348ff8-model-cache\") pod \"stop-feature-test-kserve-58d6cbd986-4bs5t\" (UID: \"4f31729a-6bdf-40a6-b4c1-1e05d0348ff8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-4bs5t"
Sep 18 20:01:32.427239 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:01:32.427207    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f31729a-6bdf-40a6-b4c1-1e05d0348ff8-home\") pod \"stop-feature-test-kserve-58d6cbd986-4bs5t\" (UID: \"4f31729a-6bdf-40a6-b4c1-1e05d0348ff8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-4bs5t"
Sep 18 20:01:32.427345 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:01:32.427327    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f31729a-6bdf-40a6-b4c1-1e05d0348ff8-cabundle-cert\") pod \"stop-feature-test-kserve-58d6cbd986-4bs5t\" (UID: \"4f31729a-6bdf-40a6-b4c1-1e05d0348ff8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-4bs5t"
Sep 18 20:01:32.429425 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:01:32.429404    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f31729a-6bdf-40a6-b4c1-1e05d0348ff8-dshm\") pod \"stop-feature-test-kserve-58d6cbd986-4bs5t\" (UID: \"4f31729a-6bdf-40a6-b4c1-1e05d0348ff8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-4bs5t"
Sep 18 20:01:32.429821 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:01:32.429800    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f31729a-6bdf-40a6-b4c1-1e05d0348ff8-tls-certs\") pod \"stop-feature-test-kserve-58d6cbd986-4bs5t\" (UID: \"4f31729a-6bdf-40a6-b4c1-1e05d0348ff8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-4bs5t"
Sep 18 20:01:32.442005 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:01:32.441959    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlfhd\" (UniqueName: \"kubernetes.io/projected/4f31729a-6bdf-40a6-b4c1-1e05d0348ff8-kube-api-access-xlfhd\") pod \"stop-feature-test-kserve-58d6cbd986-4bs5t\" (UID: \"4f31729a-6bdf-40a6-b4c1-1e05d0348ff8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-4bs5t"
Sep 18 20:01:32.534011 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:01:32.533896    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-4bs5t"
Sep 18 20:01:32.819523 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:01:32.819355    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-4bs5t"]
Sep 18 20:01:32.821774 ip-10-0-129-30 kubenswrapper[2577]: W0918 20:01:32.821743    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f31729a_6bdf_40a6_b4c1_1e05d0348ff8.slice/crio-7d2293b85d89266b4011a311511b48c5de42979c570d45c6402dbd3f5219a08d WatchSource:0}: Error finding container 7d2293b85d89266b4011a311511b48c5de42979c570d45c6402dbd3f5219a08d: Status 404 returned error can't find the container with id 7d2293b85d89266b4011a311511b48c5de42979c570d45c6402dbd3f5219a08d
Sep 18 20:01:32.997359 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:01:32.997318    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-4bs5t" event={"ID":"4f31729a-6bdf-40a6-b4c1-1e05d0348ff8","Type":"ContainerStarted","Data":"3d970ea8859830259254ef66093f5136fa22b7edc4d0133960167b372f9fb32d"}
Sep 18 20:01:32.997534 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:01:32.997381    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-4bs5t" event={"ID":"4f31729a-6bdf-40a6-b4c1-1e05d0348ff8","Type":"ContainerStarted","Data":"7d2293b85d89266b4011a311511b48c5de42979c570d45c6402dbd3f5219a08d"}
Sep 18 20:01:36.011545 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:01:36.011510    2577 generic.go:356] "Generic (PLEG): container finished" podID="4f31729a-6bdf-40a6-b4c1-1e05d0348ff8" containerID="3d970ea8859830259254ef66093f5136fa22b7edc4d0133960167b372f9fb32d" exitCode=0
Sep 18 20:01:36.011945 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:01:36.011550    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-4bs5t" event={"ID":"4f31729a-6bdf-40a6-b4c1-1e05d0348ff8","Type":"ContainerDied","Data":"3d970ea8859830259254ef66093f5136fa22b7edc4d0133960167b372f9fb32d"}
Sep 18 20:01:37.018352 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:01:37.018316    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-4bs5t" event={"ID":"4f31729a-6bdf-40a6-b4c1-1e05d0348ff8","Type":"ContainerStarted","Data":"07bfcfcc5b3b51bbb88c63bcb532acfa344a2457c2d5b8a69ee4b07de34b72e8"}
Sep 18 20:01:37.047867 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:01:37.047815    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-4bs5t" podStartSLOduration=2.910150892 podStartE2EDuration="5.047799628s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.137648736s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:01:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:01:37.045575325 +0000 UTC m=+2867.897875695" watchObservedRunningTime="2026-09-18 20:01:37.047799628 +0000 UTC m=+2867.900100041"
Sep 18 20:01:37.534653 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:01:37.534600    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-4bs5t"
Sep 18 20:01:42.534319 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:01:42.534281    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-4bs5t"
Sep 18 20:01:42.535969 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:01:42.535931    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-4bs5t" podUID="4f31729a-6bdf-40a6-b4c1-1e05d0348ff8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Sep 18 20:01:52.534488 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:01:52.534446    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-4bs5t" podUID="4f31729a-6bdf-40a6-b4c1-1e05d0348ff8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Sep 18 20:02:02.534684 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:02.534635    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-4bs5t" podUID="4f31729a-6bdf-40a6-b4c1-1e05d0348ff8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Sep 18 20:02:12.535083 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:12.535038    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-4bs5t" podUID="4f31729a-6bdf-40a6-b4c1-1e05d0348ff8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Sep 18 20:02:22.534723 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:22.534671    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-4bs5t" podUID="4f31729a-6bdf-40a6-b4c1-1e05d0348ff8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Sep 18 20:02:32.535079 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:32.535028    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-4bs5t" podUID="4f31729a-6bdf-40a6-b4c1-1e05d0348ff8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Sep 18 20:02:42.534742 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:42.534699    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-4bs5t" podUID="4f31729a-6bdf-40a6-b4c1-1e05d0348ff8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Sep 18 20:02:44.290939 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:44.290903    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-qmhbh"]
Sep 18 20:02:44.294160 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:44.294138    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-qmhbh"
Sep 18 20:02:44.299845 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:44.299781    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-gw-ca-root-cert\""
Sep 18 20:02:44.300341 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:44.300321    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 18 20:02:44.300898 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:44.300326    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 18 20:02:44.300898 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:44.300392    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-gateway-1-openshift-default-dockercfg-gb4lm\""
Sep 18 20:02:44.314437 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:44.314409    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-qmhbh"]
Sep 18 20:02:44.377249 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:44.377210    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2235b1aa-df85-42e6-b986-44a0f92473f5-istio-token\") pod \"router-gateway-1-openshift-default-df55df88d-qmhbh\" (UID: \"2235b1aa-df85-42e6-b986-44a0f92473f5\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-qmhbh"
Sep 18 20:02:44.377249 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:44.377249    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2235b1aa-df85-42e6-b986-44a0f92473f5-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-df55df88d-qmhbh\" (UID: \"2235b1aa-df85-42e6-b986-44a0f92473f5\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-qmhbh"
Sep 18 20:02:44.377506 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:44.377320    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2235b1aa-df85-42e6-b986-44a0f92473f5-istio-podinfo\") pod \"router-gateway-1-openshift-default-df55df88d-qmhbh\" (UID: \"2235b1aa-df85-42e6-b986-44a0f92473f5\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-qmhbh"
Sep 18 20:02:44.377506 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:44.377364    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2235b1aa-df85-42e6-b986-44a0f92473f5-credential-socket\") pod \"router-gateway-1-openshift-default-df55df88d-qmhbh\" (UID: \"2235b1aa-df85-42e6-b986-44a0f92473f5\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-qmhbh"
Sep 18 20:02:44.377506 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:44.377387    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2235b1aa-df85-42e6-b986-44a0f92473f5-istio-data\") pod \"router-gateway-1-openshift-default-df55df88d-qmhbh\" (UID: \"2235b1aa-df85-42e6-b986-44a0f92473f5\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-qmhbh"
Sep 18 20:02:44.377506 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:44.377446    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxjwp\" (UniqueName: \"kubernetes.io/projected/2235b1aa-df85-42e6-b986-44a0f92473f5-kube-api-access-hxjwp\") pod \"router-gateway-1-openshift-default-df55df88d-qmhbh\" (UID: \"2235b1aa-df85-42e6-b986-44a0f92473f5\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-qmhbh"
Sep 18 20:02:44.377506 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:44.377480    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2235b1aa-df85-42e6-b986-44a0f92473f5-workload-socket\") pod \"router-gateway-1-openshift-default-df55df88d-qmhbh\" (UID: \"2235b1aa-df85-42e6-b986-44a0f92473f5\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-qmhbh"
Sep 18 20:02:44.377721 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:44.377512    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2235b1aa-df85-42e6-b986-44a0f92473f5-workload-certs\") pod \"router-gateway-1-openshift-default-df55df88d-qmhbh\" (UID: \"2235b1aa-df85-42e6-b986-44a0f92473f5\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-qmhbh"
Sep 18 20:02:44.377721 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:44.377536    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2235b1aa-df85-42e6-b986-44a0f92473f5-istio-envoy\") pod \"router-gateway-1-openshift-default-df55df88d-qmhbh\" (UID: \"2235b1aa-df85-42e6-b986-44a0f92473f5\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-qmhbh"
Sep 18 20:02:44.478125 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:44.478076    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2235b1aa-df85-42e6-b986-44a0f92473f5-istio-podinfo\") pod \"router-gateway-1-openshift-default-df55df88d-qmhbh\" (UID: \"2235b1aa-df85-42e6-b986-44a0f92473f5\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-qmhbh"
Sep 18 20:02:44.478299 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:44.478168    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2235b1aa-df85-42e6-b986-44a0f92473f5-credential-socket\") pod \"router-gateway-1-openshift-default-df55df88d-qmhbh\" (UID: \"2235b1aa-df85-42e6-b986-44a0f92473f5\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-qmhbh"
Sep 18 20:02:44.478299 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:44.478200    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2235b1aa-df85-42e6-b986-44a0f92473f5-istio-data\") pod \"router-gateway-1-openshift-default-df55df88d-qmhbh\" (UID: \"2235b1aa-df85-42e6-b986-44a0f92473f5\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-qmhbh"
Sep 18 20:02:44.478299 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:44.478254    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxjwp\" (UniqueName: \"kubernetes.io/projected/2235b1aa-df85-42e6-b986-44a0f92473f5-kube-api-access-hxjwp\") pod \"router-gateway-1-openshift-default-df55df88d-qmhbh\" (UID: \"2235b1aa-df85-42e6-b986-44a0f92473f5\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-qmhbh"
Sep 18 20:02:44.478299 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:44.478291    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2235b1aa-df85-42e6-b986-44a0f92473f5-workload-socket\") pod \"router-gateway-1-openshift-default-df55df88d-qmhbh\" (UID: \"2235b1aa-df85-42e6-b986-44a0f92473f5\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-qmhbh"
Sep 18 20:02:44.478573 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:44.478322    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2235b1aa-df85-42e6-b986-44a0f92473f5-workload-certs\") pod \"router-gateway-1-openshift-default-df55df88d-qmhbh\" (UID: \"2235b1aa-df85-42e6-b986-44a0f92473f5\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-qmhbh"
Sep 18 20:02:44.478573 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:44.478344    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2235b1aa-df85-42e6-b986-44a0f92473f5-istio-envoy\") pod \"router-gateway-1-openshift-default-df55df88d-qmhbh\" (UID: \"2235b1aa-df85-42e6-b986-44a0f92473f5\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-qmhbh"
Sep 18 20:02:44.478573 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:44.478406    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2235b1aa-df85-42e6-b986-44a0f92473f5-istio-token\") pod \"router-gateway-1-openshift-default-df55df88d-qmhbh\" (UID: \"2235b1aa-df85-42e6-b986-44a0f92473f5\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-qmhbh"
Sep 18 20:02:44.478573 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:44.478435    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2235b1aa-df85-42e6-b986-44a0f92473f5-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-df55df88d-qmhbh\" (UID: \"2235b1aa-df85-42e6-b986-44a0f92473f5\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-qmhbh"
Sep 18 20:02:44.478788 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:44.478654    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2235b1aa-df85-42e6-b986-44a0f92473f5-istio-data\") pod \"router-gateway-1-openshift-default-df55df88d-qmhbh\" (UID: \"2235b1aa-df85-42e6-b986-44a0f92473f5\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-qmhbh"
Sep 18 20:02:44.478788 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:44.478705    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2235b1aa-df85-42e6-b986-44a0f92473f5-credential-socket\") pod \"router-gateway-1-openshift-default-df55df88d-qmhbh\" (UID: \"2235b1aa-df85-42e6-b986-44a0f92473f5\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-qmhbh"
Sep 18 20:02:44.479061 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:44.479043    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2235b1aa-df85-42e6-b986-44a0f92473f5-workload-socket\") pod \"router-gateway-1-openshift-default-df55df88d-qmhbh\" (UID: \"2235b1aa-df85-42e6-b986-44a0f92473f5\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-qmhbh"
Sep 18 20:02:44.479147 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:44.479070    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2235b1aa-df85-42e6-b986-44a0f92473f5-workload-certs\") pod \"router-gateway-1-openshift-default-df55df88d-qmhbh\" (UID: \"2235b1aa-df85-42e6-b986-44a0f92473f5\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-qmhbh"
Sep 18 20:02:44.479552 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:44.479525    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2235b1aa-df85-42e6-b986-44a0f92473f5-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-df55df88d-qmhbh\" (UID: \"2235b1aa-df85-42e6-b986-44a0f92473f5\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-qmhbh"
Sep 18 20:02:44.480673 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:44.480653    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2235b1aa-df85-42e6-b986-44a0f92473f5-istio-podinfo\") pod \"router-gateway-1-openshift-default-df55df88d-qmhbh\" (UID: \"2235b1aa-df85-42e6-b986-44a0f92473f5\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-qmhbh"
Sep 18 20:02:44.480819 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:44.480798    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2235b1aa-df85-42e6-b986-44a0f92473f5-istio-envoy\") pod \"router-gateway-1-openshift-default-df55df88d-qmhbh\" (UID: \"2235b1aa-df85-42e6-b986-44a0f92473f5\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-qmhbh"
Sep 18 20:02:44.486780 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:44.486749    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2235b1aa-df85-42e6-b986-44a0f92473f5-istio-token\") pod \"router-gateway-1-openshift-default-df55df88d-qmhbh\" (UID: \"2235b1aa-df85-42e6-b986-44a0f92473f5\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-qmhbh"
Sep 18 20:02:44.486921 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:44.486901    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxjwp\" (UniqueName: \"kubernetes.io/projected/2235b1aa-df85-42e6-b986-44a0f92473f5-kube-api-access-hxjwp\") pod \"router-gateway-1-openshift-default-df55df88d-qmhbh\" (UID: \"2235b1aa-df85-42e6-b986-44a0f92473f5\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-qmhbh"
Sep 18 20:02:44.605713 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:44.605627    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-qmhbh"
Sep 18 20:02:44.896176 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:44.896138    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-qmhbh"]
Sep 18 20:02:44.896801 ip-10-0-129-30 kubenswrapper[2577]: W0918 20:02:44.896773    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2235b1aa_df85_42e6_b986_44a0f92473f5.slice/crio-96f828a92857211ddd645e069e5c7133d67a6c484da2e0d442ac855e5d4518ca WatchSource:0}: Error finding container 96f828a92857211ddd645e069e5c7133d67a6c484da2e0d442ac855e5d4518ca: Status 404 returned error can't find the container with id 96f828a92857211ddd645e069e5c7133d67a6c484da2e0d442ac855e5d4518ca
Sep 18 20:02:44.898862 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:44.898832    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:02:44.898925 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:44.898894    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:02:44.898925 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:44.898922    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:02:45.340466 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:45.340352    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-qmhbh" event={"ID":"2235b1aa-df85-42e6-b986-44a0f92473f5","Type":"ContainerStarted","Data":"5789135a1d6cac1707474a44e6c088b9d41a31af9f66b38edb3748f7d082e2a6"}
Sep 18 20:02:45.340466 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:45.340430    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-qmhbh" event={"ID":"2235b1aa-df85-42e6-b986-44a0f92473f5","Type":"ContainerStarted","Data":"96f828a92857211ddd645e069e5c7133d67a6c484da2e0d442ac855e5d4518ca"}
Sep 18 20:02:45.372141 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:45.366940    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-qmhbh" podStartSLOduration=1.3669190279999999 podStartE2EDuration="1.366919028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:02:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:02:45.363874855 +0000 UTC m=+2936.216175224" watchObservedRunningTime="2026-09-18 20:02:45.366919028 +0000 UTC m=+2936.219219399"
Sep 18 20:02:45.606427 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:45.606327    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-qmhbh"
Sep 18 20:02:45.611276 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:45.611251    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-qmhbh"
Sep 18 20:02:46.345301 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:46.345271    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-qmhbh"
Sep 18 20:02:46.346327 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:46.346308    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-qmhbh"
Sep 18 20:02:52.535362 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:52.535308    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-4bs5t" podUID="4f31729a-6bdf-40a6-b4c1-1e05d0348ff8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Sep 18 20:02:56.398002 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:56.397962    2577 generic.go:356] "Generic (PLEG): container finished" podID="2235b1aa-df85-42e6-b986-44a0f92473f5" containerID="5789135a1d6cac1707474a44e6c088b9d41a31af9f66b38edb3748f7d082e2a6" exitCode=0
Sep 18 20:02:56.398442 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:56.398015    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-qmhbh" event={"ID":"2235b1aa-df85-42e6-b986-44a0f92473f5","Type":"ContainerDied","Data":"5789135a1d6cac1707474a44e6c088b9d41a31af9f66b38edb3748f7d082e2a6"}
Sep 18 20:02:56.398442 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:56.398374    2577 scope.go:117] "RemoveContainer" containerID="5789135a1d6cac1707474a44e6c088b9d41a31af9f66b38edb3748f7d082e2a6"
Sep 18 20:02:56.399382 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:56.399349    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:02:56.399487 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:56.399440    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:02:56.399553 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:56.399489    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:02:57.405088 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:57.405053    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-qmhbh" event={"ID":"2235b1aa-df85-42e6-b986-44a0f92473f5","Type":"ContainerStarted","Data":"331da3ac42b426d971fa094d71bcabce50839575e12163f54b6f500f88a8fc4e"}
Sep 18 20:02:57.606630 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:57.606593    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-qmhbh"
Sep 18 20:02:57.607761 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:57.607733    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-qmhbh" podUID="2235b1aa-df85-42e6-b986-44a0f92473f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Sep 18 20:02:58.606279 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:58.606228    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-qmhbh" podUID="2235b1aa-df85-42e6-b986-44a0f92473f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Sep 18 20:02:59.606787 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:02:59.606742    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-qmhbh" podUID="2235b1aa-df85-42e6-b986-44a0f92473f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Sep 18 20:03:00.607142 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:00.607092    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-qmhbh" podUID="2235b1aa-df85-42e6-b986-44a0f92473f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Sep 18 20:03:01.345270 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:01.345222    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-qmhbh"
Sep 18 20:03:01.607160 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:01.607052    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-qmhbh" podUID="2235b1aa-df85-42e6-b986-44a0f92473f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Sep 18 20:03:02.534891 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:02.534784    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-4bs5t" podUID="4f31729a-6bdf-40a6-b4c1-1e05d0348ff8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Sep 18 20:03:02.610648 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:02.610612    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-qmhbh"
Sep 18 20:03:02.611848 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:02.611823    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-qmhbh"
Sep 18 20:03:12.534498 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:12.534455    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-4bs5t" podUID="4f31729a-6bdf-40a6-b4c1-1e05d0348ff8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Sep 18 20:03:22.535307 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:22.535261    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-4bs5t" podUID="4f31729a-6bdf-40a6-b4c1-1e05d0348ff8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Sep 18 20:03:32.544578 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:32.544545    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-4bs5t"
Sep 18 20:03:32.552599 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:32.552570    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-4bs5t"
Sep 18 20:03:34.394765 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:34.394727    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-4bs5t"]
Sep 18 20:03:49.418287 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:49.418218    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-4bs5t" podUID="4f31729a-6bdf-40a6-b4c1-1e05d0348ff8" containerName="main" containerID="cri-o://07bfcfcc5b3b51bbb88c63bcb532acfa344a2457c2d5b8a69ee4b07de34b72e8" gracePeriod=45
Sep 18 20:03:49.545167 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:49.545129    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-4bs5t" podUID="4f31729a-6bdf-40a6-b4c1-1e05d0348ff8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Sep 18 20:03:50.164532 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:50.164508    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-4bs5t"
Sep 18 20:03:50.278343 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:50.278259    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f31729a-6bdf-40a6-b4c1-1e05d0348ff8-tmp-dir\") pod \"4f31729a-6bdf-40a6-b4c1-1e05d0348ff8\" (UID: \"4f31729a-6bdf-40a6-b4c1-1e05d0348ff8\") "
Sep 18 20:03:50.278343 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:50.278303    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f31729a-6bdf-40a6-b4c1-1e05d0348ff8-model-cache\") pod \"4f31729a-6bdf-40a6-b4c1-1e05d0348ff8\" (UID: \"4f31729a-6bdf-40a6-b4c1-1e05d0348ff8\") "
Sep 18 20:03:50.278343 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:50.278336    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f31729a-6bdf-40a6-b4c1-1e05d0348ff8-dshm\") pod \"4f31729a-6bdf-40a6-b4c1-1e05d0348ff8\" (UID: \"4f31729a-6bdf-40a6-b4c1-1e05d0348ff8\") "
Sep 18 20:03:50.278649 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:50.278364    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f31729a-6bdf-40a6-b4c1-1e05d0348ff8-cabundle-cert\") pod \"4f31729a-6bdf-40a6-b4c1-1e05d0348ff8\" (UID: \"4f31729a-6bdf-40a6-b4c1-1e05d0348ff8\") "
Sep 18 20:03:50.278649 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:50.278379    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f31729a-6bdf-40a6-b4c1-1e05d0348ff8-home\") pod \"4f31729a-6bdf-40a6-b4c1-1e05d0348ff8\" (UID: \"4f31729a-6bdf-40a6-b4c1-1e05d0348ff8\") "
Sep 18 20:03:50.278649 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:50.278401    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xlfhd\" (UniqueName: \"kubernetes.io/projected/4f31729a-6bdf-40a6-b4c1-1e05d0348ff8-kube-api-access-xlfhd\") pod \"4f31729a-6bdf-40a6-b4c1-1e05d0348ff8\" (UID: \"4f31729a-6bdf-40a6-b4c1-1e05d0348ff8\") "
Sep 18 20:03:50.278649 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:50.278443    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f31729a-6bdf-40a6-b4c1-1e05d0348ff8-tls-certs\") pod \"4f31729a-6bdf-40a6-b4c1-1e05d0348ff8\" (UID: \"4f31729a-6bdf-40a6-b4c1-1e05d0348ff8\") "
Sep 18 20:03:50.278649 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:50.278507    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f31729a-6bdf-40a6-b4c1-1e05d0348ff8-kserve-provision-location\") pod \"4f31729a-6bdf-40a6-b4c1-1e05d0348ff8\" (UID: \"4f31729a-6bdf-40a6-b4c1-1e05d0348ff8\") "
Sep 18 20:03:50.278649 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:50.278566    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f31729a-6bdf-40a6-b4c1-1e05d0348ff8-model-cache" (OuterVolumeSpecName: "model-cache") pod "4f31729a-6bdf-40a6-b4c1-1e05d0348ff8" (UID: "4f31729a-6bdf-40a6-b4c1-1e05d0348ff8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:03:50.278906 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:50.278748    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f31729a-6bdf-40a6-b4c1-1e05d0348ff8-model-cache\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:03:50.278906 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:50.278754    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f31729a-6bdf-40a6-b4c1-1e05d0348ff8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4f31729a-6bdf-40a6-b4c1-1e05d0348ff8" (UID: "4f31729a-6bdf-40a6-b4c1-1e05d0348ff8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:03:50.279180 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:50.279149    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f31729a-6bdf-40a6-b4c1-1e05d0348ff8-home" (OuterVolumeSpecName: "home") pod "4f31729a-6bdf-40a6-b4c1-1e05d0348ff8" (UID: "4f31729a-6bdf-40a6-b4c1-1e05d0348ff8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:03:50.280942 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:50.280912    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f31729a-6bdf-40a6-b4c1-1e05d0348ff8-dshm" (OuterVolumeSpecName: "dshm") pod "4f31729a-6bdf-40a6-b4c1-1e05d0348ff8" (UID: "4f31729a-6bdf-40a6-b4c1-1e05d0348ff8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:03:50.281467 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:50.281440    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f31729a-6bdf-40a6-b4c1-1e05d0348ff8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4f31729a-6bdf-40a6-b4c1-1e05d0348ff8" (UID: "4f31729a-6bdf-40a6-b4c1-1e05d0348ff8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:03:50.281604 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:50.281464    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f31729a-6bdf-40a6-b4c1-1e05d0348ff8-kube-api-access-xlfhd" (OuterVolumeSpecName: "kube-api-access-xlfhd") pod "4f31729a-6bdf-40a6-b4c1-1e05d0348ff8" (UID: "4f31729a-6bdf-40a6-b4c1-1e05d0348ff8"). InnerVolumeSpecName "kube-api-access-xlfhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:03:50.290551 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:50.290524    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f31729a-6bdf-40a6-b4c1-1e05d0348ff8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4f31729a-6bdf-40a6-b4c1-1e05d0348ff8" (UID: "4f31729a-6bdf-40a6-b4c1-1e05d0348ff8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:03:50.302700 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:50.302677    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f31729a-6bdf-40a6-b4c1-1e05d0348ff8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4f31729a-6bdf-40a6-b4c1-1e05d0348ff8" (UID: "4f31729a-6bdf-40a6-b4c1-1e05d0348ff8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:03:50.379449 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:50.379416    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f31729a-6bdf-40a6-b4c1-1e05d0348ff8-cabundle-cert\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:03:50.379449 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:50.379440    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f31729a-6bdf-40a6-b4c1-1e05d0348ff8-home\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:03:50.379449 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:50.379449    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlfhd\" (UniqueName: \"kubernetes.io/projected/4f31729a-6bdf-40a6-b4c1-1e05d0348ff8-kube-api-access-xlfhd\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:03:50.379664 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:50.379478    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f31729a-6bdf-40a6-b4c1-1e05d0348ff8-tls-certs\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:03:50.379664 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:50.379488    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f31729a-6bdf-40a6-b4c1-1e05d0348ff8-kserve-provision-location\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:03:50.379664 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:50.379496    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f31729a-6bdf-40a6-b4c1-1e05d0348ff8-tmp-dir\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:03:50.379664 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:50.379505    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f31729a-6bdf-40a6-b4c1-1e05d0348ff8-dshm\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:03:50.659203 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:50.659170    2577 generic.go:356] "Generic (PLEG): container finished" podID="4f31729a-6bdf-40a6-b4c1-1e05d0348ff8" containerID="07bfcfcc5b3b51bbb88c63bcb532acfa344a2457c2d5b8a69ee4b07de34b72e8" exitCode=0
Sep 18 20:03:50.659624 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:50.659225    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-4bs5t"
Sep 18 20:03:50.659624 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:50.659233    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-4bs5t" event={"ID":"4f31729a-6bdf-40a6-b4c1-1e05d0348ff8","Type":"ContainerDied","Data":"07bfcfcc5b3b51bbb88c63bcb532acfa344a2457c2d5b8a69ee4b07de34b72e8"}
Sep 18 20:03:50.659624 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:50.659276    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-4bs5t" event={"ID":"4f31729a-6bdf-40a6-b4c1-1e05d0348ff8","Type":"ContainerDied","Data":"7d2293b85d89266b4011a311511b48c5de42979c570d45c6402dbd3f5219a08d"}
Sep 18 20:03:50.659624 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:50.659298    2577 scope.go:117] "RemoveContainer" containerID="07bfcfcc5b3b51bbb88c63bcb532acfa344a2457c2d5b8a69ee4b07de34b72e8"
Sep 18 20:03:50.668763 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:50.668749    2577 scope.go:117] "RemoveContainer" containerID="3d970ea8859830259254ef66093f5136fa22b7edc4d0133960167b372f9fb32d"
Sep 18 20:03:50.676729 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:50.676712    2577 scope.go:117] "RemoveContainer" containerID="07bfcfcc5b3b51bbb88c63bcb532acfa344a2457c2d5b8a69ee4b07de34b72e8"
Sep 18 20:03:50.677050 ip-10-0-129-30 kubenswrapper[2577]: E0918 20:03:50.677030    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07bfcfcc5b3b51bbb88c63bcb532acfa344a2457c2d5b8a69ee4b07de34b72e8\": container with ID starting with 07bfcfcc5b3b51bbb88c63bcb532acfa344a2457c2d5b8a69ee4b07de34b72e8 not found: ID does not exist" containerID="07bfcfcc5b3b51bbb88c63bcb532acfa344a2457c2d5b8a69ee4b07de34b72e8"
Sep 18 20:03:50.677128 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:50.677058    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07bfcfcc5b3b51bbb88c63bcb532acfa344a2457c2d5b8a69ee4b07de34b72e8"} err="failed to get container status \"07bfcfcc5b3b51bbb88c63bcb532acfa344a2457c2d5b8a69ee4b07de34b72e8\": rpc error: code = NotFound desc = could not find container \"07bfcfcc5b3b51bbb88c63bcb532acfa344a2457c2d5b8a69ee4b07de34b72e8\": container with ID starting with 07bfcfcc5b3b51bbb88c63bcb532acfa344a2457c2d5b8a69ee4b07de34b72e8 not found: ID does not exist"
Sep 18 20:03:50.677128 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:50.677079    2577 scope.go:117] "RemoveContainer" containerID="3d970ea8859830259254ef66093f5136fa22b7edc4d0133960167b372f9fb32d"
Sep 18 20:03:50.677357 ip-10-0-129-30 kubenswrapper[2577]: E0918 20:03:50.677338    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d970ea8859830259254ef66093f5136fa22b7edc4d0133960167b372f9fb32d\": container with ID starting with 3d970ea8859830259254ef66093f5136fa22b7edc4d0133960167b372f9fb32d not found: ID does not exist" containerID="3d970ea8859830259254ef66093f5136fa22b7edc4d0133960167b372f9fb32d"
Sep 18 20:03:50.677405 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:50.677360    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d970ea8859830259254ef66093f5136fa22b7edc4d0133960167b372f9fb32d"} err="failed to get container status \"3d970ea8859830259254ef66093f5136fa22b7edc4d0133960167b372f9fb32d\": rpc error: code = NotFound desc = could not find container \"3d970ea8859830259254ef66093f5136fa22b7edc4d0133960167b372f9fb32d\": container with ID starting with 3d970ea8859830259254ef66093f5136fa22b7edc4d0133960167b372f9fb32d not found: ID does not exist"
Sep 18 20:03:50.682114 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:50.682092    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-4bs5t"]
Sep 18 20:03:50.687871 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:50.687847    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-4bs5t"]
Sep 18 20:03:51.349919 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:51.349882    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-hxm9h"]
Sep 18 20:03:51.350448 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:51.350425    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f31729a-6bdf-40a6-b4c1-1e05d0348ff8" containerName="main"
Sep 18 20:03:51.350506 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:51.350453    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f31729a-6bdf-40a6-b4c1-1e05d0348ff8" containerName="main"
Sep 18 20:03:51.350506 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:51.350480    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f31729a-6bdf-40a6-b4c1-1e05d0348ff8" containerName="storage-initializer"
Sep 18 20:03:51.350506 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:51.350489    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f31729a-6bdf-40a6-b4c1-1e05d0348ff8" containerName="storage-initializer"
Sep 18 20:03:51.350611 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:51.350581    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f31729a-6bdf-40a6-b4c1-1e05d0348ff8" containerName="main"
Sep 18 20:03:51.354050 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:51.354030    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-hxm9h"
Sep 18 20:03:51.357106 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:51.357081    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:03:51.357255 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:51.357156    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 18 20:03:51.357255 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:51.357167    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 18 20:03:51.357255 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:51.357156    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-dc84j\""
Sep 18 20:03:51.357255 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:51.357174    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Sep 18 20:03:51.358885 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:51.358867    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 18 20:03:51.369453 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:51.369429    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-hxm9h"]
Sep 18 20:03:51.439257 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:51.439226    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f31729a-6bdf-40a6-b4c1-1e05d0348ff8" path="/var/lib/kubelet/pods/4f31729a-6bdf-40a6-b4c1-1e05d0348ff8/volumes"
Sep 18 20:03:51.489817 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:51.489774    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5e2aa4f-47de-4d5f-b372-095cd3afcbe0-dshm\") pod \"stop-feature-test-kserve-58d6cbd986-hxm9h\" (UID: \"c5e2aa4f-47de-4d5f-b372-095cd3afcbe0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-hxm9h"
Sep 18 20:03:51.490045 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:51.489823    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx6bq\" (UniqueName: \"kubernetes.io/projected/c5e2aa4f-47de-4d5f-b372-095cd3afcbe0-kube-api-access-fx6bq\") pod \"stop-feature-test-kserve-58d6cbd986-hxm9h\" (UID: \"c5e2aa4f-47de-4d5f-b372-095cd3afcbe0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-hxm9h"
Sep 18 20:03:51.490045 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:51.489888    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5e2aa4f-47de-4d5f-b372-095cd3afcbe0-model-cache\") pod \"stop-feature-test-kserve-58d6cbd986-hxm9h\" (UID: \"c5e2aa4f-47de-4d5f-b372-095cd3afcbe0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-hxm9h"
Sep 18 20:03:51.490045 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:51.489947    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5e2aa4f-47de-4d5f-b372-095cd3afcbe0-tmp-dir\") pod \"stop-feature-test-kserve-58d6cbd986-hxm9h\" (UID: \"c5e2aa4f-47de-4d5f-b372-095cd3afcbe0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-hxm9h"
Sep 18 20:03:51.490045 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:51.489980    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5e2aa4f-47de-4d5f-b372-095cd3afcbe0-cabundle-cert\") pod \"stop-feature-test-kserve-58d6cbd986-hxm9h\" (UID: \"c5e2aa4f-47de-4d5f-b372-095cd3afcbe0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-hxm9h"
Sep 18 20:03:51.490328 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:51.490086    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5e2aa4f-47de-4d5f-b372-095cd3afcbe0-home\") pod \"stop-feature-test-kserve-58d6cbd986-hxm9h\" (UID: \"c5e2aa4f-47de-4d5f-b372-095cd3afcbe0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-hxm9h"
Sep 18 20:03:51.490328 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:51.490138    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5e2aa4f-47de-4d5f-b372-095cd3afcbe0-kserve-provision-location\") pod \"stop-feature-test-kserve-58d6cbd986-hxm9h\" (UID: \"c5e2aa4f-47de-4d5f-b372-095cd3afcbe0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-hxm9h"
Sep 18 20:03:51.490328 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:51.490178    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5e2aa4f-47de-4d5f-b372-095cd3afcbe0-tls-certs\") pod \"stop-feature-test-kserve-58d6cbd986-hxm9h\" (UID: \"c5e2aa4f-47de-4d5f-b372-095cd3afcbe0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-hxm9h"
Sep 18 20:03:51.590743 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:51.590708    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5e2aa4f-47de-4d5f-b372-095cd3afcbe0-tls-certs\") pod \"stop-feature-test-kserve-58d6cbd986-hxm9h\" (UID: \"c5e2aa4f-47de-4d5f-b372-095cd3afcbe0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-hxm9h"
Sep 18 20:03:51.590947 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:51.590882    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5e2aa4f-47de-4d5f-b372-095cd3afcbe0-dshm\") pod \"stop-feature-test-kserve-58d6cbd986-hxm9h\" (UID: \"c5e2aa4f-47de-4d5f-b372-095cd3afcbe0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-hxm9h"
Sep 18 20:03:51.590947 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:51.590924    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx6bq\" (UniqueName: \"kubernetes.io/projected/c5e2aa4f-47de-4d5f-b372-095cd3afcbe0-kube-api-access-fx6bq\") pod \"stop-feature-test-kserve-58d6cbd986-hxm9h\" (UID: \"c5e2aa4f-47de-4d5f-b372-095cd3afcbe0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-hxm9h"
Sep 18 20:03:51.591107 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:51.590969    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5e2aa4f-47de-4d5f-b372-095cd3afcbe0-model-cache\") pod \"stop-feature-test-kserve-58d6cbd986-hxm9h\" (UID: \"c5e2aa4f-47de-4d5f-b372-095cd3afcbe0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-hxm9h"
Sep 18 20:03:51.591107 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:51.591071    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5e2aa4f-47de-4d5f-b372-095cd3afcbe0-tmp-dir\") pod \"stop-feature-test-kserve-58d6cbd986-hxm9h\" (UID: \"c5e2aa4f-47de-4d5f-b372-095cd3afcbe0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-hxm9h"
Sep 18 20:03:51.591215 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:51.591109    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5e2aa4f-47de-4d5f-b372-095cd3afcbe0-cabundle-cert\") pod \"stop-feature-test-kserve-58d6cbd986-hxm9h\" (UID: \"c5e2aa4f-47de-4d5f-b372-095cd3afcbe0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-hxm9h"
Sep 18 20:03:51.591215 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:51.591142    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5e2aa4f-47de-4d5f-b372-095cd3afcbe0-home\") pod \"stop-feature-test-kserve-58d6cbd986-hxm9h\" (UID: \"c5e2aa4f-47de-4d5f-b372-095cd3afcbe0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-hxm9h"
Sep 18 20:03:51.591215 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:51.591174    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5e2aa4f-47de-4d5f-b372-095cd3afcbe0-kserve-provision-location\") pod \"stop-feature-test-kserve-58d6cbd986-hxm9h\" (UID: \"c5e2aa4f-47de-4d5f-b372-095cd3afcbe0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-hxm9h"
Sep 18 20:03:51.591918 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:51.591883    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5e2aa4f-47de-4d5f-b372-095cd3afcbe0-tmp-dir\") pod \"stop-feature-test-kserve-58d6cbd986-hxm9h\" (UID: \"c5e2aa4f-47de-4d5f-b372-095cd3afcbe0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-hxm9h"
Sep 18 20:03:51.592076 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:51.591943    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5e2aa4f-47de-4d5f-b372-095cd3afcbe0-home\") pod \"stop-feature-test-kserve-58d6cbd986-hxm9h\" (UID: \"c5e2aa4f-47de-4d5f-b372-095cd3afcbe0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-hxm9h"
Sep 18 20:03:51.592076 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:51.591945    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5e2aa4f-47de-4d5f-b372-095cd3afcbe0-model-cache\") pod \"stop-feature-test-kserve-58d6cbd986-hxm9h\" (UID: \"c5e2aa4f-47de-4d5f-b372-095cd3afcbe0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-hxm9h"
Sep 18 20:03:51.592202 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:51.592178    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5e2aa4f-47de-4d5f-b372-095cd3afcbe0-kserve-provision-location\") pod \"stop-feature-test-kserve-58d6cbd986-hxm9h\" (UID: \"c5e2aa4f-47de-4d5f-b372-095cd3afcbe0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-hxm9h"
Sep 18 20:03:51.592334 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:51.592311    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5e2aa4f-47de-4d5f-b372-095cd3afcbe0-cabundle-cert\") pod \"stop-feature-test-kserve-58d6cbd986-hxm9h\" (UID: \"c5e2aa4f-47de-4d5f-b372-095cd3afcbe0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-hxm9h"
Sep 18 20:03:51.593970 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:51.593945    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5e2aa4f-47de-4d5f-b372-095cd3afcbe0-tls-certs\") pod \"stop-feature-test-kserve-58d6cbd986-hxm9h\" (UID: \"c5e2aa4f-47de-4d5f-b372-095cd3afcbe0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-hxm9h"
Sep 18 20:03:51.599555 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:51.599511    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5e2aa4f-47de-4d5f-b372-095cd3afcbe0-dshm\") pod \"stop-feature-test-kserve-58d6cbd986-hxm9h\" (UID: \"c5e2aa4f-47de-4d5f-b372-095cd3afcbe0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-hxm9h"
Sep 18 20:03:51.601548 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:51.601481    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx6bq\" (UniqueName: \"kubernetes.io/projected/c5e2aa4f-47de-4d5f-b372-095cd3afcbe0-kube-api-access-fx6bq\") pod \"stop-feature-test-kserve-58d6cbd986-hxm9h\" (UID: \"c5e2aa4f-47de-4d5f-b372-095cd3afcbe0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-hxm9h"
Sep 18 20:03:51.664559 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:51.664527    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-hxm9h"
Sep 18 20:03:51.720018 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:51.719003    2577 scope.go:117] "RemoveContainer" containerID="4db2a2200457d3887656f278572c9e9ea0b371d044d462a8fc23de67c908f1af"
Sep 18 20:03:51.731104 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:51.731070    2577 scope.go:117] "RemoveContainer" containerID="71a94759ad1e4fc006cd2f755ac31d49d7420085e92138070ba5d15a100e9b2a"
Sep 18 20:03:51.740080 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:51.739975    2577 scope.go:117] "RemoveContainer" containerID="28f1707b8a65fc8f6197ef620edd0fbf598a948577c508b9ef169e9ab490ff4f"
Sep 18 20:03:51.947733 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:51.947705    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-hxm9h"]
Sep 18 20:03:51.949615 ip-10-0-129-30 kubenswrapper[2577]: W0918 20:03:51.949587    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5e2aa4f_47de_4d5f_b372_095cd3afcbe0.slice/crio-b8485e624acb87c324485344cb5ae42e40fced83a63142e49b6b65c619c8dcee WatchSource:0}: Error finding container b8485e624acb87c324485344cb5ae42e40fced83a63142e49b6b65c619c8dcee: Status 404 returned error can't find the container with id b8485e624acb87c324485344cb5ae42e40fced83a63142e49b6b65c619c8dcee
Sep 18 20:03:52.671975 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:52.671940    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-hxm9h" event={"ID":"c5e2aa4f-47de-4d5f-b372-095cd3afcbe0","Type":"ContainerStarted","Data":"733aa6c0addd3403a4160dd40f41b2094857f2f7e337c14bac37e295460d8e25"}
Sep 18 20:03:52.672391 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:52.672016    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-hxm9h" event={"ID":"c5e2aa4f-47de-4d5f-b372-095cd3afcbe0","Type":"ContainerStarted","Data":"b8485e624acb87c324485344cb5ae42e40fced83a63142e49b6b65c619c8dcee"}
Sep 18 20:03:54.682427 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:54.682396    2577 generic.go:356] "Generic (PLEG): container finished" podID="c5e2aa4f-47de-4d5f-b372-095cd3afcbe0" containerID="733aa6c0addd3403a4160dd40f41b2094857f2f7e337c14bac37e295460d8e25" exitCode=0
Sep 18 20:03:54.682824 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:54.682445    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-hxm9h" event={"ID":"c5e2aa4f-47de-4d5f-b372-095cd3afcbe0","Type":"ContainerDied","Data":"733aa6c0addd3403a4160dd40f41b2094857f2f7e337c14bac37e295460d8e25"}
Sep 18 20:03:55.689312 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:55.689282    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-hxm9h" event={"ID":"c5e2aa4f-47de-4d5f-b372-095cd3afcbe0","Type":"ContainerStarted","Data":"4082523115da12b3c1b23bb478956773a1145cdbd043976d265c1b1f999b2d6f"}
Sep 18 20:03:55.722564 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:55.722511    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-hxm9h" podStartSLOduration=2.588708879 podStartE2EDuration="4.722493189s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.13378431s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:03:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:03:55.719959315 +0000 UTC m=+3006.572259685" watchObservedRunningTime="2026-09-18 20:03:55.722493189 +0000 UTC m=+3006.574793559"
Sep 18 20:03:56.665103 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:03:56.665070    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-hxm9h"
Sep 18 20:04:01.664736 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:04:01.664688    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-hxm9h"
Sep 18 20:04:01.666414 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:04:01.666386    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-hxm9h" podUID="c5e2aa4f-47de-4d5f-b372-095cd3afcbe0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Sep 18 20:04:11.665256 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:04:11.665217    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-hxm9h" podUID="c5e2aa4f-47de-4d5f-b372-095cd3afcbe0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Sep 18 20:04:21.665889 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:04:21.665840    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-hxm9h" podUID="c5e2aa4f-47de-4d5f-b372-095cd3afcbe0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Sep 18 20:04:31.665075 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:04:31.665035    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-hxm9h" podUID="c5e2aa4f-47de-4d5f-b372-095cd3afcbe0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Sep 18 20:04:41.665068 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:04:41.665021    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-hxm9h" podUID="c5e2aa4f-47de-4d5f-b372-095cd3afcbe0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Sep 18 20:04:51.666019 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:04:51.665952    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-hxm9h" podUID="c5e2aa4f-47de-4d5f-b372-095cd3afcbe0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Sep 18 20:05:01.665784 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:05:01.665740    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-hxm9h" podUID="c5e2aa4f-47de-4d5f-b372-095cd3afcbe0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Sep 18 20:05:11.665310 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:05:11.665256    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-hxm9h" podUID="c5e2aa4f-47de-4d5f-b372-095cd3afcbe0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Sep 18 20:05:21.665639 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:05:21.665591    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-hxm9h" podUID="c5e2aa4f-47de-4d5f-b372-095cd3afcbe0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Sep 18 20:05:31.665230 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:05:31.665190    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-hxm9h" podUID="c5e2aa4f-47de-4d5f-b372-095cd3afcbe0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Sep 18 20:05:41.665464 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:05:41.665418    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-hxm9h" podUID="c5e2aa4f-47de-4d5f-b372-095cd3afcbe0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Sep 18 20:05:51.675553 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:05:51.675516    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-hxm9h"
Sep 18 20:05:51.683347 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:05:51.683322    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-hxm9h"
Sep 18 20:05:55.372122 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:05:55.372085    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-qmhbh"]
Sep 18 20:05:55.372594 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:05:55.372290    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-qmhbh" podUID="2235b1aa-df85-42e6-b986-44a0f92473f5" containerName="istio-proxy" containerID="cri-o://331da3ac42b426d971fa094d71bcabce50839575e12163f54b6f500f88a8fc4e" gracePeriod=30
Sep 18 20:06:00.619703 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:00.619675    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-qmhbh"
Sep 18 20:06:00.698255 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:00.698222    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hxjwp\" (UniqueName: \"kubernetes.io/projected/2235b1aa-df85-42e6-b986-44a0f92473f5-kube-api-access-hxjwp\") pod \"2235b1aa-df85-42e6-b986-44a0f92473f5\" (UID: \"2235b1aa-df85-42e6-b986-44a0f92473f5\") "
Sep 18 20:06:00.698255 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:00.698262    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2235b1aa-df85-42e6-b986-44a0f92473f5-workload-certs\") pod \"2235b1aa-df85-42e6-b986-44a0f92473f5\" (UID: \"2235b1aa-df85-42e6-b986-44a0f92473f5\") "
Sep 18 20:06:00.698500 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:00.698325    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2235b1aa-df85-42e6-b986-44a0f92473f5-istio-data\") pod \"2235b1aa-df85-42e6-b986-44a0f92473f5\" (UID: \"2235b1aa-df85-42e6-b986-44a0f92473f5\") "
Sep 18 20:06:00.698500 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:00.698343    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2235b1aa-df85-42e6-b986-44a0f92473f5-istio-envoy\") pod \"2235b1aa-df85-42e6-b986-44a0f92473f5\" (UID: \"2235b1aa-df85-42e6-b986-44a0f92473f5\") "
Sep 18 20:06:00.698500 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:00.698377    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2235b1aa-df85-42e6-b986-44a0f92473f5-istio-podinfo\") pod \"2235b1aa-df85-42e6-b986-44a0f92473f5\" (UID: \"2235b1aa-df85-42e6-b986-44a0f92473f5\") "
Sep 18 20:06:00.698500 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:00.698428    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2235b1aa-df85-42e6-b986-44a0f92473f5-istiod-ca-cert\") pod \"2235b1aa-df85-42e6-b986-44a0f92473f5\" (UID: \"2235b1aa-df85-42e6-b986-44a0f92473f5\") "
Sep 18 20:06:00.698500 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:00.698451    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2235b1aa-df85-42e6-b986-44a0f92473f5-workload-socket\") pod \"2235b1aa-df85-42e6-b986-44a0f92473f5\" (UID: \"2235b1aa-df85-42e6-b986-44a0f92473f5\") "
Sep 18 20:06:00.698500 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:00.698501    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2235b1aa-df85-42e6-b986-44a0f92473f5-istio-token\") pod \"2235b1aa-df85-42e6-b986-44a0f92473f5\" (UID: \"2235b1aa-df85-42e6-b986-44a0f92473f5\") "
Sep 18 20:06:00.698827 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:00.698553    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2235b1aa-df85-42e6-b986-44a0f92473f5-credential-socket\") pod \"2235b1aa-df85-42e6-b986-44a0f92473f5\" (UID: \"2235b1aa-df85-42e6-b986-44a0f92473f5\") "
Sep 18 20:06:00.698827 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:00.698648    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2235b1aa-df85-42e6-b986-44a0f92473f5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2235b1aa-df85-42e6-b986-44a0f92473f5" (UID: "2235b1aa-df85-42e6-b986-44a0f92473f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:06:00.698827 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:00.698660    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2235b1aa-df85-42e6-b986-44a0f92473f5-istio-data" (OuterVolumeSpecName: "istio-data") pod "2235b1aa-df85-42e6-b986-44a0f92473f5" (UID: "2235b1aa-df85-42e6-b986-44a0f92473f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:06:00.699007 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:00.698903    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2235b1aa-df85-42e6-b986-44a0f92473f5-workload-certs\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:06:00.699007 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:00.698927    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2235b1aa-df85-42e6-b986-44a0f92473f5-istio-data\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:06:00.699007 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:00.698959    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2235b1aa-df85-42e6-b986-44a0f92473f5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2235b1aa-df85-42e6-b986-44a0f92473f5" (UID: "2235b1aa-df85-42e6-b986-44a0f92473f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:06:00.699191 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:00.698921    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2235b1aa-df85-42e6-b986-44a0f92473f5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2235b1aa-df85-42e6-b986-44a0f92473f5" (UID: "2235b1aa-df85-42e6-b986-44a0f92473f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:06:00.699270 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:00.699184    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2235b1aa-df85-42e6-b986-44a0f92473f5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2235b1aa-df85-42e6-b986-44a0f92473f5" (UID: "2235b1aa-df85-42e6-b986-44a0f92473f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:06:00.701053 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:00.701024    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2235b1aa-df85-42e6-b986-44a0f92473f5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2235b1aa-df85-42e6-b986-44a0f92473f5" (UID: "2235b1aa-df85-42e6-b986-44a0f92473f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:06:00.701180 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:00.701072    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2235b1aa-df85-42e6-b986-44a0f92473f5-kube-api-access-hxjwp" (OuterVolumeSpecName: "kube-api-access-hxjwp") pod "2235b1aa-df85-42e6-b986-44a0f92473f5" (UID: "2235b1aa-df85-42e6-b986-44a0f92473f5"). InnerVolumeSpecName "kube-api-access-hxjwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:06:00.701239 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:00.701174    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2235b1aa-df85-42e6-b986-44a0f92473f5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2235b1aa-df85-42e6-b986-44a0f92473f5" (UID: "2235b1aa-df85-42e6-b986-44a0f92473f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 20:06:00.701370 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:00.701342    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2235b1aa-df85-42e6-b986-44a0f92473f5-istio-token" (OuterVolumeSpecName: "istio-token") pod "2235b1aa-df85-42e6-b986-44a0f92473f5" (UID: "2235b1aa-df85-42e6-b986-44a0f92473f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:06:00.799720 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:00.799680    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2235b1aa-df85-42e6-b986-44a0f92473f5-credential-socket\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:06:00.799720 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:00.799712    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxjwp\" (UniqueName: \"kubernetes.io/projected/2235b1aa-df85-42e6-b986-44a0f92473f5-kube-api-access-hxjwp\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:06:00.799720 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:00.799722    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2235b1aa-df85-42e6-b986-44a0f92473f5-istio-envoy\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:06:00.799720 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:00.799731    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2235b1aa-df85-42e6-b986-44a0f92473f5-istio-podinfo\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:06:00.800027 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:00.799739    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2235b1aa-df85-42e6-b986-44a0f92473f5-istiod-ca-cert\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:06:00.800027 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:00.799748    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2235b1aa-df85-42e6-b986-44a0f92473f5-workload-socket\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:06:00.800027 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:00.799756    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2235b1aa-df85-42e6-b986-44a0f92473f5-istio-token\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:06:01.285520 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:01.285480    2577 generic.go:356] "Generic (PLEG): container finished" podID="2235b1aa-df85-42e6-b986-44a0f92473f5" containerID="331da3ac42b426d971fa094d71bcabce50839575e12163f54b6f500f88a8fc4e" exitCode=0
Sep 18 20:06:01.285726 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:01.285533    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-qmhbh"
Sep 18 20:06:01.285726 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:01.285564    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-qmhbh" event={"ID":"2235b1aa-df85-42e6-b986-44a0f92473f5","Type":"ContainerDied","Data":"331da3ac42b426d971fa094d71bcabce50839575e12163f54b6f500f88a8fc4e"}
Sep 18 20:06:01.285726 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:01.285618    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-qmhbh" event={"ID":"2235b1aa-df85-42e6-b986-44a0f92473f5","Type":"ContainerDied","Data":"96f828a92857211ddd645e069e5c7133d67a6c484da2e0d442ac855e5d4518ca"}
Sep 18 20:06:01.285726 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:01.285641    2577 scope.go:117] "RemoveContainer" containerID="331da3ac42b426d971fa094d71bcabce50839575e12163f54b6f500f88a8fc4e"
Sep 18 20:06:01.295868 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:01.295850    2577 scope.go:117] "RemoveContainer" containerID="5789135a1d6cac1707474a44e6c088b9d41a31af9f66b38edb3748f7d082e2a6"
Sep 18 20:06:01.304228 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:01.304205    2577 scope.go:117] "RemoveContainer" containerID="331da3ac42b426d971fa094d71bcabce50839575e12163f54b6f500f88a8fc4e"
Sep 18 20:06:01.304540 ip-10-0-129-30 kubenswrapper[2577]: E0918 20:06:01.304513    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"331da3ac42b426d971fa094d71bcabce50839575e12163f54b6f500f88a8fc4e\": container with ID starting with 331da3ac42b426d971fa094d71bcabce50839575e12163f54b6f500f88a8fc4e not found: ID does not exist" containerID="331da3ac42b426d971fa094d71bcabce50839575e12163f54b6f500f88a8fc4e"
Sep 18 20:06:01.304584 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:01.304553    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"331da3ac42b426d971fa094d71bcabce50839575e12163f54b6f500f88a8fc4e"} err="failed to get container status \"331da3ac42b426d971fa094d71bcabce50839575e12163f54b6f500f88a8fc4e\": rpc error: code = NotFound desc = could not find container \"331da3ac42b426d971fa094d71bcabce50839575e12163f54b6f500f88a8fc4e\": container with ID starting with 331da3ac42b426d971fa094d71bcabce50839575e12163f54b6f500f88a8fc4e not found: ID does not exist"
Sep 18 20:06:01.304584 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:01.304575    2577 scope.go:117] "RemoveContainer" containerID="5789135a1d6cac1707474a44e6c088b9d41a31af9f66b38edb3748f7d082e2a6"
Sep 18 20:06:01.304815 ip-10-0-129-30 kubenswrapper[2577]: E0918 20:06:01.304796    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5789135a1d6cac1707474a44e6c088b9d41a31af9f66b38edb3748f7d082e2a6\": container with ID starting with 5789135a1d6cac1707474a44e6c088b9d41a31af9f66b38edb3748f7d082e2a6 not found: ID does not exist" containerID="5789135a1d6cac1707474a44e6c088b9d41a31af9f66b38edb3748f7d082e2a6"
Sep 18 20:06:01.304865 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:01.304821    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5789135a1d6cac1707474a44e6c088b9d41a31af9f66b38edb3748f7d082e2a6"} err="failed to get container status \"5789135a1d6cac1707474a44e6c088b9d41a31af9f66b38edb3748f7d082e2a6\": rpc error: code = NotFound desc = could not find container \"5789135a1d6cac1707474a44e6c088b9d41a31af9f66b38edb3748f7d082e2a6\": container with ID starting with 5789135a1d6cac1707474a44e6c088b9d41a31af9f66b38edb3748f7d082e2a6 not found: ID does not exist"
Sep 18 20:06:01.310660 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:01.310634    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-qmhbh"]
Sep 18 20:06:01.317375 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:01.317350    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-qmhbh"]
Sep 18 20:06:01.444208 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:01.444170    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2235b1aa-df85-42e6-b986-44a0f92473f5" path="/var/lib/kubelet/pods/2235b1aa-df85-42e6-b986-44a0f92473f5/volumes"
Sep 18 20:06:14.529420 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:14.529183    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-hxm9h"]
Sep 18 20:06:29.551442 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:29.551395    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-hxm9h" podUID="c5e2aa4f-47de-4d5f-b372-095cd3afcbe0" containerName="main" containerID="cri-o://4082523115da12b3c1b23bb478956773a1145cdbd043976d265c1b1f999b2d6f" gracePeriod=45
Sep 18 20:06:29.676912 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:29.676872    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-hxm9h" podUID="c5e2aa4f-47de-4d5f-b372-095cd3afcbe0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Sep 18 20:06:30.226253 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:30.226226    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-hxm9h"
Sep 18 20:06:30.271620 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:30.271585    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5e2aa4f-47de-4d5f-b372-095cd3afcbe0-model-cache\") pod \"c5e2aa4f-47de-4d5f-b372-095cd3afcbe0\" (UID: \"c5e2aa4f-47de-4d5f-b372-095cd3afcbe0\") "
Sep 18 20:06:30.271792 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:30.271634    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5e2aa4f-47de-4d5f-b372-095cd3afcbe0-kserve-provision-location\") pod \"c5e2aa4f-47de-4d5f-b372-095cd3afcbe0\" (UID: \"c5e2aa4f-47de-4d5f-b372-095cd3afcbe0\") "
Sep 18 20:06:30.271792 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:30.271670    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5e2aa4f-47de-4d5f-b372-095cd3afcbe0-cabundle-cert\") pod \"c5e2aa4f-47de-4d5f-b372-095cd3afcbe0\" (UID: \"c5e2aa4f-47de-4d5f-b372-095cd3afcbe0\") "
Sep 18 20:06:30.271792 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:30.271733    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5e2aa4f-47de-4d5f-b372-095cd3afcbe0-tls-certs\") pod \"c5e2aa4f-47de-4d5f-b372-095cd3afcbe0\" (UID: \"c5e2aa4f-47de-4d5f-b372-095cd3afcbe0\") "
Sep 18 20:06:30.271792 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:30.271781    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fx6bq\" (UniqueName: \"kubernetes.io/projected/c5e2aa4f-47de-4d5f-b372-095cd3afcbe0-kube-api-access-fx6bq\") pod \"c5e2aa4f-47de-4d5f-b372-095cd3afcbe0\" (UID: \"c5e2aa4f-47de-4d5f-b372-095cd3afcbe0\") "
Sep 18 20:06:30.272060 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:30.271813    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5e2aa4f-47de-4d5f-b372-095cd3afcbe0-dshm\") pod \"c5e2aa4f-47de-4d5f-b372-095cd3afcbe0\" (UID: \"c5e2aa4f-47de-4d5f-b372-095cd3afcbe0\") "
Sep 18 20:06:30.272060 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:30.271818    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e2aa4f-47de-4d5f-b372-095cd3afcbe0-model-cache" (OuterVolumeSpecName: "model-cache") pod "c5e2aa4f-47de-4d5f-b372-095cd3afcbe0" (UID: "c5e2aa4f-47de-4d5f-b372-095cd3afcbe0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:06:30.272060 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:30.271838    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5e2aa4f-47de-4d5f-b372-095cd3afcbe0-home\") pod \"c5e2aa4f-47de-4d5f-b372-095cd3afcbe0\" (UID: \"c5e2aa4f-47de-4d5f-b372-095cd3afcbe0\") "
Sep 18 20:06:30.272060 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:30.271863    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5e2aa4f-47de-4d5f-b372-095cd3afcbe0-tmp-dir\") pod \"c5e2aa4f-47de-4d5f-b372-095cd3afcbe0\" (UID: \"c5e2aa4f-47de-4d5f-b372-095cd3afcbe0\") "
Sep 18 20:06:30.272272 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:30.272071    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5e2aa4f-47de-4d5f-b372-095cd3afcbe0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c5e2aa4f-47de-4d5f-b372-095cd3afcbe0" (UID: "c5e2aa4f-47de-4d5f-b372-095cd3afcbe0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:06:30.272272 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:30.272154    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5e2aa4f-47de-4d5f-b372-095cd3afcbe0-model-cache\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:06:30.272272 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:30.272173    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5e2aa4f-47de-4d5f-b372-095cd3afcbe0-cabundle-cert\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:06:30.272676 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:30.272648    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e2aa4f-47de-4d5f-b372-095cd3afcbe0-home" (OuterVolumeSpecName: "home") pod "c5e2aa4f-47de-4d5f-b372-095cd3afcbe0" (UID: "c5e2aa4f-47de-4d5f-b372-095cd3afcbe0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:06:30.274161 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:30.274131    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5e2aa4f-47de-4d5f-b372-095cd3afcbe0-kube-api-access-fx6bq" (OuterVolumeSpecName: "kube-api-access-fx6bq") pod "c5e2aa4f-47de-4d5f-b372-095cd3afcbe0" (UID: "c5e2aa4f-47de-4d5f-b372-095cd3afcbe0"). InnerVolumeSpecName "kube-api-access-fx6bq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:06:30.274366 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:30.274336    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5e2aa4f-47de-4d5f-b372-095cd3afcbe0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c5e2aa4f-47de-4d5f-b372-095cd3afcbe0" (UID: "c5e2aa4f-47de-4d5f-b372-095cd3afcbe0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:06:30.274508 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:30.274493    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e2aa4f-47de-4d5f-b372-095cd3afcbe0-dshm" (OuterVolumeSpecName: "dshm") pod "c5e2aa4f-47de-4d5f-b372-095cd3afcbe0" (UID: "c5e2aa4f-47de-4d5f-b372-095cd3afcbe0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:06:30.292030 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:30.291999    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e2aa4f-47de-4d5f-b372-095cd3afcbe0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c5e2aa4f-47de-4d5f-b372-095cd3afcbe0" (UID: "c5e2aa4f-47de-4d5f-b372-095cd3afcbe0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:06:30.303517 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:30.303493    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e2aa4f-47de-4d5f-b372-095cd3afcbe0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c5e2aa4f-47de-4d5f-b372-095cd3afcbe0" (UID: "c5e2aa4f-47de-4d5f-b372-095cd3afcbe0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:06:30.372814 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:30.372785    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5e2aa4f-47de-4d5f-b372-095cd3afcbe0-kserve-provision-location\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:06:30.372814 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:30.372815    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5e2aa4f-47de-4d5f-b372-095cd3afcbe0-tls-certs\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:06:30.373014 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:30.372831    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx6bq\" (UniqueName: \"kubernetes.io/projected/c5e2aa4f-47de-4d5f-b372-095cd3afcbe0-kube-api-access-fx6bq\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:06:30.373014 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:30.372842    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5e2aa4f-47de-4d5f-b372-095cd3afcbe0-dshm\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:06:30.373014 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:30.372852    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5e2aa4f-47de-4d5f-b372-095cd3afcbe0-home\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:06:30.373014 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:30.372863    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5e2aa4f-47de-4d5f-b372-095cd3afcbe0-tmp-dir\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:06:30.423129 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:30.423098    2577 generic.go:356] "Generic (PLEG): container finished" podID="c5e2aa4f-47de-4d5f-b372-095cd3afcbe0" containerID="4082523115da12b3c1b23bb478956773a1145cdbd043976d265c1b1f999b2d6f" exitCode=0
Sep 18 20:06:30.423263 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:30.423146    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-hxm9h" event={"ID":"c5e2aa4f-47de-4d5f-b372-095cd3afcbe0","Type":"ContainerDied","Data":"4082523115da12b3c1b23bb478956773a1145cdbd043976d265c1b1f999b2d6f"}
Sep 18 20:06:30.423263 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:30.423156    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-hxm9h"
Sep 18 20:06:30.423263 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:30.423238    2577 scope.go:117] "RemoveContainer" containerID="4082523115da12b3c1b23bb478956773a1145cdbd043976d265c1b1f999b2d6f"
Sep 18 20:06:30.423426 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:30.423227    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-hxm9h" event={"ID":"c5e2aa4f-47de-4d5f-b372-095cd3afcbe0","Type":"ContainerDied","Data":"b8485e624acb87c324485344cb5ae42e40fced83a63142e49b6b65c619c8dcee"}
Sep 18 20:06:30.432004 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:30.431970    2577 scope.go:117] "RemoveContainer" containerID="733aa6c0addd3403a4160dd40f41b2094857f2f7e337c14bac37e295460d8e25"
Sep 18 20:06:30.439260 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:30.439242    2577 scope.go:117] "RemoveContainer" containerID="4082523115da12b3c1b23bb478956773a1145cdbd043976d265c1b1f999b2d6f"
Sep 18 20:06:30.439511 ip-10-0-129-30 kubenswrapper[2577]: E0918 20:06:30.439492    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4082523115da12b3c1b23bb478956773a1145cdbd043976d265c1b1f999b2d6f\": container with ID starting with 4082523115da12b3c1b23bb478956773a1145cdbd043976d265c1b1f999b2d6f not found: ID does not exist" containerID="4082523115da12b3c1b23bb478956773a1145cdbd043976d265c1b1f999b2d6f"
Sep 18 20:06:30.439574 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:30.439517    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4082523115da12b3c1b23bb478956773a1145cdbd043976d265c1b1f999b2d6f"} err="failed to get container status \"4082523115da12b3c1b23bb478956773a1145cdbd043976d265c1b1f999b2d6f\": rpc error: code = NotFound desc = could not find container \"4082523115da12b3c1b23bb478956773a1145cdbd043976d265c1b1f999b2d6f\": container with ID starting with 4082523115da12b3c1b23bb478956773a1145cdbd043976d265c1b1f999b2d6f not found: ID does not exist"
Sep 18 20:06:30.439574 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:30.439532    2577 scope.go:117] "RemoveContainer" containerID="733aa6c0addd3403a4160dd40f41b2094857f2f7e337c14bac37e295460d8e25"
Sep 18 20:06:30.439763 ip-10-0-129-30 kubenswrapper[2577]: E0918 20:06:30.439747    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"733aa6c0addd3403a4160dd40f41b2094857f2f7e337c14bac37e295460d8e25\": container with ID starting with 733aa6c0addd3403a4160dd40f41b2094857f2f7e337c14bac37e295460d8e25 not found: ID does not exist" containerID="733aa6c0addd3403a4160dd40f41b2094857f2f7e337c14bac37e295460d8e25"
Sep 18 20:06:30.439803 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:30.439768    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"733aa6c0addd3403a4160dd40f41b2094857f2f7e337c14bac37e295460d8e25"} err="failed to get container status \"733aa6c0addd3403a4160dd40f41b2094857f2f7e337c14bac37e295460d8e25\": rpc error: code = NotFound desc = could not find container \"733aa6c0addd3403a4160dd40f41b2094857f2f7e337c14bac37e295460d8e25\": container with ID starting with 733aa6c0addd3403a4160dd40f41b2094857f2f7e337c14bac37e295460d8e25 not found: ID does not exist"
Sep 18 20:06:30.453027 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:30.452976    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-hxm9h"]
Sep 18 20:06:30.458239 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:30.458216    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d6cbd986-hxm9h"]
Sep 18 20:06:31.437795 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:06:31.437762    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5e2aa4f-47de-4d5f-b372-095cd3afcbe0" path="/var/lib/kubelet/pods/c5e2aa4f-47de-4d5f-b372-095cd3afcbe0/volumes"
Sep 18 20:07:09.290924 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:07:09.290890    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-86b55c56d-2ff8d"]
Sep 18 20:07:09.291429 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:07:09.291210    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5e2aa4f-47de-4d5f-b372-095cd3afcbe0" containerName="storage-initializer"
Sep 18 20:07:09.291429 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:07:09.291222    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5e2aa4f-47de-4d5f-b372-095cd3afcbe0" containerName="storage-initializer"
Sep 18 20:07:09.291429 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:07:09.291237    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2235b1aa-df85-42e6-b986-44a0f92473f5" containerName="istio-proxy"
Sep 18 20:07:09.291429 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:07:09.291243    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2235b1aa-df85-42e6-b986-44a0f92473f5" containerName="istio-proxy"
Sep 18 20:07:09.291429 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:07:09.291254    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2235b1aa-df85-42e6-b986-44a0f92473f5" containerName="istio-proxy"
Sep 18 20:07:09.291429 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:07:09.291260    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2235b1aa-df85-42e6-b986-44a0f92473f5" containerName="istio-proxy"
Sep 18 20:07:09.291429 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:07:09.291269    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5e2aa4f-47de-4d5f-b372-095cd3afcbe0" containerName="main"
Sep 18 20:07:09.291429 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:07:09.291274    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5e2aa4f-47de-4d5f-b372-095cd3afcbe0" containerName="main"
Sep 18 20:07:09.291429 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:07:09.291324    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5e2aa4f-47de-4d5f-b372-095cd3afcbe0" containerName="main"
Sep 18 20:07:09.291429 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:07:09.291332    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2235b1aa-df85-42e6-b986-44a0f92473f5" containerName="istio-proxy"
Sep 18 20:07:09.291429 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:07:09.291356    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2235b1aa-df85-42e6-b986-44a0f92473f5" containerName="istio-proxy"
Sep 18 20:07:09.295730 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:07:09.295713    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-86b55c56d-2ff8d"
Sep 18 20:07:09.299144 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:07:09.299120    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"seaweedfs-s3-creds\""
Sep 18 20:07:09.299403 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:07:09.299386    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"default-dockercfg-xgfv7\""
Sep 18 20:07:09.299666 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:07:09.299640    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"kube-root-ca.crt\""
Sep 18 20:07:09.300681 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:07:09.300668    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"openshift-service-ca.crt\""
Sep 18 20:07:09.300836 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:07:09.300808    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:07:09.300941 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:07:09.300841    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Sep 18 20:07:09.317847 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:07:09.317816    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-86b55c56d-2ff8d"]
Sep 18 20:07:09.413267 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:07:09.413227    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2208afc9-cf15-4106-af0f-a52569203a69-home\") pod \"lora-single-adapter-test-kserve-86b55c56d-2ff8d\" (UID: \"2208afc9-cf15-4106-af0f-a52569203a69\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-86b55c56d-2ff8d"
Sep 18 20:07:09.413455 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:07:09.413284    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2208afc9-cf15-4106-af0f-a52569203a69-dshm\") pod \"lora-single-adapter-test-kserve-86b55c56d-2ff8d\" (UID: \"2208afc9-cf15-4106-af0f-a52569203a69\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-86b55c56d-2ff8d"
Sep 18 20:07:09.413455 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:07:09.413330    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2208afc9-cf15-4106-af0f-a52569203a69-cabundle-cert\") pod \"lora-single-adapter-test-kserve-86b55c56d-2ff8d\" (UID: \"2208afc9-cf15-4106-af0f-a52569203a69\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-86b55c56d-2ff8d"
Sep 18 20:07:09.413455 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:07:09.413348    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2208afc9-cf15-4106-af0f-a52569203a69-tmp-dir\") pod \"lora-single-adapter-test-kserve-86b55c56d-2ff8d\" (UID: \"2208afc9-cf15-4106-af0f-a52569203a69\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-86b55c56d-2ff8d"
Sep 18 20:07:09.413455 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:07:09.413421    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m7kj\" (UniqueName: \"kubernetes.io/projected/2208afc9-cf15-4106-af0f-a52569203a69-kube-api-access-6m7kj\") pod \"lora-single-adapter-test-kserve-86b55c56d-2ff8d\" (UID: \"2208afc9-cf15-4106-af0f-a52569203a69\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-86b55c56d-2ff8d"
Sep 18 20:07:09.413650 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:07:09.413457    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2208afc9-cf15-4106-af0f-a52569203a69-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-86b55c56d-2ff8d\" (UID: \"2208afc9-cf15-4106-af0f-a52569203a69\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-86b55c56d-2ff8d"
Sep 18 20:07:09.413650 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:07:09.413498    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2208afc9-cf15-4106-af0f-a52569203a69-model-cache\") pod \"lora-single-adapter-test-kserve-86b55c56d-2ff8d\" (UID: \"2208afc9-cf15-4106-af0f-a52569203a69\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-86b55c56d-2ff8d"
Sep 18 20:07:09.413650 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:07:09.413516    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2208afc9-cf15-4106-af0f-a52569203a69-tls-certs\") pod \"lora-single-adapter-test-kserve-86b55c56d-2ff8d\" (UID: \"2208afc9-cf15-4106-af0f-a52569203a69\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-86b55c56d-2ff8d"
Sep 18 20:07:09.514344 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:07:09.514308    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m7kj\" (UniqueName: \"kubernetes.io/projected/2208afc9-cf15-4106-af0f-a52569203a69-kube-api-access-6m7kj\") pod \"lora-single-adapter-test-kserve-86b55c56d-2ff8d\" (UID: \"2208afc9-cf15-4106-af0f-a52569203a69\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-86b55c56d-2ff8d"
Sep 18 20:07:09.514344 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:07:09.514350    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2208afc9-cf15-4106-af0f-a52569203a69-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-86b55c56d-2ff8d\" (UID: \"2208afc9-cf15-4106-af0f-a52569203a69\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-86b55c56d-2ff8d"
Sep 18 20:07:09.514619 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:07:09.514387    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2208afc9-cf15-4106-af0f-a52569203a69-model-cache\") pod \"lora-single-adapter-test-kserve-86b55c56d-2ff8d\" (UID: \"2208afc9-cf15-4106-af0f-a52569203a69\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-86b55c56d-2ff8d"
Sep 18 20:07:09.514619 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:07:09.514406    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2208afc9-cf15-4106-af0f-a52569203a69-tls-certs\") pod \"lora-single-adapter-test-kserve-86b55c56d-2ff8d\" (UID: \"2208afc9-cf15-4106-af0f-a52569203a69\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-86b55c56d-2ff8d"
Sep 18 20:07:09.514619 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:07:09.514446    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2208afc9-cf15-4106-af0f-a52569203a69-home\") pod \"lora-single-adapter-test-kserve-86b55c56d-2ff8d\" (UID: \"2208afc9-cf15-4106-af0f-a52569203a69\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-86b55c56d-2ff8d"
Sep 18 20:07:09.514619 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:07:09.514474    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2208afc9-cf15-4106-af0f-a52569203a69-dshm\") pod \"lora-single-adapter-test-kserve-86b55c56d-2ff8d\" (UID: \"2208afc9-cf15-4106-af0f-a52569203a69\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-86b55c56d-2ff8d"
Sep 18 20:07:09.514619 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:07:09.514500    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2208afc9-cf15-4106-af0f-a52569203a69-cabundle-cert\") pod \"lora-single-adapter-test-kserve-86b55c56d-2ff8d\" (UID: \"2208afc9-cf15-4106-af0f-a52569203a69\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-86b55c56d-2ff8d"
Sep 18 20:07:09.514619 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:07:09.514527    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2208afc9-cf15-4106-af0f-a52569203a69-tmp-dir\") pod \"lora-single-adapter-test-kserve-86b55c56d-2ff8d\" (UID: \"2208afc9-cf15-4106-af0f-a52569203a69\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-86b55c56d-2ff8d"
Sep 18 20:07:09.514905 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:07:09.514880    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2208afc9-cf15-4106-af0f-a52569203a69-model-cache\") pod \"lora-single-adapter-test-kserve-86b55c56d-2ff8d\" (UID: \"2208afc9-cf15-4106-af0f-a52569203a69\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-86b55c56d-2ff8d"
Sep 18 20:07:09.514970 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:07:09.514913    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2208afc9-cf15-4106-af0f-a52569203a69-home\") pod \"lora-single-adapter-test-kserve-86b55c56d-2ff8d\" (UID: \"2208afc9-cf15-4106-af0f-a52569203a69\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-86b55c56d-2ff8d"
Sep 18 20:07:09.514970 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:07:09.514931    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2208afc9-cf15-4106-af0f-a52569203a69-tmp-dir\") pod \"lora-single-adapter-test-kserve-86b55c56d-2ff8d\" (UID: \"2208afc9-cf15-4106-af0f-a52569203a69\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-86b55c56d-2ff8d"
Sep 18 20:07:09.515126 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:07:09.515017    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2208afc9-cf15-4106-af0f-a52569203a69-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-86b55c56d-2ff8d\" (UID: \"2208afc9-cf15-4106-af0f-a52569203a69\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-86b55c56d-2ff8d"
Sep 18 20:07:09.515392 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:07:09.515369    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2208afc9-cf15-4106-af0f-a52569203a69-cabundle-cert\") pod \"lora-single-adapter-test-kserve-86b55c56d-2ff8d\" (UID: \"2208afc9-cf15-4106-af0f-a52569203a69\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-86b55c56d-2ff8d"
Sep 18 20:07:09.516932 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:07:09.516909    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2208afc9-cf15-4106-af0f-a52569203a69-dshm\") pod \"lora-single-adapter-test-kserve-86b55c56d-2ff8d\" (UID: \"2208afc9-cf15-4106-af0f-a52569203a69\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-86b55c56d-2ff8d"
Sep 18 20:07:09.517212 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:07:09.517196    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2208afc9-cf15-4106-af0f-a52569203a69-tls-certs\") pod \"lora-single-adapter-test-kserve-86b55c56d-2ff8d\" (UID: \"2208afc9-cf15-4106-af0f-a52569203a69\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-86b55c56d-2ff8d"
Sep 18 20:07:09.524003 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:07:09.523965    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m7kj\" (UniqueName: \"kubernetes.io/projected/2208afc9-cf15-4106-af0f-a52569203a69-kube-api-access-6m7kj\") pod \"lora-single-adapter-test-kserve-86b55c56d-2ff8d\" (UID: \"2208afc9-cf15-4106-af0f-a52569203a69\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-86b55c56d-2ff8d"
Sep 18 20:07:09.606833 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:07:09.606736    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-86b55c56d-2ff8d"
Sep 18 20:07:09.893279 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:07:09.893239    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-86b55c56d-2ff8d"]
Sep 18 20:07:09.898226 ip-10-0-129-30 kubenswrapper[2577]: W0918 20:07:09.898182    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2208afc9_cf15_4106_af0f_a52569203a69.slice/crio-239e15ef1a7527504287924a5e4c5bd0bc4ddad6ee62dff1b19247563f096621 WatchSource:0}: Error finding container 239e15ef1a7527504287924a5e4c5bd0bc4ddad6ee62dff1b19247563f096621: Status 404 returned error can't find the container with id 239e15ef1a7527504287924a5e4c5bd0bc4ddad6ee62dff1b19247563f096621
Sep 18 20:07:10.592391 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:07:10.592352    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-86b55c56d-2ff8d" event={"ID":"2208afc9-cf15-4106-af0f-a52569203a69","Type":"ContainerStarted","Data":"e694f86b7b8eabccb0bd3827387c7f0c405a529e21e44abcba861dcb937d95fd"}
Sep 18 20:07:10.592800 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:07:10.592424    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-86b55c56d-2ff8d" event={"ID":"2208afc9-cf15-4106-af0f-a52569203a69","Type":"ContainerStarted","Data":"239e15ef1a7527504287924a5e4c5bd0bc4ddad6ee62dff1b19247563f096621"}
Sep 18 20:07:13.608415 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:07:13.608380    2577 generic.go:356] "Generic (PLEG): container finished" podID="2208afc9-cf15-4106-af0f-a52569203a69" containerID="e694f86b7b8eabccb0bd3827387c7f0c405a529e21e44abcba861dcb937d95fd" exitCode=0
Sep 18 20:07:13.608851 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:07:13.608451    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-86b55c56d-2ff8d" event={"ID":"2208afc9-cf15-4106-af0f-a52569203a69","Type":"ContainerDied","Data":"e694f86b7b8eabccb0bd3827387c7f0c405a529e21e44abcba861dcb937d95fd"}
Sep 18 20:07:14.614624 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:07:14.614592    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-86b55c56d-2ff8d" event={"ID":"2208afc9-cf15-4106-af0f-a52569203a69","Type":"ContainerStarted","Data":"9f92c8f2281f3d6f95323fdb93f724d552dc725c1ddd71b62138ee1a376f7e52"}
Sep 18 20:07:14.651256 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:07:14.651201    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-86b55c56d-2ff8d" podStartSLOduration=2.954998123 podStartE2EDuration="5.651185965s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.696187842s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:07:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:07:14.649591923 +0000 UTC m=+3205.501892295" watchObservedRunningTime="2026-09-18 20:07:14.651185965 +0000 UTC m=+3205.503486336"
Sep 18 20:07:15.607476 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:07:15.607444    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-86b55c56d-2ff8d"
Sep 18 20:07:19.607182 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:07:19.607139    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-86b55c56d-2ff8d"
Sep 18 20:07:19.608708 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:07:19.608666    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-86b55c56d-2ff8d" podUID="2208afc9-cf15-4106-af0f-a52569203a69" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Sep 18 20:07:29.607468 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:07:29.607425    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-86b55c56d-2ff8d" podUID="2208afc9-cf15-4106-af0f-a52569203a69" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Sep 18 20:07:39.608215 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:07:39.608174    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-86b55c56d-2ff8d" podUID="2208afc9-cf15-4106-af0f-a52569203a69" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Sep 18 20:07:49.608033 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:07:49.607981    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-86b55c56d-2ff8d" podUID="2208afc9-cf15-4106-af0f-a52569203a69" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Sep 18 20:07:59.607881 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:07:59.607833    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-86b55c56d-2ff8d" podUID="2208afc9-cf15-4106-af0f-a52569203a69" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Sep 18 20:08:09.607801 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:08:09.607747    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-86b55c56d-2ff8d" podUID="2208afc9-cf15-4106-af0f-a52569203a69" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Sep 18 20:08:19.607373 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:08:19.607325    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-86b55c56d-2ff8d" podUID="2208afc9-cf15-4106-af0f-a52569203a69" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Sep 18 20:08:29.618270 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:08:29.618235    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-86b55c56d-2ff8d"
Sep 18 20:08:29.626963 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:08:29.626936    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-86b55c56d-2ff8d"
Sep 18 20:08:55.045818 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:08:55.045785    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-86b55c56d-2ff8d_2208afc9-cf15-4106-af0f-a52569203a69/storage-initializer/0.log"
Sep 18 20:08:55.061283 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:08:55.061261    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-86b55c56d-2ff8d_2208afc9-cf15-4106-af0f-a52569203a69/main/0.log"
Sep 18 20:08:57.851157 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:08:57.851123    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-86b55c56d-2ff8d"]
Sep 18 20:09:12.873428 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:09:12.873333    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-86b55c56d-2ff8d" podUID="2208afc9-cf15-4106-af0f-a52569203a69" containerName="main" containerID="cri-o://9f92c8f2281f3d6f95323fdb93f724d552dc725c1ddd71b62138ee1a376f7e52" gracePeriod=45
Sep 18 20:09:13.518578 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:09:13.518553    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-86b55c56d-2ff8d"
Sep 18 20:09:13.641481 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:09:13.641453    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2208afc9-cf15-4106-af0f-a52569203a69-tls-certs\") pod \"2208afc9-cf15-4106-af0f-a52569203a69\" (UID: \"2208afc9-cf15-4106-af0f-a52569203a69\") "
Sep 18 20:09:13.641689 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:09:13.641496    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2208afc9-cf15-4106-af0f-a52569203a69-kserve-provision-location\") pod \"2208afc9-cf15-4106-af0f-a52569203a69\" (UID: \"2208afc9-cf15-4106-af0f-a52569203a69\") "
Sep 18 20:09:13.641689 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:09:13.641543    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6m7kj\" (UniqueName: \"kubernetes.io/projected/2208afc9-cf15-4106-af0f-a52569203a69-kube-api-access-6m7kj\") pod \"2208afc9-cf15-4106-af0f-a52569203a69\" (UID: \"2208afc9-cf15-4106-af0f-a52569203a69\") "
Sep 18 20:09:13.641689 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:09:13.641572    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2208afc9-cf15-4106-af0f-a52569203a69-home\") pod \"2208afc9-cf15-4106-af0f-a52569203a69\" (UID: \"2208afc9-cf15-4106-af0f-a52569203a69\") "
Sep 18 20:09:13.641689 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:09:13.641596    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2208afc9-cf15-4106-af0f-a52569203a69-tmp-dir\") pod \"2208afc9-cf15-4106-af0f-a52569203a69\" (UID: \"2208afc9-cf15-4106-af0f-a52569203a69\") "
Sep 18 20:09:13.641689 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:09:13.641635    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2208afc9-cf15-4106-af0f-a52569203a69-dshm\") pod \"2208afc9-cf15-4106-af0f-a52569203a69\" (UID: \"2208afc9-cf15-4106-af0f-a52569203a69\") "
Sep 18 20:09:13.641960 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:09:13.641704    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2208afc9-cf15-4106-af0f-a52569203a69-cabundle-cert\") pod \"2208afc9-cf15-4106-af0f-a52569203a69\" (UID: \"2208afc9-cf15-4106-af0f-a52569203a69\") "
Sep 18 20:09:13.641960 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:09:13.641754    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2208afc9-cf15-4106-af0f-a52569203a69-model-cache\") pod \"2208afc9-cf15-4106-af0f-a52569203a69\" (UID: \"2208afc9-cf15-4106-af0f-a52569203a69\") "
Sep 18 20:09:13.642101 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:09:13.642037    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2208afc9-cf15-4106-af0f-a52569203a69-home" (OuterVolumeSpecName: "home") pod "2208afc9-cf15-4106-af0f-a52569203a69" (UID: "2208afc9-cf15-4106-af0f-a52569203a69"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:09:13.642267 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:09:13.642240    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2208afc9-cf15-4106-af0f-a52569203a69-model-cache" (OuterVolumeSpecName: "model-cache") pod "2208afc9-cf15-4106-af0f-a52569203a69" (UID: "2208afc9-cf15-4106-af0f-a52569203a69"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:09:13.642544 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:09:13.642519    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2208afc9-cf15-4106-af0f-a52569203a69-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2208afc9-cf15-4106-af0f-a52569203a69" (UID: "2208afc9-cf15-4106-af0f-a52569203a69"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:09:13.643977 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:09:13.643956    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2208afc9-cf15-4106-af0f-a52569203a69-kube-api-access-6m7kj" (OuterVolumeSpecName: "kube-api-access-6m7kj") pod "2208afc9-cf15-4106-af0f-a52569203a69" (UID: "2208afc9-cf15-4106-af0f-a52569203a69"). InnerVolumeSpecName "kube-api-access-6m7kj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:09:13.644210 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:09:13.644178    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2208afc9-cf15-4106-af0f-a52569203a69-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2208afc9-cf15-4106-af0f-a52569203a69" (UID: "2208afc9-cf15-4106-af0f-a52569203a69"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:09:13.644210 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:09:13.644201    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2208afc9-cf15-4106-af0f-a52569203a69-dshm" (OuterVolumeSpecName: "dshm") pod "2208afc9-cf15-4106-af0f-a52569203a69" (UID: "2208afc9-cf15-4106-af0f-a52569203a69"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:09:13.659348 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:09:13.659308    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2208afc9-cf15-4106-af0f-a52569203a69-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2208afc9-cf15-4106-af0f-a52569203a69" (UID: "2208afc9-cf15-4106-af0f-a52569203a69"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:09:13.672022 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:09:13.671972    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2208afc9-cf15-4106-af0f-a52569203a69-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2208afc9-cf15-4106-af0f-a52569203a69" (UID: "2208afc9-cf15-4106-af0f-a52569203a69"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:09:13.743520 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:09:13.743485    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2208afc9-cf15-4106-af0f-a52569203a69-tmp-dir\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:09:13.743520 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:09:13.743515    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2208afc9-cf15-4106-af0f-a52569203a69-dshm\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:09:13.743520 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:09:13.743524    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2208afc9-cf15-4106-af0f-a52569203a69-cabundle-cert\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:09:13.743755 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:09:13.743534    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2208afc9-cf15-4106-af0f-a52569203a69-model-cache\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:09:13.743755 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:09:13.743542    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2208afc9-cf15-4106-af0f-a52569203a69-tls-certs\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:09:13.743755 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:09:13.743551    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2208afc9-cf15-4106-af0f-a52569203a69-kserve-provision-location\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:09:13.743755 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:09:13.743559    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m7kj\" (UniqueName: \"kubernetes.io/projected/2208afc9-cf15-4106-af0f-a52569203a69-kube-api-access-6m7kj\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:09:13.743755 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:09:13.743567    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2208afc9-cf15-4106-af0f-a52569203a69-home\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:09:14.172096 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:09:14.172058    2577 generic.go:356] "Generic (PLEG): container finished" podID="2208afc9-cf15-4106-af0f-a52569203a69" containerID="9f92c8f2281f3d6f95323fdb93f724d552dc725c1ddd71b62138ee1a376f7e52" exitCode=0
Sep 18 20:09:14.172503 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:09:14.172104    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-86b55c56d-2ff8d" event={"ID":"2208afc9-cf15-4106-af0f-a52569203a69","Type":"ContainerDied","Data":"9f92c8f2281f3d6f95323fdb93f724d552dc725c1ddd71b62138ee1a376f7e52"}
Sep 18 20:09:14.172503 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:09:14.172115    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-86b55c56d-2ff8d"
Sep 18 20:09:14.172503 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:09:14.172161    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-86b55c56d-2ff8d" event={"ID":"2208afc9-cf15-4106-af0f-a52569203a69","Type":"ContainerDied","Data":"239e15ef1a7527504287924a5e4c5bd0bc4ddad6ee62dff1b19247563f096621"}
Sep 18 20:09:14.172503 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:09:14.172184    2577 scope.go:117] "RemoveContainer" containerID="9f92c8f2281f3d6f95323fdb93f724d552dc725c1ddd71b62138ee1a376f7e52"
Sep 18 20:09:14.181522 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:09:14.181500    2577 scope.go:117] "RemoveContainer" containerID="e694f86b7b8eabccb0bd3827387c7f0c405a529e21e44abcba861dcb937d95fd"
Sep 18 20:09:14.189392 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:09:14.189374    2577 scope.go:117] "RemoveContainer" containerID="9f92c8f2281f3d6f95323fdb93f724d552dc725c1ddd71b62138ee1a376f7e52"
Sep 18 20:09:14.189672 ip-10-0-129-30 kubenswrapper[2577]: E0918 20:09:14.189647    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f92c8f2281f3d6f95323fdb93f724d552dc725c1ddd71b62138ee1a376f7e52\": container with ID starting with 9f92c8f2281f3d6f95323fdb93f724d552dc725c1ddd71b62138ee1a376f7e52 not found: ID does not exist" containerID="9f92c8f2281f3d6f95323fdb93f724d552dc725c1ddd71b62138ee1a376f7e52"
Sep 18 20:09:14.189718 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:09:14.189681    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f92c8f2281f3d6f95323fdb93f724d552dc725c1ddd71b62138ee1a376f7e52"} err="failed to get container status \"9f92c8f2281f3d6f95323fdb93f724d552dc725c1ddd71b62138ee1a376f7e52\": rpc error: code = NotFound desc = could not find container \"9f92c8f2281f3d6f95323fdb93f724d552dc725c1ddd71b62138ee1a376f7e52\": container with ID starting with 9f92c8f2281f3d6f95323fdb93f724d552dc725c1ddd71b62138ee1a376f7e52 not found: ID does not exist"
Sep 18 20:09:14.189718 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:09:14.189701    2577 scope.go:117] "RemoveContainer" containerID="e694f86b7b8eabccb0bd3827387c7f0c405a529e21e44abcba861dcb937d95fd"
Sep 18 20:09:14.191068 ip-10-0-129-30 kubenswrapper[2577]: E0918 20:09:14.189933    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e694f86b7b8eabccb0bd3827387c7f0c405a529e21e44abcba861dcb937d95fd\": container with ID starting with e694f86b7b8eabccb0bd3827387c7f0c405a529e21e44abcba861dcb937d95fd not found: ID does not exist" containerID="e694f86b7b8eabccb0bd3827387c7f0c405a529e21e44abcba861dcb937d95fd"
Sep 18 20:09:14.191068 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:09:14.190383    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e694f86b7b8eabccb0bd3827387c7f0c405a529e21e44abcba861dcb937d95fd"} err="failed to get container status \"e694f86b7b8eabccb0bd3827387c7f0c405a529e21e44abcba861dcb937d95fd\": rpc error: code = NotFound desc = could not find container \"e694f86b7b8eabccb0bd3827387c7f0c405a529e21e44abcba861dcb937d95fd\": container with ID starting with e694f86b7b8eabccb0bd3827387c7f0c405a529e21e44abcba861dcb937d95fd not found: ID does not exist"
Sep 18 20:09:14.199209 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:09:14.199186    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-86b55c56d-2ff8d"]
Sep 18 20:09:14.201397 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:09:14.201377    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-86b55c56d-2ff8d"]
Sep 18 20:09:15.438644 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:09:15.438610    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2208afc9-cf15-4106-af0f-a52569203a69" path="/var/lib/kubelet/pods/2208afc9-cf15-4106-af0f-a52569203a69/volumes"
Sep 18 20:10:02.366661 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:10:02.366627    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l"]
Sep 18 20:10:02.367266 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:10:02.367131    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2208afc9-cf15-4106-af0f-a52569203a69" containerName="storage-initializer"
Sep 18 20:10:02.367266 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:10:02.367152    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2208afc9-cf15-4106-af0f-a52569203a69" containerName="storage-initializer"
Sep 18 20:10:02.367266 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:10:02.367163    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2208afc9-cf15-4106-af0f-a52569203a69" containerName="main"
Sep 18 20:10:02.367266 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:10:02.367173    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2208afc9-cf15-4106-af0f-a52569203a69" containerName="main"
Sep 18 20:10:02.367266 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:10:02.367253    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2208afc9-cf15-4106-af0f-a52569203a69" containerName="main"
Sep 18 20:10:02.370398 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:10:02.370375    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l"
Sep 18 20:10:02.376110 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:10:02.376075    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 18 20:10:02.376241 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:10:02.376216    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-xz8hh\""
Sep 18 20:10:02.376366 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:10:02.376330    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 18 20:10:02.376484 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:10:02.376399    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 18 20:10:02.376484 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:10:02.376435    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:10:02.376600 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:10:02.376539    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 18 20:10:02.376655 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:10:02.376630    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-dockercfg-72g92\""
Sep 18 20:10:02.462505 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:10:02.462471    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/335a7571-db3b-43bd-b4ad-ae33f17f6797-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-6458d7559c-w886l\" (UID: \"335a7571-db3b-43bd-b4ad-ae33f17f6797\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l"
Sep 18 20:10:02.462672 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:10:02.462557    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/335a7571-db3b-43bd-b4ad-ae33f17f6797-dshm\") pod \"custom-route-timeout-pd-test-kserve-6458d7559c-w886l\" (UID: \"335a7571-db3b-43bd-b4ad-ae33f17f6797\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l"
Sep 18 20:10:02.462672 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:10:02.462643    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcxjd\" (UniqueName: \"kubernetes.io/projected/335a7571-db3b-43bd-b4ad-ae33f17f6797-kube-api-access-tcxjd\") pod \"custom-route-timeout-pd-test-kserve-6458d7559c-w886l\" (UID: \"335a7571-db3b-43bd-b4ad-ae33f17f6797\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l"
Sep 18 20:10:02.462672 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:10:02.462664    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/335a7571-db3b-43bd-b4ad-ae33f17f6797-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-6458d7559c-w886l\" (UID: \"335a7571-db3b-43bd-b4ad-ae33f17f6797\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l"
Sep 18 20:10:02.462785 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:10:02.462685    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/335a7571-db3b-43bd-b4ad-ae33f17f6797-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-6458d7559c-w886l\" (UID: \"335a7571-db3b-43bd-b4ad-ae33f17f6797\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l"
Sep 18 20:10:02.462785 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:10:02.462710    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/335a7571-db3b-43bd-b4ad-ae33f17f6797-home\") pod \"custom-route-timeout-pd-test-kserve-6458d7559c-w886l\" (UID: \"335a7571-db3b-43bd-b4ad-ae33f17f6797\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l"
Sep 18 20:10:02.462785 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:10:02.462732    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/335a7571-db3b-43bd-b4ad-ae33f17f6797-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-6458d7559c-w886l\" (UID: \"335a7571-db3b-43bd-b4ad-ae33f17f6797\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l"
Sep 18 20:10:02.462785 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:10:02.462756    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/335a7571-db3b-43bd-b4ad-ae33f17f6797-model-cache\") pod \"custom-route-timeout-pd-test-kserve-6458d7559c-w886l\" (UID: \"335a7571-db3b-43bd-b4ad-ae33f17f6797\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l"
Sep 18 20:10:02.468573 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:10:02.468548    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l"]
Sep 18 20:10:02.563584 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:10:02.563554    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/335a7571-db3b-43bd-b4ad-ae33f17f6797-model-cache\") pod \"custom-route-timeout-pd-test-kserve-6458d7559c-w886l\" (UID: \"335a7571-db3b-43bd-b4ad-ae33f17f6797\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l"
Sep 18 20:10:02.563793 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:10:02.563599    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/335a7571-db3b-43bd-b4ad-ae33f17f6797-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-6458d7559c-w886l\" (UID: \"335a7571-db3b-43bd-b4ad-ae33f17f6797\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l"
Sep 18 20:10:02.563793 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:10:02.563625    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/335a7571-db3b-43bd-b4ad-ae33f17f6797-dshm\") pod \"custom-route-timeout-pd-test-kserve-6458d7559c-w886l\" (UID: \"335a7571-db3b-43bd-b4ad-ae33f17f6797\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l"
Sep 18 20:10:02.563793 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:10:02.563671    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcxjd\" (UniqueName: \"kubernetes.io/projected/335a7571-db3b-43bd-b4ad-ae33f17f6797-kube-api-access-tcxjd\") pod \"custom-route-timeout-pd-test-kserve-6458d7559c-w886l\" (UID: \"335a7571-db3b-43bd-b4ad-ae33f17f6797\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l"
Sep 18 20:10:02.563793 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:10:02.563689    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/335a7571-db3b-43bd-b4ad-ae33f17f6797-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-6458d7559c-w886l\" (UID: \"335a7571-db3b-43bd-b4ad-ae33f17f6797\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l"
Sep 18 20:10:02.563793 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:10:02.563709    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/335a7571-db3b-43bd-b4ad-ae33f17f6797-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-6458d7559c-w886l\" (UID: \"335a7571-db3b-43bd-b4ad-ae33f17f6797\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l"
Sep 18 20:10:02.563793 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:10:02.563742    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/335a7571-db3b-43bd-b4ad-ae33f17f6797-home\") pod \"custom-route-timeout-pd-test-kserve-6458d7559c-w886l\" (UID: \"335a7571-db3b-43bd-b4ad-ae33f17f6797\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l"
Sep 18 20:10:02.563793 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:10:02.563765    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/335a7571-db3b-43bd-b4ad-ae33f17f6797-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-6458d7559c-w886l\" (UID: \"335a7571-db3b-43bd-b4ad-ae33f17f6797\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l"
Sep 18 20:10:02.564207 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:10:02.564078    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/335a7571-db3b-43bd-b4ad-ae33f17f6797-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-6458d7559c-w886l\" (UID: \"335a7571-db3b-43bd-b4ad-ae33f17f6797\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l"
Sep 18 20:10:02.564207 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:10:02.564110    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/335a7571-db3b-43bd-b4ad-ae33f17f6797-model-cache\") pod \"custom-route-timeout-pd-test-kserve-6458d7559c-w886l\" (UID: \"335a7571-db3b-43bd-b4ad-ae33f17f6797\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l"
Sep 18 20:10:02.564397 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:10:02.564370    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/335a7571-db3b-43bd-b4ad-ae33f17f6797-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-6458d7559c-w886l\" (UID: \"335a7571-db3b-43bd-b4ad-ae33f17f6797\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l"
Sep 18 20:10:02.564539 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:10:02.564453    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/335a7571-db3b-43bd-b4ad-ae33f17f6797-home\") pod \"custom-route-timeout-pd-test-kserve-6458d7559c-w886l\" (UID: \"335a7571-db3b-43bd-b4ad-ae33f17f6797\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l"
Sep 18 20:10:02.564539 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:10:02.564454    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/335a7571-db3b-43bd-b4ad-ae33f17f6797-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-6458d7559c-w886l\" (UID: \"335a7571-db3b-43bd-b4ad-ae33f17f6797\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l"
Sep 18 20:10:02.566224 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:10:02.566198    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/335a7571-db3b-43bd-b4ad-ae33f17f6797-dshm\") pod \"custom-route-timeout-pd-test-kserve-6458d7559c-w886l\" (UID: \"335a7571-db3b-43bd-b4ad-ae33f17f6797\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l"
Sep 18 20:10:02.566360 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:10:02.566342    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/335a7571-db3b-43bd-b4ad-ae33f17f6797-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-6458d7559c-w886l\" (UID: \"335a7571-db3b-43bd-b4ad-ae33f17f6797\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l"
Sep 18 20:10:02.573683 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:10:02.573660    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcxjd\" (UniqueName: \"kubernetes.io/projected/335a7571-db3b-43bd-b4ad-ae33f17f6797-kube-api-access-tcxjd\") pod \"custom-route-timeout-pd-test-kserve-6458d7559c-w886l\" (UID: \"335a7571-db3b-43bd-b4ad-ae33f17f6797\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l"
Sep 18 20:10:02.680376 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:10:02.680279    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l"
Sep 18 20:10:02.982242 ip-10-0-129-30 kubenswrapper[2577]: W0918 20:10:02.982209    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod335a7571_db3b_43bd_b4ad_ae33f17f6797.slice/crio-7d80611f7a7c986109e44eb967c0157850ac99c7c2492915a84c5a798cc66482 WatchSource:0}: Error finding container 7d80611f7a7c986109e44eb967c0157850ac99c7c2492915a84c5a798cc66482: Status 404 returned error can't find the container with id 7d80611f7a7c986109e44eb967c0157850ac99c7c2492915a84c5a798cc66482
Sep 18 20:10:02.982538 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:10:02.982509    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l"]
Sep 18 20:10:03.382766 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:10:03.382725    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l" event={"ID":"335a7571-db3b-43bd-b4ad-ae33f17f6797","Type":"ContainerStarted","Data":"3cd4ba68801df71b4ff4f607f16a988853f46ac7edf95600aebd0a6b72827973"}
Sep 18 20:10:03.383215 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:10:03.382795    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l" event={"ID":"335a7571-db3b-43bd-b4ad-ae33f17f6797","Type":"ContainerStarted","Data":"7d80611f7a7c986109e44eb967c0157850ac99c7c2492915a84c5a798cc66482"}
Sep 18 20:10:03.383215 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:10:03.382847    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l"
Sep 18 20:10:04.388975 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:10:04.388932    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l" event={"ID":"335a7571-db3b-43bd-b4ad-ae33f17f6797","Type":"ContainerStarted","Data":"6f15426adabafab0ae7bcf81372d81555b5c1420208ebf2fa99796ee1f77610a"}
Sep 18 20:10:06.398393 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:10:06.398360    2577 generic.go:356] "Generic (PLEG): container finished" podID="335a7571-db3b-43bd-b4ad-ae33f17f6797" containerID="6f15426adabafab0ae7bcf81372d81555b5c1420208ebf2fa99796ee1f77610a" exitCode=0
Sep 18 20:10:06.398786 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:10:06.398417    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l" event={"ID":"335a7571-db3b-43bd-b4ad-ae33f17f6797","Type":"ContainerDied","Data":"6f15426adabafab0ae7bcf81372d81555b5c1420208ebf2fa99796ee1f77610a"}
Sep 18 20:10:07.411061 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:10:07.411018    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l" event={"ID":"335a7571-db3b-43bd-b4ad-ae33f17f6797","Type":"ContainerStarted","Data":"7a8818fe552f01f67448023d5d3c36274f3222657809931be342e0ac2e5c8285"}
Sep 18 20:10:07.446182 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:10:07.446120    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l" podStartSLOduration=3.287990254 podStartE2EDuration="5.446101004s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.15811075s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:10:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:10:07.443287287 +0000 UTC m=+3378.295587655" watchObservedRunningTime="2026-09-18 20:10:07.446101004 +0000 UTC m=+3378.298401376"
Sep 18 20:10:12.680652 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:10:12.680612    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l"
Sep 18 20:10:12.681133 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:10:12.680698    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l"
Sep 18 20:10:12.681892 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:10:12.681866    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l" podUID="335a7571-db3b-43bd-b4ad-ae33f17f6797" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8001/health\": dial tcp 10.132.0.48:8001: connect: connection refused"
Sep 18 20:10:22.681637 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:10:22.681583    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l" podUID="335a7571-db3b-43bd-b4ad-ae33f17f6797" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8001/health\": dial tcp 10.132.0.48:8001: connect: connection refused"
Sep 18 20:10:22.694125 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:10:22.694102    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l"
Sep 18 20:10:32.681171 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:10:32.681047    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l" podUID="335a7571-db3b-43bd-b4ad-ae33f17f6797" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8001/health\": dial tcp 10.132.0.48:8001: connect: connection refused"
Sep 18 20:10:42.681429 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:10:42.681379    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l" podUID="335a7571-db3b-43bd-b4ad-ae33f17f6797" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8001/health\": dial tcp 10.132.0.48:8001: connect: connection refused"
Sep 18 20:10:52.681518 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:10:52.681451    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l" podUID="335a7571-db3b-43bd-b4ad-ae33f17f6797" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8001/health\": dial tcp 10.132.0.48:8001: connect: connection refused"
Sep 18 20:11:02.681475 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:11:02.681345    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l" podUID="335a7571-db3b-43bd-b4ad-ae33f17f6797" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8001/health\": dial tcp 10.132.0.48:8001: connect: connection refused"
Sep 18 20:11:12.680736 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:11:12.680678    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l" podUID="335a7571-db3b-43bd-b4ad-ae33f17f6797" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8001/health\": dial tcp 10.132.0.48:8001: connect: connection refused"
Sep 18 20:11:22.681425 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:11:22.681370    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l" podUID="335a7571-db3b-43bd-b4ad-ae33f17f6797" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8001/health\": dial tcp 10.132.0.48:8001: connect: connection refused"
Sep 18 20:11:32.681483 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:11:32.681430    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l" podUID="335a7571-db3b-43bd-b4ad-ae33f17f6797" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8001/health\": dial tcp 10.132.0.48:8001: connect: connection refused"
Sep 18 20:11:42.681247 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:11:42.681197    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l" podUID="335a7571-db3b-43bd-b4ad-ae33f17f6797" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8001/health\": dial tcp 10.132.0.48:8001: connect: connection refused"
Sep 18 20:11:52.681387 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:11:52.681336    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l" podUID="335a7571-db3b-43bd-b4ad-ae33f17f6797" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8001/health\": dial tcp 10.132.0.48:8001: connect: connection refused"
Sep 18 20:12:02.690918 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:12:02.690828    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l"
Sep 18 20:12:02.702517 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:12:02.702493    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l"
Sep 18 20:12:21.145861 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:12:21.145823    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l"]
Sep 18 20:12:36.168748 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:12:36.168699    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l" podUID="335a7571-db3b-43bd-b4ad-ae33f17f6797" containerName="main" containerID="cri-o://7a8818fe552f01f67448023d5d3c36274f3222657809931be342e0ac2e5c8285" gracePeriod=45
Sep 18 20:12:36.784751 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:12:36.784709    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l" podUID="335a7571-db3b-43bd-b4ad-ae33f17f6797" containerName="llm-d-routing-sidecar" containerID="cri-o://3cd4ba68801df71b4ff4f607f16a988853f46ac7edf95600aebd0a6b72827973" gracePeriod=45
Sep 18 20:12:37.026460 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:12:37.026435    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l"
Sep 18 20:12:37.085139 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:12:37.085112    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/335a7571-db3b-43bd-b4ad-ae33f17f6797-dshm\") pod \"335a7571-db3b-43bd-b4ad-ae33f17f6797\" (UID: \"335a7571-db3b-43bd-b4ad-ae33f17f6797\") "
Sep 18 20:12:37.085310 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:12:37.085156    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/335a7571-db3b-43bd-b4ad-ae33f17f6797-tls-certs\") pod \"335a7571-db3b-43bd-b4ad-ae33f17f6797\" (UID: \"335a7571-db3b-43bd-b4ad-ae33f17f6797\") "
Sep 18 20:12:37.085310 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:12:37.085183    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/335a7571-db3b-43bd-b4ad-ae33f17f6797-cabundle-cert\") pod \"335a7571-db3b-43bd-b4ad-ae33f17f6797\" (UID: \"335a7571-db3b-43bd-b4ad-ae33f17f6797\") "
Sep 18 20:12:37.085310 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:12:37.085199    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/335a7571-db3b-43bd-b4ad-ae33f17f6797-tmp-dir\") pod \"335a7571-db3b-43bd-b4ad-ae33f17f6797\" (UID: \"335a7571-db3b-43bd-b4ad-ae33f17f6797\") "
Sep 18 20:12:37.085310 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:12:37.085256    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tcxjd\" (UniqueName: \"kubernetes.io/projected/335a7571-db3b-43bd-b4ad-ae33f17f6797-kube-api-access-tcxjd\") pod \"335a7571-db3b-43bd-b4ad-ae33f17f6797\" (UID: \"335a7571-db3b-43bd-b4ad-ae33f17f6797\") "
Sep 18 20:12:37.085310 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:12:37.085283    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/335a7571-db3b-43bd-b4ad-ae33f17f6797-kserve-provision-location\") pod \"335a7571-db3b-43bd-b4ad-ae33f17f6797\" (UID: \"335a7571-db3b-43bd-b4ad-ae33f17f6797\") "
Sep 18 20:12:37.085603 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:12:37.085546    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/335a7571-db3b-43bd-b4ad-ae33f17f6797-model-cache\") pod \"335a7571-db3b-43bd-b4ad-ae33f17f6797\" (UID: \"335a7571-db3b-43bd-b4ad-ae33f17f6797\") "
Sep 18 20:12:37.085675 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:12:37.085604    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/335a7571-db3b-43bd-b4ad-ae33f17f6797-home\") pod \"335a7571-db3b-43bd-b4ad-ae33f17f6797\" (UID: \"335a7571-db3b-43bd-b4ad-ae33f17f6797\") "
Sep 18 20:12:37.085771 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:12:37.085730    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/335a7571-db3b-43bd-b4ad-ae33f17f6797-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "335a7571-db3b-43bd-b4ad-ae33f17f6797" (UID: "335a7571-db3b-43bd-b4ad-ae33f17f6797"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:12:37.085916 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:12:37.085894    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/335a7571-db3b-43bd-b4ad-ae33f17f6797-cabundle-cert\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:12:37.086150 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:12:37.086091    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/335a7571-db3b-43bd-b4ad-ae33f17f6797-model-cache" (OuterVolumeSpecName: "model-cache") pod "335a7571-db3b-43bd-b4ad-ae33f17f6797" (UID: "335a7571-db3b-43bd-b4ad-ae33f17f6797"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:12:37.086563 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:12:37.086538    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/335a7571-db3b-43bd-b4ad-ae33f17f6797-home" (OuterVolumeSpecName: "home") pod "335a7571-db3b-43bd-b4ad-ae33f17f6797" (UID: "335a7571-db3b-43bd-b4ad-ae33f17f6797"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:12:37.087680 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:12:37.087649    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/335a7571-db3b-43bd-b4ad-ae33f17f6797-kube-api-access-tcxjd" (OuterVolumeSpecName: "kube-api-access-tcxjd") pod "335a7571-db3b-43bd-b4ad-ae33f17f6797" (UID: "335a7571-db3b-43bd-b4ad-ae33f17f6797"). InnerVolumeSpecName "kube-api-access-tcxjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:12:37.087781 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:12:37.087674    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/335a7571-db3b-43bd-b4ad-ae33f17f6797-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "335a7571-db3b-43bd-b4ad-ae33f17f6797" (UID: "335a7571-db3b-43bd-b4ad-ae33f17f6797"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:12:37.087781 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:12:37.087689    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/335a7571-db3b-43bd-b4ad-ae33f17f6797-dshm" (OuterVolumeSpecName: "dshm") pod "335a7571-db3b-43bd-b4ad-ae33f17f6797" (UID: "335a7571-db3b-43bd-b4ad-ae33f17f6797"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:12:37.106268 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:12:37.105341    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/335a7571-db3b-43bd-b4ad-ae33f17f6797-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "335a7571-db3b-43bd-b4ad-ae33f17f6797" (UID: "335a7571-db3b-43bd-b4ad-ae33f17f6797"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:12:37.110320 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:12:37.110292    2577 generic.go:356] "Generic (PLEG): container finished" podID="335a7571-db3b-43bd-b4ad-ae33f17f6797" containerID="7a8818fe552f01f67448023d5d3c36274f3222657809931be342e0ac2e5c8285" exitCode=0
Sep 18 20:12:37.110320 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:12:37.110317    2577 generic.go:356] "Generic (PLEG): container finished" podID="335a7571-db3b-43bd-b4ad-ae33f17f6797" containerID="3cd4ba68801df71b4ff4f607f16a988853f46ac7edf95600aebd0a6b72827973" exitCode=0
Sep 18 20:12:37.110504 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:12:37.110358    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l" event={"ID":"335a7571-db3b-43bd-b4ad-ae33f17f6797","Type":"ContainerDied","Data":"7a8818fe552f01f67448023d5d3c36274f3222657809931be342e0ac2e5c8285"}
Sep 18 20:12:37.110504 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:12:37.110366    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l"
Sep 18 20:12:37.110504 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:12:37.110443    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l" event={"ID":"335a7571-db3b-43bd-b4ad-ae33f17f6797","Type":"ContainerDied","Data":"3cd4ba68801df71b4ff4f607f16a988853f46ac7edf95600aebd0a6b72827973"}
Sep 18 20:12:37.110504 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:12:37.110473    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l" event={"ID":"335a7571-db3b-43bd-b4ad-ae33f17f6797","Type":"ContainerDied","Data":"7d80611f7a7c986109e44eb967c0157850ac99c7c2492915a84c5a798cc66482"}
Sep 18 20:12:37.110504 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:12:37.110503    2577 scope.go:117] "RemoveContainer" containerID="7a8818fe552f01f67448023d5d3c36274f3222657809931be342e0ac2e5c8285"
Sep 18 20:12:37.118709 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:12:37.118684    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/335a7571-db3b-43bd-b4ad-ae33f17f6797-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "335a7571-db3b-43bd-b4ad-ae33f17f6797" (UID: "335a7571-db3b-43bd-b4ad-ae33f17f6797"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:12:37.119298 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:12:37.119283    2577 scope.go:117] "RemoveContainer" containerID="6f15426adabafab0ae7bcf81372d81555b5c1420208ebf2fa99796ee1f77610a"
Sep 18 20:12:37.126899 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:12:37.126881    2577 scope.go:117] "RemoveContainer" containerID="3cd4ba68801df71b4ff4f607f16a988853f46ac7edf95600aebd0a6b72827973"
Sep 18 20:12:37.133635 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:12:37.133616    2577 scope.go:117] "RemoveContainer" containerID="7a8818fe552f01f67448023d5d3c36274f3222657809931be342e0ac2e5c8285"
Sep 18 20:12:37.133882 ip-10-0-129-30 kubenswrapper[2577]: E0918 20:12:37.133863    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a8818fe552f01f67448023d5d3c36274f3222657809931be342e0ac2e5c8285\": container with ID starting with 7a8818fe552f01f67448023d5d3c36274f3222657809931be342e0ac2e5c8285 not found: ID does not exist" containerID="7a8818fe552f01f67448023d5d3c36274f3222657809931be342e0ac2e5c8285"
Sep 18 20:12:37.133936 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:12:37.133891    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a8818fe552f01f67448023d5d3c36274f3222657809931be342e0ac2e5c8285"} err="failed to get container status \"7a8818fe552f01f67448023d5d3c36274f3222657809931be342e0ac2e5c8285\": rpc error: code = NotFound desc = could not find container \"7a8818fe552f01f67448023d5d3c36274f3222657809931be342e0ac2e5c8285\": container with ID starting with 7a8818fe552f01f67448023d5d3c36274f3222657809931be342e0ac2e5c8285 not found: ID does not exist"
Sep 18 20:12:37.133936 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:12:37.133911    2577 scope.go:117] "RemoveContainer" containerID="6f15426adabafab0ae7bcf81372d81555b5c1420208ebf2fa99796ee1f77610a"
Sep 18 20:12:37.134185 ip-10-0-129-30 kubenswrapper[2577]: E0918 20:12:37.134157    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f15426adabafab0ae7bcf81372d81555b5c1420208ebf2fa99796ee1f77610a\": container with ID starting with 6f15426adabafab0ae7bcf81372d81555b5c1420208ebf2fa99796ee1f77610a not found: ID does not exist" containerID="6f15426adabafab0ae7bcf81372d81555b5c1420208ebf2fa99796ee1f77610a"
Sep 18 20:12:37.134290 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:12:37.134183    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f15426adabafab0ae7bcf81372d81555b5c1420208ebf2fa99796ee1f77610a"} err="failed to get container status \"6f15426adabafab0ae7bcf81372d81555b5c1420208ebf2fa99796ee1f77610a\": rpc error: code = NotFound desc = could not find container \"6f15426adabafab0ae7bcf81372d81555b5c1420208ebf2fa99796ee1f77610a\": container with ID starting with 6f15426adabafab0ae7bcf81372d81555b5c1420208ebf2fa99796ee1f77610a not found: ID does not exist"
Sep 18 20:12:37.134290 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:12:37.134198    2577 scope.go:117] "RemoveContainer" containerID="3cd4ba68801df71b4ff4f607f16a988853f46ac7edf95600aebd0a6b72827973"
Sep 18 20:12:37.134459 ip-10-0-129-30 kubenswrapper[2577]: E0918 20:12:37.134442    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cd4ba68801df71b4ff4f607f16a988853f46ac7edf95600aebd0a6b72827973\": container with ID starting with 3cd4ba68801df71b4ff4f607f16a988853f46ac7edf95600aebd0a6b72827973 not found: ID does not exist" containerID="3cd4ba68801df71b4ff4f607f16a988853f46ac7edf95600aebd0a6b72827973"
Sep 18 20:12:37.134503 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:12:37.134468    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cd4ba68801df71b4ff4f607f16a988853f46ac7edf95600aebd0a6b72827973"} err="failed to get container status \"3cd4ba68801df71b4ff4f607f16a988853f46ac7edf95600aebd0a6b72827973\": rpc error: code = NotFound desc = could not find container \"3cd4ba68801df71b4ff4f607f16a988853f46ac7edf95600aebd0a6b72827973\": container with ID starting with 3cd4ba68801df71b4ff4f607f16a988853f46ac7edf95600aebd0a6b72827973 not found: ID does not exist"
Sep 18 20:12:37.134503 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:12:37.134484    2577 scope.go:117] "RemoveContainer" containerID="7a8818fe552f01f67448023d5d3c36274f3222657809931be342e0ac2e5c8285"
Sep 18 20:12:37.134721 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:12:37.134703    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a8818fe552f01f67448023d5d3c36274f3222657809931be342e0ac2e5c8285"} err="failed to get container status \"7a8818fe552f01f67448023d5d3c36274f3222657809931be342e0ac2e5c8285\": rpc error: code = NotFound desc = could not find container \"7a8818fe552f01f67448023d5d3c36274f3222657809931be342e0ac2e5c8285\": container with ID starting with 7a8818fe552f01f67448023d5d3c36274f3222657809931be342e0ac2e5c8285 not found: ID does not exist"
Sep 18 20:12:37.134775 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:12:37.134721    2577 scope.go:117] "RemoveContainer" containerID="6f15426adabafab0ae7bcf81372d81555b5c1420208ebf2fa99796ee1f77610a"
Sep 18 20:12:37.134948 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:12:37.134933    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f15426adabafab0ae7bcf81372d81555b5c1420208ebf2fa99796ee1f77610a"} err="failed to get container status \"6f15426adabafab0ae7bcf81372d81555b5c1420208ebf2fa99796ee1f77610a\": rpc error: code = NotFound desc = could not find container \"6f15426adabafab0ae7bcf81372d81555b5c1420208ebf2fa99796ee1f77610a\": container with ID starting with 6f15426adabafab0ae7bcf81372d81555b5c1420208ebf2fa99796ee1f77610a not found: ID does not exist"
Sep 18 20:12:37.135015 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:12:37.134949    2577 scope.go:117] "RemoveContainer" containerID="3cd4ba68801df71b4ff4f607f16a988853f46ac7edf95600aebd0a6b72827973"
Sep 18 20:12:37.135199 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:12:37.135181    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cd4ba68801df71b4ff4f607f16a988853f46ac7edf95600aebd0a6b72827973"} err="failed to get container status \"3cd4ba68801df71b4ff4f607f16a988853f46ac7edf95600aebd0a6b72827973\": rpc error: code = NotFound desc = could not find container \"3cd4ba68801df71b4ff4f607f16a988853f46ac7edf95600aebd0a6b72827973\": container with ID starting with 3cd4ba68801df71b4ff4f607f16a988853f46ac7edf95600aebd0a6b72827973 not found: ID does not exist"
Sep 18 20:12:37.186867 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:12:37.186843    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/335a7571-db3b-43bd-b4ad-ae33f17f6797-tmp-dir\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:12:37.186867 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:12:37.186864    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcxjd\" (UniqueName: \"kubernetes.io/projected/335a7571-db3b-43bd-b4ad-ae33f17f6797-kube-api-access-tcxjd\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:12:37.187216 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:12:37.186875    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/335a7571-db3b-43bd-b4ad-ae33f17f6797-kserve-provision-location\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:12:37.187216 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:12:37.186884    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/335a7571-db3b-43bd-b4ad-ae33f17f6797-model-cache\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:12:37.187216 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:12:37.186892    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/335a7571-db3b-43bd-b4ad-ae33f17f6797-home\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:12:37.187216 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:12:37.186899    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/335a7571-db3b-43bd-b4ad-ae33f17f6797-dshm\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:12:37.187216 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:12:37.186907    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/335a7571-db3b-43bd-b4ad-ae33f17f6797-tls-certs\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:12:37.439177 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:12:37.439143    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l"]
Sep 18 20:12:37.440771 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:12:37.440750    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6458d7559c-w886l"]
Sep 18 20:12:39.438771 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:12:39.438737    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="335a7571-db3b-43bd-b4ad-ae33f17f6797" path="/var/lib/kubelet/pods/335a7571-db3b-43bd-b4ad-ae33f17f6797/volumes"
Sep 18 20:13:13.350260 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:13.350229    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-nshgc"]
Sep 18 20:13:13.351853 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:13.350543    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="335a7571-db3b-43bd-b4ad-ae33f17f6797" containerName="storage-initializer"
Sep 18 20:13:13.351853 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:13.350554    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="335a7571-db3b-43bd-b4ad-ae33f17f6797" containerName="storage-initializer"
Sep 18 20:13:13.351853 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:13.350561    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="335a7571-db3b-43bd-b4ad-ae33f17f6797" containerName="main"
Sep 18 20:13:13.351853 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:13.350567    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="335a7571-db3b-43bd-b4ad-ae33f17f6797" containerName="main"
Sep 18 20:13:13.351853 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:13.350575    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="335a7571-db3b-43bd-b4ad-ae33f17f6797" containerName="llm-d-routing-sidecar"
Sep 18 20:13:13.351853 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:13.350581    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="335a7571-db3b-43bd-b4ad-ae33f17f6797" containerName="llm-d-routing-sidecar"
Sep 18 20:13:13.351853 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:13.350643    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="335a7571-db3b-43bd-b4ad-ae33f17f6797" containerName="llm-d-routing-sidecar"
Sep 18 20:13:13.351853 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:13.350655    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="335a7571-db3b-43bd-b4ad-ae33f17f6797" containerName="main"
Sep 18 20:13:13.352501 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:13.352485    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-nshgc"
Sep 18 20:13:13.357759 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:13.357738    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-gw-ca-root-cert\""
Sep 18 20:13:13.358377 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:13.358352    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 18 20:13:13.360594 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:13.360495    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 18 20:13:13.365090 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:13.365068    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-gateway-2-openshift-default-dockercfg-drckc\""
Sep 18 20:13:13.370783 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:13.370760    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-nshgc"]
Sep 18 20:13:13.402630 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:13.402595    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db789071-22a5-41ef-b596-d6c08644eee3-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-nshgc\" (UID: \"db789071-22a5-41ef-b596-d6c08644eee3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-nshgc"
Sep 18 20:13:13.402630 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:13.402629    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldp2d\" (UniqueName: \"kubernetes.io/projected/db789071-22a5-41ef-b596-d6c08644eee3-kube-api-access-ldp2d\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-nshgc\" (UID: \"db789071-22a5-41ef-b596-d6c08644eee3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-nshgc"
Sep 18 20:13:13.402796 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:13.402654    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db789071-22a5-41ef-b596-d6c08644eee3-istio-podinfo\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-nshgc\" (UID: \"db789071-22a5-41ef-b596-d6c08644eee3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-nshgc"
Sep 18 20:13:13.402796 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:13.402698    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db789071-22a5-41ef-b596-d6c08644eee3-istio-data\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-nshgc\" (UID: \"db789071-22a5-41ef-b596-d6c08644eee3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-nshgc"
Sep 18 20:13:13.402796 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:13.402720    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db789071-22a5-41ef-b596-d6c08644eee3-workload-certs\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-nshgc\" (UID: \"db789071-22a5-41ef-b596-d6c08644eee3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-nshgc"
Sep 18 20:13:13.402796 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:13.402739    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db789071-22a5-41ef-b596-d6c08644eee3-workload-socket\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-nshgc\" (UID: \"db789071-22a5-41ef-b596-d6c08644eee3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-nshgc"
Sep 18 20:13:13.402796 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:13.402769    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db789071-22a5-41ef-b596-d6c08644eee3-istio-envoy\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-nshgc\" (UID: \"db789071-22a5-41ef-b596-d6c08644eee3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-nshgc"
Sep 18 20:13:13.402796 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:13.402793    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db789071-22a5-41ef-b596-d6c08644eee3-istio-token\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-nshgc\" (UID: \"db789071-22a5-41ef-b596-d6c08644eee3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-nshgc"
Sep 18 20:13:13.403031 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:13.402838    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db789071-22a5-41ef-b596-d6c08644eee3-credential-socket\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-nshgc\" (UID: \"db789071-22a5-41ef-b596-d6c08644eee3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-nshgc"
Sep 18 20:13:13.503902 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:13.503867    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db789071-22a5-41ef-b596-d6c08644eee3-workload-socket\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-nshgc\" (UID: \"db789071-22a5-41ef-b596-d6c08644eee3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-nshgc"
Sep 18 20:13:13.504119 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:13.503924    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db789071-22a5-41ef-b596-d6c08644eee3-istio-envoy\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-nshgc\" (UID: \"db789071-22a5-41ef-b596-d6c08644eee3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-nshgc"
Sep 18 20:13:13.504119 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:13.503958    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db789071-22a5-41ef-b596-d6c08644eee3-istio-token\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-nshgc\" (UID: \"db789071-22a5-41ef-b596-d6c08644eee3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-nshgc"
Sep 18 20:13:13.504119 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:13.503977    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db789071-22a5-41ef-b596-d6c08644eee3-credential-socket\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-nshgc\" (UID: \"db789071-22a5-41ef-b596-d6c08644eee3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-nshgc"
Sep 18 20:13:13.504119 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:13.504026    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db789071-22a5-41ef-b596-d6c08644eee3-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-nshgc\" (UID: \"db789071-22a5-41ef-b596-d6c08644eee3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-nshgc"
Sep 18 20:13:13.504119 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:13.504050    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldp2d\" (UniqueName: \"kubernetes.io/projected/db789071-22a5-41ef-b596-d6c08644eee3-kube-api-access-ldp2d\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-nshgc\" (UID: \"db789071-22a5-41ef-b596-d6c08644eee3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-nshgc"
Sep 18 20:13:13.504119 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:13.504090    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db789071-22a5-41ef-b596-d6c08644eee3-istio-podinfo\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-nshgc\" (UID: \"db789071-22a5-41ef-b596-d6c08644eee3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-nshgc"
Sep 18 20:13:13.504468 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:13.504130    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db789071-22a5-41ef-b596-d6c08644eee3-istio-data\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-nshgc\" (UID: \"db789071-22a5-41ef-b596-d6c08644eee3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-nshgc"
Sep 18 20:13:13.504468 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:13.504169    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db789071-22a5-41ef-b596-d6c08644eee3-workload-certs\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-nshgc\" (UID: \"db789071-22a5-41ef-b596-d6c08644eee3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-nshgc"
Sep 18 20:13:13.504468 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:13.504310    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db789071-22a5-41ef-b596-d6c08644eee3-workload-socket\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-nshgc\" (UID: \"db789071-22a5-41ef-b596-d6c08644eee3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-nshgc"
Sep 18 20:13:13.504468 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:13.504355    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db789071-22a5-41ef-b596-d6c08644eee3-credential-socket\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-nshgc\" (UID: \"db789071-22a5-41ef-b596-d6c08644eee3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-nshgc"
Sep 18 20:13:13.504699 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:13.504508    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db789071-22a5-41ef-b596-d6c08644eee3-workload-certs\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-nshgc\" (UID: \"db789071-22a5-41ef-b596-d6c08644eee3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-nshgc"
Sep 18 20:13:13.504699 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:13.504609    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db789071-22a5-41ef-b596-d6c08644eee3-istio-data\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-nshgc\" (UID: \"db789071-22a5-41ef-b596-d6c08644eee3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-nshgc"
Sep 18 20:13:13.504879 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:13.504841    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db789071-22a5-41ef-b596-d6c08644eee3-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-nshgc\" (UID: \"db789071-22a5-41ef-b596-d6c08644eee3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-nshgc"
Sep 18 20:13:13.506413 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:13.506390    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db789071-22a5-41ef-b596-d6c08644eee3-istio-envoy\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-nshgc\" (UID: \"db789071-22a5-41ef-b596-d6c08644eee3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-nshgc"
Sep 18 20:13:13.506660 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:13.506644    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db789071-22a5-41ef-b596-d6c08644eee3-istio-podinfo\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-nshgc\" (UID: \"db789071-22a5-41ef-b596-d6c08644eee3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-nshgc"
Sep 18 20:13:13.515238 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:13.515218    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db789071-22a5-41ef-b596-d6c08644eee3-istio-token\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-nshgc\" (UID: \"db789071-22a5-41ef-b596-d6c08644eee3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-nshgc"
Sep 18 20:13:13.518294 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:13.518268    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldp2d\" (UniqueName: \"kubernetes.io/projected/db789071-22a5-41ef-b596-d6c08644eee3-kube-api-access-ldp2d\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-nshgc\" (UID: \"db789071-22a5-41ef-b596-d6c08644eee3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-nshgc"
Sep 18 20:13:13.665053 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:13.665014    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-nshgc"
Sep 18 20:13:13.958355 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:13.958268    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-nshgc"]
Sep 18 20:13:13.961317 ip-10-0-129-30 kubenswrapper[2577]: W0918 20:13:13.961286    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb789071_22a5_41ef_b596_d6c08644eee3.slice/crio-795be480b51eff5943ac970dd14c693c0e95ba81e77337eab5c02d69f9440117 WatchSource:0}: Error finding container 795be480b51eff5943ac970dd14c693c0e95ba81e77337eab5c02d69f9440117: Status 404 returned error can't find the container with id 795be480b51eff5943ac970dd14c693c0e95ba81e77337eab5c02d69f9440117
Sep 18 20:13:13.969270 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:13.963748    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:13:13.969270 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:13.963924    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:13:13.969270 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:13.964069    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:13:14.277594 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:14.277507    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-nshgc" event={"ID":"db789071-22a5-41ef-b596-d6c08644eee3","Type":"ContainerStarted","Data":"a714588055635845a2a379d101eca48c7b24f14edda9decb5b8f482db440364d"}
Sep 18 20:13:14.277762 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:14.277609    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-nshgc" event={"ID":"db789071-22a5-41ef-b596-d6c08644eee3","Type":"ContainerStarted","Data":"795be480b51eff5943ac970dd14c693c0e95ba81e77337eab5c02d69f9440117"}
Sep 18 20:13:14.305012 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:14.304932    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-nshgc" podStartSLOduration=1.304917395 podStartE2EDuration="1.304917395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:13:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:13:14.302815312 +0000 UTC m=+3565.155115719" watchObservedRunningTime="2026-09-18 20:13:14.304917395 +0000 UTC m=+3565.157217842"
Sep 18 20:13:14.665484 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:14.665455    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-nshgc"
Sep 18 20:13:15.670646 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:15.670620    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-nshgc"
Sep 18 20:13:16.286580 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:16.286556    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-nshgc"
Sep 18 20:13:16.287531 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:16.287511    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-nshgc"
Sep 18 20:13:24.350090 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:24.350050    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz"]
Sep 18 20:13:24.353663 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:24.353644    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz"
Sep 18 20:13:24.356958 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:24.356934    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-lncpr\""
Sep 18 20:13:24.357093 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:24.356933    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 18 20:13:24.357257 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:24.357241    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 18 20:13:24.357335 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:24.357289    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:13:24.357397 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:24.357353    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-dockercfg-4hlxz\""
Sep 18 20:13:24.379721 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:24.379679    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz"]
Sep 18 20:13:24.413593 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:24.413554    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e508f5f3-f08f-4bf8-b1e1-57aea0366324-model-cache\") pod \"router-with-refs-pd-test-kserve-6d976dfb47-lglqz\" (UID: \"e508f5f3-f08f-4bf8-b1e1-57aea0366324\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz"
Sep 18 20:13:24.413763 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:24.413618    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e508f5f3-f08f-4bf8-b1e1-57aea0366324-tls-certs\") pod \"router-with-refs-pd-test-kserve-6d976dfb47-lglqz\" (UID: \"e508f5f3-f08f-4bf8-b1e1-57aea0366324\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz"
Sep 18 20:13:24.413763 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:24.413665    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e508f5f3-f08f-4bf8-b1e1-57aea0366324-tmp-dir\") pod \"router-with-refs-pd-test-kserve-6d976dfb47-lglqz\" (UID: \"e508f5f3-f08f-4bf8-b1e1-57aea0366324\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz"
Sep 18 20:13:24.413763 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:24.413707    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e508f5f3-f08f-4bf8-b1e1-57aea0366324-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-6d976dfb47-lglqz\" (UID: \"e508f5f3-f08f-4bf8-b1e1-57aea0366324\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz"
Sep 18 20:13:24.413763 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:24.413748    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e508f5f3-f08f-4bf8-b1e1-57aea0366324-home\") pod \"router-with-refs-pd-test-kserve-6d976dfb47-lglqz\" (UID: \"e508f5f3-f08f-4bf8-b1e1-57aea0366324\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz"
Sep 18 20:13:24.414007 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:24.413865    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkdcj\" (UniqueName: \"kubernetes.io/projected/e508f5f3-f08f-4bf8-b1e1-57aea0366324-kube-api-access-jkdcj\") pod \"router-with-refs-pd-test-kserve-6d976dfb47-lglqz\" (UID: \"e508f5f3-f08f-4bf8-b1e1-57aea0366324\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz"
Sep 18 20:13:24.414007 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:24.413915    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e508f5f3-f08f-4bf8-b1e1-57aea0366324-dshm\") pod \"router-with-refs-pd-test-kserve-6d976dfb47-lglqz\" (UID: \"e508f5f3-f08f-4bf8-b1e1-57aea0366324\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz"
Sep 18 20:13:24.414007 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:24.413969    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e508f5f3-f08f-4bf8-b1e1-57aea0366324-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-6d976dfb47-lglqz\" (UID: \"e508f5f3-f08f-4bf8-b1e1-57aea0366324\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz"
Sep 18 20:13:24.514462 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:24.514420    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e508f5f3-f08f-4bf8-b1e1-57aea0366324-tls-certs\") pod \"router-with-refs-pd-test-kserve-6d976dfb47-lglqz\" (UID: \"e508f5f3-f08f-4bf8-b1e1-57aea0366324\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz"
Sep 18 20:13:24.514651 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:24.514505    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e508f5f3-f08f-4bf8-b1e1-57aea0366324-tmp-dir\") pod \"router-with-refs-pd-test-kserve-6d976dfb47-lglqz\" (UID: \"e508f5f3-f08f-4bf8-b1e1-57aea0366324\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz"
Sep 18 20:13:24.514651 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:24.514549    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e508f5f3-f08f-4bf8-b1e1-57aea0366324-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-6d976dfb47-lglqz\" (UID: \"e508f5f3-f08f-4bf8-b1e1-57aea0366324\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz"
Sep 18 20:13:24.514651 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:24.514604    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e508f5f3-f08f-4bf8-b1e1-57aea0366324-home\") pod \"router-with-refs-pd-test-kserve-6d976dfb47-lglqz\" (UID: \"e508f5f3-f08f-4bf8-b1e1-57aea0366324\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz"
Sep 18 20:13:24.514836 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:24.514663    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkdcj\" (UniqueName: \"kubernetes.io/projected/e508f5f3-f08f-4bf8-b1e1-57aea0366324-kube-api-access-jkdcj\") pod \"router-with-refs-pd-test-kserve-6d976dfb47-lglqz\" (UID: \"e508f5f3-f08f-4bf8-b1e1-57aea0366324\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz"
Sep 18 20:13:24.514836 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:24.514717    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e508f5f3-f08f-4bf8-b1e1-57aea0366324-dshm\") pod \"router-with-refs-pd-test-kserve-6d976dfb47-lglqz\" (UID: \"e508f5f3-f08f-4bf8-b1e1-57aea0366324\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz"
Sep 18 20:13:24.514836 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:24.514758    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e508f5f3-f08f-4bf8-b1e1-57aea0366324-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-6d976dfb47-lglqz\" (UID: \"e508f5f3-f08f-4bf8-b1e1-57aea0366324\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz"
Sep 18 20:13:24.515040 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:24.514939    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e508f5f3-f08f-4bf8-b1e1-57aea0366324-tmp-dir\") pod \"router-with-refs-pd-test-kserve-6d976dfb47-lglqz\" (UID: \"e508f5f3-f08f-4bf8-b1e1-57aea0366324\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz"
Sep 18 20:13:24.515120 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:24.515058    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e508f5f3-f08f-4bf8-b1e1-57aea0366324-model-cache\") pod \"router-with-refs-pd-test-kserve-6d976dfb47-lglqz\" (UID: \"e508f5f3-f08f-4bf8-b1e1-57aea0366324\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz"
Sep 18 20:13:24.515400 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:24.515373    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e508f5f3-f08f-4bf8-b1e1-57aea0366324-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-6d976dfb47-lglqz\" (UID: \"e508f5f3-f08f-4bf8-b1e1-57aea0366324\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz"
Sep 18 20:13:24.515543 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:24.515441    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e508f5f3-f08f-4bf8-b1e1-57aea0366324-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-6d976dfb47-lglqz\" (UID: \"e508f5f3-f08f-4bf8-b1e1-57aea0366324\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz"
Sep 18 20:13:24.515543 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:24.515455    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e508f5f3-f08f-4bf8-b1e1-57aea0366324-home\") pod \"router-with-refs-pd-test-kserve-6d976dfb47-lglqz\" (UID: \"e508f5f3-f08f-4bf8-b1e1-57aea0366324\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz"
Sep 18 20:13:24.515750 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:24.515724    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e508f5f3-f08f-4bf8-b1e1-57aea0366324-model-cache\") pod \"router-with-refs-pd-test-kserve-6d976dfb47-lglqz\" (UID: \"e508f5f3-f08f-4bf8-b1e1-57aea0366324\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz"
Sep 18 20:13:24.517236 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:24.517215    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e508f5f3-f08f-4bf8-b1e1-57aea0366324-tls-certs\") pod \"router-with-refs-pd-test-kserve-6d976dfb47-lglqz\" (UID: \"e508f5f3-f08f-4bf8-b1e1-57aea0366324\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz"
Sep 18 20:13:24.517605 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:24.517586    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e508f5f3-f08f-4bf8-b1e1-57aea0366324-dshm\") pod \"router-with-refs-pd-test-kserve-6d976dfb47-lglqz\" (UID: \"e508f5f3-f08f-4bf8-b1e1-57aea0366324\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz"
Sep 18 20:13:24.532939 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:24.532917    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkdcj\" (UniqueName: \"kubernetes.io/projected/e508f5f3-f08f-4bf8-b1e1-57aea0366324-kube-api-access-jkdcj\") pod \"router-with-refs-pd-test-kserve-6d976dfb47-lglqz\" (UID: \"e508f5f3-f08f-4bf8-b1e1-57aea0366324\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz"
Sep 18 20:13:24.664248 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:24.664213    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz"
Sep 18 20:13:24.960175 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:24.960148    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz"]
Sep 18 20:13:24.962480 ip-10-0-129-30 kubenswrapper[2577]: W0918 20:13:24.962451    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode508f5f3_f08f_4bf8_b1e1_57aea0366324.slice/crio-1b3c7ff5cd0a84b526dde57bbdc864c09901e52bfd33a769ab983b33a3968ebd WatchSource:0}: Error finding container 1b3c7ff5cd0a84b526dde57bbdc864c09901e52bfd33a769ab983b33a3968ebd: Status 404 returned error can't find the container with id 1b3c7ff5cd0a84b526dde57bbdc864c09901e52bfd33a769ab983b33a3968ebd
Sep 18 20:13:25.338409 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:25.331392    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz" event={"ID":"e508f5f3-f08f-4bf8-b1e1-57aea0366324","Type":"ContainerStarted","Data":"63fcd01e138d48b9a69adf997e56c11fbce2c54b3f3be278c1e986a6fa5a08c7"}
Sep 18 20:13:25.338409 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:25.331455    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz" event={"ID":"e508f5f3-f08f-4bf8-b1e1-57aea0366324","Type":"ContainerStarted","Data":"1b3c7ff5cd0a84b526dde57bbdc864c09901e52bfd33a769ab983b33a3968ebd"}
Sep 18 20:13:25.338409 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:25.331503    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz"
Sep 18 20:13:26.339857 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:26.339809    2577 generic.go:356] "Generic (PLEG): container finished" podID="db789071-22a5-41ef-b596-d6c08644eee3" containerID="a714588055635845a2a379d101eca48c7b24f14edda9decb5b8f482db440364d" exitCode=0
Sep 18 20:13:26.340085 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:26.339890    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-nshgc" event={"ID":"db789071-22a5-41ef-b596-d6c08644eee3","Type":"ContainerDied","Data":"a714588055635845a2a379d101eca48c7b24f14edda9decb5b8f482db440364d"}
Sep 18 20:13:26.341376 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:26.341353    2577 scope.go:117] "RemoveContainer" containerID="a714588055635845a2a379d101eca48c7b24f14edda9decb5b8f482db440364d"
Sep 18 20:13:26.342564 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:26.342522    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz" event={"ID":"e508f5f3-f08f-4bf8-b1e1-57aea0366324","Type":"ContainerStarted","Data":"1f550c179047398c566d01c6cc6824377a2cbf9379a45b7c34fbdcc2d7855838"}
Sep 18 20:13:26.343555 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:26.343517    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:13:26.343666 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:26.343613    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:13:26.343742 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:26.343662    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 20:13:27.353806 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:27.353769    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-nshgc" event={"ID":"db789071-22a5-41ef-b596-d6c08644eee3","Type":"ContainerStarted","Data":"47a9ceb7e0051629b33d24f92cba3350f0f7e75e92db8450f8f9056b8e85192a"}
Sep 18 20:13:27.666164 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:27.666126    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-nshgc"
Sep 18 20:13:27.667114 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:27.667084    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-nshgc" podUID="db789071-22a5-41ef-b596-d6c08644eee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 18 20:13:28.359831 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:28.359797    2577 generic.go:356] "Generic (PLEG): container finished" podID="e508f5f3-f08f-4bf8-b1e1-57aea0366324" containerID="1f550c179047398c566d01c6cc6824377a2cbf9379a45b7c34fbdcc2d7855838" exitCode=0
Sep 18 20:13:28.360373 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:28.359868    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz" event={"ID":"e508f5f3-f08f-4bf8-b1e1-57aea0366324","Type":"ContainerDied","Data":"1f550c179047398c566d01c6cc6824377a2cbf9379a45b7c34fbdcc2d7855838"}
Sep 18 20:13:28.665556 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:28.665517    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-nshgc" podUID="db789071-22a5-41ef-b596-d6c08644eee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 18 20:13:29.367736 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:29.367687    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz" event={"ID":"e508f5f3-f08f-4bf8-b1e1-57aea0366324","Type":"ContainerStarted","Data":"cbd99f2e4b312f9aee2c367459dfa8b449ef371e7ce5217515d45db3c47f033f"}
Sep 18 20:13:29.401145 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:29.401094    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz" podStartSLOduration=3.10000409 podStartE2EDuration="5.401074861s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.301070771s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:13:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:13:29.398601527 +0000 UTC m=+3580.250901898" watchObservedRunningTime="2026-09-18 20:13:29.401074861 +0000 UTC m=+3580.253375231"
Sep 18 20:13:29.665624 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:29.665529    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-nshgc" podUID="db789071-22a5-41ef-b596-d6c08644eee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 18 20:13:30.666288 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:30.666247    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-nshgc" podUID="db789071-22a5-41ef-b596-d6c08644eee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 18 20:13:31.286558 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:31.286523    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-nshgc"
Sep 18 20:13:31.665798 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:31.665756    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-nshgc" podUID="db789071-22a5-41ef-b596-d6c08644eee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 18 20:13:32.670292 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:32.670255    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-nshgc"
Sep 18 20:13:32.671391 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:32.671370    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-nshgc"
Sep 18 20:13:34.665117 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:34.665081    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz"
Sep 18 20:13:34.665490 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:34.665141    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz"
Sep 18 20:13:34.666320 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:34.666288    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz" podUID="e508f5f3-f08f-4bf8-b1e1-57aea0366324" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8001/health\": dial tcp 10.132.0.50:8001: connect: connection refused"
Sep 18 20:13:44.664834 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:44.664793    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz" podUID="e508f5f3-f08f-4bf8-b1e1-57aea0366324" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8001/health\": dial tcp 10.132.0.50:8001: connect: connection refused"
Sep 18 20:13:44.682542 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:44.682516    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz"
Sep 18 20:13:54.665352 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:13:54.665293    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz" podUID="e508f5f3-f08f-4bf8-b1e1-57aea0366324" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8001/health\": dial tcp 10.132.0.50:8001: connect: connection refused"
Sep 18 20:14:04.664953 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:14:04.664905    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz" podUID="e508f5f3-f08f-4bf8-b1e1-57aea0366324" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8001/health\": dial tcp 10.132.0.50:8001: connect: connection refused"
Sep 18 20:14:14.665327 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:14:14.665282    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz" podUID="e508f5f3-f08f-4bf8-b1e1-57aea0366324" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8001/health\": dial tcp 10.132.0.50:8001: connect: connection refused"
Sep 18 20:14:24.664896 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:14:24.664830    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz" podUID="e508f5f3-f08f-4bf8-b1e1-57aea0366324" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8001/health\": dial tcp 10.132.0.50:8001: connect: connection refused"
Sep 18 20:14:34.664982 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:14:34.664936    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz" podUID="e508f5f3-f08f-4bf8-b1e1-57aea0366324" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8001/health\": dial tcp 10.132.0.50:8001: connect: connection refused"
Sep 18 20:14:44.665388 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:14:44.665335    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz" podUID="e508f5f3-f08f-4bf8-b1e1-57aea0366324" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8001/health\": dial tcp 10.132.0.50:8001: connect: connection refused"
Sep 18 20:14:54.665162 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:14:54.665121    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz" podUID="e508f5f3-f08f-4bf8-b1e1-57aea0366324" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8001/health\": dial tcp 10.132.0.50:8001: connect: connection refused"
Sep 18 20:15:04.664838 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:15:04.664735    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz" podUID="e508f5f3-f08f-4bf8-b1e1-57aea0366324" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8001/health\": dial tcp 10.132.0.50:8001: connect: connection refused"
Sep 18 20:15:14.664977 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:15:14.664932    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz" podUID="e508f5f3-f08f-4bf8-b1e1-57aea0366324" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8001/health\": dial tcp 10.132.0.50:8001: connect: connection refused"
Sep 18 20:15:24.674782 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:15:24.674751    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz"
Sep 18 20:15:24.691453 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:15:24.691429    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz"
Sep 18 20:15:53.462025 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:15:53.461963    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz"]
Sep 18 20:16:08.484047 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:08.484001    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz" podUID="e508f5f3-f08f-4bf8-b1e1-57aea0366324" containerName="main" containerID="cri-o://cbd99f2e4b312f9aee2c367459dfa8b449ef371e7ce5217515d45db3c47f033f" gracePeriod=45
Sep 18 20:16:09.028046 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:09.027980    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz" podUID="e508f5f3-f08f-4bf8-b1e1-57aea0366324" containerName="llm-d-routing-sidecar" containerID="cri-o://63fcd01e138d48b9a69adf997e56c11fbce2c54b3f3be278c1e986a6fa5a08c7" gracePeriod=45
Sep 18 20:16:09.144311 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:09.144278    2577 generic.go:356] "Generic (PLEG): container finished" podID="e508f5f3-f08f-4bf8-b1e1-57aea0366324" containerID="cbd99f2e4b312f9aee2c367459dfa8b449ef371e7ce5217515d45db3c47f033f" exitCode=0
Sep 18 20:16:09.144465 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:09.144318    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz" event={"ID":"e508f5f3-f08f-4bf8-b1e1-57aea0366324","Type":"ContainerDied","Data":"cbd99f2e4b312f9aee2c367459dfa8b449ef371e7ce5217515d45db3c47f033f"}
Sep 18 20:16:09.276448 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:09.276423    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz"
Sep 18 20:16:09.436923 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:09.436896    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e508f5f3-f08f-4bf8-b1e1-57aea0366324-kserve-provision-location\") pod \"e508f5f3-f08f-4bf8-b1e1-57aea0366324\" (UID: \"e508f5f3-f08f-4bf8-b1e1-57aea0366324\") "
Sep 18 20:16:09.437140 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:09.436942    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e508f5f3-f08f-4bf8-b1e1-57aea0366324-tls-certs\") pod \"e508f5f3-f08f-4bf8-b1e1-57aea0366324\" (UID: \"e508f5f3-f08f-4bf8-b1e1-57aea0366324\") "
Sep 18 20:16:09.437140 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:09.436977    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e508f5f3-f08f-4bf8-b1e1-57aea0366324-cabundle-cert\") pod \"e508f5f3-f08f-4bf8-b1e1-57aea0366324\" (UID: \"e508f5f3-f08f-4bf8-b1e1-57aea0366324\") "
Sep 18 20:16:09.437140 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:09.437036    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e508f5f3-f08f-4bf8-b1e1-57aea0366324-tmp-dir\") pod \"e508f5f3-f08f-4bf8-b1e1-57aea0366324\" (UID: \"e508f5f3-f08f-4bf8-b1e1-57aea0366324\") "
Sep 18 20:16:09.437140 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:09.437123    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e508f5f3-f08f-4bf8-b1e1-57aea0366324-model-cache\") pod \"e508f5f3-f08f-4bf8-b1e1-57aea0366324\" (UID: \"e508f5f3-f08f-4bf8-b1e1-57aea0366324\") "
Sep 18 20:16:09.437377 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:09.437156    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e508f5f3-f08f-4bf8-b1e1-57aea0366324-home\") pod \"e508f5f3-f08f-4bf8-b1e1-57aea0366324\" (UID: \"e508f5f3-f08f-4bf8-b1e1-57aea0366324\") "
Sep 18 20:16:09.437377 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:09.437187    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkdcj\" (UniqueName: \"kubernetes.io/projected/e508f5f3-f08f-4bf8-b1e1-57aea0366324-kube-api-access-jkdcj\") pod \"e508f5f3-f08f-4bf8-b1e1-57aea0366324\" (UID: \"e508f5f3-f08f-4bf8-b1e1-57aea0366324\") "
Sep 18 20:16:09.437377 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:09.437241    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e508f5f3-f08f-4bf8-b1e1-57aea0366324-dshm\") pod \"e508f5f3-f08f-4bf8-b1e1-57aea0366324\" (UID: \"e508f5f3-f08f-4bf8-b1e1-57aea0366324\") "
Sep 18 20:16:09.437377 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:09.437344    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e508f5f3-f08f-4bf8-b1e1-57aea0366324-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e508f5f3-f08f-4bf8-b1e1-57aea0366324" (UID: "e508f5f3-f08f-4bf8-b1e1-57aea0366324"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:16:09.437581 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:09.437486    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e508f5f3-f08f-4bf8-b1e1-57aea0366324-cabundle-cert\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:16:09.438040 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:09.437846    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e508f5f3-f08f-4bf8-b1e1-57aea0366324-home" (OuterVolumeSpecName: "home") pod "e508f5f3-f08f-4bf8-b1e1-57aea0366324" (UID: "e508f5f3-f08f-4bf8-b1e1-57aea0366324"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:16:09.438202 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:09.438092    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e508f5f3-f08f-4bf8-b1e1-57aea0366324-model-cache" (OuterVolumeSpecName: "model-cache") pod "e508f5f3-f08f-4bf8-b1e1-57aea0366324" (UID: "e508f5f3-f08f-4bf8-b1e1-57aea0366324"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:16:09.440053 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:09.439928    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e508f5f3-f08f-4bf8-b1e1-57aea0366324-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e508f5f3-f08f-4bf8-b1e1-57aea0366324" (UID: "e508f5f3-f08f-4bf8-b1e1-57aea0366324"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:16:09.440053 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:09.439977    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e508f5f3-f08f-4bf8-b1e1-57aea0366324-kube-api-access-jkdcj" (OuterVolumeSpecName: "kube-api-access-jkdcj") pod "e508f5f3-f08f-4bf8-b1e1-57aea0366324" (UID: "e508f5f3-f08f-4bf8-b1e1-57aea0366324"). InnerVolumeSpecName "kube-api-access-jkdcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:16:09.440218 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:09.440068    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e508f5f3-f08f-4bf8-b1e1-57aea0366324-dshm" (OuterVolumeSpecName: "dshm") pod "e508f5f3-f08f-4bf8-b1e1-57aea0366324" (UID: "e508f5f3-f08f-4bf8-b1e1-57aea0366324"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:16:09.449015 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:09.448978    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e508f5f3-f08f-4bf8-b1e1-57aea0366324-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e508f5f3-f08f-4bf8-b1e1-57aea0366324" (UID: "e508f5f3-f08f-4bf8-b1e1-57aea0366324"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:16:09.463183 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:09.463160    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e508f5f3-f08f-4bf8-b1e1-57aea0366324-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e508f5f3-f08f-4bf8-b1e1-57aea0366324" (UID: "e508f5f3-f08f-4bf8-b1e1-57aea0366324"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:16:09.537933 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:09.537849    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e508f5f3-f08f-4bf8-b1e1-57aea0366324-model-cache\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:16:09.537933 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:09.537883    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e508f5f3-f08f-4bf8-b1e1-57aea0366324-home\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:16:09.537933 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:09.537892    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkdcj\" (UniqueName: \"kubernetes.io/projected/e508f5f3-f08f-4bf8-b1e1-57aea0366324-kube-api-access-jkdcj\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:16:09.537933 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:09.537902    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e508f5f3-f08f-4bf8-b1e1-57aea0366324-dshm\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:16:09.537933 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:09.537911    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e508f5f3-f08f-4bf8-b1e1-57aea0366324-kserve-provision-location\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:16:09.537933 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:09.537920    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e508f5f3-f08f-4bf8-b1e1-57aea0366324-tls-certs\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:16:09.537933 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:09.537929    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e508f5f3-f08f-4bf8-b1e1-57aea0366324-tmp-dir\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:16:10.151287 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:10.151249    2577 generic.go:356] "Generic (PLEG): container finished" podID="e508f5f3-f08f-4bf8-b1e1-57aea0366324" containerID="63fcd01e138d48b9a69adf997e56c11fbce2c54b3f3be278c1e986a6fa5a08c7" exitCode=0
Sep 18 20:16:10.151545 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:10.151317    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz"
Sep 18 20:16:10.151545 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:10.151332    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz" event={"ID":"e508f5f3-f08f-4bf8-b1e1-57aea0366324","Type":"ContainerDied","Data":"63fcd01e138d48b9a69adf997e56c11fbce2c54b3f3be278c1e986a6fa5a08c7"}
Sep 18 20:16:10.151545 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:10.151391    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz" event={"ID":"e508f5f3-f08f-4bf8-b1e1-57aea0366324","Type":"ContainerDied","Data":"1b3c7ff5cd0a84b526dde57bbdc864c09901e52bfd33a769ab983b33a3968ebd"}
Sep 18 20:16:10.151545 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:10.151423    2577 scope.go:117] "RemoveContainer" containerID="cbd99f2e4b312f9aee2c367459dfa8b449ef371e7ce5217515d45db3c47f033f"
Sep 18 20:16:10.161075 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:10.161061    2577 scope.go:117] "RemoveContainer" containerID="1f550c179047398c566d01c6cc6824377a2cbf9379a45b7c34fbdcc2d7855838"
Sep 18 20:16:10.169023 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:10.168998    2577 scope.go:117] "RemoveContainer" containerID="63fcd01e138d48b9a69adf997e56c11fbce2c54b3f3be278c1e986a6fa5a08c7"
Sep 18 20:16:10.176428 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:10.176401    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz"]
Sep 18 20:16:10.176903 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:10.176881    2577 scope.go:117] "RemoveContainer" containerID="cbd99f2e4b312f9aee2c367459dfa8b449ef371e7ce5217515d45db3c47f033f"
Sep 18 20:16:10.177285 ip-10-0-129-30 kubenswrapper[2577]: E0918 20:16:10.177231    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbd99f2e4b312f9aee2c367459dfa8b449ef371e7ce5217515d45db3c47f033f\": container with ID starting with cbd99f2e4b312f9aee2c367459dfa8b449ef371e7ce5217515d45db3c47f033f not found: ID does not exist" containerID="cbd99f2e4b312f9aee2c367459dfa8b449ef371e7ce5217515d45db3c47f033f"
Sep 18 20:16:10.177285 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:10.177265    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbd99f2e4b312f9aee2c367459dfa8b449ef371e7ce5217515d45db3c47f033f"} err="failed to get container status \"cbd99f2e4b312f9aee2c367459dfa8b449ef371e7ce5217515d45db3c47f033f\": rpc error: code = NotFound desc = could not find container \"cbd99f2e4b312f9aee2c367459dfa8b449ef371e7ce5217515d45db3c47f033f\": container with ID starting with cbd99f2e4b312f9aee2c367459dfa8b449ef371e7ce5217515d45db3c47f033f not found: ID does not exist"
Sep 18 20:16:10.177285 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:10.177283    2577 scope.go:117] "RemoveContainer" containerID="1f550c179047398c566d01c6cc6824377a2cbf9379a45b7c34fbdcc2d7855838"
Sep 18 20:16:10.178234 ip-10-0-129-30 kubenswrapper[2577]: E0918 20:16:10.177793    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f550c179047398c566d01c6cc6824377a2cbf9379a45b7c34fbdcc2d7855838\": container with ID starting with 1f550c179047398c566d01c6cc6824377a2cbf9379a45b7c34fbdcc2d7855838 not found: ID does not exist" containerID="1f550c179047398c566d01c6cc6824377a2cbf9379a45b7c34fbdcc2d7855838"
Sep 18 20:16:10.178234 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:10.177857    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f550c179047398c566d01c6cc6824377a2cbf9379a45b7c34fbdcc2d7855838"} err="failed to get container status \"1f550c179047398c566d01c6cc6824377a2cbf9379a45b7c34fbdcc2d7855838\": rpc error: code = NotFound desc = could not find container \"1f550c179047398c566d01c6cc6824377a2cbf9379a45b7c34fbdcc2d7855838\": container with ID starting with 1f550c179047398c566d01c6cc6824377a2cbf9379a45b7c34fbdcc2d7855838 not found: ID does not exist"
Sep 18 20:16:10.178234 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:10.177881    2577 scope.go:117] "RemoveContainer" containerID="63fcd01e138d48b9a69adf997e56c11fbce2c54b3f3be278c1e986a6fa5a08c7"
Sep 18 20:16:10.178447 ip-10-0-129-30 kubenswrapper[2577]: E0918 20:16:10.178310    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63fcd01e138d48b9a69adf997e56c11fbce2c54b3f3be278c1e986a6fa5a08c7\": container with ID starting with 63fcd01e138d48b9a69adf997e56c11fbce2c54b3f3be278c1e986a6fa5a08c7 not found: ID does not exist" containerID="63fcd01e138d48b9a69adf997e56c11fbce2c54b3f3be278c1e986a6fa5a08c7"
Sep 18 20:16:10.178447 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:10.178332    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63fcd01e138d48b9a69adf997e56c11fbce2c54b3f3be278c1e986a6fa5a08c7"} err="failed to get container status \"63fcd01e138d48b9a69adf997e56c11fbce2c54b3f3be278c1e986a6fa5a08c7\": rpc error: code = NotFound desc = could not find container \"63fcd01e138d48b9a69adf997e56c11fbce2c54b3f3be278c1e986a6fa5a08c7\": container with ID starting with 63fcd01e138d48b9a69adf997e56c11fbce2c54b3f3be278c1e986a6fa5a08c7 not found: ID does not exist"
Sep 18 20:16:10.179832 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:10.179811    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6d976dfb47-lglqz"]
Sep 18 20:16:11.439267 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:11.439227    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e508f5f3-f08f-4bf8-b1e1-57aea0366324" path="/var/lib/kubelet/pods/e508f5f3-f08f-4bf8-b1e1-57aea0366324/volumes"
Sep 18 20:16:48.494049 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:48.494012    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-nshgc"]
Sep 18 20:16:48.494608 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:48.494253    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-nshgc" podUID="db789071-22a5-41ef-b596-d6c08644eee3" containerName="istio-proxy" containerID="cri-o://47a9ceb7e0051629b33d24f92cba3350f0f7e75e92db8450f8f9056b8e85192a" gracePeriod=30
Sep 18 20:16:53.734466 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:53.734437    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-nshgc"
Sep 18 20:16:53.924238 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:53.924203    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ldp2d\" (UniqueName: \"kubernetes.io/projected/db789071-22a5-41ef-b596-d6c08644eee3-kube-api-access-ldp2d\") pod \"db789071-22a5-41ef-b596-d6c08644eee3\" (UID: \"db789071-22a5-41ef-b596-d6c08644eee3\") "
Sep 18 20:16:53.924421 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:53.924250    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db789071-22a5-41ef-b596-d6c08644eee3-istio-data\") pod \"db789071-22a5-41ef-b596-d6c08644eee3\" (UID: \"db789071-22a5-41ef-b596-d6c08644eee3\") "
Sep 18 20:16:53.924421 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:53.924282    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db789071-22a5-41ef-b596-d6c08644eee3-istio-podinfo\") pod \"db789071-22a5-41ef-b596-d6c08644eee3\" (UID: \"db789071-22a5-41ef-b596-d6c08644eee3\") "
Sep 18 20:16:53.924421 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:53.924306    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db789071-22a5-41ef-b596-d6c08644eee3-istio-envoy\") pod \"db789071-22a5-41ef-b596-d6c08644eee3\" (UID: \"db789071-22a5-41ef-b596-d6c08644eee3\") "
Sep 18 20:16:53.924566 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:53.924420    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db789071-22a5-41ef-b596-d6c08644eee3-credential-socket\") pod \"db789071-22a5-41ef-b596-d6c08644eee3\" (UID: \"db789071-22a5-41ef-b596-d6c08644eee3\") "
Sep 18 20:16:53.924566 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:53.924475    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db789071-22a5-41ef-b596-d6c08644eee3-istio-token\") pod \"db789071-22a5-41ef-b596-d6c08644eee3\" (UID: \"db789071-22a5-41ef-b596-d6c08644eee3\") "
Sep 18 20:16:53.924566 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:53.924532    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db789071-22a5-41ef-b596-d6c08644eee3-istiod-ca-cert\") pod \"db789071-22a5-41ef-b596-d6c08644eee3\" (UID: \"db789071-22a5-41ef-b596-d6c08644eee3\") "
Sep 18 20:16:53.924737 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:53.924572    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db789071-22a5-41ef-b596-d6c08644eee3-workload-certs\") pod \"db789071-22a5-41ef-b596-d6c08644eee3\" (UID: \"db789071-22a5-41ef-b596-d6c08644eee3\") "
Sep 18 20:16:53.924737 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:53.924693    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db789071-22a5-41ef-b596-d6c08644eee3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "db789071-22a5-41ef-b596-d6c08644eee3" (UID: "db789071-22a5-41ef-b596-d6c08644eee3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:16:53.924863 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:53.924787    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db789071-22a5-41ef-b596-d6c08644eee3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "db789071-22a5-41ef-b596-d6c08644eee3" (UID: "db789071-22a5-41ef-b596-d6c08644eee3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:16:53.924932 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:53.924858    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db789071-22a5-41ef-b596-d6c08644eee3-workload-socket\") pod \"db789071-22a5-41ef-b596-d6c08644eee3\" (UID: \"db789071-22a5-41ef-b596-d6c08644eee3\") "
Sep 18 20:16:53.924932 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:53.924878    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db789071-22a5-41ef-b596-d6c08644eee3-istio-data" (OuterVolumeSpecName: "istio-data") pod "db789071-22a5-41ef-b596-d6c08644eee3" (UID: "db789071-22a5-41ef-b596-d6c08644eee3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:16:53.924932 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:53.924880    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db789071-22a5-41ef-b596-d6c08644eee3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "db789071-22a5-41ef-b596-d6c08644eee3" (UID: "db789071-22a5-41ef-b596-d6c08644eee3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:16:53.925127 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:53.925103    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db789071-22a5-41ef-b596-d6c08644eee3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "db789071-22a5-41ef-b596-d6c08644eee3" (UID: "db789071-22a5-41ef-b596-d6c08644eee3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:16:53.925299 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:53.925281    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db789071-22a5-41ef-b596-d6c08644eee3-istio-data\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:16:53.925364 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:53.925300    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db789071-22a5-41ef-b596-d6c08644eee3-credential-socket\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:16:53.925364 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:53.925309    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db789071-22a5-41ef-b596-d6c08644eee3-istiod-ca-cert\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:16:53.925364 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:53.925322    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db789071-22a5-41ef-b596-d6c08644eee3-workload-certs\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:16:53.925364 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:53.925332    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db789071-22a5-41ef-b596-d6c08644eee3-workload-socket\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:16:53.926716 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:53.926689    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db789071-22a5-41ef-b596-d6c08644eee3-istio-token" (OuterVolumeSpecName: "istio-token") pod "db789071-22a5-41ef-b596-d6c08644eee3" (UID: "db789071-22a5-41ef-b596-d6c08644eee3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:16:53.926880 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:53.926865    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db789071-22a5-41ef-b596-d6c08644eee3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "db789071-22a5-41ef-b596-d6c08644eee3" (UID: "db789071-22a5-41ef-b596-d6c08644eee3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:16:53.926976 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:53.926962    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db789071-22a5-41ef-b596-d6c08644eee3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "db789071-22a5-41ef-b596-d6c08644eee3" (UID: "db789071-22a5-41ef-b596-d6c08644eee3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 20:16:53.927048 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:53.927006    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db789071-22a5-41ef-b596-d6c08644eee3-kube-api-access-ldp2d" (OuterVolumeSpecName: "kube-api-access-ldp2d") pod "db789071-22a5-41ef-b596-d6c08644eee3" (UID: "db789071-22a5-41ef-b596-d6c08644eee3"). InnerVolumeSpecName "kube-api-access-ldp2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:16:54.025728 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:54.025692    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldp2d\" (UniqueName: \"kubernetes.io/projected/db789071-22a5-41ef-b596-d6c08644eee3-kube-api-access-ldp2d\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:16:54.025728 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:54.025721    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db789071-22a5-41ef-b596-d6c08644eee3-istio-podinfo\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:16:54.025728 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:54.025731    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db789071-22a5-41ef-b596-d6c08644eee3-istio-envoy\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:16:54.025947 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:54.025738    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db789071-22a5-41ef-b596-d6c08644eee3-istio-token\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:16:54.347069 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:54.347033    2577 generic.go:356] "Generic (PLEG): container finished" podID="db789071-22a5-41ef-b596-d6c08644eee3" containerID="47a9ceb7e0051629b33d24f92cba3350f0f7e75e92db8450f8f9056b8e85192a" exitCode=0
Sep 18 20:16:54.347246 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:54.347092    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-nshgc"
Sep 18 20:16:54.347246 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:54.347125    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-nshgc" event={"ID":"db789071-22a5-41ef-b596-d6c08644eee3","Type":"ContainerDied","Data":"47a9ceb7e0051629b33d24f92cba3350f0f7e75e92db8450f8f9056b8e85192a"}
Sep 18 20:16:54.347246 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:54.347185    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-nshgc" event={"ID":"db789071-22a5-41ef-b596-d6c08644eee3","Type":"ContainerDied","Data":"795be480b51eff5943ac970dd14c693c0e95ba81e77337eab5c02d69f9440117"}
Sep 18 20:16:54.347246 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:54.347208    2577 scope.go:117] "RemoveContainer" containerID="47a9ceb7e0051629b33d24f92cba3350f0f7e75e92db8450f8f9056b8e85192a"
Sep 18 20:16:54.356110 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:54.356096    2577 scope.go:117] "RemoveContainer" containerID="a714588055635845a2a379d101eca48c7b24f14edda9decb5b8f482db440364d"
Sep 18 20:16:54.365019 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:54.364976    2577 scope.go:117] "RemoveContainer" containerID="47a9ceb7e0051629b33d24f92cba3350f0f7e75e92db8450f8f9056b8e85192a"
Sep 18 20:16:54.365307 ip-10-0-129-30 kubenswrapper[2577]: E0918 20:16:54.365289    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47a9ceb7e0051629b33d24f92cba3350f0f7e75e92db8450f8f9056b8e85192a\": container with ID starting with 47a9ceb7e0051629b33d24f92cba3350f0f7e75e92db8450f8f9056b8e85192a not found: ID does not exist" containerID="47a9ceb7e0051629b33d24f92cba3350f0f7e75e92db8450f8f9056b8e85192a"
Sep 18 20:16:54.365353 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:54.365316    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47a9ceb7e0051629b33d24f92cba3350f0f7e75e92db8450f8f9056b8e85192a"} err="failed to get container status \"47a9ceb7e0051629b33d24f92cba3350f0f7e75e92db8450f8f9056b8e85192a\": rpc error: code = NotFound desc = could not find container \"47a9ceb7e0051629b33d24f92cba3350f0f7e75e92db8450f8f9056b8e85192a\": container with ID starting with 47a9ceb7e0051629b33d24f92cba3350f0f7e75e92db8450f8f9056b8e85192a not found: ID does not exist"
Sep 18 20:16:54.365353 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:54.365335    2577 scope.go:117] "RemoveContainer" containerID="a714588055635845a2a379d101eca48c7b24f14edda9decb5b8f482db440364d"
Sep 18 20:16:54.365599 ip-10-0-129-30 kubenswrapper[2577]: E0918 20:16:54.365578    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a714588055635845a2a379d101eca48c7b24f14edda9decb5b8f482db440364d\": container with ID starting with a714588055635845a2a379d101eca48c7b24f14edda9decb5b8f482db440364d not found: ID does not exist" containerID="a714588055635845a2a379d101eca48c7b24f14edda9decb5b8f482db440364d"
Sep 18 20:16:54.365719 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:54.365603    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a714588055635845a2a379d101eca48c7b24f14edda9decb5b8f482db440364d"} err="failed to get container status \"a714588055635845a2a379d101eca48c7b24f14edda9decb5b8f482db440364d\": rpc error: code = NotFound desc = could not find container \"a714588055635845a2a379d101eca48c7b24f14edda9decb5b8f482db440364d\": container with ID starting with a714588055635845a2a379d101eca48c7b24f14edda9decb5b8f482db440364d not found: ID does not exist"
Sep 18 20:16:54.370860 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:54.370838    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-nshgc"]
Sep 18 20:16:54.376733 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:54.376703    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-nshgc"]
Sep 18 20:16:55.438134 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:55.438103    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db789071-22a5-41ef-b596-d6c08644eee3" path="/var/lib/kubelet/pods/db789071-22a5-41ef-b596-d6c08644eee3/volumes"
Sep 18 20:16:58.214466 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:58.214432    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r"]
Sep 18 20:16:58.214840 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:58.214733    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db789071-22a5-41ef-b596-d6c08644eee3" containerName="istio-proxy"
Sep 18 20:16:58.214840 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:58.214743    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="db789071-22a5-41ef-b596-d6c08644eee3" containerName="istio-proxy"
Sep 18 20:16:58.214840 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:58.214755    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e508f5f3-f08f-4bf8-b1e1-57aea0366324" containerName="storage-initializer"
Sep 18 20:16:58.214840 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:58.214761    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e508f5f3-f08f-4bf8-b1e1-57aea0366324" containerName="storage-initializer"
Sep 18 20:16:58.214840 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:58.214778    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e508f5f3-f08f-4bf8-b1e1-57aea0366324" containerName="main"
Sep 18 20:16:58.214840 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:58.214784    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e508f5f3-f08f-4bf8-b1e1-57aea0366324" containerName="main"
Sep 18 20:16:58.214840 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:58.214792    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e508f5f3-f08f-4bf8-b1e1-57aea0366324" containerName="llm-d-routing-sidecar"
Sep 18 20:16:58.214840 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:58.214797    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e508f5f3-f08f-4bf8-b1e1-57aea0366324" containerName="llm-d-routing-sidecar"
Sep 18 20:16:58.214840 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:58.214842    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e508f5f3-f08f-4bf8-b1e1-57aea0366324" containerName="main"
Sep 18 20:16:58.215158 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:58.214855    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="db789071-22a5-41ef-b596-d6c08644eee3" containerName="istio-proxy"
Sep 18 20:16:58.215158 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:58.214861    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e508f5f3-f08f-4bf8-b1e1-57aea0366324" containerName="llm-d-routing-sidecar"
Sep 18 20:16:58.215158 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:58.214868    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="db789071-22a5-41ef-b596-d6c08644eee3" containerName="istio-proxy"
Sep 18 20:16:58.215158 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:58.214922    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db789071-22a5-41ef-b596-d6c08644eee3" containerName="istio-proxy"
Sep 18 20:16:58.215158 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:58.214927    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="db789071-22a5-41ef-b596-d6c08644eee3" containerName="istio-proxy"
Sep 18 20:16:58.217563 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:58.217545    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r"
Sep 18 20:16:58.220697 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:58.220664    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Sep 18 20:16:58.220848 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:58.220810    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"default-dockercfg-l7zz4\""
Sep 18 20:16:58.220933 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:58.220850    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:16:58.220933 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:58.220858    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"openshift-service-ca.crt\""
Sep 18 20:16:58.221085 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:58.220933    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"seaweedfs-s3-creds\""
Sep 18 20:16:58.221085 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:58.220937    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"kube-root-ca.crt\""
Sep 18 20:16:58.233097 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:58.233069    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r"]
Sep 18 20:16:58.363593 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:58.363553    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c74cc46-19a2-4b10-bcc8-d24fd72d7dae-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r\" (UID: \"7c74cc46-19a2-4b10-bcc8-d24fd72d7dae\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r"
Sep 18 20:16:58.363808 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:58.363617    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c74cc46-19a2-4b10-bcc8-d24fd72d7dae-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r\" (UID: \"7c74cc46-19a2-4b10-bcc8-d24fd72d7dae\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r"
Sep 18 20:16:58.363808 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:58.363635    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c74cc46-19a2-4b10-bcc8-d24fd72d7dae-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r\" (UID: \"7c74cc46-19a2-4b10-bcc8-d24fd72d7dae\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r"
Sep 18 20:16:58.363808 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:58.363678    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c74cc46-19a2-4b10-bcc8-d24fd72d7dae-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r\" (UID: \"7c74cc46-19a2-4b10-bcc8-d24fd72d7dae\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r"
Sep 18 20:16:58.363808 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:58.363718    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c74cc46-19a2-4b10-bcc8-d24fd72d7dae-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r\" (UID: \"7c74cc46-19a2-4b10-bcc8-d24fd72d7dae\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r"
Sep 18 20:16:58.363808 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:58.363746    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqxgx\" (UniqueName: \"kubernetes.io/projected/7c74cc46-19a2-4b10-bcc8-d24fd72d7dae-kube-api-access-nqxgx\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r\" (UID: \"7c74cc46-19a2-4b10-bcc8-d24fd72d7dae\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r"
Sep 18 20:16:58.363808 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:58.363798    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c74cc46-19a2-4b10-bcc8-d24fd72d7dae-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r\" (UID: \"7c74cc46-19a2-4b10-bcc8-d24fd72d7dae\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r"
Sep 18 20:16:58.364120 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:58.363837    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c74cc46-19a2-4b10-bcc8-d24fd72d7dae-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r\" (UID: \"7c74cc46-19a2-4b10-bcc8-d24fd72d7dae\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r"
Sep 18 20:16:58.464268 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:58.464228    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c74cc46-19a2-4b10-bcc8-d24fd72d7dae-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r\" (UID: \"7c74cc46-19a2-4b10-bcc8-d24fd72d7dae\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r"
Sep 18 20:16:58.464425 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:58.464285    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c74cc46-19a2-4b10-bcc8-d24fd72d7dae-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r\" (UID: \"7c74cc46-19a2-4b10-bcc8-d24fd72d7dae\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r"
Sep 18 20:16:58.464425 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:58.464302    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c74cc46-19a2-4b10-bcc8-d24fd72d7dae-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r\" (UID: \"7c74cc46-19a2-4b10-bcc8-d24fd72d7dae\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r"
Sep 18 20:16:58.464425 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:58.464320    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c74cc46-19a2-4b10-bcc8-d24fd72d7dae-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r\" (UID: \"7c74cc46-19a2-4b10-bcc8-d24fd72d7dae\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r"
Sep 18 20:16:58.464425 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:58.464339    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c74cc46-19a2-4b10-bcc8-d24fd72d7dae-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r\" (UID: \"7c74cc46-19a2-4b10-bcc8-d24fd72d7dae\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r"
Sep 18 20:16:58.464425 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:58.464355    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqxgx\" (UniqueName: \"kubernetes.io/projected/7c74cc46-19a2-4b10-bcc8-d24fd72d7dae-kube-api-access-nqxgx\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r\" (UID: \"7c74cc46-19a2-4b10-bcc8-d24fd72d7dae\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r"
Sep 18 20:16:58.464425 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:58.464377    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c74cc46-19a2-4b10-bcc8-d24fd72d7dae-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r\" (UID: \"7c74cc46-19a2-4b10-bcc8-d24fd72d7dae\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r"
Sep 18 20:16:58.464425 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:58.464410    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c74cc46-19a2-4b10-bcc8-d24fd72d7dae-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r\" (UID: \"7c74cc46-19a2-4b10-bcc8-d24fd72d7dae\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r"
Sep 18 20:16:58.464855 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:58.464797    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c74cc46-19a2-4b10-bcc8-d24fd72d7dae-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r\" (UID: \"7c74cc46-19a2-4b10-bcc8-d24fd72d7dae\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r"
Sep 18 20:16:58.464920 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:58.464854    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c74cc46-19a2-4b10-bcc8-d24fd72d7dae-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r\" (UID: \"7c74cc46-19a2-4b10-bcc8-d24fd72d7dae\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r"
Sep 18 20:16:58.464920 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:58.464883    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c74cc46-19a2-4b10-bcc8-d24fd72d7dae-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r\" (UID: \"7c74cc46-19a2-4b10-bcc8-d24fd72d7dae\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r"
Sep 18 20:16:58.464920 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:58.464912    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c74cc46-19a2-4b10-bcc8-d24fd72d7dae-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r\" (UID: \"7c74cc46-19a2-4b10-bcc8-d24fd72d7dae\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r"
Sep 18 20:16:58.465403 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:58.465385    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c74cc46-19a2-4b10-bcc8-d24fd72d7dae-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r\" (UID: \"7c74cc46-19a2-4b10-bcc8-d24fd72d7dae\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r"
Sep 18 20:16:58.466722 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:58.466688    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c74cc46-19a2-4b10-bcc8-d24fd72d7dae-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r\" (UID: \"7c74cc46-19a2-4b10-bcc8-d24fd72d7dae\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r"
Sep 18 20:16:58.466816 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:58.466805    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c74cc46-19a2-4b10-bcc8-d24fd72d7dae-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r\" (UID: \"7c74cc46-19a2-4b10-bcc8-d24fd72d7dae\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r"
Sep 18 20:16:58.472632 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:58.472613    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqxgx\" (UniqueName: \"kubernetes.io/projected/7c74cc46-19a2-4b10-bcc8-d24fd72d7dae-kube-api-access-nqxgx\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r\" (UID: \"7c74cc46-19a2-4b10-bcc8-d24fd72d7dae\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r"
Sep 18 20:16:58.527404 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:58.527378    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r"
Sep 18 20:16:58.809893 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:58.809865    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r"]
Sep 18 20:16:58.812070 ip-10-0-129-30 kubenswrapper[2577]: W0918 20:16:58.812039    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c74cc46_19a2_4b10_bcc8_d24fd72d7dae.slice/crio-0d75ea43d00925f867b1012b750c7ec4d29a4b4f1634b621964cdff2ff5fba91 WatchSource:0}: Error finding container 0d75ea43d00925f867b1012b750c7ec4d29a4b4f1634b621964cdff2ff5fba91: Status 404 returned error can't find the container with id 0d75ea43d00925f867b1012b750c7ec4d29a4b4f1634b621964cdff2ff5fba91
Sep 18 20:16:59.372606 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:59.372560    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r" event={"ID":"7c74cc46-19a2-4b10-bcc8-d24fd72d7dae","Type":"ContainerStarted","Data":"ab525e3b20110b5230da5f8e389b4a1abf5b0c63d69a76261fda5dc00b1d38da"}
Sep 18 20:16:59.373075 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:16:59.372623    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r" event={"ID":"7c74cc46-19a2-4b10-bcc8-d24fd72d7dae","Type":"ContainerStarted","Data":"0d75ea43d00925f867b1012b750c7ec4d29a4b4f1634b621964cdff2ff5fba91"}
Sep 18 20:17:01.383366 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:17:01.383331    2577 generic.go:356] "Generic (PLEG): container finished" podID="7c74cc46-19a2-4b10-bcc8-d24fd72d7dae" containerID="ab525e3b20110b5230da5f8e389b4a1abf5b0c63d69a76261fda5dc00b1d38da" exitCode=0
Sep 18 20:17:01.383772 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:17:01.383394    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r" event={"ID":"7c74cc46-19a2-4b10-bcc8-d24fd72d7dae","Type":"ContainerDied","Data":"ab525e3b20110b5230da5f8e389b4a1abf5b0c63d69a76261fda5dc00b1d38da"}
Sep 18 20:17:02.390422 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:17:02.390380    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r" event={"ID":"7c74cc46-19a2-4b10-bcc8-d24fd72d7dae","Type":"ContainerStarted","Data":"e3b381d9977f8600e6a9060238a0b3597624f9f964d596ecdd09eb5910714927"}
Sep 18 20:17:02.417930 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:17:02.417834    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r" podStartSLOduration=2.221151605 podStartE2EDuration="4.417818184s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.196666579s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:16:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:17:02.416014705 +0000 UTC m=+3793.268315076" watchObservedRunningTime="2026-09-18 20:17:02.417818184 +0000 UTC m=+3793.270118553"
Sep 18 20:17:02.527498 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:17:02.527461    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r"
Sep 18 20:17:08.528390 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:17:08.528351    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r"
Sep 18 20:17:08.529879 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:17:08.529852    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r" podUID="7c74cc46-19a2-4b10-bcc8-d24fd72d7dae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 18 20:17:18.528339 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:17:18.528283    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r" podUID="7c74cc46-19a2-4b10-bcc8-d24fd72d7dae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 18 20:17:28.528236 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:17:28.528188    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r" podUID="7c74cc46-19a2-4b10-bcc8-d24fd72d7dae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 18 20:17:38.528160 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:17:38.528118    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r" podUID="7c74cc46-19a2-4b10-bcc8-d24fd72d7dae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 18 20:17:48.528238 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:17:48.528195    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r" podUID="7c74cc46-19a2-4b10-bcc8-d24fd72d7dae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 18 20:17:58.528242 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:17:58.528198    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r" podUID="7c74cc46-19a2-4b10-bcc8-d24fd72d7dae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 18 20:18:08.528009 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:18:08.527892    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r" podUID="7c74cc46-19a2-4b10-bcc8-d24fd72d7dae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 18 20:18:18.528608 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:18:18.528561    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r" podUID="7c74cc46-19a2-4b10-bcc8-d24fd72d7dae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 18 20:18:28.528348 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:18:28.528303    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r" podUID="7c74cc46-19a2-4b10-bcc8-d24fd72d7dae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 18 20:18:38.527821 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:18:38.527772    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r" podUID="7c74cc46-19a2-4b10-bcc8-d24fd72d7dae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 18 20:18:48.528155 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:18:48.528113    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r" podUID="7c74cc46-19a2-4b10-bcc8-d24fd72d7dae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 18 20:18:58.538116 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:18:58.538076    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r"
Sep 18 20:18:58.545698 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:18:58.545673    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r"
Sep 18 20:19:02.766751 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:02.766721    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r"]
Sep 18 20:19:17.788636 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:17.788568    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r" podUID="7c74cc46-19a2-4b10-bcc8-d24fd72d7dae" containerName="main" containerID="cri-o://e3b381d9977f8600e6a9060238a0b3597624f9f964d596ecdd09eb5910714927" gracePeriod=45
Sep 18 20:19:18.537176 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:18.537145    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r"
Sep 18 20:19:18.633475 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:18.633440    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c74cc46-19a2-4b10-bcc8-d24fd72d7dae-kserve-provision-location\") pod \"7c74cc46-19a2-4b10-bcc8-d24fd72d7dae\" (UID: \"7c74cc46-19a2-4b10-bcc8-d24fd72d7dae\") "
Sep 18 20:19:18.633675 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:18.633485    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c74cc46-19a2-4b10-bcc8-d24fd72d7dae-tmp-dir\") pod \"7c74cc46-19a2-4b10-bcc8-d24fd72d7dae\" (UID: \"7c74cc46-19a2-4b10-bcc8-d24fd72d7dae\") "
Sep 18 20:19:18.633675 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:18.633536    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c74cc46-19a2-4b10-bcc8-d24fd72d7dae-tls-certs\") pod \"7c74cc46-19a2-4b10-bcc8-d24fd72d7dae\" (UID: \"7c74cc46-19a2-4b10-bcc8-d24fd72d7dae\") "
Sep 18 20:19:18.633675 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:18.633564    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c74cc46-19a2-4b10-bcc8-d24fd72d7dae-dshm\") pod \"7c74cc46-19a2-4b10-bcc8-d24fd72d7dae\" (UID: \"7c74cc46-19a2-4b10-bcc8-d24fd72d7dae\") "
Sep 18 20:19:18.633675 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:18.633603    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nqxgx\" (UniqueName: \"kubernetes.io/projected/7c74cc46-19a2-4b10-bcc8-d24fd72d7dae-kube-api-access-nqxgx\") pod \"7c74cc46-19a2-4b10-bcc8-d24fd72d7dae\" (UID: \"7c74cc46-19a2-4b10-bcc8-d24fd72d7dae\") "
Sep 18 20:19:18.633675 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:18.633625    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c74cc46-19a2-4b10-bcc8-d24fd72d7dae-cabundle-cert\") pod \"7c74cc46-19a2-4b10-bcc8-d24fd72d7dae\" (UID: \"7c74cc46-19a2-4b10-bcc8-d24fd72d7dae\") "
Sep 18 20:19:18.633675 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:18.633660    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c74cc46-19a2-4b10-bcc8-d24fd72d7dae-home\") pod \"7c74cc46-19a2-4b10-bcc8-d24fd72d7dae\" (UID: \"7c74cc46-19a2-4b10-bcc8-d24fd72d7dae\") "
Sep 18 20:19:18.634017 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:18.633680    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c74cc46-19a2-4b10-bcc8-d24fd72d7dae-model-cache\") pod \"7c74cc46-19a2-4b10-bcc8-d24fd72d7dae\" (UID: \"7c74cc46-19a2-4b10-bcc8-d24fd72d7dae\") "
Sep 18 20:19:18.634150 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:18.634116    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c74cc46-19a2-4b10-bcc8-d24fd72d7dae-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7c74cc46-19a2-4b10-bcc8-d24fd72d7dae" (UID: "7c74cc46-19a2-4b10-bcc8-d24fd72d7dae"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:19:18.634453 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:18.634273    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c74cc46-19a2-4b10-bcc8-d24fd72d7dae-model-cache" (OuterVolumeSpecName: "model-cache") pod "7c74cc46-19a2-4b10-bcc8-d24fd72d7dae" (UID: "7c74cc46-19a2-4b10-bcc8-d24fd72d7dae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:19:18.634576 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:18.634500    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c74cc46-19a2-4b10-bcc8-d24fd72d7dae-home" (OuterVolumeSpecName: "home") pod "7c74cc46-19a2-4b10-bcc8-d24fd72d7dae" (UID: "7c74cc46-19a2-4b10-bcc8-d24fd72d7dae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:19:18.636118 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:18.636081    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c74cc46-19a2-4b10-bcc8-d24fd72d7dae-kube-api-access-nqxgx" (OuterVolumeSpecName: "kube-api-access-nqxgx") pod "7c74cc46-19a2-4b10-bcc8-d24fd72d7dae" (UID: "7c74cc46-19a2-4b10-bcc8-d24fd72d7dae"). InnerVolumeSpecName "kube-api-access-nqxgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:19:18.636223 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:18.636184    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c74cc46-19a2-4b10-bcc8-d24fd72d7dae-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7c74cc46-19a2-4b10-bcc8-d24fd72d7dae" (UID: "7c74cc46-19a2-4b10-bcc8-d24fd72d7dae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:19:18.636278 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:18.636244    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c74cc46-19a2-4b10-bcc8-d24fd72d7dae-dshm" (OuterVolumeSpecName: "dshm") pod "7c74cc46-19a2-4b10-bcc8-d24fd72d7dae" (UID: "7c74cc46-19a2-4b10-bcc8-d24fd72d7dae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:19:18.647073 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:18.647047    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c74cc46-19a2-4b10-bcc8-d24fd72d7dae-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7c74cc46-19a2-4b10-bcc8-d24fd72d7dae" (UID: "7c74cc46-19a2-4b10-bcc8-d24fd72d7dae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:19:18.658647 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:18.658624    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c74cc46-19a2-4b10-bcc8-d24fd72d7dae-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7c74cc46-19a2-4b10-bcc8-d24fd72d7dae" (UID: "7c74cc46-19a2-4b10-bcc8-d24fd72d7dae"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:19:18.734813 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:18.734775    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqxgx\" (UniqueName: \"kubernetes.io/projected/7c74cc46-19a2-4b10-bcc8-d24fd72d7dae-kube-api-access-nqxgx\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:19:18.734813 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:18.734803    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c74cc46-19a2-4b10-bcc8-d24fd72d7dae-cabundle-cert\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:19:18.734813 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:18.734814    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c74cc46-19a2-4b10-bcc8-d24fd72d7dae-home\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:19:18.735061 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:18.734822    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c74cc46-19a2-4b10-bcc8-d24fd72d7dae-model-cache\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:19:18.735061 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:18.734836    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c74cc46-19a2-4b10-bcc8-d24fd72d7dae-kserve-provision-location\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:19:18.735061 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:18.734844    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c74cc46-19a2-4b10-bcc8-d24fd72d7dae-tmp-dir\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:19:18.735061 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:18.734855    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c74cc46-19a2-4b10-bcc8-d24fd72d7dae-tls-certs\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:19:18.735061 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:18.734863    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c74cc46-19a2-4b10-bcc8-d24fd72d7dae-dshm\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:19:19.026222 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:19.026136    2577 generic.go:356] "Generic (PLEG): container finished" podID="7c74cc46-19a2-4b10-bcc8-d24fd72d7dae" containerID="e3b381d9977f8600e6a9060238a0b3597624f9f964d596ecdd09eb5910714927" exitCode=0
Sep 18 20:19:19.026222 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:19.026201    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r"
Sep 18 20:19:19.026796 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:19.026204    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r" event={"ID":"7c74cc46-19a2-4b10-bcc8-d24fd72d7dae","Type":"ContainerDied","Data":"e3b381d9977f8600e6a9060238a0b3597624f9f964d596ecdd09eb5910714927"}
Sep 18 20:19:19.026796 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:19.026341    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r" event={"ID":"7c74cc46-19a2-4b10-bcc8-d24fd72d7dae","Type":"ContainerDied","Data":"0d75ea43d00925f867b1012b750c7ec4d29a4b4f1634b621964cdff2ff5fba91"}
Sep 18 20:19:19.026796 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:19.026363    2577 scope.go:117] "RemoveContainer" containerID="e3b381d9977f8600e6a9060238a0b3597624f9f964d596ecdd09eb5910714927"
Sep 18 20:19:19.035607 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:19.035586    2577 scope.go:117] "RemoveContainer" containerID="ab525e3b20110b5230da5f8e389b4a1abf5b0c63d69a76261fda5dc00b1d38da"
Sep 18 20:19:19.042924 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:19.042906    2577 scope.go:117] "RemoveContainer" containerID="e3b381d9977f8600e6a9060238a0b3597624f9f964d596ecdd09eb5910714927"
Sep 18 20:19:19.043186 ip-10-0-129-30 kubenswrapper[2577]: E0918 20:19:19.043166    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3b381d9977f8600e6a9060238a0b3597624f9f964d596ecdd09eb5910714927\": container with ID starting with e3b381d9977f8600e6a9060238a0b3597624f9f964d596ecdd09eb5910714927 not found: ID does not exist" containerID="e3b381d9977f8600e6a9060238a0b3597624f9f964d596ecdd09eb5910714927"
Sep 18 20:19:19.043242 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:19.043195    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3b381d9977f8600e6a9060238a0b3597624f9f964d596ecdd09eb5910714927"} err="failed to get container status \"e3b381d9977f8600e6a9060238a0b3597624f9f964d596ecdd09eb5910714927\": rpc error: code = NotFound desc = could not find container \"e3b381d9977f8600e6a9060238a0b3597624f9f964d596ecdd09eb5910714927\": container with ID starting with e3b381d9977f8600e6a9060238a0b3597624f9f964d596ecdd09eb5910714927 not found: ID does not exist"
Sep 18 20:19:19.043242 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:19.043213    2577 scope.go:117] "RemoveContainer" containerID="ab525e3b20110b5230da5f8e389b4a1abf5b0c63d69a76261fda5dc00b1d38da"
Sep 18 20:19:19.043442 ip-10-0-129-30 kubenswrapper[2577]: E0918 20:19:19.043422    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab525e3b20110b5230da5f8e389b4a1abf5b0c63d69a76261fda5dc00b1d38da\": container with ID starting with ab525e3b20110b5230da5f8e389b4a1abf5b0c63d69a76261fda5dc00b1d38da not found: ID does not exist" containerID="ab525e3b20110b5230da5f8e389b4a1abf5b0c63d69a76261fda5dc00b1d38da"
Sep 18 20:19:19.043494 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:19.043452    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab525e3b20110b5230da5f8e389b4a1abf5b0c63d69a76261fda5dc00b1d38da"} err="failed to get container status \"ab525e3b20110b5230da5f8e389b4a1abf5b0c63d69a76261fda5dc00b1d38da\": rpc error: code = NotFound desc = could not find container \"ab525e3b20110b5230da5f8e389b4a1abf5b0c63d69a76261fda5dc00b1d38da\": container with ID starting with ab525e3b20110b5230da5f8e389b4a1abf5b0c63d69a76261fda5dc00b1d38da not found: ID does not exist"
Sep 18 20:19:19.051682 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:19.051660    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r"]
Sep 18 20:19:19.056514 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:19.056491    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-96b5dbc44-scl9r"]
Sep 18 20:19:19.438231 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:19.438197    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c74cc46-19a2-4b10-bcc8-d24fd72d7dae" path="/var/lib/kubelet/pods/7c74cc46-19a2-4b10-bcc8-d24fd72d7dae/volumes"
Sep 18 20:19:35.302513 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:35.302428    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 18 20:19:35.303095 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:35.302883    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c74cc46-19a2-4b10-bcc8-d24fd72d7dae" containerName="main"
Sep 18 20:19:35.303095 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:35.302901    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c74cc46-19a2-4b10-bcc8-d24fd72d7dae" containerName="main"
Sep 18 20:19:35.303095 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:35.302928    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c74cc46-19a2-4b10-bcc8-d24fd72d7dae" containerName="storage-initializer"
Sep 18 20:19:35.303095 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:35.302937    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c74cc46-19a2-4b10-bcc8-d24fd72d7dae" containerName="storage-initializer"
Sep 18 20:19:35.303095 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:35.303057    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c74cc46-19a2-4b10-bcc8-d24fd72d7dae" containerName="main"
Sep 18 20:19:35.306070 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:35.306045    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 20:19:35.309082 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:35.309052    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 18 20:19:35.310180 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:35.310154    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:19:35.310466 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:35.310322    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-j5f7g\""
Sep 18 20:19:35.310466 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:35.310338    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 18 20:19:35.310466 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:35.310329    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 18 20:19:35.310466 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:35.310442    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-tklqs\""
Sep 18 20:19:35.310741 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:35.310465    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 18 20:19:35.341855 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:35.341832    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 18 20:19:35.376428 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:35.376402    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e1a2759-c819-4ec9-8a5f-ee73436fe01b-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"7e1a2759-c819-4ec9-8a5f-ee73436fe01b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 20:19:35.376543 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:35.376444    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e1a2759-c819-4ec9-8a5f-ee73436fe01b-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"7e1a2759-c819-4ec9-8a5f-ee73436fe01b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 20:19:35.376543 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:35.376464    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e1a2759-c819-4ec9-8a5f-ee73436fe01b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"7e1a2759-c819-4ec9-8a5f-ee73436fe01b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 20:19:35.376624 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:35.376538    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e1a2759-c819-4ec9-8a5f-ee73436fe01b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"7e1a2759-c819-4ec9-8a5f-ee73436fe01b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 20:19:35.376624 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:35.376606    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e1a2759-c819-4ec9-8a5f-ee73436fe01b-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"7e1a2759-c819-4ec9-8a5f-ee73436fe01b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 20:19:35.376691 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:35.376640    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e1a2759-c819-4ec9-8a5f-ee73436fe01b-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"7e1a2759-c819-4ec9-8a5f-ee73436fe01b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 20:19:35.376735 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:35.376714    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e1a2759-c819-4ec9-8a5f-ee73436fe01b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"7e1a2759-c819-4ec9-8a5f-ee73436fe01b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 20:19:35.376772 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:35.376734    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5mzd\" (UniqueName: \"kubernetes.io/projected/7e1a2759-c819-4ec9-8a5f-ee73436fe01b-kube-api-access-g5mzd\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"7e1a2759-c819-4ec9-8a5f-ee73436fe01b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 20:19:35.477685 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:35.477647    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e1a2759-c819-4ec9-8a5f-ee73436fe01b-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"7e1a2759-c819-4ec9-8a5f-ee73436fe01b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 20:19:35.477685 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:35.477687    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e1a2759-c819-4ec9-8a5f-ee73436fe01b-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"7e1a2759-c819-4ec9-8a5f-ee73436fe01b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 20:19:35.477939 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:35.477780    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e1a2759-c819-4ec9-8a5f-ee73436fe01b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"7e1a2759-c819-4ec9-8a5f-ee73436fe01b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 20:19:35.477939 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:35.477808    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5mzd\" (UniqueName: \"kubernetes.io/projected/7e1a2759-c819-4ec9-8a5f-ee73436fe01b-kube-api-access-g5mzd\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"7e1a2759-c819-4ec9-8a5f-ee73436fe01b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 20:19:35.477939 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:35.477856    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e1a2759-c819-4ec9-8a5f-ee73436fe01b-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"7e1a2759-c819-4ec9-8a5f-ee73436fe01b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 20:19:35.478150 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:35.478124    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e1a2759-c819-4ec9-8a5f-ee73436fe01b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"7e1a2759-c819-4ec9-8a5f-ee73436fe01b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 20:19:35.478215 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:35.478149    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e1a2759-c819-4ec9-8a5f-ee73436fe01b-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"7e1a2759-c819-4ec9-8a5f-ee73436fe01b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 20:19:35.478215 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:35.478198    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e1a2759-c819-4ec9-8a5f-ee73436fe01b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"7e1a2759-c819-4ec9-8a5f-ee73436fe01b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 20:19:35.478325 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:35.478214    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e1a2759-c819-4ec9-8a5f-ee73436fe01b-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"7e1a2759-c819-4ec9-8a5f-ee73436fe01b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 20:19:35.478325 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:35.478230    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e1a2759-c819-4ec9-8a5f-ee73436fe01b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"7e1a2759-c819-4ec9-8a5f-ee73436fe01b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 20:19:35.478434 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:35.478382    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e1a2759-c819-4ec9-8a5f-ee73436fe01b-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"7e1a2759-c819-4ec9-8a5f-ee73436fe01b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 20:19:35.478491 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:35.478443    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e1a2759-c819-4ec9-8a5f-ee73436fe01b-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"7e1a2759-c819-4ec9-8a5f-ee73436fe01b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 20:19:35.478491 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:35.478450    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e1a2759-c819-4ec9-8a5f-ee73436fe01b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"7e1a2759-c819-4ec9-8a5f-ee73436fe01b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 20:19:35.480048 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:35.480025    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e1a2759-c819-4ec9-8a5f-ee73436fe01b-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"7e1a2759-c819-4ec9-8a5f-ee73436fe01b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 20:19:35.480517 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:35.480497    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e1a2759-c819-4ec9-8a5f-ee73436fe01b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"7e1a2759-c819-4ec9-8a5f-ee73436fe01b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 20:19:35.486489 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:35.486468    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5mzd\" (UniqueName: \"kubernetes.io/projected/7e1a2759-c819-4ec9-8a5f-ee73436fe01b-kube-api-access-g5mzd\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"7e1a2759-c819-4ec9-8a5f-ee73436fe01b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 20:19:35.616920 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:35.616886    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 20:19:35.898201 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:35.898083    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 18 20:19:35.900608 ip-10-0-129-30 kubenswrapper[2577]: W0918 20:19:35.900579    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e1a2759_c819_4ec9_8a5f_ee73436fe01b.slice/crio-4e610af923753e47c8205f59c356c7384651f1b64819830dc496191ce9875ce0 WatchSource:0}: Error finding container 4e610af923753e47c8205f59c356c7384651f1b64819830dc496191ce9875ce0: Status 404 returned error can't find the container with id 4e610af923753e47c8205f59c356c7384651f1b64819830dc496191ce9875ce0
Sep 18 20:19:36.106830 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:36.106787    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"7e1a2759-c819-4ec9-8a5f-ee73436fe01b","Type":"ContainerStarted","Data":"6465cbb141d41650620b9a2fac6c5d99172ffbdfb664ac5beb982ba30bef22d9"}
Sep 18 20:19:36.107061 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:36.106860    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"7e1a2759-c819-4ec9-8a5f-ee73436fe01b","Type":"ContainerStarted","Data":"4e610af923753e47c8205f59c356c7384651f1b64819830dc496191ce9875ce0"}
Sep 18 20:19:38.121612 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:38.121574    2577 generic.go:356] "Generic (PLEG): container finished" podID="7e1a2759-c819-4ec9-8a5f-ee73436fe01b" containerID="6465cbb141d41650620b9a2fac6c5d99172ffbdfb664ac5beb982ba30bef22d9" exitCode=0
Sep 18 20:19:38.122056 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:38.121641    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"7e1a2759-c819-4ec9-8a5f-ee73436fe01b","Type":"ContainerDied","Data":"6465cbb141d41650620b9a2fac6c5d99172ffbdfb664ac5beb982ba30bef22d9"}
Sep 18 20:19:39.129347 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:39.129312    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"7e1a2759-c819-4ec9-8a5f-ee73436fe01b","Type":"ContainerStarted","Data":"038e0fc1bbfc1096099e781c5028944a5ab5e3da146b89fce1bc9ab39183cbfa"}
Sep 18 20:19:39.154134 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:39.154069    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=2.015840773 podStartE2EDuration="4.154048795s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.138208022s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:19:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:19:39.151375857 +0000 UTC m=+3950.003676229" watchObservedRunningTime="2026-09-18 20:19:39.154048795 +0000 UTC m=+3950.006349166"
Sep 18 20:19:39.617478 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:39.617442    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 20:19:45.617395 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:45.617354    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 20:19:45.618940 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:45.618913    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="7e1a2759-c819-4ec9-8a5f-ee73436fe01b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Sep 18 20:19:55.617470 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:19:55.617425    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="7e1a2759-c819-4ec9-8a5f-ee73436fe01b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Sep 18 20:20:05.617924 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:20:05.617882    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="7e1a2759-c819-4ec9-8a5f-ee73436fe01b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Sep 18 20:20:15.617927 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:20:15.617879    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="7e1a2759-c819-4ec9-8a5f-ee73436fe01b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Sep 18 20:20:25.618202 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:20:25.618158    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="7e1a2759-c819-4ec9-8a5f-ee73436fe01b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Sep 18 20:20:35.618180 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:20:35.618127    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="7e1a2759-c819-4ec9-8a5f-ee73436fe01b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Sep 18 20:20:45.618416 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:20:45.618361    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="7e1a2759-c819-4ec9-8a5f-ee73436fe01b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Sep 18 20:20:55.617364 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:20:55.617316    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="7e1a2759-c819-4ec9-8a5f-ee73436fe01b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Sep 18 20:21:05.618197 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:21:05.618092    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="7e1a2759-c819-4ec9-8a5f-ee73436fe01b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Sep 18 20:21:15.628114 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:21:15.628079    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 20:21:15.635811 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:21:15.635786    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 20:21:21.370339 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:21:21.370305    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 18 20:21:36.392258 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:21:36.392213    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="7e1a2759-c819-4ec9-8a5f-ee73436fe01b" containerName="main" containerID="cri-o://038e0fc1bbfc1096099e781c5028944a5ab5e3da146b89fce1bc9ab39183cbfa" gracePeriod=45
Sep 18 20:21:36.628778 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:21:36.628734    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="7e1a2759-c819-4ec9-8a5f-ee73436fe01b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Sep 18 20:21:37.145410 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:21:37.145382    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 20:21:37.183367 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:21:37.183283    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e1a2759-c819-4ec9-8a5f-ee73436fe01b-cabundle-cert\") pod \"7e1a2759-c819-4ec9-8a5f-ee73436fe01b\" (UID: \"7e1a2759-c819-4ec9-8a5f-ee73436fe01b\") "
Sep 18 20:21:37.183367 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:21:37.183320    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e1a2759-c819-4ec9-8a5f-ee73436fe01b-model-cache\") pod \"7e1a2759-c819-4ec9-8a5f-ee73436fe01b\" (UID: \"7e1a2759-c819-4ec9-8a5f-ee73436fe01b\") "
Sep 18 20:21:37.183367 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:21:37.183354    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e1a2759-c819-4ec9-8a5f-ee73436fe01b-kserve-provision-location\") pod \"7e1a2759-c819-4ec9-8a5f-ee73436fe01b\" (UID: \"7e1a2759-c819-4ec9-8a5f-ee73436fe01b\") "
Sep 18 20:21:37.183671 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:21:37.183429    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e1a2759-c819-4ec9-8a5f-ee73436fe01b-tls-certs\") pod \"7e1a2759-c819-4ec9-8a5f-ee73436fe01b\" (UID: \"7e1a2759-c819-4ec9-8a5f-ee73436fe01b\") "
Sep 18 20:21:37.183671 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:21:37.183484    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e1a2759-c819-4ec9-8a5f-ee73436fe01b-tmp-dir\") pod \"7e1a2759-c819-4ec9-8a5f-ee73436fe01b\" (UID: \"7e1a2759-c819-4ec9-8a5f-ee73436fe01b\") "
Sep 18 20:21:37.183671 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:21:37.183505    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e1a2759-c819-4ec9-8a5f-ee73436fe01b-home\") pod \"7e1a2759-c819-4ec9-8a5f-ee73436fe01b\" (UID: \"7e1a2759-c819-4ec9-8a5f-ee73436fe01b\") "
Sep 18 20:21:37.183671 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:21:37.183527    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g5mzd\" (UniqueName: \"kubernetes.io/projected/7e1a2759-c819-4ec9-8a5f-ee73436fe01b-kube-api-access-g5mzd\") pod \"7e1a2759-c819-4ec9-8a5f-ee73436fe01b\" (UID: \"7e1a2759-c819-4ec9-8a5f-ee73436fe01b\") "
Sep 18 20:21:37.183671 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:21:37.183578    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e1a2759-c819-4ec9-8a5f-ee73436fe01b-dshm\") pod \"7e1a2759-c819-4ec9-8a5f-ee73436fe01b\" (UID: \"7e1a2759-c819-4ec9-8a5f-ee73436fe01b\") "
Sep 18 20:21:37.183671 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:21:37.183635    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e1a2759-c819-4ec9-8a5f-ee73436fe01b-model-cache" (OuterVolumeSpecName: "model-cache") pod "7e1a2759-c819-4ec9-8a5f-ee73436fe01b" (UID: "7e1a2759-c819-4ec9-8a5f-ee73436fe01b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:21:37.184016 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:21:37.183727    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e1a2759-c819-4ec9-8a5f-ee73436fe01b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7e1a2759-c819-4ec9-8a5f-ee73436fe01b" (UID: "7e1a2759-c819-4ec9-8a5f-ee73436fe01b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:21:37.184016 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:21:37.183980    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e1a2759-c819-4ec9-8a5f-ee73436fe01b-cabundle-cert\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:21:37.184136 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:21:37.184022    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e1a2759-c819-4ec9-8a5f-ee73436fe01b-model-cache\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:21:37.184281 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:21:37.184232    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e1a2759-c819-4ec9-8a5f-ee73436fe01b-home" (OuterVolumeSpecName: "home") pod "7e1a2759-c819-4ec9-8a5f-ee73436fe01b" (UID: "7e1a2759-c819-4ec9-8a5f-ee73436fe01b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:21:37.186512 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:21:37.186459    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e1a2759-c819-4ec9-8a5f-ee73436fe01b-dshm" (OuterVolumeSpecName: "dshm") pod "7e1a2759-c819-4ec9-8a5f-ee73436fe01b" (UID: "7e1a2759-c819-4ec9-8a5f-ee73436fe01b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:21:37.186620 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:21:37.186591    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e1a2759-c819-4ec9-8a5f-ee73436fe01b-kube-api-access-g5mzd" (OuterVolumeSpecName: "kube-api-access-g5mzd") pod "7e1a2759-c819-4ec9-8a5f-ee73436fe01b" (UID: "7e1a2759-c819-4ec9-8a5f-ee73436fe01b"). InnerVolumeSpecName "kube-api-access-g5mzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:21:37.186694 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:21:37.186677    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e1a2759-c819-4ec9-8a5f-ee73436fe01b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7e1a2759-c819-4ec9-8a5f-ee73436fe01b" (UID: "7e1a2759-c819-4ec9-8a5f-ee73436fe01b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:21:37.196749 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:21:37.196720    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e1a2759-c819-4ec9-8a5f-ee73436fe01b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7e1a2759-c819-4ec9-8a5f-ee73436fe01b" (UID: "7e1a2759-c819-4ec9-8a5f-ee73436fe01b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:21:37.209127 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:21:37.209095    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e1a2759-c819-4ec9-8a5f-ee73436fe01b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7e1a2759-c819-4ec9-8a5f-ee73436fe01b" (UID: "7e1a2759-c819-4ec9-8a5f-ee73436fe01b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:21:37.285461 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:21:37.285422    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e1a2759-c819-4ec9-8a5f-ee73436fe01b-tls-certs\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:21:37.285461 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:21:37.285455    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e1a2759-c819-4ec9-8a5f-ee73436fe01b-tmp-dir\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:21:37.285461 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:21:37.285463    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e1a2759-c819-4ec9-8a5f-ee73436fe01b-home\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:21:37.285712 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:21:37.285474    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5mzd\" (UniqueName: \"kubernetes.io/projected/7e1a2759-c819-4ec9-8a5f-ee73436fe01b-kube-api-access-g5mzd\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:21:37.285712 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:21:37.285483    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e1a2759-c819-4ec9-8a5f-ee73436fe01b-dshm\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:21:37.285712 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:21:37.285491    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e1a2759-c819-4ec9-8a5f-ee73436fe01b-kserve-provision-location\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:21:37.691969 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:21:37.691932    2577 generic.go:356] "Generic (PLEG): container finished" podID="7e1a2759-c819-4ec9-8a5f-ee73436fe01b" containerID="038e0fc1bbfc1096099e781c5028944a5ab5e3da146b89fce1bc9ab39183cbfa" exitCode=0
Sep 18 20:21:37.692417 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:21:37.691999    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 20:21:37.692417 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:21:37.692005    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"7e1a2759-c819-4ec9-8a5f-ee73436fe01b","Type":"ContainerDied","Data":"038e0fc1bbfc1096099e781c5028944a5ab5e3da146b89fce1bc9ab39183cbfa"}
Sep 18 20:21:37.692417 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:21:37.692054    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"7e1a2759-c819-4ec9-8a5f-ee73436fe01b","Type":"ContainerDied","Data":"4e610af923753e47c8205f59c356c7384651f1b64819830dc496191ce9875ce0"}
Sep 18 20:21:37.692417 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:21:37.692076    2577 scope.go:117] "RemoveContainer" containerID="038e0fc1bbfc1096099e781c5028944a5ab5e3da146b89fce1bc9ab39183cbfa"
Sep 18 20:21:37.700486 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:21:37.700468    2577 scope.go:117] "RemoveContainer" containerID="6465cbb141d41650620b9a2fac6c5d99172ffbdfb664ac5beb982ba30bef22d9"
Sep 18 20:21:37.708531 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:21:37.708501    2577 scope.go:117] "RemoveContainer" containerID="038e0fc1bbfc1096099e781c5028944a5ab5e3da146b89fce1bc9ab39183cbfa"
Sep 18 20:21:37.708830 ip-10-0-129-30 kubenswrapper[2577]: E0918 20:21:37.708806    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"038e0fc1bbfc1096099e781c5028944a5ab5e3da146b89fce1bc9ab39183cbfa\": container with ID starting with 038e0fc1bbfc1096099e781c5028944a5ab5e3da146b89fce1bc9ab39183cbfa not found: ID does not exist" containerID="038e0fc1bbfc1096099e781c5028944a5ab5e3da146b89fce1bc9ab39183cbfa"
Sep 18 20:21:37.708885 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:21:37.708843    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"038e0fc1bbfc1096099e781c5028944a5ab5e3da146b89fce1bc9ab39183cbfa"} err="failed to get container status \"038e0fc1bbfc1096099e781c5028944a5ab5e3da146b89fce1bc9ab39183cbfa\": rpc error: code = NotFound desc = could not find container \"038e0fc1bbfc1096099e781c5028944a5ab5e3da146b89fce1bc9ab39183cbfa\": container with ID starting with 038e0fc1bbfc1096099e781c5028944a5ab5e3da146b89fce1bc9ab39183cbfa not found: ID does not exist"
Sep 18 20:21:37.708885 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:21:37.708871    2577 scope.go:117] "RemoveContainer" containerID="6465cbb141d41650620b9a2fac6c5d99172ffbdfb664ac5beb982ba30bef22d9"
Sep 18 20:21:37.709152 ip-10-0-129-30 kubenswrapper[2577]: E0918 20:21:37.709134    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6465cbb141d41650620b9a2fac6c5d99172ffbdfb664ac5beb982ba30bef22d9\": container with ID starting with 6465cbb141d41650620b9a2fac6c5d99172ffbdfb664ac5beb982ba30bef22d9 not found: ID does not exist" containerID="6465cbb141d41650620b9a2fac6c5d99172ffbdfb664ac5beb982ba30bef22d9"
Sep 18 20:21:37.709206 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:21:37.709159    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6465cbb141d41650620b9a2fac6c5d99172ffbdfb664ac5beb982ba30bef22d9"} err="failed to get container status \"6465cbb141d41650620b9a2fac6c5d99172ffbdfb664ac5beb982ba30bef22d9\": rpc error: code = NotFound desc = could not find container \"6465cbb141d41650620b9a2fac6c5d99172ffbdfb664ac5beb982ba30bef22d9\": container with ID starting with 6465cbb141d41650620b9a2fac6c5d99172ffbdfb664ac5beb982ba30bef22d9 not found: ID does not exist"
Sep 18 20:21:37.711797 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:21:37.711775    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 18 20:21:37.715536 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:21:37.715514    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 18 20:21:39.438762 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:21:39.438732    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e1a2759-c819-4ec9-8a5f-ee73436fe01b" path="/var/lib/kubelet/pods/7e1a2759-c819-4ec9-8a5f-ee73436fe01b/volumes"
Sep 18 20:22:17.407028 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:17.406977    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2d2ld"]
Sep 18 20:22:17.407555 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:17.407309    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e1a2759-c819-4ec9-8a5f-ee73436fe01b" containerName="storage-initializer"
Sep 18 20:22:17.407555 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:17.407320    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e1a2759-c819-4ec9-8a5f-ee73436fe01b" containerName="storage-initializer"
Sep 18 20:22:17.407555 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:17.407328    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e1a2759-c819-4ec9-8a5f-ee73436fe01b" containerName="main"
Sep 18 20:22:17.407555 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:17.407333    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e1a2759-c819-4ec9-8a5f-ee73436fe01b" containerName="main"
Sep 18 20:22:17.407555 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:17.407415    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e1a2759-c819-4ec9-8a5f-ee73436fe01b" containerName="main"
Sep 18 20:22:17.410329 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:17.410310    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2d2ld"
Sep 18 20:22:17.414183 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:17.414154    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Sep 18 20:22:17.414306 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:17.414198    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Sep 18 20:22:17.414306 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:17.414235    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Sep 18 20:22:17.414306 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:17.414160    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-82nbc\""
Sep 18 20:22:17.427979 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:17.427954    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2d2ld"]
Sep 18 20:22:17.440449 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:17.440423    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cee82511-d973-455f-8673-586b89baf347-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-2d2ld\" (UID: \"cee82511-d973-455f-8673-586b89baf347\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2d2ld"
Sep 18 20:22:17.440584 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:17.440462    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np8bw\" (UniqueName: \"kubernetes.io/projected/cee82511-d973-455f-8673-586b89baf347-kube-api-access-np8bw\") pod \"scheduler-inline-config-test-kserve-8655bc757-2d2ld\" (UID: \"cee82511-d973-455f-8673-586b89baf347\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2d2ld"
Sep 18 20:22:17.440584 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:17.440481    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cee82511-d973-455f-8673-586b89baf347-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-2d2ld\" (UID: \"cee82511-d973-455f-8673-586b89baf347\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2d2ld"
Sep 18 20:22:17.440715 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:17.440580    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cee82511-d973-455f-8673-586b89baf347-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-2d2ld\" (UID: \"cee82511-d973-455f-8673-586b89baf347\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2d2ld"
Sep 18 20:22:17.440715 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:17.440606    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cee82511-d973-455f-8673-586b89baf347-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-2d2ld\" (UID: \"cee82511-d973-455f-8673-586b89baf347\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2d2ld"
Sep 18 20:22:17.440715 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:17.440637    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cee82511-d973-455f-8673-586b89baf347-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-2d2ld\" (UID: \"cee82511-d973-455f-8673-586b89baf347\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2d2ld"
Sep 18 20:22:17.541524 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:17.541483    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cee82511-d973-455f-8673-586b89baf347-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-2d2ld\" (UID: \"cee82511-d973-455f-8673-586b89baf347\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2d2ld"
Sep 18 20:22:17.541524 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:17.541522    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cee82511-d973-455f-8673-586b89baf347-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-2d2ld\" (UID: \"cee82511-d973-455f-8673-586b89baf347\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2d2ld"
Sep 18 20:22:17.541775 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:17.541567    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cee82511-d973-455f-8673-586b89baf347-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-2d2ld\" (UID: \"cee82511-d973-455f-8673-586b89baf347\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2d2ld"
Sep 18 20:22:17.541775 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:17.541631    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cee82511-d973-455f-8673-586b89baf347-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-2d2ld\" (UID: \"cee82511-d973-455f-8673-586b89baf347\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2d2ld"
Sep 18 20:22:17.541775 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:17.541677    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np8bw\" (UniqueName: \"kubernetes.io/projected/cee82511-d973-455f-8673-586b89baf347-kube-api-access-np8bw\") pod \"scheduler-inline-config-test-kserve-8655bc757-2d2ld\" (UID: \"cee82511-d973-455f-8673-586b89baf347\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2d2ld"
Sep 18 20:22:17.541775 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:17.541706    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cee82511-d973-455f-8673-586b89baf347-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-2d2ld\" (UID: \"cee82511-d973-455f-8673-586b89baf347\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2d2ld"
Sep 18 20:22:17.542015 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:17.541973    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cee82511-d973-455f-8673-586b89baf347-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-2d2ld\" (UID: \"cee82511-d973-455f-8673-586b89baf347\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2d2ld"
Sep 18 20:22:17.542213 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:17.542058    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cee82511-d973-455f-8673-586b89baf347-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-2d2ld\" (UID: \"cee82511-d973-455f-8673-586b89baf347\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2d2ld"
Sep 18 20:22:17.542287 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:17.542097    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cee82511-d973-455f-8673-586b89baf347-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-2d2ld\" (UID: \"cee82511-d973-455f-8673-586b89baf347\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2d2ld"
Sep 18 20:22:17.543942 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:17.543920    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cee82511-d973-455f-8673-586b89baf347-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-2d2ld\" (UID: \"cee82511-d973-455f-8673-586b89baf347\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2d2ld"
Sep 18 20:22:17.544155 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:17.544138    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cee82511-d973-455f-8673-586b89baf347-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-2d2ld\" (UID: \"cee82511-d973-455f-8673-586b89baf347\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2d2ld"
Sep 18 20:22:17.550312 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:17.550289    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-np8bw\" (UniqueName: \"kubernetes.io/projected/cee82511-d973-455f-8673-586b89baf347-kube-api-access-np8bw\") pod \"scheduler-inline-config-test-kserve-8655bc757-2d2ld\" (UID: \"cee82511-d973-455f-8673-586b89baf347\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2d2ld"
Sep 18 20:22:17.721491 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:17.721385    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2d2ld"
Sep 18 20:22:17.995735 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:17.995637    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2d2ld"]
Sep 18 20:22:17.998460 ip-10-0-129-30 kubenswrapper[2577]: W0918 20:22:17.998430    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcee82511_d973_455f_8673_586b89baf347.slice/crio-f292fd7b217ca13a149c8aa8969341c346208a990b5b503ca3d1881c64375079 WatchSource:0}: Error finding container f292fd7b217ca13a149c8aa8969341c346208a990b5b503ca3d1881c64375079: Status 404 returned error can't find the container with id f292fd7b217ca13a149c8aa8969341c346208a990b5b503ca3d1881c64375079
Sep 18 20:22:18.864470 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:18.864433    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2d2ld" event={"ID":"cee82511-d973-455f-8673-586b89baf347","Type":"ContainerStarted","Data":"ed37cd1232876fec638a5fbfabc2e1b878d54d039995a9fa14ef8e48bfe42462"}
Sep 18 20:22:18.864889 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:18.864484    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2d2ld" event={"ID":"cee82511-d973-455f-8673-586b89baf347","Type":"ContainerStarted","Data":"f292fd7b217ca13a149c8aa8969341c346208a990b5b503ca3d1881c64375079"}
Sep 18 20:22:18.906108 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:18.906059    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2d2ld" podStartSLOduration=1.9060441479999999 podStartE2EDuration="1.906044148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:22:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:22:18.905815417 +0000 UTC m=+4109.758115787" watchObservedRunningTime="2026-09-18 20:22:18.906044148 +0000 UTC m=+4109.758344518"
Sep 18 20:22:19.721660 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:19.721625    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2d2ld"
Sep 18 20:22:27.722567 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:27.722535    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2d2ld"
Sep 18 20:22:27.735063 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:27.735033    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2d2ld"
Sep 18 20:22:27.914611 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:27.914580    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2d2ld"
Sep 18 20:22:51.334322 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:51.334285    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2d2ld"]
Sep 18 20:22:51.343243 ip-10-0-129-30 kubenswrapper[2577]: E0918 20:22:51.343204    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 20:22:51.343243 ip-10-0-129-30 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:22:51.343243 ip-10-0-129-30 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 18 20:22:51.343243 ip-10-0-129-30 kubenswrapper[2577]:  > containerID="ed37cd1232876fec638a5fbfabc2e1b878d54d039995a9fa14ef8e48bfe42462" cmd=["/bin/sleep","15"]
Sep 18 20:22:51.343481 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:51.343261    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2d2ld" message="" err=<
Sep 18 20:22:51.343481 ip-10-0-129-30 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:22:51.343481 ip-10-0-129-30 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 18 20:22:51.343481 ip-10-0-129-30 kubenswrapper[2577]:  >
Sep 18 20:22:51.343481 ip-10-0-129-30 kubenswrapper[2577]: E0918 20:22:51.343273    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 20:22:51.343481 ip-10-0-129-30 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:22:51.343481 ip-10-0-129-30 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 18 20:22:51.343481 ip-10-0-129-30 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2d2ld" podUID="cee82511-d973-455f-8673-586b89baf347" containerName="main" containerID="cri-o://ed37cd1232876fec638a5fbfabc2e1b878d54d039995a9fa14ef8e48bfe42462"
Sep 18 20:22:51.343481 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:51.343317    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2d2ld" podUID="cee82511-d973-455f-8673-586b89baf347" containerName="main" containerID="cri-o://ed37cd1232876fec638a5fbfabc2e1b878d54d039995a9fa14ef8e48bfe42462" gracePeriod=60
Sep 18 20:22:51.903726 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:51.903678    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2d2ld" podUID="cee82511-d973-455f-8673-586b89baf347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Sep 18 20:22:52.011974 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:52.011941    2577 generic.go:356] "Generic (PLEG): container finished" podID="cee82511-d973-455f-8673-586b89baf347" containerID="ed37cd1232876fec638a5fbfabc2e1b878d54d039995a9fa14ef8e48bfe42462" exitCode=0
Sep 18 20:22:52.012159 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:52.012026    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2d2ld" event={"ID":"cee82511-d973-455f-8673-586b89baf347","Type":"ContainerDied","Data":"ed37cd1232876fec638a5fbfabc2e1b878d54d039995a9fa14ef8e48bfe42462"}
Sep 18 20:22:52.078024 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:52.077980    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2d2ld"
Sep 18 20:22:52.250267 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:52.250179    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cee82511-d973-455f-8673-586b89baf347-tls-certs\") pod \"cee82511-d973-455f-8673-586b89baf347\" (UID: \"cee82511-d973-455f-8673-586b89baf347\") "
Sep 18 20:22:52.250267 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:52.250233    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cee82511-d973-455f-8673-586b89baf347-tmp-dir\") pod \"cee82511-d973-455f-8673-586b89baf347\" (UID: \"cee82511-d973-455f-8673-586b89baf347\") "
Sep 18 20:22:52.250267 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:52.250253    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cee82511-d973-455f-8673-586b89baf347-model-cache\") pod \"cee82511-d973-455f-8673-586b89baf347\" (UID: \"cee82511-d973-455f-8673-586b89baf347\") "
Sep 18 20:22:52.250557 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:52.250296    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-np8bw\" (UniqueName: \"kubernetes.io/projected/cee82511-d973-455f-8673-586b89baf347-kube-api-access-np8bw\") pod \"cee82511-d973-455f-8673-586b89baf347\" (UID: \"cee82511-d973-455f-8673-586b89baf347\") "
Sep 18 20:22:52.250557 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:52.250336    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cee82511-d973-455f-8673-586b89baf347-dshm\") pod \"cee82511-d973-455f-8673-586b89baf347\" (UID: \"cee82511-d973-455f-8673-586b89baf347\") "
Sep 18 20:22:52.250557 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:52.250380    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cee82511-d973-455f-8673-586b89baf347-home\") pod \"cee82511-d973-455f-8673-586b89baf347\" (UID: \"cee82511-d973-455f-8673-586b89baf347\") "
Sep 18 20:22:52.250557 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:52.250514    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cee82511-d973-455f-8673-586b89baf347-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cee82511-d973-455f-8673-586b89baf347" (UID: "cee82511-d973-455f-8673-586b89baf347"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:22:52.250794 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:52.250582    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cee82511-d973-455f-8673-586b89baf347-model-cache" (OuterVolumeSpecName: "model-cache") pod "cee82511-d973-455f-8673-586b89baf347" (UID: "cee82511-d973-455f-8673-586b89baf347"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:22:52.250794 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:52.250707    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cee82511-d973-455f-8673-586b89baf347-home" (OuterVolumeSpecName: "home") pod "cee82511-d973-455f-8673-586b89baf347" (UID: "cee82511-d973-455f-8673-586b89baf347"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:22:52.250794 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:52.250766    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cee82511-d973-455f-8673-586b89baf347-tmp-dir\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:22:52.250794 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:52.250779    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cee82511-d973-455f-8673-586b89baf347-model-cache\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:22:52.252634 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:52.252607    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cee82511-d973-455f-8673-586b89baf347-dshm" (OuterVolumeSpecName: "dshm") pod "cee82511-d973-455f-8673-586b89baf347" (UID: "cee82511-d973-455f-8673-586b89baf347"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:22:52.252634 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:52.252620    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cee82511-d973-455f-8673-586b89baf347-kube-api-access-np8bw" (OuterVolumeSpecName: "kube-api-access-np8bw") pod "cee82511-d973-455f-8673-586b89baf347" (UID: "cee82511-d973-455f-8673-586b89baf347"). InnerVolumeSpecName "kube-api-access-np8bw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:22:52.252918 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:52.252898    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cee82511-d973-455f-8673-586b89baf347-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cee82511-d973-455f-8673-586b89baf347" (UID: "cee82511-d973-455f-8673-586b89baf347"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:22:52.351413 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:52.351382    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cee82511-d973-455f-8673-586b89baf347-dshm\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:22:52.351413 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:52.351407    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cee82511-d973-455f-8673-586b89baf347-home\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:22:52.351413 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:52.351416    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cee82511-d973-455f-8673-586b89baf347-tls-certs\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:22:52.351834 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:52.351424    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-np8bw\" (UniqueName: \"kubernetes.io/projected/cee82511-d973-455f-8673-586b89baf347-kube-api-access-np8bw\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:22:53.017891 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:53.017857    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2d2ld" event={"ID":"cee82511-d973-455f-8673-586b89baf347","Type":"ContainerDied","Data":"f292fd7b217ca13a149c8aa8969341c346208a990b5b503ca3d1881c64375079"}
Sep 18 20:22:53.017891 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:53.017887    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2d2ld"
Sep 18 20:22:53.018139 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:53.017917    2577 scope.go:117] "RemoveContainer" containerID="ed37cd1232876fec638a5fbfabc2e1b878d54d039995a9fa14ef8e48bfe42462"
Sep 18 20:22:53.043147 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:53.043120    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2d2ld"]
Sep 18 20:22:53.049716 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:53.049693    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2d2ld"]
Sep 18 20:22:53.438147 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:53.438117    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cee82511-d973-455f-8673-586b89baf347" path="/var/lib/kubelet/pods/cee82511-d973-455f-8673-586b89baf347/volumes"
Sep 18 20:22:55.949647 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:55.949608    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-z8bm8"]
Sep 18 20:22:55.950098 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:55.949924    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cee82511-d973-455f-8673-586b89baf347" containerName="main"
Sep 18 20:22:55.950098 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:55.949935    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cee82511-d973-455f-8673-586b89baf347" containerName="main"
Sep 18 20:22:55.950098 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:55.950015    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cee82511-d973-455f-8673-586b89baf347" containerName="main"
Sep 18 20:22:55.954277 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:55.954258    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-z8bm8"
Sep 18 20:22:55.957460 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:55.957432    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:22:55.957589 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:55.957459    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Sep 18 20:22:55.957589 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:55.957511    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 18 20:22:55.957589 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:55.957537    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 18 20:22:55.957589 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:55.957546    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-b9l6g\""
Sep 18 20:22:55.957589 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:55.957536    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 18 20:22:55.965625 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:55.965600    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-z8bm8"]
Sep 18 20:22:56.085032 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:56.084969    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r82s\" (UniqueName: \"kubernetes.io/projected/a7f8e100-3db8-4119-9aeb-4f1dd435d69d-kube-api-access-7r82s\") pod \"prestop-hook-test-kserve-7db57ffd58-z8bm8\" (UID: \"a7f8e100-3db8-4119-9aeb-4f1dd435d69d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-z8bm8"
Sep 18 20:22:56.085032 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:56.085035    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7f8e100-3db8-4119-9aeb-4f1dd435d69d-tmp-dir\") pod \"prestop-hook-test-kserve-7db57ffd58-z8bm8\" (UID: \"a7f8e100-3db8-4119-9aeb-4f1dd435d69d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-z8bm8"
Sep 18 20:22:56.085320 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:56.085146    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7f8e100-3db8-4119-9aeb-4f1dd435d69d-tls-certs\") pod \"prestop-hook-test-kserve-7db57ffd58-z8bm8\" (UID: \"a7f8e100-3db8-4119-9aeb-4f1dd435d69d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-z8bm8"
Sep 18 20:22:56.085320 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:56.085196    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7f8e100-3db8-4119-9aeb-4f1dd435d69d-kserve-provision-location\") pod \"prestop-hook-test-kserve-7db57ffd58-z8bm8\" (UID: \"a7f8e100-3db8-4119-9aeb-4f1dd435d69d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-z8bm8"
Sep 18 20:22:56.085320 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:56.085254    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7f8e100-3db8-4119-9aeb-4f1dd435d69d-dshm\") pod \"prestop-hook-test-kserve-7db57ffd58-z8bm8\" (UID: \"a7f8e100-3db8-4119-9aeb-4f1dd435d69d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-z8bm8"
Sep 18 20:22:56.085320 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:56.085283    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7f8e100-3db8-4119-9aeb-4f1dd435d69d-home\") pod \"prestop-hook-test-kserve-7db57ffd58-z8bm8\" (UID: \"a7f8e100-3db8-4119-9aeb-4f1dd435d69d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-z8bm8"
Sep 18 20:22:56.085320 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:56.085300    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7f8e100-3db8-4119-9aeb-4f1dd435d69d-model-cache\") pod \"prestop-hook-test-kserve-7db57ffd58-z8bm8\" (UID: \"a7f8e100-3db8-4119-9aeb-4f1dd435d69d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-z8bm8"
Sep 18 20:22:56.085320 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:56.085320    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7f8e100-3db8-4119-9aeb-4f1dd435d69d-cabundle-cert\") pod \"prestop-hook-test-kserve-7db57ffd58-z8bm8\" (UID: \"a7f8e100-3db8-4119-9aeb-4f1dd435d69d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-z8bm8"
Sep 18 20:22:56.185905 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:56.185871    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7f8e100-3db8-4119-9aeb-4f1dd435d69d-tmp-dir\") pod \"prestop-hook-test-kserve-7db57ffd58-z8bm8\" (UID: \"a7f8e100-3db8-4119-9aeb-4f1dd435d69d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-z8bm8"
Sep 18 20:22:56.186108 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:56.185922    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7f8e100-3db8-4119-9aeb-4f1dd435d69d-tls-certs\") pod \"prestop-hook-test-kserve-7db57ffd58-z8bm8\" (UID: \"a7f8e100-3db8-4119-9aeb-4f1dd435d69d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-z8bm8"
Sep 18 20:22:56.186108 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:56.185944    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7f8e100-3db8-4119-9aeb-4f1dd435d69d-kserve-provision-location\") pod \"prestop-hook-test-kserve-7db57ffd58-z8bm8\" (UID: \"a7f8e100-3db8-4119-9aeb-4f1dd435d69d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-z8bm8"
Sep 18 20:22:56.186240 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:56.186123    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7f8e100-3db8-4119-9aeb-4f1dd435d69d-dshm\") pod \"prestop-hook-test-kserve-7db57ffd58-z8bm8\" (UID: \"a7f8e100-3db8-4119-9aeb-4f1dd435d69d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-z8bm8"
Sep 18 20:22:56.186240 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:56.186171    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7f8e100-3db8-4119-9aeb-4f1dd435d69d-home\") pod \"prestop-hook-test-kserve-7db57ffd58-z8bm8\" (UID: \"a7f8e100-3db8-4119-9aeb-4f1dd435d69d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-z8bm8"
Sep 18 20:22:56.186344 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:56.186294    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7f8e100-3db8-4119-9aeb-4f1dd435d69d-model-cache\") pod \"prestop-hook-test-kserve-7db57ffd58-z8bm8\" (UID: \"a7f8e100-3db8-4119-9aeb-4f1dd435d69d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-z8bm8"
Sep 18 20:22:56.186344 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:56.186313    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7f8e100-3db8-4119-9aeb-4f1dd435d69d-kserve-provision-location\") pod \"prestop-hook-test-kserve-7db57ffd58-z8bm8\" (UID: \"a7f8e100-3db8-4119-9aeb-4f1dd435d69d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-z8bm8"
Sep 18 20:22:56.186344 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:56.186330    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7f8e100-3db8-4119-9aeb-4f1dd435d69d-cabundle-cert\") pod \"prestop-hook-test-kserve-7db57ffd58-z8bm8\" (UID: \"a7f8e100-3db8-4119-9aeb-4f1dd435d69d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-z8bm8"
Sep 18 20:22:56.186509 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:56.186365    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7f8e100-3db8-4119-9aeb-4f1dd435d69d-tmp-dir\") pod \"prestop-hook-test-kserve-7db57ffd58-z8bm8\" (UID: \"a7f8e100-3db8-4119-9aeb-4f1dd435d69d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-z8bm8"
Sep 18 20:22:56.186509 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:56.186444    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r82s\" (UniqueName: \"kubernetes.io/projected/a7f8e100-3db8-4119-9aeb-4f1dd435d69d-kube-api-access-7r82s\") pod \"prestop-hook-test-kserve-7db57ffd58-z8bm8\" (UID: \"a7f8e100-3db8-4119-9aeb-4f1dd435d69d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-z8bm8"
Sep 18 20:22:56.186509 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:56.186468    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7f8e100-3db8-4119-9aeb-4f1dd435d69d-home\") pod \"prestop-hook-test-kserve-7db57ffd58-z8bm8\" (UID: \"a7f8e100-3db8-4119-9aeb-4f1dd435d69d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-z8bm8"
Sep 18 20:22:56.186696 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:56.186673    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7f8e100-3db8-4119-9aeb-4f1dd435d69d-model-cache\") pod \"prestop-hook-test-kserve-7db57ffd58-z8bm8\" (UID: \"a7f8e100-3db8-4119-9aeb-4f1dd435d69d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-z8bm8"
Sep 18 20:22:56.187009 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:56.186965    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7f8e100-3db8-4119-9aeb-4f1dd435d69d-cabundle-cert\") pod \"prestop-hook-test-kserve-7db57ffd58-z8bm8\" (UID: \"a7f8e100-3db8-4119-9aeb-4f1dd435d69d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-z8bm8"
Sep 18 20:22:56.188426 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:56.188405    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7f8e100-3db8-4119-9aeb-4f1dd435d69d-dshm\") pod \"prestop-hook-test-kserve-7db57ffd58-z8bm8\" (UID: \"a7f8e100-3db8-4119-9aeb-4f1dd435d69d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-z8bm8"
Sep 18 20:22:56.188711 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:56.188680    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7f8e100-3db8-4119-9aeb-4f1dd435d69d-tls-certs\") pod \"prestop-hook-test-kserve-7db57ffd58-z8bm8\" (UID: \"a7f8e100-3db8-4119-9aeb-4f1dd435d69d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-z8bm8"
Sep 18 20:22:56.196429 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:56.196406    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r82s\" (UniqueName: \"kubernetes.io/projected/a7f8e100-3db8-4119-9aeb-4f1dd435d69d-kube-api-access-7r82s\") pod \"prestop-hook-test-kserve-7db57ffd58-z8bm8\" (UID: \"a7f8e100-3db8-4119-9aeb-4f1dd435d69d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-z8bm8"
Sep 18 20:22:56.265464 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:56.265381    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-z8bm8"
Sep 18 20:22:56.575904 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:56.575825    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-z8bm8"]
Sep 18 20:22:56.585636 ip-10-0-129-30 kubenswrapper[2577]: W0918 20:22:56.585601    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7f8e100_3db8_4119_9aeb_4f1dd435d69d.slice/crio-68bb12ed7409c9250b3a39346b8ae561af0dbb70b504653520b2872640b006f8 WatchSource:0}: Error finding container 68bb12ed7409c9250b3a39346b8ae561af0dbb70b504653520b2872640b006f8: Status 404 returned error can't find the container with id 68bb12ed7409c9250b3a39346b8ae561af0dbb70b504653520b2872640b006f8
Sep 18 20:22:57.037955 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:57.037916    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-z8bm8" event={"ID":"a7f8e100-3db8-4119-9aeb-4f1dd435d69d","Type":"ContainerStarted","Data":"7d6bd56ccc45a3bef7cbf37b0ccf643277b25cb07475a8e6f93b3a2a1a49aa9e"}
Sep 18 20:22:57.038521 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:57.037973    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-z8bm8" event={"ID":"a7f8e100-3db8-4119-9aeb-4f1dd435d69d","Type":"ContainerStarted","Data":"68bb12ed7409c9250b3a39346b8ae561af0dbb70b504653520b2872640b006f8"}
Sep 18 20:22:59.049551 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:59.049452    2577 generic.go:356] "Generic (PLEG): container finished" podID="a7f8e100-3db8-4119-9aeb-4f1dd435d69d" containerID="7d6bd56ccc45a3bef7cbf37b0ccf643277b25cb07475a8e6f93b3a2a1a49aa9e" exitCode=0
Sep 18 20:22:59.049551 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:22:59.049532    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-z8bm8" event={"ID":"a7f8e100-3db8-4119-9aeb-4f1dd435d69d","Type":"ContainerDied","Data":"7d6bd56ccc45a3bef7cbf37b0ccf643277b25cb07475a8e6f93b3a2a1a49aa9e"}
Sep 18 20:23:00.055865 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:23:00.055826    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-z8bm8" event={"ID":"a7f8e100-3db8-4119-9aeb-4f1dd435d69d","Type":"ContainerStarted","Data":"721292c9fb7de8ffa6a2a4667d74b439c2d65462567cde9b38e2c1c21f128ef4"}
Sep 18 20:23:00.084296 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:23:00.084244    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-z8bm8" podStartSLOduration=3.000102649 podStartE2EDuration="5.084228077s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.084125428s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:22:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:23:00.082326762 +0000 UTC m=+4150.934627133" watchObservedRunningTime="2026-09-18 20:23:00.084228077 +0000 UTC m=+4150.936528446"
Sep 18 20:23:00.265522 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:23:00.265487    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-z8bm8"
Sep 18 20:23:06.266091 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:23:06.266046    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-z8bm8"
Sep 18 20:23:06.267538 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:23:06.267508    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-z8bm8" podUID="a7f8e100-3db8-4119-9aeb-4f1dd435d69d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 18 20:23:15.224373 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:23:15.224334    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-z8bm8"]
Sep 18 20:23:30.248133 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:23:30.248093    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-z8bm8" podUID="a7f8e100-3db8-4119-9aeb-4f1dd435d69d" containerName="main" containerID="cri-o://721292c9fb7de8ffa6a2a4667d74b439c2d65462567cde9b38e2c1c21f128ef4" gracePeriod=45
Sep 18 20:24:15.424477 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:15.424443    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-7db57ffd58-z8bm8_a7f8e100-3db8-4119-9aeb-4f1dd435d69d/main/0.log"
Sep 18 20:24:15.424928 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:15.424799    2577 generic.go:356] "Generic (PLEG): container finished" podID="a7f8e100-3db8-4119-9aeb-4f1dd435d69d" containerID="721292c9fb7de8ffa6a2a4667d74b439c2d65462567cde9b38e2c1c21f128ef4" exitCode=137
Sep 18 20:24:15.424928 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:15.424851    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-z8bm8" event={"ID":"a7f8e100-3db8-4119-9aeb-4f1dd435d69d","Type":"ContainerDied","Data":"721292c9fb7de8ffa6a2a4667d74b439c2d65462567cde9b38e2c1c21f128ef4"}
Sep 18 20:24:15.882309 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:15.882282    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-7db57ffd58-z8bm8_a7f8e100-3db8-4119-9aeb-4f1dd435d69d/main/0.log"
Sep 18 20:24:15.885013 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:15.884981    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-z8bm8"
Sep 18 20:24:16.010514 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:16.010474    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7f8e100-3db8-4119-9aeb-4f1dd435d69d-kserve-provision-location\") pod \"a7f8e100-3db8-4119-9aeb-4f1dd435d69d\" (UID: \"a7f8e100-3db8-4119-9aeb-4f1dd435d69d\") "
Sep 18 20:24:16.010514 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:16.010514    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7f8e100-3db8-4119-9aeb-4f1dd435d69d-model-cache\") pod \"a7f8e100-3db8-4119-9aeb-4f1dd435d69d\" (UID: \"a7f8e100-3db8-4119-9aeb-4f1dd435d69d\") "
Sep 18 20:24:16.010790 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:16.010552    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7f8e100-3db8-4119-9aeb-4f1dd435d69d-tls-certs\") pod \"a7f8e100-3db8-4119-9aeb-4f1dd435d69d\" (UID: \"a7f8e100-3db8-4119-9aeb-4f1dd435d69d\") "
Sep 18 20:24:16.010790 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:16.010578    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7f8e100-3db8-4119-9aeb-4f1dd435d69d-dshm\") pod \"a7f8e100-3db8-4119-9aeb-4f1dd435d69d\" (UID: \"a7f8e100-3db8-4119-9aeb-4f1dd435d69d\") "
Sep 18 20:24:16.010790 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:16.010647    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7f8e100-3db8-4119-9aeb-4f1dd435d69d-cabundle-cert\") pod \"a7f8e100-3db8-4119-9aeb-4f1dd435d69d\" (UID: \"a7f8e100-3db8-4119-9aeb-4f1dd435d69d\") "
Sep 18 20:24:16.010790 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:16.010672    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7f8e100-3db8-4119-9aeb-4f1dd435d69d-tmp-dir\") pod \"a7f8e100-3db8-4119-9aeb-4f1dd435d69d\" (UID: \"a7f8e100-3db8-4119-9aeb-4f1dd435d69d\") "
Sep 18 20:24:16.010790 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:16.010696    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7r82s\" (UniqueName: \"kubernetes.io/projected/a7f8e100-3db8-4119-9aeb-4f1dd435d69d-kube-api-access-7r82s\") pod \"a7f8e100-3db8-4119-9aeb-4f1dd435d69d\" (UID: \"a7f8e100-3db8-4119-9aeb-4f1dd435d69d\") "
Sep 18 20:24:16.010790 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:16.010720    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7f8e100-3db8-4119-9aeb-4f1dd435d69d-home\") pod \"a7f8e100-3db8-4119-9aeb-4f1dd435d69d\" (UID: \"a7f8e100-3db8-4119-9aeb-4f1dd435d69d\") "
Sep 18 20:24:16.011140 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:16.010897    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7f8e100-3db8-4119-9aeb-4f1dd435d69d-model-cache" (OuterVolumeSpecName: "model-cache") pod "a7f8e100-3db8-4119-9aeb-4f1dd435d69d" (UID: "a7f8e100-3db8-4119-9aeb-4f1dd435d69d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:24:16.011140 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:16.011106    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7f8e100-3db8-4119-9aeb-4f1dd435d69d-model-cache\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:24:16.011226 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:16.011148    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7f8e100-3db8-4119-9aeb-4f1dd435d69d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a7f8e100-3db8-4119-9aeb-4f1dd435d69d" (UID: "a7f8e100-3db8-4119-9aeb-4f1dd435d69d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:24:16.011356 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:16.011270    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7f8e100-3db8-4119-9aeb-4f1dd435d69d-home" (OuterVolumeSpecName: "home") pod "a7f8e100-3db8-4119-9aeb-4f1dd435d69d" (UID: "a7f8e100-3db8-4119-9aeb-4f1dd435d69d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:24:16.011571 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:16.011544    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7f8e100-3db8-4119-9aeb-4f1dd435d69d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a7f8e100-3db8-4119-9aeb-4f1dd435d69d" (UID: "a7f8e100-3db8-4119-9aeb-4f1dd435d69d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:24:16.013002 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:16.012969    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7f8e100-3db8-4119-9aeb-4f1dd435d69d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a7f8e100-3db8-4119-9aeb-4f1dd435d69d" (UID: "a7f8e100-3db8-4119-9aeb-4f1dd435d69d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:24:16.013179 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:16.013152    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7f8e100-3db8-4119-9aeb-4f1dd435d69d-dshm" (OuterVolumeSpecName: "dshm") pod "a7f8e100-3db8-4119-9aeb-4f1dd435d69d" (UID: "a7f8e100-3db8-4119-9aeb-4f1dd435d69d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:24:16.013547 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:16.013501    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7f8e100-3db8-4119-9aeb-4f1dd435d69d-kube-api-access-7r82s" (OuterVolumeSpecName: "kube-api-access-7r82s") pod "a7f8e100-3db8-4119-9aeb-4f1dd435d69d" (UID: "a7f8e100-3db8-4119-9aeb-4f1dd435d69d"). InnerVolumeSpecName "kube-api-access-7r82s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:24:16.035783 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:16.035749    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7f8e100-3db8-4119-9aeb-4f1dd435d69d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a7f8e100-3db8-4119-9aeb-4f1dd435d69d" (UID: "a7f8e100-3db8-4119-9aeb-4f1dd435d69d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:24:16.112299 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:16.112265    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7f8e100-3db8-4119-9aeb-4f1dd435d69d-cabundle-cert\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:24:16.112299 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:16.112295    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7f8e100-3db8-4119-9aeb-4f1dd435d69d-tmp-dir\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:24:16.112299 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:16.112304    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r82s\" (UniqueName: \"kubernetes.io/projected/a7f8e100-3db8-4119-9aeb-4f1dd435d69d-kube-api-access-7r82s\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:24:16.112538 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:16.112314    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7f8e100-3db8-4119-9aeb-4f1dd435d69d-home\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:24:16.112538 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:16.112323    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7f8e100-3db8-4119-9aeb-4f1dd435d69d-kserve-provision-location\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:24:16.112538 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:16.112332    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7f8e100-3db8-4119-9aeb-4f1dd435d69d-tls-certs\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:24:16.112538 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:16.112340    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7f8e100-3db8-4119-9aeb-4f1dd435d69d-dshm\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:24:16.431090 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:16.431058    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-7db57ffd58-z8bm8_a7f8e100-3db8-4119-9aeb-4f1dd435d69d/main/0.log"
Sep 18 20:24:16.431497 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:16.431467    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-z8bm8" event={"ID":"a7f8e100-3db8-4119-9aeb-4f1dd435d69d","Type":"ContainerDied","Data":"68bb12ed7409c9250b3a39346b8ae561af0dbb70b504653520b2872640b006f8"}
Sep 18 20:24:16.431549 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:16.431517    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-z8bm8"
Sep 18 20:24:16.431549 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:16.431527    2577 scope.go:117] "RemoveContainer" containerID="721292c9fb7de8ffa6a2a4667d74b439c2d65462567cde9b38e2c1c21f128ef4"
Sep 18 20:24:16.441776 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:16.441489    2577 scope.go:117] "RemoveContainer" containerID="7d6bd56ccc45a3bef7cbf37b0ccf643277b25cb07475a8e6f93b3a2a1a49aa9e"
Sep 18 20:24:16.458034 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:16.457983    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-z8bm8"]
Sep 18 20:24:16.462399 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:16.462379    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-z8bm8"]
Sep 18 20:24:17.438288 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:17.438253    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7f8e100-3db8-4119-9aeb-4f1dd435d69d" path="/var/lib/kubelet/pods/a7f8e100-3db8-4119-9aeb-4f1dd435d69d/volumes"
Sep 18 20:24:58.642420 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:58.642385    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-ts5f4"]
Sep 18 20:24:58.642906 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:58.642719    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7f8e100-3db8-4119-9aeb-4f1dd435d69d" containerName="main"
Sep 18 20:24:58.642906 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:58.642730    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7f8e100-3db8-4119-9aeb-4f1dd435d69d" containerName="main"
Sep 18 20:24:58.642906 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:58.642749    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7f8e100-3db8-4119-9aeb-4f1dd435d69d" containerName="storage-initializer"
Sep 18 20:24:58.642906 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:58.642755    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7f8e100-3db8-4119-9aeb-4f1dd435d69d" containerName="storage-initializer"
Sep 18 20:24:58.642906 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:58.642807    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7f8e100-3db8-4119-9aeb-4f1dd435d69d" containerName="main"
Sep 18 20:24:58.646828 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:58.646809    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-ts5f4"
Sep 18 20:24:58.651684 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:58.651658    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 18 20:24:58.652389 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:58.652367    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 18 20:24:58.653630 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:58.653605    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 18 20:24:58.653746 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:58.653619    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-p2kjg\""
Sep 18 20:24:58.664692 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:58.664669    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-ts5f4"]
Sep 18 20:24:58.778458 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:58.778423    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2c02c46-afb1-4060-9ac8-db18dc1dd2e9-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-ts5f4\" (UID: \"d2c02c46-afb1-4060-9ac8-db18dc1dd2e9\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-ts5f4"
Sep 18 20:24:58.778458 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:58.778459    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2c02c46-afb1-4060-9ac8-db18dc1dd2e9-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-ts5f4\" (UID: \"d2c02c46-afb1-4060-9ac8-db18dc1dd2e9\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-ts5f4"
Sep 18 20:24:58.778697 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:58.778536    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2c02c46-afb1-4060-9ac8-db18dc1dd2e9-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-ts5f4\" (UID: \"d2c02c46-afb1-4060-9ac8-db18dc1dd2e9\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-ts5f4"
Sep 18 20:24:58.778697 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:58.778605    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2c02c46-afb1-4060-9ac8-db18dc1dd2e9-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-ts5f4\" (UID: \"d2c02c46-afb1-4060-9ac8-db18dc1dd2e9\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-ts5f4"
Sep 18 20:24:58.778697 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:58.778649    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5dfw\" (UniqueName: \"kubernetes.io/projected/d2c02c46-afb1-4060-9ac8-db18dc1dd2e9-kube-api-access-z5dfw\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-ts5f4\" (UID: \"d2c02c46-afb1-4060-9ac8-db18dc1dd2e9\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-ts5f4"
Sep 18 20:24:58.778697 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:58.778674    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2c02c46-afb1-4060-9ac8-db18dc1dd2e9-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-ts5f4\" (UID: \"d2c02c46-afb1-4060-9ac8-db18dc1dd2e9\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-ts5f4"
Sep 18 20:24:58.879779 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:58.879735    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5dfw\" (UniqueName: \"kubernetes.io/projected/d2c02c46-afb1-4060-9ac8-db18dc1dd2e9-kube-api-access-z5dfw\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-ts5f4\" (UID: \"d2c02c46-afb1-4060-9ac8-db18dc1dd2e9\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-ts5f4"
Sep 18 20:24:58.880019 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:58.879792    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2c02c46-afb1-4060-9ac8-db18dc1dd2e9-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-ts5f4\" (UID: \"d2c02c46-afb1-4060-9ac8-db18dc1dd2e9\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-ts5f4"
Sep 18 20:24:58.880019 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:58.879825    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2c02c46-afb1-4060-9ac8-db18dc1dd2e9-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-ts5f4\" (UID: \"d2c02c46-afb1-4060-9ac8-db18dc1dd2e9\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-ts5f4"
Sep 18 20:24:58.880019 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:58.879847    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2c02c46-afb1-4060-9ac8-db18dc1dd2e9-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-ts5f4\" (UID: \"d2c02c46-afb1-4060-9ac8-db18dc1dd2e9\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-ts5f4"
Sep 18 20:24:58.880019 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:58.879894    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2c02c46-afb1-4060-9ac8-db18dc1dd2e9-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-ts5f4\" (UID: \"d2c02c46-afb1-4060-9ac8-db18dc1dd2e9\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-ts5f4"
Sep 18 20:24:58.880019 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:58.879960    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2c02c46-afb1-4060-9ac8-db18dc1dd2e9-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-ts5f4\" (UID: \"d2c02c46-afb1-4060-9ac8-db18dc1dd2e9\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-ts5f4"
Sep 18 20:24:58.880322 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:58.880269    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2c02c46-afb1-4060-9ac8-db18dc1dd2e9-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-ts5f4\" (UID: \"d2c02c46-afb1-4060-9ac8-db18dc1dd2e9\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-ts5f4"
Sep 18 20:24:58.880386 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:58.880314    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2c02c46-afb1-4060-9ac8-db18dc1dd2e9-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-ts5f4\" (UID: \"d2c02c46-afb1-4060-9ac8-db18dc1dd2e9\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-ts5f4"
Sep 18 20:24:58.880386 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:58.880345    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2c02c46-afb1-4060-9ac8-db18dc1dd2e9-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-ts5f4\" (UID: \"d2c02c46-afb1-4060-9ac8-db18dc1dd2e9\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-ts5f4"
Sep 18 20:24:58.882263 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:58.882240    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2c02c46-afb1-4060-9ac8-db18dc1dd2e9-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-ts5f4\" (UID: \"d2c02c46-afb1-4060-9ac8-db18dc1dd2e9\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-ts5f4"
Sep 18 20:24:58.882576 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:58.882558    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2c02c46-afb1-4060-9ac8-db18dc1dd2e9-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-ts5f4\" (UID: \"d2c02c46-afb1-4060-9ac8-db18dc1dd2e9\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-ts5f4"
Sep 18 20:24:58.891573 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:58.891549    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5dfw\" (UniqueName: \"kubernetes.io/projected/d2c02c46-afb1-4060-9ac8-db18dc1dd2e9-kube-api-access-z5dfw\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-ts5f4\" (UID: \"d2c02c46-afb1-4060-9ac8-db18dc1dd2e9\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-ts5f4"
Sep 18 20:24:58.957372 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:58.957286    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-ts5f4"
Sep 18 20:24:59.251208 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:59.251119    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-ts5f4"]
Sep 18 20:24:59.255765 ip-10-0-129-30 kubenswrapper[2577]: W0918 20:24:59.255736    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2c02c46_afb1_4060_9ac8_db18dc1dd2e9.slice/crio-8b8d3b871b67fa240b741dbffa8cf64a38bdb32439b300372e5b82cf96eec132 WatchSource:0}: Error finding container 8b8d3b871b67fa240b741dbffa8cf64a38bdb32439b300372e5b82cf96eec132: Status 404 returned error can't find the container with id 8b8d3b871b67fa240b741dbffa8cf64a38bdb32439b300372e5b82cf96eec132
Sep 18 20:24:59.622929 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:59.622890    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-ts5f4" event={"ID":"d2c02c46-afb1-4060-9ac8-db18dc1dd2e9","Type":"ContainerStarted","Data":"d1125425a63512f4a5cb971549b9963f57e761e23afa29e1bcc6ca233ece0dbc"}
Sep 18 20:24:59.623124 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:59.622945    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-ts5f4" event={"ID":"d2c02c46-afb1-4060-9ac8-db18dc1dd2e9","Type":"ContainerStarted","Data":"8b8d3b871b67fa240b741dbffa8cf64a38bdb32439b300372e5b82cf96eec132"}
Sep 18 20:24:59.656551 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:59.656497    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-ts5f4" podStartSLOduration=1.656479943 podStartE2EDuration="1.656479943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:24:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:24:59.655285924 +0000 UTC m=+4270.507586296" watchObservedRunningTime="2026-09-18 20:24:59.656479943 +0000 UTC m=+4270.508780313"
Sep 18 20:24:59.958441 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:24:59.958343    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-ts5f4"
Sep 18 20:25:07.065082 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:25:07.065032    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-ts5f4"]
Sep 18 20:25:07.078890 ip-10-0-129-30 kubenswrapper[2577]: E0918 20:25:07.078849    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 20:25:07.078890 ip-10-0-129-30 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:25:07.078890 ip-10-0-129-30 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 18 20:25:07.078890 ip-10-0-129-30 kubenswrapper[2577]:  > containerID="d1125425a63512f4a5cb971549b9963f57e761e23afa29e1bcc6ca233ece0dbc" cmd=["/bin/sleep","15"]
Sep 18 20:25:07.079223 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:25:07.078914    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-ts5f4" message="" err=<
Sep 18 20:25:07.079223 ip-10-0-129-30 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:25:07.079223 ip-10-0-129-30 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 18 20:25:07.079223 ip-10-0-129-30 kubenswrapper[2577]:  >
Sep 18 20:25:07.079223 ip-10-0-129-30 kubenswrapper[2577]: E0918 20:25:07.078930    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 20:25:07.079223 ip-10-0-129-30 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:25:07.079223 ip-10-0-129-30 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 18 20:25:07.079223 ip-10-0-129-30 kubenswrapper[2577]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-ts5f4" podUID="d2c02c46-afb1-4060-9ac8-db18dc1dd2e9" containerName="main" containerID="cri-o://d1125425a63512f4a5cb971549b9963f57e761e23afa29e1bcc6ca233ece0dbc"
Sep 18 20:25:07.079223 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:25:07.078983    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-ts5f4" podUID="d2c02c46-afb1-4060-9ac8-db18dc1dd2e9" containerName="main" containerID="cri-o://d1125425a63512f4a5cb971549b9963f57e761e23afa29e1bcc6ca233ece0dbc" gracePeriod=60
Sep 18 20:25:07.329254 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:25:07.329187    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-ts5f4"
Sep 18 20:25:07.458403 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:25:07.458372    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2c02c46-afb1-4060-9ac8-db18dc1dd2e9-home\") pod \"d2c02c46-afb1-4060-9ac8-db18dc1dd2e9\" (UID: \"d2c02c46-afb1-4060-9ac8-db18dc1dd2e9\") "
Sep 18 20:25:07.458403 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:25:07.458407    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z5dfw\" (UniqueName: \"kubernetes.io/projected/d2c02c46-afb1-4060-9ac8-db18dc1dd2e9-kube-api-access-z5dfw\") pod \"d2c02c46-afb1-4060-9ac8-db18dc1dd2e9\" (UID: \"d2c02c46-afb1-4060-9ac8-db18dc1dd2e9\") "
Sep 18 20:25:07.458641 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:25:07.458441    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2c02c46-afb1-4060-9ac8-db18dc1dd2e9-tls-certs\") pod \"d2c02c46-afb1-4060-9ac8-db18dc1dd2e9\" (UID: \"d2c02c46-afb1-4060-9ac8-db18dc1dd2e9\") "
Sep 18 20:25:07.458641 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:25:07.458472    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2c02c46-afb1-4060-9ac8-db18dc1dd2e9-tmp-dir\") pod \"d2c02c46-afb1-4060-9ac8-db18dc1dd2e9\" (UID: \"d2c02c46-afb1-4060-9ac8-db18dc1dd2e9\") "
Sep 18 20:25:07.458641 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:25:07.458526    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2c02c46-afb1-4060-9ac8-db18dc1dd2e9-dshm\") pod \"d2c02c46-afb1-4060-9ac8-db18dc1dd2e9\" (UID: \"d2c02c46-afb1-4060-9ac8-db18dc1dd2e9\") "
Sep 18 20:25:07.458641 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:25:07.458609    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2c02c46-afb1-4060-9ac8-db18dc1dd2e9-model-cache\") pod \"d2c02c46-afb1-4060-9ac8-db18dc1dd2e9\" (UID: \"d2c02c46-afb1-4060-9ac8-db18dc1dd2e9\") "
Sep 18 20:25:07.458881 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:25:07.458668    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c02c46-afb1-4060-9ac8-db18dc1dd2e9-home" (OuterVolumeSpecName: "home") pod "d2c02c46-afb1-4060-9ac8-db18dc1dd2e9" (UID: "d2c02c46-afb1-4060-9ac8-db18dc1dd2e9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:25:07.458881 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:25:07.458761    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c02c46-afb1-4060-9ac8-db18dc1dd2e9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d2c02c46-afb1-4060-9ac8-db18dc1dd2e9" (UID: "d2c02c46-afb1-4060-9ac8-db18dc1dd2e9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:25:07.458961 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:25:07.458881    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c02c46-afb1-4060-9ac8-db18dc1dd2e9-model-cache" (OuterVolumeSpecName: "model-cache") pod "d2c02c46-afb1-4060-9ac8-db18dc1dd2e9" (UID: "d2c02c46-afb1-4060-9ac8-db18dc1dd2e9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:25:07.458961 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:25:07.458935    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2c02c46-afb1-4060-9ac8-db18dc1dd2e9-home\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:25:07.458961 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:25:07.458952    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2c02c46-afb1-4060-9ac8-db18dc1dd2e9-tmp-dir\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:25:07.460795 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:25:07.460769    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2c02c46-afb1-4060-9ac8-db18dc1dd2e9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d2c02c46-afb1-4060-9ac8-db18dc1dd2e9" (UID: "d2c02c46-afb1-4060-9ac8-db18dc1dd2e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:25:07.460911 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:25:07.460803    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2c02c46-afb1-4060-9ac8-db18dc1dd2e9-kube-api-access-z5dfw" (OuterVolumeSpecName: "kube-api-access-z5dfw") pod "d2c02c46-afb1-4060-9ac8-db18dc1dd2e9" (UID: "d2c02c46-afb1-4060-9ac8-db18dc1dd2e9"). InnerVolumeSpecName "kube-api-access-z5dfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:25:07.460911 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:25:07.460870    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c02c46-afb1-4060-9ac8-db18dc1dd2e9-dshm" (OuterVolumeSpecName: "dshm") pod "d2c02c46-afb1-4060-9ac8-db18dc1dd2e9" (UID: "d2c02c46-afb1-4060-9ac8-db18dc1dd2e9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:25:07.560135 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:25:07.560098    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2c02c46-afb1-4060-9ac8-db18dc1dd2e9-model-cache\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:25:07.560135 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:25:07.560126    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5dfw\" (UniqueName: \"kubernetes.io/projected/d2c02c46-afb1-4060-9ac8-db18dc1dd2e9-kube-api-access-z5dfw\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:25:07.560135 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:25:07.560137    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2c02c46-afb1-4060-9ac8-db18dc1dd2e9-tls-certs\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:25:07.560135 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:25:07.560148    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2c02c46-afb1-4060-9ac8-db18dc1dd2e9-dshm\") on node \"ip-10-0-129-30.ec2.internal\" DevicePath \"\""
Sep 18 20:25:07.657130 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:25:07.657089    2577 generic.go:356] "Generic (PLEG): container finished" podID="d2c02c46-afb1-4060-9ac8-db18dc1dd2e9" containerID="d1125425a63512f4a5cb971549b9963f57e761e23afa29e1bcc6ca233ece0dbc" exitCode=0
Sep 18 20:25:07.657309 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:25:07.657139    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-ts5f4"
Sep 18 20:25:07.657309 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:25:07.657176    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-ts5f4" event={"ID":"d2c02c46-afb1-4060-9ac8-db18dc1dd2e9","Type":"ContainerDied","Data":"d1125425a63512f4a5cb971549b9963f57e761e23afa29e1bcc6ca233ece0dbc"}
Sep 18 20:25:07.657309 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:25:07.657237    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-ts5f4" event={"ID":"d2c02c46-afb1-4060-9ac8-db18dc1dd2e9","Type":"ContainerDied","Data":"8b8d3b871b67fa240b741dbffa8cf64a38bdb32439b300372e5b82cf96eec132"}
Sep 18 20:25:07.657309 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:25:07.657263    2577 scope.go:117] "RemoveContainer" containerID="d1125425a63512f4a5cb971549b9963f57e761e23afa29e1bcc6ca233ece0dbc"
Sep 18 20:25:07.670707 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:25:07.667169    2577 scope.go:117] "RemoveContainer" containerID="d1125425a63512f4a5cb971549b9963f57e761e23afa29e1bcc6ca233ece0dbc"
Sep 18 20:25:07.670707 ip-10-0-129-30 kubenswrapper[2577]: E0918 20:25:07.667536    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1125425a63512f4a5cb971549b9963f57e761e23afa29e1bcc6ca233ece0dbc\": container with ID starting with d1125425a63512f4a5cb971549b9963f57e761e23afa29e1bcc6ca233ece0dbc not found: ID does not exist" containerID="d1125425a63512f4a5cb971549b9963f57e761e23afa29e1bcc6ca233ece0dbc"
Sep 18 20:25:07.670707 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:25:07.667575    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1125425a63512f4a5cb971549b9963f57e761e23afa29e1bcc6ca233ece0dbc"} err="failed to get container status \"d1125425a63512f4a5cb971549b9963f57e761e23afa29e1bcc6ca233ece0dbc\": rpc error: code = NotFound desc = could not find container \"d1125425a63512f4a5cb971549b9963f57e761e23afa29e1bcc6ca233ece0dbc\": container with ID starting with d1125425a63512f4a5cb971549b9963f57e761e23afa29e1bcc6ca233ece0dbc not found: ID does not exist"
Sep 18 20:25:07.701058 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:25:07.701028    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-ts5f4"]
Sep 18 20:25:07.720929 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:25:07.720902    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-ts5f4"]
Sep 18 20:25:09.438576 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:25:09.438542    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2c02c46-afb1-4060-9ac8-db18dc1dd2e9" path="/var/lib/kubelet/pods/d2c02c46-afb1-4060-9ac8-db18dc1dd2e9/volumes"
Sep 18 20:26:22.527948 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:22.527881    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx_625e8845-a6ab-4281-838a-b4051ae7ad5d/istio-proxy/0.log"
Sep 18 20:26:22.530332 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:22.530312    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx_625e8845-a6ab-4281-838a-b4051ae7ad5d/istio-proxy/1.log"
Sep 18 20:26:23.545200 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:23.545165    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx_625e8845-a6ab-4281-838a-b4051ae7ad5d/istio-proxy/0.log"
Sep 18 20:26:23.548210 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:23.548191    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx_625e8845-a6ab-4281-838a-b4051ae7ad5d/istio-proxy/1.log"
Sep 18 20:26:24.513136 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:24.513108    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-zx9fc_2733a01c-8640-4c51-a877-3ee099deed3b/kuadrant-console-plugin/0.log"
Sep 18 20:26:24.635962 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:24.635935    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-hq9st_6aae2a34-975a-4871-b0a8-c1e7326f5d12/limitador/0.log"
Sep 18 20:26:29.979917 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:29.979880    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2m5d7_5ebd2878-a3f1-467a-b25f-29124e9d5e7b/global-pull-secret-syncer/0.log"
Sep 18 20:26:30.169973 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:30.169934    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-s6qfb_fd53f3e2-ce65-4c01-9a2b-10803d702989/konnectivity-agent/0.log"
Sep 18 20:26:30.249922 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:30.249842    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-30.ec2.internal_5e6343628ef74873cac4c6411141685f/haproxy/0.log"
Sep 18 20:26:35.237733 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:35.237707    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-zx9fc_2733a01c-8640-4c51-a877-3ee099deed3b/kuadrant-console-plugin/0.log"
Sep 18 20:26:35.428557 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:35.428526    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-hq9st_6aae2a34-975a-4871-b0a8-c1e7326f5d12/limitador/0.log"
Sep 18 20:26:36.941450 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:36.941416    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b682942d-7069-404b-918d-e83dfeb960cf/alertmanager/0.log"
Sep 18 20:26:36.997354 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:36.997280    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b682942d-7069-404b-918d-e83dfeb960cf/config-reloader/0.log"
Sep 18 20:26:37.048163 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:37.048136    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b682942d-7069-404b-918d-e83dfeb960cf/kube-rbac-proxy-web/0.log"
Sep 18 20:26:37.117137 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:37.117102    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b682942d-7069-404b-918d-e83dfeb960cf/kube-rbac-proxy/0.log"
Sep 18 20:26:37.177247 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:37.177211    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b682942d-7069-404b-918d-e83dfeb960cf/kube-rbac-proxy-metric/0.log"
Sep 18 20:26:37.282044 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:37.281940    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b682942d-7069-404b-918d-e83dfeb960cf/prom-label-proxy/0.log"
Sep 18 20:26:37.377458 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:37.377420    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b682942d-7069-404b-918d-e83dfeb960cf/init-config-reloader/0.log"
Sep 18 20:26:37.679384 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:37.679351    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-9b97dfbcc-xz9kd_e612908b-7c17-483f-8c5d-9480888ceddf/metrics-server/0.log"
Sep 18 20:26:37.899032 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:37.899005    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9xg4r_58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd/node-exporter/0.log"
Sep 18 20:26:37.930667 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:37.930643    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9xg4r_58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd/kube-rbac-proxy/0.log"
Sep 18 20:26:37.967133 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:37.967111    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9xg4r_58eefe10-4fdc-4fff-a8b1-1ccb6f1744fd/init-textfile/0.log"
Sep 18 20:26:38.246155 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:38.246076    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c0b6d69b-f7ce-4b85-84d5-dc6440246e3d/prometheus/0.log"
Sep 18 20:26:38.297413 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:38.297386    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c0b6d69b-f7ce-4b85-84d5-dc6440246e3d/config-reloader/0.log"
Sep 18 20:26:38.349482 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:38.349456    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c0b6d69b-f7ce-4b85-84d5-dc6440246e3d/thanos-sidecar/0.log"
Sep 18 20:26:38.409422 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:38.409392    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c0b6d69b-f7ce-4b85-84d5-dc6440246e3d/kube-rbac-proxy-web/0.log"
Sep 18 20:26:38.462556 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:38.462532    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c0b6d69b-f7ce-4b85-84d5-dc6440246e3d/kube-rbac-proxy/0.log"
Sep 18 20:26:38.516383 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:38.516305    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c0b6d69b-f7ce-4b85-84d5-dc6440246e3d/kube-rbac-proxy-thanos/0.log"
Sep 18 20:26:38.567307 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:38.567284    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c0b6d69b-f7ce-4b85-84d5-dc6440246e3d/init-config-reloader/0.log"
Sep 18 20:26:38.617742 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:38.617718    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-w7p4q_83e874e2-103a-4097-b485-253beb0f1590/prometheus-operator/0.log"
Sep 18 20:26:38.664533 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:38.664505    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-w7p4q_83e874e2-103a-4097-b485-253beb0f1590/kube-rbac-proxy/0.log"
Sep 18 20:26:39.313382 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:39.313348    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-sggjm/perf-node-gather-daemonset-j9nn5"]
Sep 18 20:26:39.313784 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:39.313653    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2c02c46-afb1-4060-9ac8-db18dc1dd2e9" containerName="main"
Sep 18 20:26:39.313784 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:39.313664    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2c02c46-afb1-4060-9ac8-db18dc1dd2e9" containerName="main"
Sep 18 20:26:39.313784 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:39.313731    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2c02c46-afb1-4060-9ac8-db18dc1dd2e9" containerName="main"
Sep 18 20:26:39.316824 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:39.316808    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-j9nn5"
Sep 18 20:26:39.320682 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:39.320660    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-sggjm\"/\"default-dockercfg-gqtsx\""
Sep 18 20:26:39.320847 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:39.320685    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sggjm\"/\"openshift-service-ca.crt\""
Sep 18 20:26:39.321654 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:39.321637    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sggjm\"/\"kube-root-ca.crt\""
Sep 18 20:26:39.327636 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:39.327612    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sggjm/perf-node-gather-daemonset-j9nn5"]
Sep 18 20:26:39.454173 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:39.454127    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f2f60f88-1938-49ea-9b90-e055f3ac87a6-sys\") pod \"perf-node-gather-daemonset-j9nn5\" (UID: \"f2f60f88-1938-49ea-9b90-e055f3ac87a6\") " pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-j9nn5"
Sep 18 20:26:39.454383 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:39.454195    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f2f60f88-1938-49ea-9b90-e055f3ac87a6-proc\") pod \"perf-node-gather-daemonset-j9nn5\" (UID: \"f2f60f88-1938-49ea-9b90-e055f3ac87a6\") " pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-j9nn5"
Sep 18 20:26:39.454383 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:39.454296    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f2f60f88-1938-49ea-9b90-e055f3ac87a6-podres\") pod \"perf-node-gather-daemonset-j9nn5\" (UID: \"f2f60f88-1938-49ea-9b90-e055f3ac87a6\") " pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-j9nn5"
Sep 18 20:26:39.454383 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:39.454331    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f2f60f88-1938-49ea-9b90-e055f3ac87a6-lib-modules\") pod \"perf-node-gather-daemonset-j9nn5\" (UID: \"f2f60f88-1938-49ea-9b90-e055f3ac87a6\") " pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-j9nn5"
Sep 18 20:26:39.454383 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:39.454359    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k29jb\" (UniqueName: \"kubernetes.io/projected/f2f60f88-1938-49ea-9b90-e055f3ac87a6-kube-api-access-k29jb\") pod \"perf-node-gather-daemonset-j9nn5\" (UID: \"f2f60f88-1938-49ea-9b90-e055f3ac87a6\") " pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-j9nn5"
Sep 18 20:26:39.555131 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:39.555092    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f2f60f88-1938-49ea-9b90-e055f3ac87a6-podres\") pod \"perf-node-gather-daemonset-j9nn5\" (UID: \"f2f60f88-1938-49ea-9b90-e055f3ac87a6\") " pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-j9nn5"
Sep 18 20:26:39.555312 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:39.555145    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f2f60f88-1938-49ea-9b90-e055f3ac87a6-lib-modules\") pod \"perf-node-gather-daemonset-j9nn5\" (UID: \"f2f60f88-1938-49ea-9b90-e055f3ac87a6\") " pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-j9nn5"
Sep 18 20:26:39.555312 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:39.555180    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k29jb\" (UniqueName: \"kubernetes.io/projected/f2f60f88-1938-49ea-9b90-e055f3ac87a6-kube-api-access-k29jb\") pod \"perf-node-gather-daemonset-j9nn5\" (UID: \"f2f60f88-1938-49ea-9b90-e055f3ac87a6\") " pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-j9nn5"
Sep 18 20:26:39.555312 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:39.555199    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f2f60f88-1938-49ea-9b90-e055f3ac87a6-sys\") pod \"perf-node-gather-daemonset-j9nn5\" (UID: \"f2f60f88-1938-49ea-9b90-e055f3ac87a6\") " pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-j9nn5"
Sep 18 20:26:39.555312 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:39.555247    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f2f60f88-1938-49ea-9b90-e055f3ac87a6-proc\") pod \"perf-node-gather-daemonset-j9nn5\" (UID: \"f2f60f88-1938-49ea-9b90-e055f3ac87a6\") " pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-j9nn5"
Sep 18 20:26:39.555312 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:39.555273    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f2f60f88-1938-49ea-9b90-e055f3ac87a6-podres\") pod \"perf-node-gather-daemonset-j9nn5\" (UID: \"f2f60f88-1938-49ea-9b90-e055f3ac87a6\") " pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-j9nn5"
Sep 18 20:26:39.555572 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:39.555372    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f2f60f88-1938-49ea-9b90-e055f3ac87a6-sys\") pod \"perf-node-gather-daemonset-j9nn5\" (UID: \"f2f60f88-1938-49ea-9b90-e055f3ac87a6\") " pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-j9nn5"
Sep 18 20:26:39.555572 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:39.555395    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f2f60f88-1938-49ea-9b90-e055f3ac87a6-proc\") pod \"perf-node-gather-daemonset-j9nn5\" (UID: \"f2f60f88-1938-49ea-9b90-e055f3ac87a6\") " pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-j9nn5"
Sep 18 20:26:39.555572 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:39.555432    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f2f60f88-1938-49ea-9b90-e055f3ac87a6-lib-modules\") pod \"perf-node-gather-daemonset-j9nn5\" (UID: \"f2f60f88-1938-49ea-9b90-e055f3ac87a6\") " pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-j9nn5"
Sep 18 20:26:39.565343 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:39.565274    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k29jb\" (UniqueName: \"kubernetes.io/projected/f2f60f88-1938-49ea-9b90-e055f3ac87a6-kube-api-access-k29jb\") pod \"perf-node-gather-daemonset-j9nn5\" (UID: \"f2f60f88-1938-49ea-9b90-e055f3ac87a6\") " pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-j9nn5"
Sep 18 20:26:39.628260 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:39.628222    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-j9nn5"
Sep 18 20:26:39.911392 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:39.911343    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sggjm/perf-node-gather-daemonset-j9nn5"]
Sep 18 20:26:39.914192 ip-10-0-129-30 kubenswrapper[2577]: W0918 20:26:39.914156    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf2f60f88_1938_49ea_9b90_e055f3ac87a6.slice/crio-7e34130e97348fd57971f9e2b23df8698a0257ca5dfada5fe1e8bc6854821d29 WatchSource:0}: Error finding container 7e34130e97348fd57971f9e2b23df8698a0257ca5dfada5fe1e8bc6854821d29: Status 404 returned error can't find the container with id 7e34130e97348fd57971f9e2b23df8698a0257ca5dfada5fe1e8bc6854821d29
Sep 18 20:26:40.051398 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:40.051369    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-j9nn5" event={"ID":"f2f60f88-1938-49ea-9b90-e055f3ac87a6","Type":"ContainerStarted","Data":"a3245ed1c6c3679ba54644f51e932b5b8cdde55f162ddc02a3e6c9c62b92329a"}
Sep 18 20:26:40.051520 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:40.051424    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-j9nn5" event={"ID":"f2f60f88-1938-49ea-9b90-e055f3ac87a6","Type":"ContainerStarted","Data":"7e34130e97348fd57971f9e2b23df8698a0257ca5dfada5fe1e8bc6854821d29"}
Sep 18 20:26:40.051520 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:40.051453    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-j9nn5"
Sep 18 20:26:40.071968 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:40.071917    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-j9nn5" podStartSLOduration=1.071904023 podStartE2EDuration="1.071904023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:26:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:26:40.069847226 +0000 UTC m=+4370.922147595" watchObservedRunningTime="2026-09-18 20:26:40.071904023 +0000 UTC m=+4370.924204393"
Sep 18 20:26:40.177296 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:40.177201    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6446c8d75d-92v7w_bf41ad7a-3615-4981-8a3e-125f973295aa/networking-console-plugin/0.log"
Sep 18 20:26:41.962352 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:41.962323    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-p7mxq_2767860b-c421-4310-8990-99b908e03b08/volume-data-source-validator/0.log"
Sep 18 20:26:43.154646 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:43.154620    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dg5zx_dbd605d1-eb07-4c9c-aea1-f1a610394801/dns/0.log"
Sep 18 20:26:43.190206 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:43.190182    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dg5zx_dbd605d1-eb07-4c9c-aea1-f1a610394801/kube-rbac-proxy/0.log"
Sep 18 20:26:43.368543 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:43.368513    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5jzcl_b7797c1c-31e6-4f63-84d2-6f3c99e195ee/dns-node-resolver/0.log"
Sep 18 20:26:44.122305 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:44.122279    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mx49g_ea018bdf-321c-4368-875d-937e82b2ed3b/node-ca/0.log"
Sep 18 20:26:45.551101 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:45.551076    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx_625e8845-a6ab-4281-838a-b4051ae7ad5d/istio-proxy/0.log"
Sep 18 20:26:45.553790 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:45.553770    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx_625e8845-a6ab-4281-838a-b4051ae7ad5d/istio-proxy/1.log"
Sep 18 20:26:46.065853 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:46.065827    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-j9nn5"
Sep 18 20:26:46.289109 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:46.289079    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mb6kf_44aa71cc-fda2-4a75-a657-23e31f95e973/serve-healthcheck-canary/0.log"
Sep 18 20:26:46.875376 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:46.875348    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2mhnp_4b200649-8dc1-4a78-9d23-4f0c0ff3fc08/kube-rbac-proxy/0.log"
Sep 18 20:26:46.916172 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:46.916147    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2mhnp_4b200649-8dc1-4a78-9d23-4f0c0ff3fc08/exporter/0.log"
Sep 18 20:26:46.951119 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:46.951094    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2mhnp_4b200649-8dc1-4a78-9d23-4f0c0ff3fc08/extractor/0.log"
Sep 18 20:26:49.715393 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:49.715287    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx_625e8845-a6ab-4281-838a-b4051ae7ad5d/istio-proxy/0.log"
Sep 18 20:26:49.718023 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:49.718004    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx_625e8845-a6ab-4281-838a-b4051ae7ad5d/istio-proxy/1.log"
Sep 18 20:26:51.024398 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:51.024368    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx_625e8845-a6ab-4281-838a-b4051ae7ad5d/istio-proxy/1.log"
Sep 18 20:26:51.024861 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:51.024839    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-k8pdx_625e8845-a6ab-4281-838a-b4051ae7ad5d/istio-proxy/0.log"
Sep 18 20:26:51.211482 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:51.211404    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-85dd87cf5-zqskb_c1b15394-b52c-4899-a52a-1dd0ea6a4823/manager/0.log"
Sep 18 20:26:51.254217 ip-10-0-129-30 kubenswrapper[2577]: I0918 20:26:51.254191    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-cfrgg_7b44ab54-569e-4c59-ae22-24d2eeafb924/openshift-lws-operator/0.log"