Sep 16 19:37:25.255075 ip-10-0-139-81 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 16 19:37:25.255089 ip-10-0-139-81 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 16 19:37:25.255100 ip-10-0-139-81 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 16 19:37:25.255455 ip-10-0-139-81 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 16 19:37:35.318605 ip-10-0-139-81 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 16 19:37:35.318624 ip-10-0-139-81 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 86a2f7eee6d84148a5baff974fd77a86 --
Sep 16 19:39:31.369092 ip-10-0-139-81 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 19:39:31.722563 ip-10-0-139-81 kubenswrapper[2557]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 19:39:31.722563 ip-10-0-139-81 kubenswrapper[2557]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 19:39:31.722563 ip-10-0-139-81 kubenswrapper[2557]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 19:39:31.722563 ip-10-0-139-81 kubenswrapper[2557]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 19:39:31.722563 ip-10-0-139-81 kubenswrapper[2557]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 19:39:31.724942 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.724849    2557 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 19:39:31.727781 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727764    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 19:39:31.727781 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727782    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 19:39:31.727868 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727786    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 19:39:31.727868 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727790    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 19:39:31.727868 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727793    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 19:39:31.727868 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727796    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 19:39:31.727868 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727799    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 19:39:31.727868 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727802    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 19:39:31.727868 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727805    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 19:39:31.727868 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727810    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 19:39:31.727868 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727813    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 19:39:31.727868 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727816    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 19:39:31.727868 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727818    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 19:39:31.727868 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727821    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 19:39:31.727868 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727824    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 19:39:31.727868 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727827    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 19:39:31.727868 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727829    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 19:39:31.727868 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727832    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 19:39:31.727868 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727848    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 19:39:31.727868 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727851    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 19:39:31.727868 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727853    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 19:39:31.727868 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727856    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 19:39:31.728337 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727859    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 19:39:31.728337 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727862    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 19:39:31.728337 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727865    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 19:39:31.728337 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727867    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 19:39:31.728337 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727870    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 19:39:31.728337 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727873    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 19:39:31.728337 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727876    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 19:39:31.728337 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727879    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 19:39:31.728337 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727882    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 19:39:31.728337 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727885    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 19:39:31.728337 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727888    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 19:39:31.728337 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727890    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 19:39:31.728337 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727893    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 19:39:31.728337 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727895    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 19:39:31.728337 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727898    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 19:39:31.728337 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727901    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 19:39:31.728337 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727904    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 19:39:31.728337 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727906    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 19:39:31.728337 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727909    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 19:39:31.728337 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727911    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 19:39:31.728866 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727914    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 19:39:31.728866 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727918    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 19:39:31.728866 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727922    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 19:39:31.728866 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727926    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 19:39:31.728866 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727929    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 19:39:31.728866 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727932    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 19:39:31.728866 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727934    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 19:39:31.728866 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727937    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 19:39:31.728866 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727939    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 19:39:31.728866 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727942    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 19:39:31.728866 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727945    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 19:39:31.728866 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727947    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 19:39:31.728866 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727950    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 19:39:31.728866 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727953    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 19:39:31.728866 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727956    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 19:39:31.728866 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727959    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 19:39:31.728866 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727961    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 19:39:31.728866 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727964    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 19:39:31.728866 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727966    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 19:39:31.729321 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727969    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 19:39:31.729321 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727972    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 19:39:31.729321 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727975    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 19:39:31.729321 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727977    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 19:39:31.729321 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727980    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 19:39:31.729321 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727982    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 19:39:31.729321 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727986    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 19:39:31.729321 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727990    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 19:39:31.729321 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727993    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 19:39:31.729321 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.727996    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 19:39:31.729321 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728000    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 19:39:31.729321 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728003    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 19:39:31.729321 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728006    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 19:39:31.729321 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728009    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 19:39:31.729321 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728012    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 19:39:31.729321 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728014    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 19:39:31.729321 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728017    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 19:39:31.729321 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728020    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 19:39:31.729321 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728022    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 19:39:31.729774 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728025    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 19:39:31.729774 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728027    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 19:39:31.729774 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728030    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 19:39:31.729774 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728032    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 19:39:31.729774 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728035    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 19:39:31.729774 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728037    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 19:39:31.729774 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728040    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 19:39:31.729774 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728044    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 19:39:31.729774 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728046    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 19:39:31.729774 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728049    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 19:39:31.729774 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728052    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 19:39:31.729774 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728055    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 19:39:31.729774 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728058    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 19:39:31.729774 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728060    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 19:39:31.729774 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728063    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 19:39:31.729774 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728065    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 19:39:31.729774 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728068    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 19:39:31.729774 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728072    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 19:39:31.729774 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728076    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 19:39:31.729774 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728785    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 19:39:31.730293 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728794    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 19:39:31.730293 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728798    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 19:39:31.730293 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728802    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 19:39:31.730293 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728805    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 19:39:31.730293 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728809    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 19:39:31.730293 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728812    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 19:39:31.730293 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728815    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 19:39:31.730293 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728818    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 19:39:31.730293 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728821    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 19:39:31.730293 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728824    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 19:39:31.730293 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728827    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 19:39:31.730293 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728829    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 19:39:31.730293 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728832    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 19:39:31.730293 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728834    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 19:39:31.730293 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728852    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 19:39:31.730293 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728855    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 19:39:31.730293 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728858    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 19:39:31.730293 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728860    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 19:39:31.730293 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728863    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 19:39:31.730781 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728868    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 19:39:31.730781 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728871    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 19:39:31.730781 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728873    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 19:39:31.730781 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728877    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 19:39:31.730781 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728880    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 19:39:31.730781 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728882    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 19:39:31.730781 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728885    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 19:39:31.730781 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728888    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 19:39:31.730781 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728890    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 19:39:31.730781 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728893    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 19:39:31.730781 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728895    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 19:39:31.730781 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728898    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 19:39:31.730781 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728901    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 19:39:31.730781 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728904    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 19:39:31.730781 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728908    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 19:39:31.730781 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728911    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 19:39:31.730781 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728914    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 19:39:31.730781 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728917    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 19:39:31.730781 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728920    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 19:39:31.731286 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728923    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 19:39:31.731286 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728925    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 19:39:31.731286 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728928    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 19:39:31.731286 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728930    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 19:39:31.731286 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728933    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 19:39:31.731286 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728936    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 19:39:31.731286 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728938    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 19:39:31.731286 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728941    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 19:39:31.731286 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728943    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 19:39:31.731286 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728946    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 19:39:31.731286 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728949    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 19:39:31.731286 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728952    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 19:39:31.731286 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728954    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 19:39:31.731286 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728957    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 19:39:31.731286 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728961    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 19:39:31.731286 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728965    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 19:39:31.731286 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728968    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 19:39:31.731286 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728970    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 19:39:31.731286 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728973    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 19:39:31.731286 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728976    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 19:39:31.731778 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728979    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 19:39:31.731778 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728981    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 19:39:31.731778 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728984    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 19:39:31.731778 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728986    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 19:39:31.731778 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728989    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 19:39:31.731778 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728991    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 19:39:31.731778 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728993    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 19:39:31.731778 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728996    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 19:39:31.731778 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.728998    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 19:39:31.731778 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729001    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 19:39:31.731778 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729003    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 19:39:31.731778 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729006    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 19:39:31.731778 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729012    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 19:39:31.731778 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729014    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 19:39:31.731778 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729018    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 19:39:31.731778 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729020    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 19:39:31.731778 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729023    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 19:39:31.731778 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729026    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 19:39:31.731778 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729029    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 19:39:31.731778 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729032    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 19:39:31.732282 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729034    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 19:39:31.732282 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729037    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 19:39:31.732282 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729039    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 19:39:31.732282 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729045    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 19:39:31.732282 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729048    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 19:39:31.732282 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729051    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 19:39:31.732282 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729053    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 19:39:31.732282 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729056    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 19:39:31.732282 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729059    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 19:39:31.732282 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729061    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 19:39:31.732282 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729064    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 19:39:31.732282 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729067    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 19:39:31.732282 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729070    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 19:39:31.732282 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729072    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 19:39:31.732282 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729075    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 19:39:31.732282 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729077    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 19:39:31.732282 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729080    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 19:39:31.732282 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729084    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 19:39:31.732282 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729087    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 19:39:31.732753 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729091    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 19:39:31.732753 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729474    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 19:39:31.732753 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729479    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 19:39:31.732753 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729483    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 19:39:31.732753 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729486    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 19:39:31.732753 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729490    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 19:39:31.732753 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729493    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 19:39:31.732753 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729496    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 19:39:31.732753 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729499    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 19:39:31.732753 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729501    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 19:39:31.732753 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729504    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 19:39:31.732753 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729507    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 19:39:31.732753 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729509    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 19:39:31.732753 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729512    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 19:39:31.732753 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729515    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 19:39:31.732753 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729517    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 19:39:31.732753 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729521    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 19:39:31.732753 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729525    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 19:39:31.732753 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729528    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 19:39:31.732753 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729530    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 19:39:31.733320 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729533    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 19:39:31.733320 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729535    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 19:39:31.733320 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729538    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 19:39:31.733320 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729541    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 19:39:31.733320 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729543    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 19:39:31.733320 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729546    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 19:39:31.733320 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729548    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 19:39:31.733320 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729551    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 19:39:31.733320 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729554    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 19:39:31.733320 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729556    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 19:39:31.733320 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729559    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 19:39:31.733320 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729562    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 19:39:31.733320 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729564    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 19:39:31.733320 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729567    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 19:39:31.733320 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729569    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 19:39:31.733320 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729572    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 19:39:31.733320 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729574    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 19:39:31.733320 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729577    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 19:39:31.733320 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729580    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 19:39:31.733790 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729582    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 19:39:31.733790 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729585    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 19:39:31.733790 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729587    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 19:39:31.733790 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729591    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 19:39:31.733790 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729594    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 19:39:31.733790 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729597    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 19:39:31.733790 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729599    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 19:39:31.733790 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729602    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 19:39:31.733790 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729605    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 19:39:31.733790 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729608    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 19:39:31.733790 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729613    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 19:39:31.733790 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729618    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 19:39:31.733790 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729621    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 19:39:31.733790 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729624    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 19:39:31.733790 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729627    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 19:39:31.733790 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729630    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 19:39:31.733790 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729633    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 19:39:31.733790 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729636    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 19:39:31.733790 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729639    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 19:39:31.734398 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729641    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 19:39:31.734398 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729644    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 19:39:31.734398 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729648    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 19:39:31.734398 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729651    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 19:39:31.734398 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729654    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 19:39:31.734398 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729657    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 19:39:31.734398 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729660    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 19:39:31.734398 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729663    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 19:39:31.734398 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729665    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 19:39:31.734398 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729668    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 19:39:31.734398 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729670    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 19:39:31.734398 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729673    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 19:39:31.734398 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729675    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 19:39:31.734398 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729678    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 19:39:31.734398 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729681    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 19:39:31.734398 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729683    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 19:39:31.734398 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729686    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 19:39:31.734398 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729689    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 19:39:31.734398 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729691    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 19:39:31.734398 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729693    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 19:39:31.734932 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729696    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 19:39:31.734932 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729698    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 19:39:31.734932 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729702    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 19:39:31.734932 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729706    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 19:39:31.734932 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729709    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 19:39:31.734932 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729711    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 19:39:31.734932 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729714    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 19:39:31.734932 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729717    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 19:39:31.734932 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729720    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 19:39:31.734932 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729722    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 19:39:31.734932 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729725    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 19:39:31.734932 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729728    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 19:39:31.734932 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729730    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 19:39:31.734932 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729733    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 19:39:31.734932 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729735    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 19:39:31.734932 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729738    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 19:39:31.734932 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729740    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 19:39:31.734932 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729743    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 19:39:31.734932 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729746    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 19:39:31.734932 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729748    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 19:39:31.735453 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729751    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 19:39:31.735453 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.729754    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 19:39:31.735453 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.729830    2557 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 19:39:31.735453 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.729882    2557 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 19:39:31.735453 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.729890    2557 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 19:39:31.735453 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.729895    2557 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 19:39:31.735453 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.729899    2557 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 19:39:31.735453 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.729903    2557 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 19:39:31.735453 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.729907    2557 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 19:39:31.735453 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.729914    2557 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 19:39:31.735453 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.729917    2557 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 19:39:31.735453 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.729920    2557 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 19:39:31.735453 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.729923    2557 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 19:39:31.735453 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.729927    2557 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 19:39:31.735453 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.729931    2557 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 19:39:31.735453 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.729935    2557 flags.go:64] FLAG: --cgroup-root=""
Sep 16 19:39:31.735453 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.729938    2557 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 19:39:31.735453 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.729941    2557 flags.go:64] FLAG: --client-ca-file=""
Sep 16 19:39:31.735453 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.729944    2557 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 19:39:31.735453 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.729947    2557 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 19:39:31.735453 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.729951    2557 flags.go:64] FLAG: --cluster-domain=""
Sep 16 19:39:31.735453 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.729954    2557 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 19:39:31.735453 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.729957    2557 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 19:39:31.735453 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.729960    2557 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 19:39:31.736167 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.729964    2557 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 19:39:31.736167 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.729968    2557 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 19:39:31.736167 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.729971    2557 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 19:39:31.736167 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.729974    2557 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 19:39:31.736167 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.729977    2557 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 19:39:31.736167 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.729980    2557 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 19:39:31.736167 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.729983    2557 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 19:39:31.736167 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.729986    2557 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 19:39:31.736167 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.729989    2557 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 19:39:31.736167 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.729992    2557 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 19:39:31.736167 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.729996    2557 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 19:39:31.736167 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730000    2557 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 19:39:31.736167 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730003    2557 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 19:39:31.736167 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730005    2557 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 19:39:31.736167 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730008    2557 flags.go:64] FLAG: --enable-server="true"
Sep 16 19:39:31.736167 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730024    2557 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 19:39:31.736167 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730032    2557 flags.go:64] FLAG: --event-burst="100"
Sep 16 19:39:31.736167 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730035    2557 flags.go:64] FLAG: --event-qps="50"
Sep 16 19:39:31.736167 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730038    2557 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 19:39:31.736167 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730041    2557 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 19:39:31.736167 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730044    2557 flags.go:64] FLAG: --eviction-hard=""
Sep 16 19:39:31.736167 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730048    2557 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 19:39:31.736167 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730052    2557 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 19:39:31.736167 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730056    2557 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 19:39:31.736167 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730059    2557 flags.go:64] FLAG: --eviction-soft=""
Sep 16 19:39:31.736772 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730062    2557 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 19:39:31.736772 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730065    2557 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 19:39:31.736772 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730068    2557 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 19:39:31.736772 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730071    2557 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 19:39:31.736772 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730074    2557 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 19:39:31.736772 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730077    2557 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 19:39:31.736772 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730080    2557 flags.go:64] FLAG: --feature-gates=""
Sep 16 19:39:31.736772 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730084    2557 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 19:39:31.736772 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730087    2557 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 19:39:31.736772 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730090    2557 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 19:39:31.736772 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730094    2557 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 19:39:31.736772 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730097    2557 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 19:39:31.736772 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730100    2557 flags.go:64] FLAG: --help="false"
Sep 16 19:39:31.736772 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730103    2557 flags.go:64] FLAG: --hostname-override="ip-10-0-139-81.ec2.internal"
Sep 16 19:39:31.736772 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730106    2557 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 19:39:31.736772 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730109    2557 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 19:39:31.736772 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730112    2557 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 19:39:31.736772 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730115    2557 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 19:39:31.736772 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730119    2557 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 19:39:31.736772 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730121    2557 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 19:39:31.736772 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730124    2557 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 19:39:31.736772 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730127    2557 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 19:39:31.736772 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730130    2557 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 19:39:31.737401 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730139    2557 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 19:39:31.737401 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730142    2557 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 19:39:31.737401 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730145    2557 flags.go:64] FLAG: --kube-reserved=""
Sep 16 19:39:31.737401 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730148    2557 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 19:39:31.737401 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730151    2557 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 19:39:31.737401 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730154    2557 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 19:39:31.737401 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730162    2557 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 19:39:31.737401 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730165    2557 flags.go:64] FLAG: --lock-file=""
Sep 16 19:39:31.737401 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730174    2557 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 19:39:31.737401 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730178    2557 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 19:39:31.737401 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730181    2557 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 19:39:31.737401 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730186    2557 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 19:39:31.737401 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730189    2557 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 19:39:31.737401 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730192    2557 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 19:39:31.737401 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730195    2557 flags.go:64] FLAG: --logging-format="text"
Sep 16 19:39:31.737401 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730199    2557 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 19:39:31.737401 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730202    2557 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 19:39:31.737401 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730205    2557 flags.go:64] FLAG: --manifest-url=""
Sep 16 19:39:31.737401 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730208    2557 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 19:39:31.737401 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730212    2557 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 19:39:31.737401 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730216    2557 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 19:39:31.737401 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730220    2557 flags.go:64] FLAG: --max-pods="110"
Sep 16 19:39:31.737401 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730223    2557 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 19:39:31.737401 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730226    2557 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 19:39:31.737401 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730229    2557 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 19:39:31.738108 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730232    2557 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 19:39:31.738108 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730235    2557 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 19:39:31.738108 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730238    2557 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 19:39:31.738108 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730241    2557 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 19:39:31.738108 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730248    2557 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 19:39:31.738108 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730251    2557 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 19:39:31.738108 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730254    2557 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 19:39:31.738108 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730257    2557 flags.go:64] FLAG: --pod-cidr=""
Sep 16 19:39:31.738108 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730261    2557 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 19:39:31.738108 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730266    2557 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 19:39:31.738108 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730269    2557 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 19:39:31.738108 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730272    2557 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 19:39:31.738108 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730275    2557 flags.go:64] FLAG: --port="10250"
Sep 16 19:39:31.738108 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730279    2557 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 19:39:31.738108 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730284    2557 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07d260c3087238ad7"
Sep 16 19:39:31.738108 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730287    2557 flags.go:64] FLAG: --qos-reserved=""
Sep 16 19:39:31.738108 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730290    2557 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 19:39:31.738108 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730292    2557 flags.go:64] FLAG: --register-node="true"
Sep 16 19:39:31.738108 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730296    2557 flags.go:64] FLAG: --register-with-taints=""
Sep 16 19:39:31.738108 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730300    2557 flags.go:64] FLAG: --registry-burst="10"
Sep 16 19:39:31.738108 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730303    2557 flags.go:64] FLAG: --registry-qps="5"
Sep 16 19:39:31.738108 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730306    2557 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 19:39:31.738108 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730309    2557 flags.go:64] FLAG: --reserved-memory=""
Sep 16 19:39:31.738108 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730313    2557 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 19:39:31.738108 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730316    2557 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 19:39:31.738721 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730319    2557 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 19:39:31.738721 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730322    2557 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 19:39:31.738721 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730325    2557 flags.go:64] FLAG: --runonce="false"
Sep 16 19:39:31.738721 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730328    2557 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 19:39:31.738721 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730331    2557 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 19:39:31.738721 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730333    2557 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 19:39:31.738721 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730336    2557 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 19:39:31.738721 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730339    2557 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 19:39:31.738721 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730342    2557 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 19:39:31.738721 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730345    2557 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 19:39:31.738721 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730348    2557 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 19:39:31.738721 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730351    2557 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 19:39:31.738721 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730354    2557 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 19:39:31.738721 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730356    2557 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 19:39:31.738721 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730359    2557 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 19:39:31.738721 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730362    2557 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 19:39:31.738721 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730373    2557 flags.go:64] FLAG: --system-cgroups=""
Sep 16 19:39:31.738721 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730376    2557 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 19:39:31.738721 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730381    2557 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 19:39:31.738721 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730385    2557 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 19:39:31.738721 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730387    2557 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 19:39:31.738721 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730396    2557 flags.go:64] FLAG: --tls-min-version=""
Sep 16 19:39:31.738721 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730399    2557 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 19:39:31.738721 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730402    2557 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 19:39:31.738721 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730404    2557 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 19:39:31.739348 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730407    2557 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 19:39:31.739348 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730410    2557 flags.go:64] FLAG: --v="2"
Sep 16 19:39:31.739348 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730418    2557 flags.go:64] FLAG: --version="false"
Sep 16 19:39:31.739348 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730422    2557 flags.go:64] FLAG: --vmodule=""
Sep 16 19:39:31.739348 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730425    2557 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 19:39:31.739348 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.730429    2557 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 19:39:31.739348 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730595    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 19:39:31.739348 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730600    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 19:39:31.739348 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730603    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 19:39:31.739348 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730606    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 19:39:31.739348 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730609    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 19:39:31.739348 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730611    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 19:39:31.739348 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730614    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 19:39:31.739348 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730617    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 19:39:31.739348 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730619    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 19:39:31.739348 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730622    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 19:39:31.739348 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730624    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 19:39:31.739348 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730627    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 19:39:31.739348 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730629    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 19:39:31.739348 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730632    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 19:39:31.739348 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730634    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 19:39:31.739944 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730637    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 19:39:31.739944 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730639    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 19:39:31.739944 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730642    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 19:39:31.739944 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730646    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 19:39:31.739944 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730651    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 19:39:31.739944 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730654    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 19:39:31.739944 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730657    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 19:39:31.739944 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730662    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 19:39:31.739944 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730665    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 19:39:31.739944 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730668    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 19:39:31.739944 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730671    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 19:39:31.739944 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730674    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 19:39:31.739944 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730676    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 19:39:31.739944 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730679    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 19:39:31.739944 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730682    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 19:39:31.739944 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730685    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 19:39:31.739944 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730687    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 19:39:31.739944 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730690    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 19:39:31.739944 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730693    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 19:39:31.740413 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730695    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 19:39:31.740413 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730698    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 19:39:31.740413 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730700    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 19:39:31.740413 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730703    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 19:39:31.740413 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730706    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 19:39:31.740413 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730709    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 19:39:31.740413 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730711    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 19:39:31.740413 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730714    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 19:39:31.740413 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730716    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 19:39:31.740413 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730720    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 19:39:31.740413 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730724    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 19:39:31.740413 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730727    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 19:39:31.740413 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730730    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 19:39:31.740413 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730733    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 19:39:31.740413 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730736    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 19:39:31.740413 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730738    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 19:39:31.740413 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730741    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 19:39:31.740413 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730745    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 19:39:31.740413 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730747    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 19:39:31.740413 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730750    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 19:39:31.740919 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730756    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 19:39:31.740919 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730759    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 19:39:31.740919 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730761    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 19:39:31.740919 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730764    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 19:39:31.740919 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730767    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 19:39:31.740919 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730769    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 19:39:31.740919 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730772    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 19:39:31.740919 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730774    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 19:39:31.740919 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730777    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 19:39:31.740919 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730779    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 19:39:31.740919 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730782    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 19:39:31.740919 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730785    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 19:39:31.740919 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730788    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 19:39:31.740919 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730791    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 19:39:31.740919 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730793    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 19:39:31.740919 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730796    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 19:39:31.740919 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730799    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 19:39:31.740919 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730801    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 19:39:31.740919 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730804    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 19:39:31.740919 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730806    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 19:39:31.741410 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730809    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 19:39:31.741410 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730811    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 19:39:31.741410 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730814    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 19:39:31.741410 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730817    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 19:39:31.741410 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730819    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 19:39:31.741410 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730822    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 19:39:31.741410 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730824    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 19:39:31.741410 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730827    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 19:39:31.741410 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730829    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 19:39:31.741410 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730832    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 19:39:31.741410 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730847    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 19:39:31.741410 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730850    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 19:39:31.741410 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730855    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 19:39:31.741410 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730857    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 19:39:31.741410 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730860    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 19:39:31.741410 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730862    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 19:39:31.741410 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730865    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 19:39:31.741410 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730868    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 19:39:31.741410 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730870    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 19:39:31.741410 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730873    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 19:39:31.741920 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730876    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 19:39:31.741920 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730878    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 19:39:31.741920 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730881    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 19:39:31.741920 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730883    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 19:39:31.741920 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.730886    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 19:39:31.741920 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.731719    2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 19:39:31.741920 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.739049    2557 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 19:39:31.741920 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.739068    2557 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 19:39:31.741920 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.739088    2557 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 19:39:31.741920 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.739094    2557 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 19:39:31.741920 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739150    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 19:39:31.741920 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739154    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 19:39:31.741920 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739157    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 19:39:31.741920 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739160    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 19:39:31.741920 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739163    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 19:39:31.742313 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739165    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 19:39:31.742313 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739168    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 19:39:31.742313 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739172    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 19:39:31.742313 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739174    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 19:39:31.742313 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739177    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 19:39:31.742313 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739180    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 19:39:31.742313 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739182    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 19:39:31.742313 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739186    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 19:39:31.742313 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739189    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 19:39:31.742313 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739192    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 19:39:31.742313 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739195    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 19:39:31.742313 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739198    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 19:39:31.742313 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739200    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 19:39:31.742313 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739203    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 19:39:31.742313 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739206    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 19:39:31.742313 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739208    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 19:39:31.742313 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739211    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 19:39:31.742313 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739214    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 19:39:31.742313 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739217    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 19:39:31.742313 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739220    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 19:39:31.742801 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739222    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 19:39:31.742801 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739225    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 19:39:31.742801 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739227    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 19:39:31.742801 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739231    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 19:39:31.742801 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739234    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 19:39:31.742801 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739237    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 19:39:31.742801 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739240    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 19:39:31.742801 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739243    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 19:39:31.742801 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739246    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 19:39:31.742801 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739248    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 19:39:31.742801 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739251    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 19:39:31.742801 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739254    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 19:39:31.742801 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739256    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 19:39:31.742801 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739259    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 19:39:31.742801 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739261    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 19:39:31.742801 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739264    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 19:39:31.742801 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739267    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 19:39:31.742801 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739269    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 19:39:31.742801 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739272    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 19:39:31.742801 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739274    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 19:39:31.743349 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739277    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 19:39:31.743349 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739280    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 19:39:31.743349 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739283    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 19:39:31.743349 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739285    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 19:39:31.743349 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739288    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 19:39:31.743349 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739290    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 19:39:31.743349 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739292    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 19:39:31.743349 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739296    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 19:39:31.743349 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739300    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 19:39:31.743349 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739303    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 19:39:31.743349 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739306    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 19:39:31.743349 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739309    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 19:39:31.743349 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739311    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 19:39:31.743349 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739315    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 19:39:31.743349 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739319    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 19:39:31.743349 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739322    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 19:39:31.743349 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739327    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 19:39:31.743349 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739330    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 19:39:31.743794 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739333    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 19:39:31.743794 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739336    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 19:39:31.743794 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739339    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 19:39:31.743794 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739341    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 19:39:31.743794 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739344    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 19:39:31.743794 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739346    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 19:39:31.743794 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739349    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 19:39:31.743794 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739352    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 19:39:31.743794 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739355    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 19:39:31.743794 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739357    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 19:39:31.743794 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739360    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 19:39:31.743794 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739363    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 19:39:31.743794 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739366    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 19:39:31.743794 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739369    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 19:39:31.743794 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739371    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 19:39:31.743794 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739374    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 19:39:31.743794 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739377    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 19:39:31.743794 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739380    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 19:39:31.743794 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739382    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 19:39:31.743794 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739385    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 19:39:31.744372 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739388    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 19:39:31.744372 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739390    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 19:39:31.744372 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739393    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 19:39:31.744372 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739396    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 19:39:31.744372 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739398    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 19:39:31.744372 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739401    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 19:39:31.744372 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739403    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 19:39:31.744372 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739406    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 19:39:31.744372 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739408    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 19:39:31.744372 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739411    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 19:39:31.744372 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739414    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 19:39:31.744372 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739418    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 19:39:31.744372 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739421    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 19:39:31.744372 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739423    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 19:39:31.744372 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739426    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 19:39:31.744372 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739428    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 19:39:31.744772 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.739434    2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 19:39:31.744772 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739552    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 19:39:31.744772 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739557    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 19:39:31.744772 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739560    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 19:39:31.744772 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739563    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 19:39:31.744772 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739566    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 19:39:31.744772 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739569    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 19:39:31.744772 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739572    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 19:39:31.744772 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739575    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 19:39:31.744772 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739577    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 19:39:31.744772 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739580    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 19:39:31.744772 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739582    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 19:39:31.744772 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739585    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 19:39:31.744772 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739588    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 19:39:31.744772 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739591    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 19:39:31.745189 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739593    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 19:39:31.745189 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739596    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 19:39:31.745189 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739599    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 19:39:31.745189 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739601    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 19:39:31.745189 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739604    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 19:39:31.745189 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739607    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 19:39:31.745189 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739609    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 19:39:31.745189 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739611    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 19:39:31.745189 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739614    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 19:39:31.745189 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739617    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 19:39:31.745189 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739619    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 19:39:31.745189 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739622    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 19:39:31.745189 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739625    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 19:39:31.745189 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739629    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 19:39:31.745189 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739632    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 19:39:31.745189 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739635    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 19:39:31.745189 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739638    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 19:39:31.745189 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739641    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 19:39:31.745189 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739643    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 19:39:31.745189 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739646    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 19:39:31.745679 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739649    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 19:39:31.745679 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739652    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 19:39:31.745679 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739655    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 19:39:31.745679 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739657    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 19:39:31.745679 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739660    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 19:39:31.745679 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739662    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 19:39:31.745679 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739665    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 19:39:31.745679 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739667    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 19:39:31.745679 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739670    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 19:39:31.745679 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739673    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 19:39:31.745679 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739676    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 19:39:31.745679 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739679    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 19:39:31.745679 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739681    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 19:39:31.745679 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739684    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 19:39:31.745679 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739686    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 19:39:31.745679 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739689    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 19:39:31.745679 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739691    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 19:39:31.745679 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739694    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 19:39:31.745679 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739696    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 19:39:31.745679 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739699    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 19:39:31.746292 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739701    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 19:39:31.746292 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739704    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 19:39:31.746292 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739707    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 19:39:31.746292 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739709    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 19:39:31.746292 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739712    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 19:39:31.746292 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739716    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 19:39:31.746292 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739719    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 19:39:31.746292 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739722    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 19:39:31.746292 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739724    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 19:39:31.746292 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739727    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 19:39:31.746292 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739729    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 19:39:31.746292 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739732    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 19:39:31.746292 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739734    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 19:39:31.746292 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739737    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 19:39:31.746292 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739740    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 19:39:31.746292 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739742    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 19:39:31.746292 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739745    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 19:39:31.746292 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739747    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 19:39:31.746292 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739750    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 19:39:31.747067 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739753    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 19:39:31.747067 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739755    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 19:39:31.747067 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739758    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 19:39:31.747067 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739760    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 19:39:31.747067 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739763    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 19:39:31.747067 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739765    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 19:39:31.747067 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739768    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 19:39:31.747067 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739770    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 19:39:31.747067 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739773    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 19:39:31.747067 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739777    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 19:39:31.747067 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739780    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 19:39:31.747067 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739783    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 19:39:31.747067 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739786    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 19:39:31.747067 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739789    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 19:39:31.747067 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739792    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 19:39:31.747067 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739795    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 19:39:31.747067 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739798    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 19:39:31.747067 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739800    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 19:39:31.747067 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739803    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 19:39:31.747620 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739805    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 19:39:31.747620 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739808    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 19:39:31.747620 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739810    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 19:39:31.747620 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739813    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 19:39:31.747620 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739816    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 19:39:31.747620 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739818    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 19:39:31.747620 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:31.739821    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 19:39:31.747620 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.739826    2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 19:39:31.747620 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.739952    2557 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 19:39:31.747620 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.741918    2557 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 19:39:31.747620 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.742819    2557 server.go:1019] "Starting client certificate rotation"
Sep 16 19:39:31.747620 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.742867    2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 19:39:31.747620 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.743514    2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 19:39:31.747982 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.745071    2557 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 19:39:31.749995 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.749818    2557 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 19:39:31.761325 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.761307    2557 log.go:25] "Validated CRI v1 runtime API"
Sep 16 19:39:31.763853 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.763828    2557 log.go:25] "Validated CRI v1 image API"
Sep 16 19:39:31.764614 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.764601    2557 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 19:39:31.766786 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.766767    2557 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7fddccff-c2ad-4e74-8865-3a4fd86aff3c:/dev/nvme0n1p3 ff32d6d1-8c4c-4064-8952-f3aa111ceb3a:/dev/nvme0n1p4]
Sep 16 19:39:31.766874 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.766785    2557 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 19:39:31.771801 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.771691    2557 manager.go:216] Machine: {Timestamp:2026-09-16 19:39:31.770548996 +0000 UTC m=+0.295937372 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099102 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2bcb9f437224ddf3eef95208ee7828 SystemUUID:ec2bcb9f-4372-24dd-f3ee-f95208ee7828 BootID:86a2f7ee-e6d8-4148-a5ba-ff974fd77a86 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d9:7b:93:cd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d9:7b:93:cd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:82:98:99:e4:42:bd Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 19:39:31.771801 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.771799    2557 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 19:39:31.771936 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.771924    2557 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 19:39:31.774566 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.774544    2557 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 19:39:31.774695 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.774569    2557 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-81.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 19:39:31.775202 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.775188    2557 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 19:39:31.775241 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.775209    2557 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 19:39:31.775241 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.775222    2557 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 19:39:31.775773 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.775760    2557 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 19:39:31.776627 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.776617    2557 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 19:39:31.780160 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.780149    2557 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 19:39:31.780279 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.780269    2557 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 19:39:31.780309 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.780298    2557 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 19:39:31.780309 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.780308    2557 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 19:39:31.780359 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.780322    2557 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 19:39:31.780359 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.780331    2557 kubelet.go:398] "Adding apiserver pod source"
Sep 16 19:39:31.780359 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.780346    2557 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 19:39:31.780652 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.780639    2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 19:39:31.780697 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.780664    2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 19:39:31.781236 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.781211    2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 19:39:31.782404 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.782388    2557 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 19:39:31.784547 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.784531    2557 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 19:39:31.784599 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.784564    2557 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 19:39:31.786773 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.786753    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 19:39:31.786865 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.786788    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 19:39:31.786865 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.786821    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 19:39:31.786865 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.786855    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 19:39:31.786978 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.786875    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 19:39:31.786978 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.786890    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 19:39:31.786978 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.786930    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 19:39:31.786978 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.786942    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 19:39:31.787136 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.786979    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 19:39:31.787136 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.786996    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 19:39:31.787136 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.787016    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 19:39:31.787136 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.787031    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 19:39:31.787883 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.787762    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 19:39:31.787883 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.787786    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 19:39:31.792359 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.792344    2557 server.go:1268] "Started kubelet"
Sep 16 19:39:31.792954 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.792438    2557 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 19:39:31.792954 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.792514    2557 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 19:39:31.792954 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.792623    2557 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 19:39:31.793158 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.793092    2557 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-81.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 19:39:31.793158 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:31.793102    2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-81.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 19:39:31.793254 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:31.793214    2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 19:39:31.793456 ip-10-0-139-81 systemd[1]: Started Kubernetes Kubelet.
Sep 16 19:39:31.797433 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.797419    2557 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 19:39:31.798074 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.798057    2557 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 19:39:31.799527 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:31.798568    2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-81.ec2.internal.18d5e46a3721ca2d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-81.ec2.internal,UID:ip-10-0-139-81.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-81.ec2.internal,},FirstTimestamp:2026-09-16 19:39:31.792267821 +0000 UTC m=+0.317656199,LastTimestamp:2026-09-16 19:39:31.792267821 +0000 UTC m=+0.317656199,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-81.ec2.internal,}"
Sep 16 19:39:31.800662 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:31.800646    2557 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 19:39:31.803034 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.803020    2557 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 19:39:31.803081 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.803033    2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 19:39:31.803197 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:31.803181    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-81.ec2.internal\" not found"
Sep 16 19:39:31.803446 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.803411    2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hcfgj"
Sep 16 19:39:31.803889 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.803871    2557 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 19:39:31.803998 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.803891    2557 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 19:39:31.803998 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.803905    2557 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 19:39:31.804112 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.804033    2557 factory.go:153] Registering CRI-O factory
Sep 16 19:39:31.804112 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.804051    2557 factory.go:223] Registration of the crio container factory successfully
Sep 16 19:39:31.804112 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.804036    2557 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 19:39:31.804112 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.804087    2557 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 19:39:31.804112 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.804104    2557 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 19:39:31.804112 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.804112    2557 factory.go:55] Registering systemd factory
Sep 16 19:39:31.804112 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.804118    2557 factory.go:223] Registration of the systemd container factory successfully
Sep 16 19:39:31.804403 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.804129    2557 factory.go:103] Registering Raw factory
Sep 16 19:39:31.804403 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.804140    2557 manager.go:1196] Started watching for new ooms in manager
Sep 16 19:39:31.804599 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.804583    2557 manager.go:349] Starting recovery of all containers
Sep 16 19:39:31.808942 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:31.808912    2557 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-81.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 19:39:31.809095 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:31.809070    2557 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 19:39:31.814473 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.814450    2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hcfgj"
Sep 16 19:39:31.817371 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.817351    2557 manager.go:354] Recovery completed
Sep 16 19:39:31.823487 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.823472    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 19:39:31.825368 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.825347    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-81.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 19:39:31.825435 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.825387    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-81.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 19:39:31.825435 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.825414    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-81.ec2.internal" event="NodeHasSufficientPID"
Sep 16 19:39:31.825959 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.825945    2557 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 19:39:31.825959 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.825960    2557 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 19:39:31.826096 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.825976    2557 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 19:39:31.826096 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.825948    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 19:39:31.826752 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.826736    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-81.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 19:39:31.826817 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.826767    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-81.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 19:39:31.826817 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.826782    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-81.ec2.internal" event="NodeHasSufficientPID"
Sep 16 19:39:31.827232 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:31.827169    2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-81.ec2.internal.18d5e46a391adef3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-81.ec2.internal,UID:ip-10-0-139-81.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-81.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-81.ec2.internal,},FirstTimestamp:2026-09-16 19:39:31.825368819 +0000 UTC m=+0.350757202,LastTimestamp:2026-09-16 19:39:31.825368819 +0000 UTC m=+0.350757202,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-81.ec2.internal,}"
Sep 16 19:39:31.827479 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.827465    2557 policy_none.go:49] "None policy: Start"
Sep 16 19:39:31.827549 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.827483    2557 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 19:39:31.827549 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.827496    2557 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 19:39:31.828658 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.828640    2557 policy_none.go:47] "Start"
Sep 16 19:39:31.869447 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.869432    2557 manager.go:337] "Starting Device Plugin manager"
Sep 16 19:39:31.875441 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:31.869470    2557 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 19:39:31.875441 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.869484    2557 server.go:87] "Starting device plugin registration server"
Sep 16 19:39:31.875441 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.869815    2557 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 19:39:31.875441 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.869827    2557 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 19:39:31.875441 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.869988    2557 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 19:39:31.875441 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.870077    2557 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 19:39:31.875441 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.870085    2557 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 19:39:31.875441 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:31.870607    2557 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 19:39:31.875441 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:31.870662    2557 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-81.ec2.internal\" not found"
Sep 16 19:39:31.929358 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.929332    2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 19:39:31.930563 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.930542    2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 19:39:31.930648 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.930595    2557 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 19:39:31.930648 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.930628    2557 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 19:39:31.930718 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:31.930675    2557 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 19:39:31.933243 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.933226    2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 19:39:31.970524 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.970495    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 19:39:31.971513 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.971496    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-81.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 19:39:31.971602 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.971524    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-81.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 19:39:31.971602 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.971534    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-81.ec2.internal" event="NodeHasSufficientPID"
Sep 16 19:39:31.971602 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.971557    2557 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-81.ec2.internal"
Sep 16 19:39:31.977911 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:31.977822    2557 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-81.ec2.internal"
Sep 16 19:39:31.977911 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:31.977877    2557 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-81.ec2.internal\": node \"ip-10-0-139-81.ec2.internal\" not found"
Sep 16 19:39:31.989574 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:31.989550    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-81.ec2.internal\" not found"
Sep 16 19:39:32.031366 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:32.031329    2557 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-81.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-81.ec2.internal"]
Sep 16 19:39:32.031488 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:32.031408    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 19:39:32.032403 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:32.032387    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-81.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 19:39:32.032469 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:32.032416    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-81.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 19:39:32.032469 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:32.032428    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-81.ec2.internal" event="NodeHasSufficientPID"
Sep 16 19:39:32.034166 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:32.034154    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 19:39:32.034312 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:32.034285    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-81.ec2.internal"
Sep 16 19:39:32.034350 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:32.034330    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 19:39:32.034885 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:32.034863    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-81.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 19:39:32.034885 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:32.034888    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-81.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 19:39:32.035027 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:32.034896    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-81.ec2.internal" event="NodeHasSufficientPID"
Sep 16 19:39:32.035027 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:32.034948    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-81.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 19:39:32.035027 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:32.034965    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-81.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 19:39:32.035027 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:32.034985    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-81.ec2.internal" event="NodeHasSufficientPID"
Sep 16 19:39:32.036871 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:32.036835    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-81.ec2.internal"
Sep 16 19:39:32.036941 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:32.036886    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 19:39:32.037546 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:32.037527    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-81.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 19:39:32.037625 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:32.037557    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-81.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 19:39:32.037625 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:32.037586    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-81.ec2.internal" event="NodeHasSufficientPID"
Sep 16 19:39:32.053382 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:32.053361    2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-81.ec2.internal\" not found" node="ip-10-0-139-81.ec2.internal"
Sep 16 19:39:32.057958 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:32.057944    2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-81.ec2.internal\" not found" node="ip-10-0-139-81.ec2.internal"
Sep 16 19:39:32.089721 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:32.089703    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-81.ec2.internal\" not found"
Sep 16 19:39:32.106418 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:32.106400    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/38ae68d3fbc6666f2f805ad0321d7dc4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-81.ec2.internal\" (UID: \"38ae68d3fbc6666f2f805ad0321d7dc4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-81.ec2.internal"
Sep 16 19:39:32.106477 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:32.106422    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/38ae68d3fbc6666f2f805ad0321d7dc4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-81.ec2.internal\" (UID: \"38ae68d3fbc6666f2f805ad0321d7dc4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-81.ec2.internal"
Sep 16 19:39:32.106477 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:32.106445    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a47431da3c311f72143c9d580d811b73-config\") pod \"kube-apiserver-proxy-ip-10-0-139-81.ec2.internal\" (UID: \"a47431da3c311f72143c9d580d811b73\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-81.ec2.internal"
Sep 16 19:39:32.190351 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:32.190326    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-81.ec2.internal\" not found"
Sep 16 19:39:32.206721 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:32.206699    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/38ae68d3fbc6666f2f805ad0321d7dc4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-81.ec2.internal\" (UID: \"38ae68d3fbc6666f2f805ad0321d7dc4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-81.ec2.internal"
Sep 16 19:39:32.206810 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:32.206727    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/38ae68d3fbc6666f2f805ad0321d7dc4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-81.ec2.internal\" (UID: \"38ae68d3fbc6666f2f805ad0321d7dc4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-81.ec2.internal"
Sep 16 19:39:32.206810 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:32.206746    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a47431da3c311f72143c9d580d811b73-config\") pod \"kube-apiserver-proxy-ip-10-0-139-81.ec2.internal\" (UID: \"a47431da3c311f72143c9d580d811b73\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-81.ec2.internal"
Sep 16 19:39:32.206810 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:32.206772    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a47431da3c311f72143c9d580d811b73-config\") pod \"kube-apiserver-proxy-ip-10-0-139-81.ec2.internal\" (UID: \"a47431da3c311f72143c9d580d811b73\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-81.ec2.internal"
Sep 16 19:39:32.206810 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:32.206793    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/38ae68d3fbc6666f2f805ad0321d7dc4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-81.ec2.internal\" (UID: \"38ae68d3fbc6666f2f805ad0321d7dc4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-81.ec2.internal"
Sep 16 19:39:32.206810 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:32.206806    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/38ae68d3fbc6666f2f805ad0321d7dc4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-81.ec2.internal\" (UID: \"38ae68d3fbc6666f2f805ad0321d7dc4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-81.ec2.internal"
Sep 16 19:39:32.291131 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:32.291054    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-81.ec2.internal\" not found"
Sep 16 19:39:32.356534 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:32.356498    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-81.ec2.internal"
Sep 16 19:39:32.361543 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:32.361529    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-81.ec2.internal"
Sep 16 19:39:32.391387 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:32.391361    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-81.ec2.internal\" not found"
Sep 16 19:39:32.494921 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:32.494866    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-81.ec2.internal\" not found"
Sep 16 19:39:32.596124 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:32.596052    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-81.ec2.internal\" not found"
Sep 16 19:39:32.696886 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:32.696831    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-81.ec2.internal\" not found"
Sep 16 19:39:32.743196 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:32.743169    2557 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 19:39:32.743640 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:32.743371    2557 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 19:39:32.796988 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:32.796958    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-81.ec2.internal\" not found"
Sep 16 19:39:32.818391 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:32.818355    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 19:34:32 +0000 UTC" deadline="2028-04-21 18:13:53.071907048 +0000 UTC"
Sep 16 19:39:32.818391 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:32.818382    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13990h34m20.253528342s"
Sep 16 19:39:32.897866 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:32.897606    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-81.ec2.internal\" not found"
Sep 16 19:39:32.963112 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:32.963076    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38ae68d3fbc6666f2f805ad0321d7dc4.slice/crio-10934270ac4b35a1cf395eb4eb5c929d3d083324655de6a9bc6f7fc62d545dba WatchSource:0}: Error finding container 10934270ac4b35a1cf395eb4eb5c929d3d083324655de6a9bc6f7fc62d545dba: Status 404 returned error can't find the container with id 10934270ac4b35a1cf395eb4eb5c929d3d083324655de6a9bc6f7fc62d545dba
Sep 16 19:39:32.963978 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:32.963954    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda47431da3c311f72143c9d580d811b73.slice/crio-640f7a3bdca81598869d363564d7f7a7e603f10334b13a726bf057078d17bf59 WatchSource:0}: Error finding container 640f7a3bdca81598869d363564d7f7a7e603f10334b13a726bf057078d17bf59: Status 404 returned error can't find the container with id 640f7a3bdca81598869d363564d7f7a7e603f10334b13a726bf057078d17bf59
Sep 16 19:39:32.969390 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:32.969372    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:39:32.998768 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:32.998736    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-81.ec2.internal\" not found"
Sep 16 19:39:33.104963 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:33.104916    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-81.ec2.internal\" not found"
Sep 16 19:39:33.205593 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:33.205498    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-81.ec2.internal\" not found"
Sep 16 19:39:33.293163 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.293133    2557 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 19:39:33.306378 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:33.306350    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-81.ec2.internal\" not found"
Sep 16 19:39:33.368457 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.368426    2557 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 19:39:33.388804 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.388774    2557 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 19:39:33.403739 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.403477    2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-81.ec2.internal"
Sep 16 19:39:33.418896 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.418867    2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 19:39:33.420055 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.419835    2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-81.ec2.internal"
Sep 16 19:39:33.428360 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.428336    2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 19:39:33.781836 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.781805    2557 apiserver.go:52] "Watching apiserver"
Sep 16 19:39:33.790985 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.790956    2557 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 19:39:33.791386 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.791359    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-drgl8","openshift-network-diagnostics/network-check-target-2cjmm","openshift-ovn-kubernetes/ovnkube-node-qbv6k","kube-system/konnectivity-agent-v57p4","kube-system/kube-apiserver-proxy-ip-10-0-139-81.ec2.internal","openshift-cluster-node-tuning-operator/tuned-shdd9","openshift-dns/node-resolver-clfb8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-81.ec2.internal","openshift-multus/multus-58kk4","openshift-multus/multus-additional-cni-plugins-xv4vd","openshift-network-operator/iptables-alerter-vqqgj","kube-system/global-pull-secret-syncer-qpp69","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7cdd","openshift-image-registry/node-ca-hhvcc"]
Sep 16 19:39:33.792966 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.792940    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-clfb8"
Sep 16 19:39:33.794188 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.794165    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vqqgj"
Sep 16 19:39:33.795930 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.795779    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 19:39:33.795930 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.795791    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wp6dc\""
Sep 16 19:39:33.795930 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.795791    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 19:39:33.796151 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.796120    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.797693 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.797071    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 19:39:33.797693 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.797095    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 19:39:33.797693 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.797071    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-65kkm\""
Sep 16 19:39:33.797693 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.797413    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 19:39:33.797693 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.797633    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v57p4"
Sep 16 19:39:33.799018 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.799000    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 19:39:33.799972 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.799252    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hhvcc"
Sep 16 19:39:33.799972 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.799358    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 19:39:33.799972 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.799589    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 19:39:33.799972 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.799659    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 19:39:33.799972 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.799798    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 19:39:33.799972 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.799826    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wlxtx\""
Sep 16 19:39:33.800365 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.800006    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 19:39:33.801468 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.800756    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-phw22\""
Sep 16 19:39:33.801468 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.800842    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 19:39:33.801468 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.801029    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 19:39:33.801468 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.801155    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-shdd9"
Sep 16 19:39:33.802619 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.802223    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-j8thk\""
Sep 16 19:39:33.802619 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.802413    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 19:39:33.802619 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.802487    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 19:39:33.803230 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.802740    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 19:39:33.803230 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.803072    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-58kk4"
Sep 16 19:39:33.803745 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.803721    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 19:39:33.803969 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.803952    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-q7qnw\""
Sep 16 19:39:33.804183 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.804167    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 19:39:33.804367 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.804344    2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 19:39:33.805090 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.805069    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7cdd"
Sep 16 19:39:33.805614 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.805595    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 19:39:33.805930 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.805887    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 19:39:33.806249 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.806171    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 19:39:33.806249 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.806198    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-kdbbc\""
Sep 16 19:39:33.807683 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.807305    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 19:39:33.807683 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.807430    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2cjmm"
Sep 16 19:39:33.807683 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:33.807543    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2cjmm" podUID="b9840a22-6dcc-4836-b682-d1d667d2e30f"
Sep 16 19:39:33.809100 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.808427    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 19:39:33.809100 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.808585    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 19:39:33.809855 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.809307    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wdgn8\""
Sep 16 19:39:33.809855 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.809779    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 19:39:33.814146 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.814124    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xv4vd"
Sep 16 19:39:33.815903 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.815727    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/98bd8e7f-1aa8-418b-ae78-208cbfb13392-host-kubelet\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.815903 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.815776    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/98bd8e7f-1aa8-418b-ae78-208cbfb13392-ovnkube-script-lib\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.815903 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.815805    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/142b9c47-4826-4fa8-b228-5500d98fee67-hosts-file\") pod \"node-resolver-clfb8\" (UID: \"142b9c47-4826-4fa8-b228-5500d98fee67\") " pod="openshift-dns/node-resolver-clfb8"
Sep 16 19:39:33.815903 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.815814    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-drgl8"
Sep 16 19:39:33.815903 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.815881    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmdth\" (UniqueName: \"kubernetes.io/projected/98bd8e7f-1aa8-418b-ae78-208cbfb13392-kube-api-access-fmdth\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.815903 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:33.815889    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-drgl8" podUID="b4ea9fa1-48c7-45be-bfcf-4594ee751086"
Sep 16 19:39:33.816233 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.815919    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0e9a620d-8ac5-4f5d-8e0f-f172664bc35e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j7cdd\" (UID: \"0e9a620d-8ac5-4f5d-8e0f-f172664bc35e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7cdd"
Sep 16 19:39:33.816233 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.815948    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0e9a620d-8ac5-4f5d-8e0f-f172664bc35e-socket-dir\") pod \"aws-ebs-csi-driver-node-j7cdd\" (UID: \"0e9a620d-8ac5-4f5d-8e0f-f172664bc35e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7cdd"
Sep 16 19:39:33.816233 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.815979    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0e9a620d-8ac5-4f5d-8e0f-f172664bc35e-registration-dir\") pod \"aws-ebs-csi-driver-node-j7cdd\" (UID: \"0e9a620d-8ac5-4f5d-8e0f-f172664bc35e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7cdd"
Sep 16 19:39:33.816233 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.816004    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0e9a620d-8ac5-4f5d-8e0f-f172664bc35e-etc-selinux\") pod \"aws-ebs-csi-driver-node-j7cdd\" (UID: \"0e9a620d-8ac5-4f5d-8e0f-f172664bc35e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7cdd"
Sep 16 19:39:33.816233 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.816029    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0e9a620d-8ac5-4f5d-8e0f-f172664bc35e-sys-fs\") pod \"aws-ebs-csi-driver-node-j7cdd\" (UID: \"0e9a620d-8ac5-4f5d-8e0f-f172664bc35e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7cdd"
Sep 16 19:39:33.816233 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.816054    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98bd8e7f-1aa8-418b-ae78-208cbfb13392-etc-openvswitch\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.816233 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.816082    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/43cea22a-1cba-450a-8a7d-86beb5de6d75-system-cni-dir\") pod \"multus-58kk4\" (UID: \"43cea22a-1cba-450a-8a7d-86beb5de6d75\") " pod="openshift-multus/multus-58kk4"
Sep 16 19:39:33.816233 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.816110    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22-etc-kubernetes\") pod \"tuned-shdd9\" (UID: \"9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22\") " pod="openshift-cluster-node-tuning-operator/tuned-shdd9"
Sep 16 19:39:33.816233 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.816134    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22-etc-sysctl-d\") pod \"tuned-shdd9\" (UID: \"9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22\") " pod="openshift-cluster-node-tuning-operator/tuned-shdd9"
Sep 16 19:39:33.816233 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.816157    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf824\" (UniqueName: \"kubernetes.io/projected/9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22-kube-api-access-rf824\") pod \"tuned-shdd9\" (UID: \"9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22\") " pod="openshift-cluster-node-tuning-operator/tuned-shdd9"
Sep 16 19:39:33.816233 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.816180    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/98bd8e7f-1aa8-418b-ae78-208cbfb13392-host-run-netns\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.816233 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.816203    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98bd8e7f-1aa8-418b-ae78-208cbfb13392-run-openvswitch\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.816621 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.816253    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bb2436af-e07f-464b-8973-1b0370888a65-host\") pod \"node-ca-hhvcc\" (UID: \"bb2436af-e07f-464b-8973-1b0370888a65\") " pod="openshift-image-registry/node-ca-hhvcc"
Sep 16 19:39:33.816621 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.816304    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/98bd8e7f-1aa8-418b-ae78-208cbfb13392-run-ovn\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.816621 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.816337    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/98bd8e7f-1aa8-418b-ae78-208cbfb13392-env-overrides\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.816621 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.816369    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bb2436af-e07f-464b-8973-1b0370888a65-serviceca\") pod \"node-ca-hhvcc\" (UID: \"bb2436af-e07f-464b-8973-1b0370888a65\") " pod="openshift-image-registry/node-ca-hhvcc"
Sep 16 19:39:33.816621 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.816413    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/43cea22a-1cba-450a-8a7d-86beb5de6d75-multus-daemon-config\") pod \"multus-58kk4\" (UID: \"43cea22a-1cba-450a-8a7d-86beb5de6d75\") " pod="openshift-multus/multus-58kk4"
Sep 16 19:39:33.816621 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.816443    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdhvd\" (UniqueName: \"kubernetes.io/projected/43cea22a-1cba-450a-8a7d-86beb5de6d75-kube-api-access-wdhvd\") pod \"multus-58kk4\" (UID: \"43cea22a-1cba-450a-8a7d-86beb5de6d75\") " pod="openshift-multus/multus-58kk4"
Sep 16 19:39:33.816621 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.816470    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sfdb\" (UniqueName: \"kubernetes.io/projected/ed745f6b-93e2-416c-a9b3-9acc437b3d9d-kube-api-access-9sfdb\") pod \"iptables-alerter-vqqgj\" (UID: \"ed745f6b-93e2-416c-a9b3-9acc437b3d9d\") " pod="openshift-network-operator/iptables-alerter-vqqgj"
Sep 16 19:39:33.816621 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.816525    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22-etc-sysconfig\") pod \"tuned-shdd9\" (UID: \"9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22\") " pod="openshift-cluster-node-tuning-operator/tuned-shdd9"
Sep 16 19:39:33.816621 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.816552    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22-sys\") pod \"tuned-shdd9\" (UID: \"9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22\") " pod="openshift-cluster-node-tuning-operator/tuned-shdd9"
Sep 16 19:39:33.816621 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.816579    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hctdf\" (UniqueName: \"kubernetes.io/projected/142b9c47-4826-4fa8-b228-5500d98fee67-kube-api-access-hctdf\") pod \"node-resolver-clfb8\" (UID: \"142b9c47-4826-4fa8-b228-5500d98fee67\") " pod="openshift-dns/node-resolver-clfb8"
Sep 16 19:39:33.816621 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.816604    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/98bd8e7f-1aa8-418b-ae78-208cbfb13392-node-log\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.817125 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.816629    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/43cea22a-1cba-450a-8a7d-86beb5de6d75-multus-socket-dir-parent\") pod \"multus-58kk4\" (UID: \"43cea22a-1cba-450a-8a7d-86beb5de6d75\") " pod="openshift-multus/multus-58kk4"
Sep 16 19:39:33.817125 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.816655    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/43cea22a-1cba-450a-8a7d-86beb5de6d75-host-run-netns\") pod \"multus-58kk4\" (UID: \"43cea22a-1cba-450a-8a7d-86beb5de6d75\") " pod="openshift-multus/multus-58kk4"
Sep 16 19:39:33.817125 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.816679    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/43cea22a-1cba-450a-8a7d-86beb5de6d75-host-run-multus-certs\") pod \"multus-58kk4\" (UID: \"43cea22a-1cba-450a-8a7d-86beb5de6d75\") " pod="openshift-multus/multus-58kk4"
Sep 16 19:39:33.817125 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.816686    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 19:39:33.817125 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.816707    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/98bd8e7f-1aa8-418b-ae78-208cbfb13392-host-cni-bin\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.817125 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.816741    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22-etc-systemd\") pod \"tuned-shdd9\" (UID: \"9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22\") " pod="openshift-cluster-node-tuning-operator/tuned-shdd9"
Sep 16 19:39:33.817125 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.816773    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22-host\") pod \"tuned-shdd9\" (UID: \"9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22\") " pod="openshift-cluster-node-tuning-operator/tuned-shdd9"
Sep 16 19:39:33.817125 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.816798    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/43cea22a-1cba-450a-8a7d-86beb5de6d75-host-var-lib-kubelet\") pod \"multus-58kk4\" (UID: \"43cea22a-1cba-450a-8a7d-86beb5de6d75\") " pod="openshift-multus/multus-58kk4"
Sep 16 19:39:33.817125 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.816823    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/98bd8e7f-1aa8-418b-ae78-208cbfb13392-systemd-units\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.817125 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.816836    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rdb8s\""
Sep 16 19:39:33.817125 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.816852    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98bd8e7f-1aa8-418b-ae78-208cbfb13392-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.817125 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.816895    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22-tmp\") pod \"tuned-shdd9\" (UID: \"9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22\") " pod="openshift-cluster-node-tuning-operator/tuned-shdd9"
Sep 16 19:39:33.817125 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.816942    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/43cea22a-1cba-450a-8a7d-86beb5de6d75-host-var-lib-cni-multus\") pod \"multus-58kk4\" (UID: \"43cea22a-1cba-450a-8a7d-86beb5de6d75\") " pod="openshift-multus/multus-58kk4"
Sep 16 19:39:33.817125 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.816989    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/98bd8e7f-1aa8-418b-ae78-208cbfb13392-host-slash\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.817125 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.817036    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/43cea22a-1cba-450a-8a7d-86beb5de6d75-host-var-lib-cni-bin\") pod \"multus-58kk4\" (UID: \"43cea22a-1cba-450a-8a7d-86beb5de6d75\") " pod="openshift-multus/multus-58kk4"
Sep 16 19:39:33.817125 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.817075    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 19:39:33.817125 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.817125    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22-etc-sysctl-conf\") pod \"tuned-shdd9\" (UID: \"9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22\") " pod="openshift-cluster-node-tuning-operator/tuned-shdd9"
Sep 16 19:39:33.817976 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.817152    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22-run\") pod \"tuned-shdd9\" (UID: \"9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22\") " pod="openshift-cluster-node-tuning-operator/tuned-shdd9"
Sep 16 19:39:33.817976 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.817202    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22-etc-tuned\") pod \"tuned-shdd9\" (UID: \"9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22\") " pod="openshift-cluster-node-tuning-operator/tuned-shdd9"
Sep 16 19:39:33.817976 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.817230    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/43cea22a-1cba-450a-8a7d-86beb5de6d75-hostroot\") pod \"multus-58kk4\" (UID: \"43cea22a-1cba-450a-8a7d-86beb5de6d75\") " pod="openshift-multus/multus-58kk4"
Sep 16 19:39:33.817976 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.817256    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/142b9c47-4826-4fa8-b228-5500d98fee67-tmp-dir\") pod \"node-resolver-clfb8\" (UID: \"142b9c47-4826-4fa8-b228-5500d98fee67\") " pod="openshift-dns/node-resolver-clfb8"
Sep 16 19:39:33.817976 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.817289    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/98bd8e7f-1aa8-418b-ae78-208cbfb13392-ovnkube-config\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.817976 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.817303    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/43cea22a-1cba-450a-8a7d-86beb5de6d75-cni-binary-copy\") pod \"multus-58kk4\" (UID: \"43cea22a-1cba-450a-8a7d-86beb5de6d75\") " pod="openshift-multus/multus-58kk4"
Sep 16 19:39:33.817976 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.817318    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/43cea22a-1cba-450a-8a7d-86beb5de6d75-host-run-k8s-cni-cncf-io\") pod \"multus-58kk4\" (UID: \"43cea22a-1cba-450a-8a7d-86beb5de6d75\") " pod="openshift-multus/multus-58kk4"
Sep 16 19:39:33.817976 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.817342    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ed745f6b-93e2-416c-a9b3-9acc437b3d9d-iptables-alerter-script\") pod \"iptables-alerter-vqqgj\" (UID: \"ed745f6b-93e2-416c-a9b3-9acc437b3d9d\") " pod="openshift-network-operator/iptables-alerter-vqqgj"
Sep 16 19:39:33.817976 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.817362    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98bd8e7f-1aa8-418b-ae78-208cbfb13392-host-run-ovn-kubernetes\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.817976 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.817377    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/98bd8e7f-1aa8-418b-ae78-208cbfb13392-ovn-node-metrics-cert\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.817976 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.817391    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vqmg\" (UniqueName: \"kubernetes.io/projected/bb2436af-e07f-464b-8973-1b0370888a65-kube-api-access-5vqmg\") pod \"node-ca-hhvcc\" (UID: \"bb2436af-e07f-464b-8973-1b0370888a65\") " pod="openshift-image-registry/node-ca-hhvcc"
Sep 16 19:39:33.817976 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.817423    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0e9a620d-8ac5-4f5d-8e0f-f172664bc35e-device-dir\") pod \"aws-ebs-csi-driver-node-j7cdd\" (UID: \"0e9a620d-8ac5-4f5d-8e0f-f172664bc35e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7cdd"
Sep 16 19:39:33.817976 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.817452    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22-etc-modprobe-d\") pod \"tuned-shdd9\" (UID: \"9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22\") " pod="openshift-cluster-node-tuning-operator/tuned-shdd9"
Sep 16 19:39:33.817976 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.817479    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22-lib-modules\") pod \"tuned-shdd9\" (UID: \"9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22\") " pod="openshift-cluster-node-tuning-operator/tuned-shdd9"
Sep 16 19:39:33.817976 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.817498    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22-var-lib-kubelet\") pod \"tuned-shdd9\" (UID: \"9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22\") " pod="openshift-cluster-node-tuning-operator/tuned-shdd9"
Sep 16 19:39:33.817976 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.817526    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/43cea22a-1cba-450a-8a7d-86beb5de6d75-multus-cni-dir\") pod \"multus-58kk4\" (UID: \"43cea22a-1cba-450a-8a7d-86beb5de6d75\") " pod="openshift-multus/multus-58kk4"
Sep 16 19:39:33.817976 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.817549    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/98bd8e7f-1aa8-418b-ae78-208cbfb13392-run-systemd\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.818799 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.817573    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98bd8e7f-1aa8-418b-ae78-208cbfb13392-var-lib-openvswitch\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.818799 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.817589    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgvpk\" (UniqueName: \"kubernetes.io/projected/0e9a620d-8ac5-4f5d-8e0f-f172664bc35e-kube-api-access-vgvpk\") pod \"aws-ebs-csi-driver-node-j7cdd\" (UID: \"0e9a620d-8ac5-4f5d-8e0f-f172664bc35e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7cdd"
Sep 16 19:39:33.818799 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.817603    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/43cea22a-1cba-450a-8a7d-86beb5de6d75-cnibin\") pod \"multus-58kk4\" (UID: \"43cea22a-1cba-450a-8a7d-86beb5de6d75\") " pod="openshift-multus/multus-58kk4"
Sep 16 19:39:33.818799 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.817628    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/98bd8e7f-1aa8-418b-ae78-208cbfb13392-log-socket\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.818799 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.817642    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/98bd8e7f-1aa8-418b-ae78-208cbfb13392-host-cni-netd\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.818799 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.817659    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/85c2f2c0-d2a8-454e-9e28-498c0c8d3c21-agent-certs\") pod \"konnectivity-agent-v57p4\" (UID: \"85c2f2c0-d2a8-454e-9e28-498c0c8d3c21\") " pod="kube-system/konnectivity-agent-v57p4"
Sep 16 19:39:33.818799 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.817673    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/85c2f2c0-d2a8-454e-9e28-498c0c8d3c21-konnectivity-ca\") pod \"konnectivity-agent-v57p4\" (UID: \"85c2f2c0-d2a8-454e-9e28-498c0c8d3c21\") " pod="kube-system/konnectivity-agent-v57p4"
Sep 16 19:39:33.818799 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.817687    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/43cea22a-1cba-450a-8a7d-86beb5de6d75-multus-conf-dir\") pod \"multus-58kk4\" (UID: \"43cea22a-1cba-450a-8a7d-86beb5de6d75\") " pod="openshift-multus/multus-58kk4"
Sep 16 19:39:33.818799 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.817701    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/43cea22a-1cba-450a-8a7d-86beb5de6d75-etc-kubernetes\") pod \"multus-58kk4\" (UID: \"43cea22a-1cba-450a-8a7d-86beb5de6d75\") " pod="openshift-multus/multus-58kk4"
Sep 16 19:39:33.818799 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.817728    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ed745f6b-93e2-416c-a9b3-9acc437b3d9d-host-slash\") pod \"iptables-alerter-vqqgj\" (UID: \"ed745f6b-93e2-416c-a9b3-9acc437b3d9d\") " pod="openshift-network-operator/iptables-alerter-vqqgj"
Sep 16 19:39:33.818799 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.818178    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qpp69"
Sep 16 19:39:33.818799 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:33.818245    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qpp69" podUID="3428bc6c-ab5c-47f1-8f1d-f5923fdacfce"
Sep 16 19:39:33.824484 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.824464    2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 19:39:33.847064 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.847037    2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-95txk"
Sep 16 19:39:33.855605 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.855585    2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-95txk"
Sep 16 19:39:33.905381 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.905347    2557 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 19:39:33.917960 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.917936    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/98bd8e7f-1aa8-418b-ae78-208cbfb13392-host-run-netns\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.918111 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.917968    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98bd8e7f-1aa8-418b-ae78-208cbfb13392-run-openvswitch\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.918111 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.918001    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bb2436af-e07f-464b-8973-1b0370888a65-host\") pod \"node-ca-hhvcc\" (UID: \"bb2436af-e07f-464b-8973-1b0370888a65\") " pod="openshift-image-registry/node-ca-hhvcc"
Sep 16 19:39:33.918111 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.918066    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bb2436af-e07f-464b-8973-1b0370888a65-host\") pod \"node-ca-hhvcc\" (UID: \"bb2436af-e07f-464b-8973-1b0370888a65\") " pod="openshift-image-registry/node-ca-hhvcc"
Sep 16 19:39:33.918111 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.918069    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/98bd8e7f-1aa8-418b-ae78-208cbfb13392-host-run-netns\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.918111 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.918072    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98bd8e7f-1aa8-418b-ae78-208cbfb13392-run-openvswitch\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.918111 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.918098    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4ea9fa1-48c7-45be-bfcf-4594ee751086-metrics-certs\") pod \"network-metrics-daemon-drgl8\" (UID: \"b4ea9fa1-48c7-45be-bfcf-4594ee751086\") " pod="openshift-multus/network-metrics-daemon-drgl8"
Sep 16 19:39:33.918405 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.918140    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3428bc6c-ab5c-47f1-8f1d-f5923fdacfce-kubelet-config\") pod \"global-pull-secret-syncer-qpp69\" (UID: \"3428bc6c-ab5c-47f1-8f1d-f5923fdacfce\") " pod="kube-system/global-pull-secret-syncer-qpp69"
Sep 16 19:39:33.918405 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.918172    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/98bd8e7f-1aa8-418b-ae78-208cbfb13392-run-ovn\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.918405 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.918200    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/98bd8e7f-1aa8-418b-ae78-208cbfb13392-env-overrides\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.918405 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.918224    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bb2436af-e07f-464b-8973-1b0370888a65-serviceca\") pod \"node-ca-hhvcc\" (UID: \"bb2436af-e07f-464b-8973-1b0370888a65\") " pod="openshift-image-registry/node-ca-hhvcc"
Sep 16 19:39:33.918405 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.918249    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/43cea22a-1cba-450a-8a7d-86beb5de6d75-multus-daemon-config\") pod \"multus-58kk4\" (UID: \"43cea22a-1cba-450a-8a7d-86beb5de6d75\") " pod="openshift-multus/multus-58kk4"
Sep 16 19:39:33.918405 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.918253    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/98bd8e7f-1aa8-418b-ae78-208cbfb13392-run-ovn\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.918405 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.918274    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdhvd\" (UniqueName: \"kubernetes.io/projected/43cea22a-1cba-450a-8a7d-86beb5de6d75-kube-api-access-wdhvd\") pod \"multus-58kk4\" (UID: \"43cea22a-1cba-450a-8a7d-86beb5de6d75\") " pod="openshift-multus/multus-58kk4"
Sep 16 19:39:33.918405 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.918301    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sfdb\" (UniqueName: \"kubernetes.io/projected/ed745f6b-93e2-416c-a9b3-9acc437b3d9d-kube-api-access-9sfdb\") pod \"iptables-alerter-vqqgj\" (UID: \"ed745f6b-93e2-416c-a9b3-9acc437b3d9d\") " pod="openshift-network-operator/iptables-alerter-vqqgj"
Sep 16 19:39:33.918405 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.918329    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fb2ab49a-ef77-4425-91b4-2b6a6ca89630-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xv4vd\" (UID: \"fb2ab49a-ef77-4425-91b4-2b6a6ca89630\") " pod="openshift-multus/multus-additional-cni-plugins-xv4vd"
Sep 16 19:39:33.918405 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.918355    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22-etc-sysconfig\") pod \"tuned-shdd9\" (UID: \"9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22\") " pod="openshift-cluster-node-tuning-operator/tuned-shdd9"
Sep 16 19:39:33.918405 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.918394    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22-sys\") pod \"tuned-shdd9\" (UID: \"9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22\") " pod="openshift-cluster-node-tuning-operator/tuned-shdd9"
Sep 16 19:39:33.918974 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.918421    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hctdf\" (UniqueName: \"kubernetes.io/projected/142b9c47-4826-4fa8-b228-5500d98fee67-kube-api-access-hctdf\") pod \"node-resolver-clfb8\" (UID: \"142b9c47-4826-4fa8-b228-5500d98fee67\") " pod="openshift-dns/node-resolver-clfb8"
Sep 16 19:39:33.918974 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.918445    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/98bd8e7f-1aa8-418b-ae78-208cbfb13392-node-log\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.918974 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.918470    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/43cea22a-1cba-450a-8a7d-86beb5de6d75-multus-socket-dir-parent\") pod \"multus-58kk4\" (UID: \"43cea22a-1cba-450a-8a7d-86beb5de6d75\") " pod="openshift-multus/multus-58kk4"
Sep 16 19:39:33.918974 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.918495    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/43cea22a-1cba-450a-8a7d-86beb5de6d75-host-run-netns\") pod \"multus-58kk4\" (UID: \"43cea22a-1cba-450a-8a7d-86beb5de6d75\") " pod="openshift-multus/multus-58kk4"
Sep 16 19:39:33.918974 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.918521    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/43cea22a-1cba-450a-8a7d-86beb5de6d75-host-run-multus-certs\") pod \"multus-58kk4\" (UID: \"43cea22a-1cba-450a-8a7d-86beb5de6d75\") " pod="openshift-multus/multus-58kk4"
Sep 16 19:39:33.918974 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.918545    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/98bd8e7f-1aa8-418b-ae78-208cbfb13392-host-cni-bin\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.918974 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.918568    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3428bc6c-ab5c-47f1-8f1d-f5923fdacfce-dbus\") pod \"global-pull-secret-syncer-qpp69\" (UID: \"3428bc6c-ab5c-47f1-8f1d-f5923fdacfce\") " pod="kube-system/global-pull-secret-syncer-qpp69"
Sep 16 19:39:33.918974 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.918598    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22-etc-systemd\") pod \"tuned-shdd9\" (UID: \"9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22\") " pod="openshift-cluster-node-tuning-operator/tuned-shdd9"
Sep 16 19:39:33.918974 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.918626    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22-host\") pod \"tuned-shdd9\" (UID: \"9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22\") " pod="openshift-cluster-node-tuning-operator/tuned-shdd9"
Sep 16 19:39:33.918974 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.918653    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/43cea22a-1cba-450a-8a7d-86beb5de6d75-host-var-lib-kubelet\") pod \"multus-58kk4\" (UID: \"43cea22a-1cba-450a-8a7d-86beb5de6d75\") " pod="openshift-multus/multus-58kk4"
Sep 16 19:39:33.918974 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.918680    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/98bd8e7f-1aa8-418b-ae78-208cbfb13392-systemd-units\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.918974 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.918709    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98bd8e7f-1aa8-418b-ae78-208cbfb13392-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.918974 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.918744    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb2ab49a-ef77-4425-91b4-2b6a6ca89630-system-cni-dir\") pod \"multus-additional-cni-plugins-xv4vd\" (UID: \"fb2ab49a-ef77-4425-91b4-2b6a6ca89630\") " pod="openshift-multus/multus-additional-cni-plugins-xv4vd"
Sep 16 19:39:33.918974 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.918770    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9clm\" (UniqueName: \"kubernetes.io/projected/b4ea9fa1-48c7-45be-bfcf-4594ee751086-kube-api-access-c9clm\") pod \"network-metrics-daemon-drgl8\" (UID: \"b4ea9fa1-48c7-45be-bfcf-4594ee751086\") " pod="openshift-multus/network-metrics-daemon-drgl8"
Sep 16 19:39:33.918974 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.918796    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22-tmp\") pod \"tuned-shdd9\" (UID: \"9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22\") " pod="openshift-cluster-node-tuning-operator/tuned-shdd9"
Sep 16 19:39:33.918974 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.918803    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bb2436af-e07f-464b-8973-1b0370888a65-serviceca\") pod \"node-ca-hhvcc\" (UID: \"bb2436af-e07f-464b-8973-1b0370888a65\") " pod="openshift-image-registry/node-ca-hhvcc"
Sep 16 19:39:33.918974 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.918824    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/43cea22a-1cba-450a-8a7d-86beb5de6d75-host-var-lib-cni-multus\") pod \"multus-58kk4\" (UID: \"43cea22a-1cba-450a-8a7d-86beb5de6d75\") " pod="openshift-multus/multus-58kk4"
Sep 16 19:39:33.919854 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.918852    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/98bd8e7f-1aa8-418b-ae78-208cbfb13392-host-slash\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.919854 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.918880    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fb2ab49a-ef77-4425-91b4-2b6a6ca89630-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xv4vd\" (UID: \"fb2ab49a-ef77-4425-91b4-2b6a6ca89630\") " pod="openshift-multus/multus-additional-cni-plugins-xv4vd"
Sep 16 19:39:33.919854 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.918926    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/43cea22a-1cba-450a-8a7d-86beb5de6d75-host-var-lib-cni-bin\") pod \"multus-58kk4\" (UID: \"43cea22a-1cba-450a-8a7d-86beb5de6d75\") " pod="openshift-multus/multus-58kk4"
Sep 16 19:39:33.919854 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.918957    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22-etc-sysctl-conf\") pod \"tuned-shdd9\" (UID: \"9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22\") " pod="openshift-cluster-node-tuning-operator/tuned-shdd9"
Sep 16 19:39:33.919854 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.918980    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22-run\") pod \"tuned-shdd9\" (UID: \"9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22\") " pod="openshift-cluster-node-tuning-operator/tuned-shdd9"
Sep 16 19:39:33.919854 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.919023    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22-etc-tuned\") pod \"tuned-shdd9\" (UID: \"9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22\") " pod="openshift-cluster-node-tuning-operator/tuned-shdd9"
Sep 16 19:39:33.919854 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.919057    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/43cea22a-1cba-450a-8a7d-86beb5de6d75-hostroot\") pod \"multus-58kk4\" (UID: \"43cea22a-1cba-450a-8a7d-86beb5de6d75\") " pod="openshift-multus/multus-58kk4"
Sep 16 19:39:33.919854 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.919082    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/142b9c47-4826-4fa8-b228-5500d98fee67-tmp-dir\") pod \"node-resolver-clfb8\" (UID: \"142b9c47-4826-4fa8-b228-5500d98fee67\") " pod="openshift-dns/node-resolver-clfb8"
Sep 16 19:39:33.919854 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.919119    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/98bd8e7f-1aa8-418b-ae78-208cbfb13392-ovnkube-config\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.919854 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.919146    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/43cea22a-1cba-450a-8a7d-86beb5de6d75-cni-binary-copy\") pod \"multus-58kk4\" (UID: \"43cea22a-1cba-450a-8a7d-86beb5de6d75\") " pod="openshift-multus/multus-58kk4"
Sep 16 19:39:33.919854 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.919171    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/43cea22a-1cba-450a-8a7d-86beb5de6d75-host-run-k8s-cni-cncf-io\") pod \"multus-58kk4\" (UID: \"43cea22a-1cba-450a-8a7d-86beb5de6d75\") " pod="openshift-multus/multus-58kk4"
Sep 16 19:39:33.919854 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.919204    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ed745f6b-93e2-416c-a9b3-9acc437b3d9d-iptables-alerter-script\") pod \"iptables-alerter-vqqgj\" (UID: \"ed745f6b-93e2-416c-a9b3-9acc437b3d9d\") " pod="openshift-network-operator/iptables-alerter-vqqgj"
Sep 16 19:39:33.919854 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.919232    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98bd8e7f-1aa8-418b-ae78-208cbfb13392-host-run-ovn-kubernetes\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.919854 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.919259    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/98bd8e7f-1aa8-418b-ae78-208cbfb13392-ovn-node-metrics-cert\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.919854 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.919285    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vqmg\" (UniqueName: \"kubernetes.io/projected/bb2436af-e07f-464b-8973-1b0370888a65-kube-api-access-5vqmg\") pod \"node-ca-hhvcc\" (UID: \"bb2436af-e07f-464b-8973-1b0370888a65\") " pod="openshift-image-registry/node-ca-hhvcc"
Sep 16 19:39:33.919854 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.919311    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0e9a620d-8ac5-4f5d-8e0f-f172664bc35e-device-dir\") pod \"aws-ebs-csi-driver-node-j7cdd\" (UID: \"0e9a620d-8ac5-4f5d-8e0f-f172664bc35e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7cdd"
Sep 16 19:39:33.919854 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.919314    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/98bd8e7f-1aa8-418b-ae78-208cbfb13392-env-overrides\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.920770 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.919338    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8s6m\" (UniqueName: \"kubernetes.io/projected/fb2ab49a-ef77-4425-91b4-2b6a6ca89630-kube-api-access-q8s6m\") pod \"multus-additional-cni-plugins-xv4vd\" (UID: \"fb2ab49a-ef77-4425-91b4-2b6a6ca89630\") " pod="openshift-multus/multus-additional-cni-plugins-xv4vd"
Sep 16 19:39:33.920770 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.919405    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22-etc-modprobe-d\") pod \"tuned-shdd9\" (UID: \"9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22\") " pod="openshift-cluster-node-tuning-operator/tuned-shdd9"
Sep 16 19:39:33.920770 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.919433    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22-lib-modules\") pod \"tuned-shdd9\" (UID: \"9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22\") " pod="openshift-cluster-node-tuning-operator/tuned-shdd9"
Sep 16 19:39:33.920770 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.919458    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22-var-lib-kubelet\") pod \"tuned-shdd9\" (UID: \"9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22\") " pod="openshift-cluster-node-tuning-operator/tuned-shdd9"
Sep 16 19:39:33.920770 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.919485    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/43cea22a-1cba-450a-8a7d-86beb5de6d75-multus-cni-dir\") pod \"multus-58kk4\" (UID: \"43cea22a-1cba-450a-8a7d-86beb5de6d75\") " pod="openshift-multus/multus-58kk4"
Sep 16 19:39:33.920770 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.919513    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/98bd8e7f-1aa8-418b-ae78-208cbfb13392-run-systemd\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.920770 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.919538    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98bd8e7f-1aa8-418b-ae78-208cbfb13392-var-lib-openvswitch\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.920770 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.919566    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgvpk\" (UniqueName: \"kubernetes.io/projected/0e9a620d-8ac5-4f5d-8e0f-f172664bc35e-kube-api-access-vgvpk\") pod \"aws-ebs-csi-driver-node-j7cdd\" (UID: \"0e9a620d-8ac5-4f5d-8e0f-f172664bc35e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7cdd"
Sep 16 19:39:33.920770 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.919597    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb2ab49a-ef77-4425-91b4-2b6a6ca89630-cnibin\") pod \"multus-additional-cni-plugins-xv4vd\" (UID: \"fb2ab49a-ef77-4425-91b4-2b6a6ca89630\") " pod="openshift-multus/multus-additional-cni-plugins-xv4vd"
Sep 16 19:39:33.920770 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.919631    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/43cea22a-1cba-450a-8a7d-86beb5de6d75-cnibin\") pod \"multus-58kk4\" (UID: \"43cea22a-1cba-450a-8a7d-86beb5de6d75\") " pod="openshift-multus/multus-58kk4"
Sep 16 19:39:33.920770 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.919657    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/98bd8e7f-1aa8-418b-ae78-208cbfb13392-log-socket\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.920770 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.919683    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/98bd8e7f-1aa8-418b-ae78-208cbfb13392-host-cni-netd\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.920770 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.919712    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/85c2f2c0-d2a8-454e-9e28-498c0c8d3c21-agent-certs\") pod \"konnectivity-agent-v57p4\" (UID: \"85c2f2c0-d2a8-454e-9e28-498c0c8d3c21\") " pod="kube-system/konnectivity-agent-v57p4"
Sep 16 19:39:33.920770 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.919738    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/85c2f2c0-d2a8-454e-9e28-498c0c8d3c21-konnectivity-ca\") pod \"konnectivity-agent-v57p4\" (UID: \"85c2f2c0-d2a8-454e-9e28-498c0c8d3c21\") " pod="kube-system/konnectivity-agent-v57p4"
Sep 16 19:39:33.920770 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.919766    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/43cea22a-1cba-450a-8a7d-86beb5de6d75-multus-conf-dir\") pod \"multus-58kk4\" (UID: \"43cea22a-1cba-450a-8a7d-86beb5de6d75\") " pod="openshift-multus/multus-58kk4"
Sep 16 19:39:33.920770 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.919796    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/43cea22a-1cba-450a-8a7d-86beb5de6d75-etc-kubernetes\") pod \"multus-58kk4\" (UID: \"43cea22a-1cba-450a-8a7d-86beb5de6d75\") " pod="openshift-multus/multus-58kk4"
Sep 16 19:39:33.920770 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.919823    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ed745f6b-93e2-416c-a9b3-9acc437b3d9d-host-slash\") pod \"iptables-alerter-vqqgj\" (UID: \"ed745f6b-93e2-416c-a9b3-9acc437b3d9d\") " pod="openshift-network-operator/iptables-alerter-vqqgj"
Sep 16 19:39:33.921619 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.919850    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/98bd8e7f-1aa8-418b-ae78-208cbfb13392-host-kubelet\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.921619 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.919864    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/43cea22a-1cba-450a-8a7d-86beb5de6d75-multus-daemon-config\") pod \"multus-58kk4\" (UID: \"43cea22a-1cba-450a-8a7d-86beb5de6d75\") " pod="openshift-multus/multus-58kk4"
Sep 16 19:39:33.921619 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.919877    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/98bd8e7f-1aa8-418b-ae78-208cbfb13392-ovnkube-script-lib\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.921619 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.919947    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/142b9c47-4826-4fa8-b228-5500d98fee67-hosts-file\") pod \"node-resolver-clfb8\" (UID: \"142b9c47-4826-4fa8-b228-5500d98fee67\") " pod="openshift-dns/node-resolver-clfb8"
Sep 16 19:39:33.921619 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.919981    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmdth\" (UniqueName: \"kubernetes.io/projected/98bd8e7f-1aa8-418b-ae78-208cbfb13392-kube-api-access-fmdth\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.921619 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.920011    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0e9a620d-8ac5-4f5d-8e0f-f172664bc35e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j7cdd\" (UID: \"0e9a620d-8ac5-4f5d-8e0f-f172664bc35e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7cdd"
Sep 16 19:39:33.921619 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.920038    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0e9a620d-8ac5-4f5d-8e0f-f172664bc35e-socket-dir\") pod \"aws-ebs-csi-driver-node-j7cdd\" (UID: \"0e9a620d-8ac5-4f5d-8e0f-f172664bc35e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7cdd"
Sep 16 19:39:33.921619 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.920064    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0e9a620d-8ac5-4f5d-8e0f-f172664bc35e-registration-dir\") pod \"aws-ebs-csi-driver-node-j7cdd\" (UID: \"0e9a620d-8ac5-4f5d-8e0f-f172664bc35e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7cdd"
Sep 16 19:39:33.921619 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.920114    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0e9a620d-8ac5-4f5d-8e0f-f172664bc35e-etc-selinux\") pod \"aws-ebs-csi-driver-node-j7cdd\" (UID: \"0e9a620d-8ac5-4f5d-8e0f-f172664bc35e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7cdd"
Sep 16 19:39:33.921619 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.920149    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0e9a620d-8ac5-4f5d-8e0f-f172664bc35e-sys-fs\") pod \"aws-ebs-csi-driver-node-j7cdd\" (UID: \"0e9a620d-8ac5-4f5d-8e0f-f172664bc35e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7cdd"
Sep 16 19:39:33.921619 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.920191    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fb2ab49a-ef77-4425-91b4-2b6a6ca89630-cni-binary-copy\") pod \"multus-additional-cni-plugins-xv4vd\" (UID: \"fb2ab49a-ef77-4425-91b4-2b6a6ca89630\") " pod="openshift-multus/multus-additional-cni-plugins-xv4vd"
Sep 16 19:39:33.921619 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.920221    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98bd8e7f-1aa8-418b-ae78-208cbfb13392-etc-openvswitch\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.921619 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.920279    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb28t\" (UniqueName: \"kubernetes.io/projected/b9840a22-6dcc-4836-b682-d1d667d2e30f-kube-api-access-xb28t\") pod \"network-check-target-2cjmm\" (UID: \"b9840a22-6dcc-4836-b682-d1d667d2e30f\") " pod="openshift-network-diagnostics/network-check-target-2cjmm"
Sep 16 19:39:33.921619 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.920309    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/43cea22a-1cba-450a-8a7d-86beb5de6d75-system-cni-dir\") pod \"multus-58kk4\" (UID: \"43cea22a-1cba-450a-8a7d-86beb5de6d75\") " pod="openshift-multus/multus-58kk4"
Sep 16 19:39:33.921619 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.920337    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb2ab49a-ef77-4425-91b4-2b6a6ca89630-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xv4vd\" (UID: \"fb2ab49a-ef77-4425-91b4-2b6a6ca89630\") " pod="openshift-multus/multus-additional-cni-plugins-xv4vd"
Sep 16 19:39:33.921619 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.920365    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3428bc6c-ab5c-47f1-8f1d-f5923fdacfce-original-pull-secret\") pod \"global-pull-secret-syncer-qpp69\" (UID: \"3428bc6c-ab5c-47f1-8f1d-f5923fdacfce\") " pod="kube-system/global-pull-secret-syncer-qpp69"
Sep 16 19:39:33.921619 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.920442    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22-etc-kubernetes\") pod \"tuned-shdd9\" (UID: \"9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22\") " pod="openshift-cluster-node-tuning-operator/tuned-shdd9"
Sep 16 19:39:33.922572 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.920486    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22-etc-sysctl-d\") pod \"tuned-shdd9\" (UID: \"9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22\") " pod="openshift-cluster-node-tuning-operator/tuned-shdd9"
Sep 16 19:39:33.922572 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.920494    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/98bd8e7f-1aa8-418b-ae78-208cbfb13392-ovnkube-script-lib\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.922572 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.920531    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf824\" (UniqueName: \"kubernetes.io/projected/9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22-kube-api-access-rf824\") pod \"tuned-shdd9\" (UID: \"9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22\") " pod="openshift-cluster-node-tuning-operator/tuned-shdd9"
Sep 16 19:39:33.922572 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.920647    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/43cea22a-1cba-450a-8a7d-86beb5de6d75-host-var-lib-cni-multus\") pod \"multus-58kk4\" (UID: \"43cea22a-1cba-450a-8a7d-86beb5de6d75\") " pod="openshift-multus/multus-58kk4"
Sep 16 19:39:33.922572 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.920979    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/98bd8e7f-1aa8-418b-ae78-208cbfb13392-host-slash\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.922572 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.921084    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/43cea22a-1cba-450a-8a7d-86beb5de6d75-host-var-lib-cni-bin\") pod \"multus-58kk4\" (UID: \"43cea22a-1cba-450a-8a7d-86beb5de6d75\") " pod="openshift-multus/multus-58kk4"
Sep 16 19:39:33.922572 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.921184    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22-etc-sysconfig\") pod \"tuned-shdd9\" (UID: \"9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22\") " pod="openshift-cluster-node-tuning-operator/tuned-shdd9"
Sep 16 19:39:33.922572 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.921201    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22-etc-sysctl-conf\") pod \"tuned-shdd9\" (UID: \"9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22\") " pod="openshift-cluster-node-tuning-operator/tuned-shdd9"
Sep 16 19:39:33.922572 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.921243    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22-sys\") pod \"tuned-shdd9\" (UID: \"9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22\") " pod="openshift-cluster-node-tuning-operator/tuned-shdd9"
Sep 16 19:39:33.922572 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.921297    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22-run\") pod \"tuned-shdd9\" (UID: \"9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22\") " pod="openshift-cluster-node-tuning-operator/tuned-shdd9"
Sep 16 19:39:33.922572 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.921319    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0e9a620d-8ac5-4f5d-8e0f-f172664bc35e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j7cdd\" (UID: \"0e9a620d-8ac5-4f5d-8e0f-f172664bc35e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7cdd"
Sep 16 19:39:33.922572 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.921306    2557 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 19:39:33.922572 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.921396    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/43cea22a-1cba-450a-8a7d-86beb5de6d75-hostroot\") pod \"multus-58kk4\" (UID: \"43cea22a-1cba-450a-8a7d-86beb5de6d75\") " pod="openshift-multus/multus-58kk4"
Sep 16 19:39:33.922572 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.921621    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0e9a620d-8ac5-4f5d-8e0f-f172664bc35e-socket-dir\") pod \"aws-ebs-csi-driver-node-j7cdd\" (UID: \"0e9a620d-8ac5-4f5d-8e0f-f172664bc35e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7cdd"
Sep 16 19:39:33.922572 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.921720    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/142b9c47-4826-4fa8-b228-5500d98fee67-tmp-dir\") pod \"node-resolver-clfb8\" (UID: \"142b9c47-4826-4fa8-b228-5500d98fee67\") " pod="openshift-dns/node-resolver-clfb8"
Sep 16 19:39:33.922572 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.921717    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0e9a620d-8ac5-4f5d-8e0f-f172664bc35e-registration-dir\") pod \"aws-ebs-csi-driver-node-j7cdd\" (UID: \"0e9a620d-8ac5-4f5d-8e0f-f172664bc35e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7cdd"
Sep 16 19:39:33.922572 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.921815    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0e9a620d-8ac5-4f5d-8e0f-f172664bc35e-etc-selinux\") pod \"aws-ebs-csi-driver-node-j7cdd\" (UID: \"0e9a620d-8ac5-4f5d-8e0f-f172664bc35e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7cdd"
Sep 16 19:39:33.922572 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.921936    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0e9a620d-8ac5-4f5d-8e0f-f172664bc35e-sys-fs\") pod \"aws-ebs-csi-driver-node-j7cdd\" (UID: \"0e9a620d-8ac5-4f5d-8e0f-f172664bc35e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7cdd"
Sep 16 19:39:33.923508 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.922014    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98bd8e7f-1aa8-418b-ae78-208cbfb13392-etc-openvswitch\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.923508 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.922122    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/43cea22a-1cba-450a-8a7d-86beb5de6d75-system-cni-dir\") pod \"multus-58kk4\" (UID: \"43cea22a-1cba-450a-8a7d-86beb5de6d75\") " pod="openshift-multus/multus-58kk4"
Sep 16 19:39:33.923508 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.922198    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22-etc-kubernetes\") pod \"tuned-shdd9\" (UID: \"9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22\") " pod="openshift-cluster-node-tuning-operator/tuned-shdd9"
Sep 16 19:39:33.923508 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.922251    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/98bd8e7f-1aa8-418b-ae78-208cbfb13392-ovnkube-config\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.923508 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.922293    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22-etc-sysctl-d\") pod \"tuned-shdd9\" (UID: \"9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22\") " pod="openshift-cluster-node-tuning-operator/tuned-shdd9"
Sep 16 19:39:33.923508 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.922496    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22-var-lib-kubelet\") pod \"tuned-shdd9\" (UID: \"9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22\") " pod="openshift-cluster-node-tuning-operator/tuned-shdd9"
Sep 16 19:39:33.923508 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.922582    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/98bd8e7f-1aa8-418b-ae78-208cbfb13392-node-log\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.923508 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.922632    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98bd8e7f-1aa8-418b-ae78-208cbfb13392-var-lib-openvswitch\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.923508 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.922637    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/98bd8e7f-1aa8-418b-ae78-208cbfb13392-run-systemd\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.923508 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.922698    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/43cea22a-1cba-450a-8a7d-86beb5de6d75-multus-socket-dir-parent\") pod \"multus-58kk4\" (UID: \"43cea22a-1cba-450a-8a7d-86beb5de6d75\") " pod="openshift-multus/multus-58kk4"
Sep 16 19:39:33.923508 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.922700    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/43cea22a-1cba-450a-8a7d-86beb5de6d75-cnibin\") pod \"multus-58kk4\" (UID: \"43cea22a-1cba-450a-8a7d-86beb5de6d75\") " pod="openshift-multus/multus-58kk4"
Sep 16 19:39:33.923508 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.922724    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/98bd8e7f-1aa8-418b-ae78-208cbfb13392-log-socket\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.923508 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.922750    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/43cea22a-1cba-450a-8a7d-86beb5de6d75-host-run-netns\") pod \"multus-58kk4\" (UID: \"43cea22a-1cba-450a-8a7d-86beb5de6d75\") " pod="openshift-multus/multus-58kk4"
Sep 16 19:39:33.923508 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.922789    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/43cea22a-1cba-450a-8a7d-86beb5de6d75-host-run-multus-certs\") pod \"multus-58kk4\" (UID: \"43cea22a-1cba-450a-8a7d-86beb5de6d75\") " pod="openshift-multus/multus-58kk4"
Sep 16 19:39:33.923508 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.922827    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ed745f6b-93e2-416c-a9b3-9acc437b3d9d-iptables-alerter-script\") pod \"iptables-alerter-vqqgj\" (UID: \"ed745f6b-93e2-416c-a9b3-9acc437b3d9d\") " pod="openshift-network-operator/iptables-alerter-vqqgj"
Sep 16 19:39:33.923508 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.922831    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/98bd8e7f-1aa8-418b-ae78-208cbfb13392-host-cni-netd\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.923508 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.922864    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/98bd8e7f-1aa8-418b-ae78-208cbfb13392-host-cni-bin\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.923508 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.922892    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98bd8e7f-1aa8-418b-ae78-208cbfb13392-host-run-ovn-kubernetes\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.924380 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.922898    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/43cea22a-1cba-450a-8a7d-86beb5de6d75-cni-binary-copy\") pod \"multus-58kk4\" (UID: \"43cea22a-1cba-450a-8a7d-86beb5de6d75\") " pod="openshift-multus/multus-58kk4"
Sep 16 19:39:33.924380 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.923006    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0e9a620d-8ac5-4f5d-8e0f-f172664bc35e-device-dir\") pod \"aws-ebs-csi-driver-node-j7cdd\" (UID: \"0e9a620d-8ac5-4f5d-8e0f-f172664bc35e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7cdd"
Sep 16 19:39:33.924380 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.923245    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22-etc-modprobe-d\") pod \"tuned-shdd9\" (UID: \"9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22\") " pod="openshift-cluster-node-tuning-operator/tuned-shdd9"
Sep 16 19:39:33.924380 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.923291    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/43cea22a-1cba-450a-8a7d-86beb5de6d75-host-run-k8s-cni-cncf-io\") pod \"multus-58kk4\" (UID: \"43cea22a-1cba-450a-8a7d-86beb5de6d75\") " pod="openshift-multus/multus-58kk4"
Sep 16 19:39:33.924380 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.922941    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22-etc-systemd\") pod \"tuned-shdd9\" (UID: \"9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22\") " pod="openshift-cluster-node-tuning-operator/tuned-shdd9"
Sep 16 19:39:33.924380 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.923336    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22-lib-modules\") pod \"tuned-shdd9\" (UID: \"9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22\") " pod="openshift-cluster-node-tuning-operator/tuned-shdd9"
Sep 16 19:39:33.924380 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.923360    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22-host\") pod \"tuned-shdd9\" (UID: \"9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22\") " pod="openshift-cluster-node-tuning-operator/tuned-shdd9"
Sep 16 19:39:33.924380 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.923386    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/43cea22a-1cba-450a-8a7d-86beb5de6d75-etc-kubernetes\") pod \"multus-58kk4\" (UID: \"43cea22a-1cba-450a-8a7d-86beb5de6d75\") " pod="openshift-multus/multus-58kk4"
Sep 16 19:39:33.924380 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.923461    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/43cea22a-1cba-450a-8a7d-86beb5de6d75-multus-conf-dir\") pod \"multus-58kk4\" (UID: \"43cea22a-1cba-450a-8a7d-86beb5de6d75\") " pod="openshift-multus/multus-58kk4"
Sep 16 19:39:33.924380 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.923550    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/98bd8e7f-1aa8-418b-ae78-208cbfb13392-systemd-units\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.924380 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.923598    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ed745f6b-93e2-416c-a9b3-9acc437b3d9d-host-slash\") pod \"iptables-alerter-vqqgj\" (UID: \"ed745f6b-93e2-416c-a9b3-9acc437b3d9d\") " pod="openshift-network-operator/iptables-alerter-vqqgj"
Sep 16 19:39:33.924380 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.923646    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98bd8e7f-1aa8-418b-ae78-208cbfb13392-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.924380 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.922583    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/43cea22a-1cba-450a-8a7d-86beb5de6d75-multus-cni-dir\") pod \"multus-58kk4\" (UID: \"43cea22a-1cba-450a-8a7d-86beb5de6d75\") " pod="openshift-multus/multus-58kk4"
Sep 16 19:39:33.924380 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.923723    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/43cea22a-1cba-450a-8a7d-86beb5de6d75-host-var-lib-kubelet\") pod \"multus-58kk4\" (UID: \"43cea22a-1cba-450a-8a7d-86beb5de6d75\") " pod="openshift-multus/multus-58kk4"
Sep 16 19:39:33.924380 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.923728    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/98bd8e7f-1aa8-418b-ae78-208cbfb13392-host-kubelet\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.924380 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.923934    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/85c2f2c0-d2a8-454e-9e28-498c0c8d3c21-konnectivity-ca\") pod \"konnectivity-agent-v57p4\" (UID: \"85c2f2c0-d2a8-454e-9e28-498c0c8d3c21\") " pod="kube-system/konnectivity-agent-v57p4"
Sep 16 19:39:33.924380 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.924022    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/142b9c47-4826-4fa8-b228-5500d98fee67-hosts-file\") pod \"node-resolver-clfb8\" (UID: \"142b9c47-4826-4fa8-b228-5500d98fee67\") " pod="openshift-dns/node-resolver-clfb8"
Sep 16 19:39:33.926252 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.926227    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22-tmp\") pod \"tuned-shdd9\" (UID: \"9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22\") " pod="openshift-cluster-node-tuning-operator/tuned-shdd9"
Sep 16 19:39:33.926354 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.926251    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22-etc-tuned\") pod \"tuned-shdd9\" (UID: \"9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22\") " pod="openshift-cluster-node-tuning-operator/tuned-shdd9"
Sep 16 19:39:33.927324 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.927297    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/85c2f2c0-d2a8-454e-9e28-498c0c8d3c21-agent-certs\") pod \"konnectivity-agent-v57p4\" (UID: \"85c2f2c0-d2a8-454e-9e28-498c0c8d3c21\") " pod="kube-system/konnectivity-agent-v57p4"
Sep 16 19:39:33.927666 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.927639    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/98bd8e7f-1aa8-418b-ae78-208cbfb13392-ovn-node-metrics-cert\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.932830 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.932800    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdhvd\" (UniqueName: \"kubernetes.io/projected/43cea22a-1cba-450a-8a7d-86beb5de6d75-kube-api-access-wdhvd\") pod \"multus-58kk4\" (UID: \"43cea22a-1cba-450a-8a7d-86beb5de6d75\") " pod="openshift-multus/multus-58kk4"
Sep 16 19:39:33.933086 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.933024    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sfdb\" (UniqueName: \"kubernetes.io/projected/ed745f6b-93e2-416c-a9b3-9acc437b3d9d-kube-api-access-9sfdb\") pod \"iptables-alerter-vqqgj\" (UID: \"ed745f6b-93e2-416c-a9b3-9acc437b3d9d\") " pod="openshift-network-operator/iptables-alerter-vqqgj"
Sep 16 19:39:33.936427 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.936389    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vqmg\" (UniqueName: \"kubernetes.io/projected/bb2436af-e07f-464b-8973-1b0370888a65-kube-api-access-5vqmg\") pod \"node-ca-hhvcc\" (UID: \"bb2436af-e07f-464b-8973-1b0370888a65\") " pod="openshift-image-registry/node-ca-hhvcc"
Sep 16 19:39:33.937381 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.937352    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmdth\" (UniqueName: \"kubernetes.io/projected/98bd8e7f-1aa8-418b-ae78-208cbfb13392-kube-api-access-fmdth\") pod \"ovnkube-node-qbv6k\" (UID: \"98bd8e7f-1aa8-418b-ae78-208cbfb13392\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:33.939715 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.939173    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf824\" (UniqueName: \"kubernetes.io/projected/9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22-kube-api-access-rf824\") pod \"tuned-shdd9\" (UID: \"9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22\") " pod="openshift-cluster-node-tuning-operator/tuned-shdd9"
Sep 16 19:39:33.940407 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.940197    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-81.ec2.internal" event={"ID":"a47431da3c311f72143c9d580d811b73","Type":"ContainerStarted","Data":"640f7a3bdca81598869d363564d7f7a7e603f10334b13a726bf057078d17bf59"}
Sep 16 19:39:33.940407 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.940277    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgvpk\" (UniqueName: \"kubernetes.io/projected/0e9a620d-8ac5-4f5d-8e0f-f172664bc35e-kube-api-access-vgvpk\") pod \"aws-ebs-csi-driver-node-j7cdd\" (UID: \"0e9a620d-8ac5-4f5d-8e0f-f172664bc35e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7cdd"
Sep 16 19:39:33.941399 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.941377    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hctdf\" (UniqueName: \"kubernetes.io/projected/142b9c47-4826-4fa8-b228-5500d98fee67-kube-api-access-hctdf\") pod \"node-resolver-clfb8\" (UID: \"142b9c47-4826-4fa8-b228-5500d98fee67\") " pod="openshift-dns/node-resolver-clfb8"
Sep 16 19:39:33.941991 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:33.941970    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-81.ec2.internal" event={"ID":"38ae68d3fbc6666f2f805ad0321d7dc4","Type":"ContainerStarted","Data":"10934270ac4b35a1cf395eb4eb5c929d3d083324655de6a9bc6f7fc62d545dba"}
Sep 16 19:39:34.021653 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:34.021612    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fb2ab49a-ef77-4425-91b4-2b6a6ca89630-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xv4vd\" (UID: \"fb2ab49a-ef77-4425-91b4-2b6a6ca89630\") " pod="openshift-multus/multus-additional-cni-plugins-xv4vd"
Sep 16 19:39:34.021841 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:34.021686    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8s6m\" (UniqueName: \"kubernetes.io/projected/fb2ab49a-ef77-4425-91b4-2b6a6ca89630-kube-api-access-q8s6m\") pod \"multus-additional-cni-plugins-xv4vd\" (UID: \"fb2ab49a-ef77-4425-91b4-2b6a6ca89630\") " pod="openshift-multus/multus-additional-cni-plugins-xv4vd"
Sep 16 19:39:34.021841 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:34.021718    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb2ab49a-ef77-4425-91b4-2b6a6ca89630-cnibin\") pod \"multus-additional-cni-plugins-xv4vd\" (UID: \"fb2ab49a-ef77-4425-91b4-2b6a6ca89630\") " pod="openshift-multus/multus-additional-cni-plugins-xv4vd"
Sep 16 19:39:34.021841 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:34.021757    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fb2ab49a-ef77-4425-91b4-2b6a6ca89630-cni-binary-copy\") pod \"multus-additional-cni-plugins-xv4vd\" (UID: \"fb2ab49a-ef77-4425-91b4-2b6a6ca89630\") " pod="openshift-multus/multus-additional-cni-plugins-xv4vd"
Sep 16 19:39:34.021841 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:34.021784    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb28t\" (UniqueName: \"kubernetes.io/projected/b9840a22-6dcc-4836-b682-d1d667d2e30f-kube-api-access-xb28t\") pod \"network-check-target-2cjmm\" (UID: \"b9840a22-6dcc-4836-b682-d1d667d2e30f\") " pod="openshift-network-diagnostics/network-check-target-2cjmm"
Sep 16 19:39:34.021841 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:34.021811    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb2ab49a-ef77-4425-91b4-2b6a6ca89630-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xv4vd\" (UID: \"fb2ab49a-ef77-4425-91b4-2b6a6ca89630\") " pod="openshift-multus/multus-additional-cni-plugins-xv4vd"
Sep 16 19:39:34.022098 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:34.021834    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3428bc6c-ab5c-47f1-8f1d-f5923fdacfce-original-pull-secret\") pod \"global-pull-secret-syncer-qpp69\" (UID: \"3428bc6c-ab5c-47f1-8f1d-f5923fdacfce\") " pod="kube-system/global-pull-secret-syncer-qpp69"
Sep 16 19:39:34.022098 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:34.021858    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb2ab49a-ef77-4425-91b4-2b6a6ca89630-cnibin\") pod \"multus-additional-cni-plugins-xv4vd\" (UID: \"fb2ab49a-ef77-4425-91b4-2b6a6ca89630\") " pod="openshift-multus/multus-additional-cni-plugins-xv4vd"
Sep 16 19:39:34.022098 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:34.021969    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 19:39:34.022098 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:34.022038    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3428bc6c-ab5c-47f1-8f1d-f5923fdacfce-original-pull-secret podName:3428bc6c-ab5c-47f1-8f1d-f5923fdacfce nodeName:}" failed. No retries permitted until 2026-09-16 19:39:34.522014824 +0000 UTC m=+3.047403212 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3428bc6c-ab5c-47f1-8f1d-f5923fdacfce-original-pull-secret") pod "global-pull-secret-syncer-qpp69" (UID: "3428bc6c-ab5c-47f1-8f1d-f5923fdacfce") : object "kube-system"/"original-pull-secret" not registered
Sep 16 19:39:34.022098 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:34.022048    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb2ab49a-ef77-4425-91b4-2b6a6ca89630-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xv4vd\" (UID: \"fb2ab49a-ef77-4425-91b4-2b6a6ca89630\") " pod="openshift-multus/multus-additional-cni-plugins-xv4vd"
Sep 16 19:39:34.022341 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:34.022101    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4ea9fa1-48c7-45be-bfcf-4594ee751086-metrics-certs\") pod \"network-metrics-daemon-drgl8\" (UID: \"b4ea9fa1-48c7-45be-bfcf-4594ee751086\") " pod="openshift-multus/network-metrics-daemon-drgl8"
Sep 16 19:39:34.022341 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:34.022140    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3428bc6c-ab5c-47f1-8f1d-f5923fdacfce-kubelet-config\") pod \"global-pull-secret-syncer-qpp69\" (UID: \"3428bc6c-ab5c-47f1-8f1d-f5923fdacfce\") " pod="kube-system/global-pull-secret-syncer-qpp69"
Sep 16 19:39:34.022341 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:34.022178    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fb2ab49a-ef77-4425-91b4-2b6a6ca89630-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xv4vd\" (UID: \"fb2ab49a-ef77-4425-91b4-2b6a6ca89630\") " pod="openshift-multus/multus-additional-cni-plugins-xv4vd"
Sep 16 19:39:34.022341 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:34.022250    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3428bc6c-ab5c-47f1-8f1d-f5923fdacfce-kubelet-config\") pod \"global-pull-secret-syncer-qpp69\" (UID: \"3428bc6c-ab5c-47f1-8f1d-f5923fdacfce\") " pod="kube-system/global-pull-secret-syncer-qpp69"
Sep 16 19:39:34.022341 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:34.022267    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 19:39:34.022341 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:34.022288    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3428bc6c-ab5c-47f1-8f1d-f5923fdacfce-dbus\") pod \"global-pull-secret-syncer-qpp69\" (UID: \"3428bc6c-ab5c-47f1-8f1d-f5923fdacfce\") " pod="kube-system/global-pull-secret-syncer-qpp69"
Sep 16 19:39:34.022341 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:34.022323    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb2ab49a-ef77-4425-91b4-2b6a6ca89630-system-cni-dir\") pod \"multus-additional-cni-plugins-xv4vd\" (UID: \"fb2ab49a-ef77-4425-91b4-2b6a6ca89630\") " pod="openshift-multus/multus-additional-cni-plugins-xv4vd"
Sep 16 19:39:34.022341 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:34.022335    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fb2ab49a-ef77-4425-91b4-2b6a6ca89630-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xv4vd\" (UID: \"fb2ab49a-ef77-4425-91b4-2b6a6ca89630\") " pod="openshift-multus/multus-additional-cni-plugins-xv4vd"
Sep 16 19:39:34.022652 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:34.022359    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb2ab49a-ef77-4425-91b4-2b6a6ca89630-system-cni-dir\") pod \"multus-additional-cni-plugins-xv4vd\" (UID: \"fb2ab49a-ef77-4425-91b4-2b6a6ca89630\") " pod="openshift-multus/multus-additional-cni-plugins-xv4vd"
Sep 16 19:39:34.022652 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:34.022336    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4ea9fa1-48c7-45be-bfcf-4594ee751086-metrics-certs podName:b4ea9fa1-48c7-45be-bfcf-4594ee751086 nodeName:}" failed. No retries permitted until 2026-09-16 19:39:34.522316161 +0000 UTC m=+3.047704543 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b4ea9fa1-48c7-45be-bfcf-4594ee751086-metrics-certs") pod "network-metrics-daemon-drgl8" (UID: "b4ea9fa1-48c7-45be-bfcf-4594ee751086") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 19:39:34.022652 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:34.022410    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9clm\" (UniqueName: \"kubernetes.io/projected/b4ea9fa1-48c7-45be-bfcf-4594ee751086-kube-api-access-c9clm\") pod \"network-metrics-daemon-drgl8\" (UID: \"b4ea9fa1-48c7-45be-bfcf-4594ee751086\") " pod="openshift-multus/network-metrics-daemon-drgl8"
Sep 16 19:39:34.022652 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:34.022414    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fb2ab49a-ef77-4425-91b4-2b6a6ca89630-cni-binary-copy\") pod \"multus-additional-cni-plugins-xv4vd\" (UID: \"fb2ab49a-ef77-4425-91b4-2b6a6ca89630\") " pod="openshift-multus/multus-additional-cni-plugins-xv4vd"
Sep 16 19:39:34.022652 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:34.022542    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3428bc6c-ab5c-47f1-8f1d-f5923fdacfce-dbus\") pod \"global-pull-secret-syncer-qpp69\" (UID: \"3428bc6c-ab5c-47f1-8f1d-f5923fdacfce\") " pod="kube-system/global-pull-secret-syncer-qpp69"
Sep 16 19:39:34.022652 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:34.022635    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fb2ab49a-ef77-4425-91b4-2b6a6ca89630-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xv4vd\" (UID: \"fb2ab49a-ef77-4425-91b4-2b6a6ca89630\") " pod="openshift-multus/multus-additional-cni-plugins-xv4vd"
Sep 16 19:39:34.036378 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:34.036308    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 19:39:34.036378 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:34.036328    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 19:39:34.036378 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:34.036339    2557 projected.go:196] Error preparing data for projected volume kube-api-access-xb28t for pod openshift-network-diagnostics/network-check-target-2cjmm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 19:39:34.036599 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:34.036397    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b9840a22-6dcc-4836-b682-d1d667d2e30f-kube-api-access-xb28t podName:b9840a22-6dcc-4836-b682-d1d667d2e30f nodeName:}" failed. No retries permitted until 2026-09-16 19:39:34.536384489 +0000 UTC m=+3.061772877 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xb28t" (UniqueName: "kubernetes.io/projected/b9840a22-6dcc-4836-b682-d1d667d2e30f-kube-api-access-xb28t") pod "network-check-target-2cjmm" (UID: "b9840a22-6dcc-4836-b682-d1d667d2e30f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 19:39:34.040582 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:34.040556    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9clm\" (UniqueName: \"kubernetes.io/projected/b4ea9fa1-48c7-45be-bfcf-4594ee751086-kube-api-access-c9clm\") pod \"network-metrics-daemon-drgl8\" (UID: \"b4ea9fa1-48c7-45be-bfcf-4594ee751086\") " pod="openshift-multus/network-metrics-daemon-drgl8"
Sep 16 19:39:34.041201 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:34.041178    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8s6m\" (UniqueName: \"kubernetes.io/projected/fb2ab49a-ef77-4425-91b4-2b6a6ca89630-kube-api-access-q8s6m\") pod \"multus-additional-cni-plugins-xv4vd\" (UID: \"fb2ab49a-ef77-4425-91b4-2b6a6ca89630\") " pod="openshift-multus/multus-additional-cni-plugins-xv4vd"
Sep 16 19:39:34.108779 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:34.108742    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-clfb8"
Sep 16 19:39:34.117294 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:34.117257    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vqqgj"
Sep 16 19:39:34.125953 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:34.125934    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:34.133498 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:34.133470    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v57p4"
Sep 16 19:39:34.140952 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:34.140929    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hhvcc"
Sep 16 19:39:34.148288 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:34.148266    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-58kk4"
Sep 16 19:39:34.156847 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:34.156821    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-shdd9"
Sep 16 19:39:34.164817 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:34.164798    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7cdd"
Sep 16 19:39:34.171589 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:34.171571    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xv4vd"
Sep 16 19:39:34.202998 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:34.202976    2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 19:39:34.526632 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:34.526542    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3428bc6c-ab5c-47f1-8f1d-f5923fdacfce-original-pull-secret\") pod \"global-pull-secret-syncer-qpp69\" (UID: \"3428bc6c-ab5c-47f1-8f1d-f5923fdacfce\") " pod="kube-system/global-pull-secret-syncer-qpp69"
Sep 16 19:39:34.526632 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:34.526586    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4ea9fa1-48c7-45be-bfcf-4594ee751086-metrics-certs\") pod \"network-metrics-daemon-drgl8\" (UID: \"b4ea9fa1-48c7-45be-bfcf-4594ee751086\") " pod="openshift-multus/network-metrics-daemon-drgl8"
Sep 16 19:39:34.526847 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:34.526715    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 19:39:34.526847 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:34.526718    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 19:39:34.526847 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:34.526787    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4ea9fa1-48c7-45be-bfcf-4594ee751086-metrics-certs podName:b4ea9fa1-48c7-45be-bfcf-4594ee751086 nodeName:}" failed. No retries permitted until 2026-09-16 19:39:35.526766895 +0000 UTC m=+4.052155278 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b4ea9fa1-48c7-45be-bfcf-4594ee751086-metrics-certs") pod "network-metrics-daemon-drgl8" (UID: "b4ea9fa1-48c7-45be-bfcf-4594ee751086") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 19:39:34.526847 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:34.526809    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3428bc6c-ab5c-47f1-8f1d-f5923fdacfce-original-pull-secret podName:3428bc6c-ab5c-47f1-8f1d-f5923fdacfce nodeName:}" failed. No retries permitted until 2026-09-16 19:39:35.526797336 +0000 UTC m=+4.052185705 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3428bc6c-ab5c-47f1-8f1d-f5923fdacfce-original-pull-secret") pod "global-pull-secret-syncer-qpp69" (UID: "3428bc6c-ab5c-47f1-8f1d-f5923fdacfce") : object "kube-system"/"original-pull-secret" not registered
Sep 16 19:39:34.627380 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:34.627334    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb28t\" (UniqueName: \"kubernetes.io/projected/b9840a22-6dcc-4836-b682-d1d667d2e30f-kube-api-access-xb28t\") pod \"network-check-target-2cjmm\" (UID: \"b9840a22-6dcc-4836-b682-d1d667d2e30f\") " pod="openshift-network-diagnostics/network-check-target-2cjmm"
Sep 16 19:39:34.627540 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:34.627514    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 19:39:34.627611 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:34.627545    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 19:39:34.627611 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:34.627556    2557 projected.go:196] Error preparing data for projected volume kube-api-access-xb28t for pod openshift-network-diagnostics/network-check-target-2cjmm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 19:39:34.627699 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:34.627622    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b9840a22-6dcc-4836-b682-d1d667d2e30f-kube-api-access-xb28t podName:b9840a22-6dcc-4836-b682-d1d667d2e30f nodeName:}" failed. No retries permitted until 2026-09-16 19:39:35.62760138 +0000 UTC m=+4.152989757 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-xb28t" (UniqueName: "kubernetes.io/projected/b9840a22-6dcc-4836-b682-d1d667d2e30f-kube-api-access-xb28t") pod "network-check-target-2cjmm" (UID: "b9840a22-6dcc-4836-b682-d1d667d2e30f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 19:39:34.858673 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:34.858574    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 19:34:34 +0000 UTC" deadline="2028-04-10 09:21:33.755476546 +0000 UTC"
Sep 16 19:39:34.858673 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:34.858623    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13717h41m58.89685738s"
Sep 16 19:39:34.905951 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:34.905920    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98bd8e7f_1aa8_418b_ae78_208cbfb13392.slice/crio-53fac940e3abbb691de7639c9b4b87412b1b119e47eb3a9eec889893daa11ced WatchSource:0}: Error finding container 53fac940e3abbb691de7639c9b4b87412b1b119e47eb3a9eec889893daa11ced: Status 404 returned error can't find the container with id 53fac940e3abbb691de7639c9b4b87412b1b119e47eb3a9eec889893daa11ced
Sep 16 19:39:34.907072 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:34.906968    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod142b9c47_4826_4fa8_b228_5500d98fee67.slice/crio-2bfef9990a0c368f80a8e9962f231a9d15c0964b8d00550c8b884fbda9109052 WatchSource:0}: Error finding container 2bfef9990a0c368f80a8e9962f231a9d15c0964b8d00550c8b884fbda9109052: Status 404 returned error can't find the container with id 2bfef9990a0c368f80a8e9962f231a9d15c0964b8d00550c8b884fbda9109052
Sep 16 19:39:34.910315 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:34.910269    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d10b5b8_04a4_4a6e_ba00_d96a5e90ac22.slice/crio-0322b108eaa5641368e35f98d65467c840a1853055c8c229ba38ab0b436a3091 WatchSource:0}: Error finding container 0322b108eaa5641368e35f98d65467c840a1853055c8c229ba38ab0b436a3091: Status 404 returned error can't find the container with id 0322b108eaa5641368e35f98d65467c840a1853055c8c229ba38ab0b436a3091
Sep 16 19:39:34.913990 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:34.913964    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb2ab49a_ef77_4425_91b4_2b6a6ca89630.slice/crio-940cd14864ce51d5d46709fa9f7d605cc7b57044da382c4519f85ecc7fe2bfb7 WatchSource:0}: Error finding container 940cd14864ce51d5d46709fa9f7d605cc7b57044da382c4519f85ecc7fe2bfb7: Status 404 returned error can't find the container with id 940cd14864ce51d5d46709fa9f7d605cc7b57044da382c4519f85ecc7fe2bfb7
Sep 16 19:39:34.914868 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:34.914847    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded745f6b_93e2_416c_a9b3_9acc437b3d9d.slice/crio-d03016e7a3420941801be6dd6afc522a1ab107bb084d635e92ac6cf625c46d94 WatchSource:0}: Error finding container d03016e7a3420941801be6dd6afc522a1ab107bb084d635e92ac6cf625c46d94: Status 404 returned error can't find the container with id d03016e7a3420941801be6dd6afc522a1ab107bb084d635e92ac6cf625c46d94
Sep 16 19:39:34.916213 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:34.916190    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43cea22a_1cba_450a_8a7d_86beb5de6d75.slice/crio-e5d03b059b8c4c003bbceb4f2f561b213266b4ffda6ac48b4eeeb87829c22a77 WatchSource:0}: Error finding container e5d03b059b8c4c003bbceb4f2f561b213266b4ffda6ac48b4eeeb87829c22a77: Status 404 returned error can't find the container with id e5d03b059b8c4c003bbceb4f2f561b213266b4ffda6ac48b4eeeb87829c22a77
Sep 16 19:39:34.917140 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:34.917120    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e9a620d_8ac5_4f5d_8e0f_f172664bc35e.slice/crio-84a16253e75a1c16d304ecb2c133f8a5baafc630ec99fcb5c3e4e43cc78a34ed WatchSource:0}: Error finding container 84a16253e75a1c16d304ecb2c133f8a5baafc630ec99fcb5c3e4e43cc78a34ed: Status 404 returned error can't find the container with id 84a16253e75a1c16d304ecb2c133f8a5baafc630ec99fcb5c3e4e43cc78a34ed
Sep 16 19:39:34.918889 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:34.918853    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb2436af_e07f_464b_8973_1b0370888a65.slice/crio-fc9d37cc9334703cecd6a22289ade6fa00abcf82ae80d7385ec17322f56c8934 WatchSource:0}: Error finding container fc9d37cc9334703cecd6a22289ade6fa00abcf82ae80d7385ec17322f56c8934: Status 404 returned error can't find the container with id fc9d37cc9334703cecd6a22289ade6fa00abcf82ae80d7385ec17322f56c8934
Sep 16 19:39:34.920037 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:39:34.919953    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85c2f2c0_d2a8_454e_9e28_498c0c8d3c21.slice/crio-454afd1479f187a8ce1fc86fd8997b69a504b831714de018fc21b40f966deff8 WatchSource:0}: Error finding container 454afd1479f187a8ce1fc86fd8997b69a504b831714de018fc21b40f966deff8: Status 404 returned error can't find the container with id 454afd1479f187a8ce1fc86fd8997b69a504b831714de018fc21b40f966deff8
Sep 16 19:39:34.945050 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:34.945018    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-clfb8" event={"ID":"142b9c47-4826-4fa8-b228-5500d98fee67","Type":"ContainerStarted","Data":"2bfef9990a0c368f80a8e9962f231a9d15c0964b8d00550c8b884fbda9109052"}
Sep 16 19:39:34.946283 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:34.946256    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k" event={"ID":"98bd8e7f-1aa8-418b-ae78-208cbfb13392","Type":"ContainerStarted","Data":"53fac940e3abbb691de7639c9b4b87412b1b119e47eb3a9eec889893daa11ced"}
Sep 16 19:39:34.947857 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:34.947824    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v57p4" event={"ID":"85c2f2c0-d2a8-454e-9e28-498c0c8d3c21","Type":"ContainerStarted","Data":"454afd1479f187a8ce1fc86fd8997b69a504b831714de018fc21b40f966deff8"}
Sep 16 19:39:34.949079 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:34.949056    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7cdd" event={"ID":"0e9a620d-8ac5-4f5d-8e0f-f172664bc35e","Type":"ContainerStarted","Data":"84a16253e75a1c16d304ecb2c133f8a5baafc630ec99fcb5c3e4e43cc78a34ed"}
Sep 16 19:39:34.951593 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:34.951568    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vqqgj" event={"ID":"ed745f6b-93e2-416c-a9b3-9acc437b3d9d","Type":"ContainerStarted","Data":"d03016e7a3420941801be6dd6afc522a1ab107bb084d635e92ac6cf625c46d94"}
Sep 16 19:39:34.955469 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:34.955442    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-shdd9" event={"ID":"9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22","Type":"ContainerStarted","Data":"0322b108eaa5641368e35f98d65467c840a1853055c8c229ba38ab0b436a3091"}
Sep 16 19:39:34.959620 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:34.959552    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xv4vd" event={"ID":"fb2ab49a-ef77-4425-91b4-2b6a6ca89630","Type":"ContainerStarted","Data":"940cd14864ce51d5d46709fa9f7d605cc7b57044da382c4519f85ecc7fe2bfb7"}
Sep 16 19:39:34.960832 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:34.960812    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hhvcc" event={"ID":"bb2436af-e07f-464b-8973-1b0370888a65","Type":"ContainerStarted","Data":"fc9d37cc9334703cecd6a22289ade6fa00abcf82ae80d7385ec17322f56c8934"}
Sep 16 19:39:34.962361 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:34.962342    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-58kk4" event={"ID":"43cea22a-1cba-450a-8a7d-86beb5de6d75","Type":"ContainerStarted","Data":"e5d03b059b8c4c003bbceb4f2f561b213266b4ffda6ac48b4eeeb87829c22a77"}
Sep 16 19:39:35.536157 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:35.536060    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3428bc6c-ab5c-47f1-8f1d-f5923fdacfce-original-pull-secret\") pod \"global-pull-secret-syncer-qpp69\" (UID: \"3428bc6c-ab5c-47f1-8f1d-f5923fdacfce\") " pod="kube-system/global-pull-secret-syncer-qpp69"
Sep 16 19:39:35.536157 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:35.536124    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4ea9fa1-48c7-45be-bfcf-4594ee751086-metrics-certs\") pod \"network-metrics-daemon-drgl8\" (UID: \"b4ea9fa1-48c7-45be-bfcf-4594ee751086\") " pod="openshift-multus/network-metrics-daemon-drgl8"
Sep 16 19:39:35.536387 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:35.536298    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 19:39:35.536387 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:35.536366    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4ea9fa1-48c7-45be-bfcf-4594ee751086-metrics-certs podName:b4ea9fa1-48c7-45be-bfcf-4594ee751086 nodeName:}" failed. No retries permitted until 2026-09-16 19:39:37.536345397 +0000 UTC m=+6.061733764 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b4ea9fa1-48c7-45be-bfcf-4594ee751086-metrics-certs") pod "network-metrics-daemon-drgl8" (UID: "b4ea9fa1-48c7-45be-bfcf-4594ee751086") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 19:39:35.536517 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:35.536463    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 19:39:35.536569 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:35.536523    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3428bc6c-ab5c-47f1-8f1d-f5923fdacfce-original-pull-secret podName:3428bc6c-ab5c-47f1-8f1d-f5923fdacfce nodeName:}" failed. No retries permitted until 2026-09-16 19:39:37.536507537 +0000 UTC m=+6.061895919 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3428bc6c-ab5c-47f1-8f1d-f5923fdacfce-original-pull-secret") pod "global-pull-secret-syncer-qpp69" (UID: "3428bc6c-ab5c-47f1-8f1d-f5923fdacfce") : object "kube-system"/"original-pull-secret" not registered
Sep 16 19:39:35.637635 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:35.637051    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb28t\" (UniqueName: \"kubernetes.io/projected/b9840a22-6dcc-4836-b682-d1d667d2e30f-kube-api-access-xb28t\") pod \"network-check-target-2cjmm\" (UID: \"b9840a22-6dcc-4836-b682-d1d667d2e30f\") " pod="openshift-network-diagnostics/network-check-target-2cjmm"
Sep 16 19:39:35.637635 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:35.637204    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 19:39:35.637635 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:35.637221    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 19:39:35.637635 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:35.637232    2557 projected.go:196] Error preparing data for projected volume kube-api-access-xb28t for pod openshift-network-diagnostics/network-check-target-2cjmm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 19:39:35.637635 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:35.637277    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b9840a22-6dcc-4836-b682-d1d667d2e30f-kube-api-access-xb28t podName:b9840a22-6dcc-4836-b682-d1d667d2e30f nodeName:}" failed. No retries permitted until 2026-09-16 19:39:37.637264196 +0000 UTC m=+6.162652561 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-xb28t" (UniqueName: "kubernetes.io/projected/b9840a22-6dcc-4836-b682-d1d667d2e30f-kube-api-access-xb28t") pod "network-check-target-2cjmm" (UID: "b9840a22-6dcc-4836-b682-d1d667d2e30f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 19:39:35.859342 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:35.859168    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 19:34:34 +0000 UTC" deadline="2028-07-04 12:16:30.207052918 +0000 UTC"
Sep 16 19:39:35.859342 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:35.859263    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15760h36m54.34779545s"
Sep 16 19:39:35.932567 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:35.932006    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2cjmm"
Sep 16 19:39:35.932567 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:35.932037    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-drgl8"
Sep 16 19:39:35.932567 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:35.932188    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qpp69"
Sep 16 19:39:35.932567 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:35.932192    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2cjmm" podUID="b9840a22-6dcc-4836-b682-d1d667d2e30f"
Sep 16 19:39:35.932567 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:35.932286    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qpp69" podUID="3428bc6c-ab5c-47f1-8f1d-f5923fdacfce"
Sep 16 19:39:35.932567 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:35.932370    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-drgl8" podUID="b4ea9fa1-48c7-45be-bfcf-4594ee751086"
Sep 16 19:39:35.986888 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:35.986672    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-81.ec2.internal" event={"ID":"a47431da3c311f72143c9d580d811b73","Type":"ContainerStarted","Data":"9a1bbcc1b062f8f4e64bf0a944856ea76dc60fc1cc997e26495c58580d6835a6"}
Sep 16 19:39:35.996228 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:35.995622    2557 generic.go:356] "Generic (PLEG): container finished" podID="38ae68d3fbc6666f2f805ad0321d7dc4" containerID="af54df94d802a8d091f74f63092d2746265e7b2317d45154a712c1d91ff22249" exitCode=0
Sep 16 19:39:35.996228 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:35.995666    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-81.ec2.internal" event={"ID":"38ae68d3fbc6666f2f805ad0321d7dc4","Type":"ContainerDied","Data":"af54df94d802a8d091f74f63092d2746265e7b2317d45154a712c1d91ff22249"}
Sep 16 19:39:36.001714 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:36.001496    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-81.ec2.internal" podStartSLOduration=3.001482633 podStartE2EDuration="3.001482633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:39:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:39:36.000267873 +0000 UTC m=+4.525656287" watchObservedRunningTime="2026-09-16 19:39:36.001482633 +0000 UTC m=+4.526871018"
Sep 16 19:39:37.005593 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:37.005183    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-81.ec2.internal" event={"ID":"38ae68d3fbc6666f2f805ad0321d7dc4","Type":"ContainerStarted","Data":"d543e84c9337fcd189f3637bfc52801b261946a3dcf448d3203ac15740c11c9b"}
Sep 16 19:39:37.553395 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:37.553349    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3428bc6c-ab5c-47f1-8f1d-f5923fdacfce-original-pull-secret\") pod \"global-pull-secret-syncer-qpp69\" (UID: \"3428bc6c-ab5c-47f1-8f1d-f5923fdacfce\") " pod="kube-system/global-pull-secret-syncer-qpp69"
Sep 16 19:39:37.553597 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:37.553417    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4ea9fa1-48c7-45be-bfcf-4594ee751086-metrics-certs\") pod \"network-metrics-daemon-drgl8\" (UID: \"b4ea9fa1-48c7-45be-bfcf-4594ee751086\") " pod="openshift-multus/network-metrics-daemon-drgl8"
Sep 16 19:39:37.553597 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:37.553591    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 19:39:37.553727 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:37.553660    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4ea9fa1-48c7-45be-bfcf-4594ee751086-metrics-certs podName:b4ea9fa1-48c7-45be-bfcf-4594ee751086 nodeName:}" failed. No retries permitted until 2026-09-16 19:39:41.553638959 +0000 UTC m=+10.079027349 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b4ea9fa1-48c7-45be-bfcf-4594ee751086-metrics-certs") pod "network-metrics-daemon-drgl8" (UID: "b4ea9fa1-48c7-45be-bfcf-4594ee751086") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 19:39:37.554114 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:37.554093    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 19:39:37.554216 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:37.554155    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3428bc6c-ab5c-47f1-8f1d-f5923fdacfce-original-pull-secret podName:3428bc6c-ab5c-47f1-8f1d-f5923fdacfce nodeName:}" failed. No retries permitted until 2026-09-16 19:39:41.554138368 +0000 UTC m=+10.079526736 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3428bc6c-ab5c-47f1-8f1d-f5923fdacfce-original-pull-secret") pod "global-pull-secret-syncer-qpp69" (UID: "3428bc6c-ab5c-47f1-8f1d-f5923fdacfce") : object "kube-system"/"original-pull-secret" not registered
Sep 16 19:39:37.654844 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:37.654801    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb28t\" (UniqueName: \"kubernetes.io/projected/b9840a22-6dcc-4836-b682-d1d667d2e30f-kube-api-access-xb28t\") pod \"network-check-target-2cjmm\" (UID: \"b9840a22-6dcc-4836-b682-d1d667d2e30f\") " pod="openshift-network-diagnostics/network-check-target-2cjmm"
Sep 16 19:39:37.655044 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:37.654990    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 19:39:37.655044 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:37.655012    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 19:39:37.655044 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:37.655025    2557 projected.go:196] Error preparing data for projected volume kube-api-access-xb28t for pod openshift-network-diagnostics/network-check-target-2cjmm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 19:39:37.655222 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:37.655088    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b9840a22-6dcc-4836-b682-d1d667d2e30f-kube-api-access-xb28t podName:b9840a22-6dcc-4836-b682-d1d667d2e30f nodeName:}" failed. No retries permitted until 2026-09-16 19:39:41.655068296 +0000 UTC m=+10.180456677 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-xb28t" (UniqueName: "kubernetes.io/projected/b9840a22-6dcc-4836-b682-d1d667d2e30f-kube-api-access-xb28t") pod "network-check-target-2cjmm" (UID: "b9840a22-6dcc-4836-b682-d1d667d2e30f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 19:39:37.932429 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:37.931952    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2cjmm"
Sep 16 19:39:37.932429 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:37.931971    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-drgl8"
Sep 16 19:39:37.932429 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:37.932085    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2cjmm" podUID="b9840a22-6dcc-4836-b682-d1d667d2e30f"
Sep 16 19:39:37.932429 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:37.932200    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-drgl8" podUID="b4ea9fa1-48c7-45be-bfcf-4594ee751086"
Sep 16 19:39:37.932429 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:37.932247    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qpp69"
Sep 16 19:39:37.932429 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:37.932332    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qpp69" podUID="3428bc6c-ab5c-47f1-8f1d-f5923fdacfce"
Sep 16 19:39:39.934217 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:39.934183    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qpp69"
Sep 16 19:39:39.934699 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:39.934314    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qpp69" podUID="3428bc6c-ab5c-47f1-8f1d-f5923fdacfce"
Sep 16 19:39:39.934774 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:39.934732    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2cjmm"
Sep 16 19:39:39.934895 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:39.934869    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2cjmm" podUID="b9840a22-6dcc-4836-b682-d1d667d2e30f"
Sep 16 19:39:39.934999 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:39.934938    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-drgl8"
Sep 16 19:39:39.935051 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:39.935025    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-drgl8" podUID="b4ea9fa1-48c7-45be-bfcf-4594ee751086"
Sep 16 19:39:41.591005 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:41.590967    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3428bc6c-ab5c-47f1-8f1d-f5923fdacfce-original-pull-secret\") pod \"global-pull-secret-syncer-qpp69\" (UID: \"3428bc6c-ab5c-47f1-8f1d-f5923fdacfce\") " pod="kube-system/global-pull-secret-syncer-qpp69"
Sep 16 19:39:41.591445 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:41.591013    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4ea9fa1-48c7-45be-bfcf-4594ee751086-metrics-certs\") pod \"network-metrics-daemon-drgl8\" (UID: \"b4ea9fa1-48c7-45be-bfcf-4594ee751086\") " pod="openshift-multus/network-metrics-daemon-drgl8"
Sep 16 19:39:41.591445 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:41.591139    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 19:39:41.591445 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:41.591204    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4ea9fa1-48c7-45be-bfcf-4594ee751086-metrics-certs podName:b4ea9fa1-48c7-45be-bfcf-4594ee751086 nodeName:}" failed. No retries permitted until 2026-09-16 19:39:49.591184731 +0000 UTC m=+18.116573097 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b4ea9fa1-48c7-45be-bfcf-4594ee751086-metrics-certs") pod "network-metrics-daemon-drgl8" (UID: "b4ea9fa1-48c7-45be-bfcf-4594ee751086") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 19:39:41.591631 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:41.591614    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 19:39:41.591685 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:41.591669    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3428bc6c-ab5c-47f1-8f1d-f5923fdacfce-original-pull-secret podName:3428bc6c-ab5c-47f1-8f1d-f5923fdacfce nodeName:}" failed. No retries permitted until 2026-09-16 19:39:49.591654348 +0000 UTC m=+18.117042716 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3428bc6c-ab5c-47f1-8f1d-f5923fdacfce-original-pull-secret") pod "global-pull-secret-syncer-qpp69" (UID: "3428bc6c-ab5c-47f1-8f1d-f5923fdacfce") : object "kube-system"/"original-pull-secret" not registered
Sep 16 19:39:41.692403 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:41.692364    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb28t\" (UniqueName: \"kubernetes.io/projected/b9840a22-6dcc-4836-b682-d1d667d2e30f-kube-api-access-xb28t\") pod \"network-check-target-2cjmm\" (UID: \"b9840a22-6dcc-4836-b682-d1d667d2e30f\") " pod="openshift-network-diagnostics/network-check-target-2cjmm"
Sep 16 19:39:41.692591 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:41.692574    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 19:39:41.692642 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:41.692601    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 19:39:41.692642 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:41.692615    2557 projected.go:196] Error preparing data for projected volume kube-api-access-xb28t for pod openshift-network-diagnostics/network-check-target-2cjmm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 19:39:41.692727 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:41.692680    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b9840a22-6dcc-4836-b682-d1d667d2e30f-kube-api-access-xb28t podName:b9840a22-6dcc-4836-b682-d1d667d2e30f nodeName:}" failed. No retries permitted until 2026-09-16 19:39:49.692661491 +0000 UTC m=+18.218049857 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-xb28t" (UniqueName: "kubernetes.io/projected/b9840a22-6dcc-4836-b682-d1d667d2e30f-kube-api-access-xb28t") pod "network-check-target-2cjmm" (UID: "b9840a22-6dcc-4836-b682-d1d667d2e30f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 19:39:41.932710 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:41.931976    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-drgl8"
Sep 16 19:39:41.932710 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:41.932094    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-drgl8" podUID="b4ea9fa1-48c7-45be-bfcf-4594ee751086"
Sep 16 19:39:41.932710 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:41.932439    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2cjmm"
Sep 16 19:39:41.932710 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:41.932519    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2cjmm" podUID="b9840a22-6dcc-4836-b682-d1d667d2e30f"
Sep 16 19:39:41.932710 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:41.932560    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qpp69"
Sep 16 19:39:41.932710 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:41.932628    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qpp69" podUID="3428bc6c-ab5c-47f1-8f1d-f5923fdacfce"
Sep 16 19:39:43.931541 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:43.931495    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2cjmm"
Sep 16 19:39:43.932054 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:43.931505    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-drgl8"
Sep 16 19:39:43.932054 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:43.931641    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qpp69"
Sep 16 19:39:43.932054 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:43.931643    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2cjmm" podUID="b9840a22-6dcc-4836-b682-d1d667d2e30f"
Sep 16 19:39:43.932054 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:43.931759    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-drgl8" podUID="b4ea9fa1-48c7-45be-bfcf-4594ee751086"
Sep 16 19:39:43.932054 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:43.931861    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qpp69" podUID="3428bc6c-ab5c-47f1-8f1d-f5923fdacfce"
Sep 16 19:39:45.932011 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:45.931918    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-drgl8"
Sep 16 19:39:45.932011 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:45.931949    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qpp69"
Sep 16 19:39:45.932505 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:45.931918    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2cjmm"
Sep 16 19:39:45.932505 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:45.932048    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-drgl8" podUID="b4ea9fa1-48c7-45be-bfcf-4594ee751086"
Sep 16 19:39:45.932505 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:45.932124    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qpp69" podUID="3428bc6c-ab5c-47f1-8f1d-f5923fdacfce"
Sep 16 19:39:45.932505 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:45.932216    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2cjmm" podUID="b9840a22-6dcc-4836-b682-d1d667d2e30f"
Sep 16 19:39:47.931290 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:47.931256    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2cjmm"
Sep 16 19:39:47.931754 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:47.931258    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-drgl8"
Sep 16 19:39:47.931754 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:47.931379    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2cjmm" podUID="b9840a22-6dcc-4836-b682-d1d667d2e30f"
Sep 16 19:39:47.931754 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:47.931393    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qpp69"
Sep 16 19:39:47.931754 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:47.931510    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-drgl8" podUID="b4ea9fa1-48c7-45be-bfcf-4594ee751086"
Sep 16 19:39:47.931754 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:47.931604    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qpp69" podUID="3428bc6c-ab5c-47f1-8f1d-f5923fdacfce"
Sep 16 19:39:49.649897 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:49.649866    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4ea9fa1-48c7-45be-bfcf-4594ee751086-metrics-certs\") pod \"network-metrics-daemon-drgl8\" (UID: \"b4ea9fa1-48c7-45be-bfcf-4594ee751086\") " pod="openshift-multus/network-metrics-daemon-drgl8"
Sep 16 19:39:49.650404 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:49.649994    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3428bc6c-ab5c-47f1-8f1d-f5923fdacfce-original-pull-secret\") pod \"global-pull-secret-syncer-qpp69\" (UID: \"3428bc6c-ab5c-47f1-8f1d-f5923fdacfce\") " pod="kube-system/global-pull-secret-syncer-qpp69"
Sep 16 19:39:49.650404 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:49.650008    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 19:39:49.650404 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:49.650078    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4ea9fa1-48c7-45be-bfcf-4594ee751086-metrics-certs podName:b4ea9fa1-48c7-45be-bfcf-4594ee751086 nodeName:}" failed. No retries permitted until 2026-09-16 19:40:05.65005825 +0000 UTC m=+34.175446616 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b4ea9fa1-48c7-45be-bfcf-4594ee751086-metrics-certs") pod "network-metrics-daemon-drgl8" (UID: "b4ea9fa1-48c7-45be-bfcf-4594ee751086") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 19:39:49.650404 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:49.650099    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 19:39:49.650404 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:49.650149    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3428bc6c-ab5c-47f1-8f1d-f5923fdacfce-original-pull-secret podName:3428bc6c-ab5c-47f1-8f1d-f5923fdacfce nodeName:}" failed. No retries permitted until 2026-09-16 19:40:05.650132963 +0000 UTC m=+34.175521349 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3428bc6c-ab5c-47f1-8f1d-f5923fdacfce-original-pull-secret") pod "global-pull-secret-syncer-qpp69" (UID: "3428bc6c-ab5c-47f1-8f1d-f5923fdacfce") : object "kube-system"/"original-pull-secret" not registered
Sep 16 19:39:49.750806 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:49.750720    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb28t\" (UniqueName: \"kubernetes.io/projected/b9840a22-6dcc-4836-b682-d1d667d2e30f-kube-api-access-xb28t\") pod \"network-check-target-2cjmm\" (UID: \"b9840a22-6dcc-4836-b682-d1d667d2e30f\") " pod="openshift-network-diagnostics/network-check-target-2cjmm"
Sep 16 19:39:49.750983 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:49.750917    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 19:39:49.750983 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:49.750946    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 19:39:49.750983 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:49.750959    2557 projected.go:196] Error preparing data for projected volume kube-api-access-xb28t for pod openshift-network-diagnostics/network-check-target-2cjmm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 19:39:49.751153 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:49.751021    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b9840a22-6dcc-4836-b682-d1d667d2e30f-kube-api-access-xb28t podName:b9840a22-6dcc-4836-b682-d1d667d2e30f nodeName:}" failed. No retries permitted until 2026-09-16 19:40:05.751002764 +0000 UTC m=+34.276391144 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-xb28t" (UniqueName: "kubernetes.io/projected/b9840a22-6dcc-4836-b682-d1d667d2e30f-kube-api-access-xb28t") pod "network-check-target-2cjmm" (UID: "b9840a22-6dcc-4836-b682-d1d667d2e30f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 19:39:49.931325 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:49.931077    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qpp69"
Sep 16 19:39:49.931325 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:49.931092    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-drgl8"
Sep 16 19:39:49.931325 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:49.931214    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2cjmm"
Sep 16 19:39:49.931325 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:49.931219    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qpp69" podUID="3428bc6c-ab5c-47f1-8f1d-f5923fdacfce"
Sep 16 19:39:49.931325 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:49.931317    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-drgl8" podUID="b4ea9fa1-48c7-45be-bfcf-4594ee751086"
Sep 16 19:39:49.931703 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:49.931403    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2cjmm" podUID="b9840a22-6dcc-4836-b682-d1d667d2e30f"
Sep 16 19:39:51.932146 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:51.932111    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-drgl8"
Sep 16 19:39:51.932606 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:51.932213    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qpp69"
Sep 16 19:39:51.932606 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:51.932231    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-drgl8" podUID="b4ea9fa1-48c7-45be-bfcf-4594ee751086"
Sep 16 19:39:51.932606 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:51.932314    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2cjmm"
Sep 16 19:39:51.932606 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:51.932314    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qpp69" podUID="3428bc6c-ab5c-47f1-8f1d-f5923fdacfce"
Sep 16 19:39:51.932606 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:51.932419    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2cjmm" podUID="b9840a22-6dcc-4836-b682-d1d667d2e30f"
Sep 16 19:39:53.932381 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:53.931932    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qpp69"
Sep 16 19:39:53.932381 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:53.931934    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-drgl8"
Sep 16 19:39:53.932381 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:53.931940    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2cjmm"
Sep 16 19:39:53.933330 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:53.932394    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qpp69" podUID="3428bc6c-ab5c-47f1-8f1d-f5923fdacfce"
Sep 16 19:39:53.933330 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:53.932492    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2cjmm" podUID="b9840a22-6dcc-4836-b682-d1d667d2e30f"
Sep 16 19:39:53.933330 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:53.932582    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-drgl8" podUID="b4ea9fa1-48c7-45be-bfcf-4594ee751086"
Sep 16 19:39:54.044497 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:54.044461    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hhvcc" event={"ID":"bb2436af-e07f-464b-8973-1b0370888a65","Type":"ContainerStarted","Data":"f751aea3cb65b22e4a193e9d548290d73f05d02257ec00cfc8f29c67481bbab8"}
Sep 16 19:39:54.046190 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:54.046152    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-58kk4" event={"ID":"43cea22a-1cba-450a-8a7d-86beb5de6d75","Type":"ContainerStarted","Data":"e6608d592b65703a61dd810f6ae66ddc0f8454dd042490a7f9874a29ec5887d7"}
Sep 16 19:39:54.047552 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:54.047522    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-clfb8" event={"ID":"142b9c47-4826-4fa8-b228-5500d98fee67","Type":"ContainerStarted","Data":"dfe9b4f5957ede4b1febe4781798c3a04dcfdac907430295da37f7d4873a1fb2"}
Sep 16 19:39:54.050702 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:54.050680    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k" event={"ID":"98bd8e7f-1aa8-418b-ae78-208cbfb13392","Type":"ContainerStarted","Data":"8ee3bc01d4da72ab3791c1b4db504d88e7993f769c58646a9219c12395ef6aac"}
Sep 16 19:39:54.050819 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:54.050749    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k" event={"ID":"98bd8e7f-1aa8-418b-ae78-208cbfb13392","Type":"ContainerStarted","Data":"2c5bd64d5e358bf48d453bc3944accfad7378798d0f8be586ea17df84f1c39d4"}
Sep 16 19:39:54.050819 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:54.050799    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k" event={"ID":"98bd8e7f-1aa8-418b-ae78-208cbfb13392","Type":"ContainerStarted","Data":"c53a54beea5cd3af914f3dfe6cf1c1dd87352df58028c49256e9bab655b33fcd"}
Sep 16 19:39:54.050956 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:54.050866    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k" event={"ID":"98bd8e7f-1aa8-418b-ae78-208cbfb13392","Type":"ContainerStarted","Data":"8feed1ed1f8dc23b47ceef0107be7bd9c0f25f47360860c8e28c99c59f292f05"}
Sep 16 19:39:54.050956 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:54.050918    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k" event={"ID":"98bd8e7f-1aa8-418b-ae78-208cbfb13392","Type":"ContainerStarted","Data":"5048d3dec701e9e9316c354c4a5a80095e061d3af3cd229223260c45d4af96e1"}
Sep 16 19:39:54.051046 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:54.050967    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k" event={"ID":"98bd8e7f-1aa8-418b-ae78-208cbfb13392","Type":"ContainerStarted","Data":"832e459517259ee51b73d84f7b2fbd77dbec1b96e04bc5164228897481b3066b"}
Sep 16 19:39:54.052064 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:54.052042    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v57p4" event={"ID":"85c2f2c0-d2a8-454e-9e28-498c0c8d3c21","Type":"ContainerStarted","Data":"8781a94b2d60a8bf219f3d601d3537e5c77a029d68ca197b5f7eaaac1298b63e"}
Sep 16 19:39:54.053409 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:54.053391    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7cdd" event={"ID":"0e9a620d-8ac5-4f5d-8e0f-f172664bc35e","Type":"ContainerStarted","Data":"cf326256a7d24325a4b140e132778475bda9a8bc81442e865f3590b0b2f151b5"}
Sep 16 19:39:54.054825 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:54.054803    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-shdd9" event={"ID":"9d10b5b8-04a4-4a6e-ba00-d96a5e90ac22","Type":"ContainerStarted","Data":"ff041e1faad487e5d1c76d23f98dfb7f509c50ef5fe6f18dd8802526143227e9"}
Sep 16 19:39:54.056218 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:54.056197    2557 generic.go:356] "Generic (PLEG): container finished" podID="fb2ab49a-ef77-4425-91b4-2b6a6ca89630" containerID="1baa8f73dbc61d24615e504b8c4923f0400bc5ac1ea0a4feced6a36955b39bac" exitCode=0
Sep 16 19:39:54.056293 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:54.056228    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xv4vd" event={"ID":"fb2ab49a-ef77-4425-91b4-2b6a6ca89630","Type":"ContainerDied","Data":"1baa8f73dbc61d24615e504b8c4923f0400bc5ac1ea0a4feced6a36955b39bac"}
Sep 16 19:39:54.061897 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:54.061831    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-81.ec2.internal" podStartSLOduration=21.06181851 podStartE2EDuration="21.06181851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:39:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:39:37.032438345 +0000 UTC m=+5.557826735" watchObservedRunningTime="2026-09-16 19:39:54.06181851 +0000 UTC m=+22.587206895"
Sep 16 19:39:54.062240 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:54.062218    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hhvcc" podStartSLOduration=3.970637173 podStartE2EDuration="22.062209219s" totalImagesPullingTime="18.091572046s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:39:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:39:54.061350393 +0000 UTC m=+22.586738779" watchObservedRunningTime="2026-09-16 19:39:54.062209219 +0000 UTC m=+22.587597649"
Sep 16 19:39:54.104018 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:54.103966    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-v57p4" podStartSLOduration=7.565005033 podStartE2EDuration="22.103951802s" totalImagesPullingTime="14.538946769s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:39:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:39:54.103490057 +0000 UTC m=+22.628878441" watchObservedRunningTime="2026-09-16 19:39:54.103951802 +0000 UTC m=+22.629340186"
Sep 16 19:39:54.143322 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:54.143271    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-shdd9" podStartSLOduration=4.028872902 podStartE2EDuration="22.143254599s" totalImagesPullingTime="18.114381697s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:39:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:39:54.142914692 +0000 UTC m=+22.668303077" watchObservedRunningTime="2026-09-16 19:39:54.143254599 +0000 UTC m=+22.668642984"
Sep 16 19:39:54.143483 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:54.143363    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-clfb8" podStartSLOduration=4.054286473 podStartE2EDuration="22.143358823s" totalImagesPullingTime="18.08907235s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:39:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:39:54.123726329 +0000 UTC m=+22.649114715" watchObservedRunningTime="2026-09-16 19:39:54.143358823 +0000 UTC m=+22.668747209"
Sep 16 19:39:54.160729 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:54.160683    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-58kk4" podStartSLOduration=4.008570901 podStartE2EDuration="22.16066823s" totalImagesPullingTime="18.152097329s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:39:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:39:54.160597492 +0000 UTC m=+22.685985877" watchObservedRunningTime="2026-09-16 19:39:54.16066823 +0000 UTC m=+22.686056614"
Sep 16 19:39:54.427738 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:54.427710    2557 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 19:39:54.884262 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:54.884158    2557 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T19:39:54.427734888Z","UUID":"fb194b85-9a25-48a8-b654-d7e469facc2a","Handler":null,"Name":"","Endpoint":""}
Sep 16 19:39:54.887000 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:54.886971    2557 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 19:39:54.887137 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:54.887007    2557 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 19:39:55.062026 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:55.061974    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7cdd" event={"ID":"0e9a620d-8ac5-4f5d-8e0f-f172664bc35e","Type":"ContainerStarted","Data":"770804a904b2996386b53dd08921bdb4deeb3ec2a42e7585aff1654999482656"}
Sep 16 19:39:55.065151 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:55.064699    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vqqgj" event={"ID":"ed745f6b-93e2-416c-a9b3-9acc437b3d9d","Type":"ContainerStarted","Data":"136ad6e0270da38242ee200d8c8d3bd62534e76a0f3a7814f38e363bdde073ae"}
Sep 16 19:39:55.082504 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:55.082446    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vqqgj" podStartSLOduration=5.001459695 podStartE2EDuration="23.082423165s" totalImagesPullingTime="18.08096347s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:39:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:39:55.081673832 +0000 UTC m=+23.607062218" watchObservedRunningTime="2026-09-16 19:39:55.082423165 +0000 UTC m=+23.607811552"
Sep 16 19:39:55.931143 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:55.931086    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2cjmm"
Sep 16 19:39:55.931339 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:55.931207    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-drgl8"
Sep 16 19:39:55.931339 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:55.931220    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2cjmm" podUID="b9840a22-6dcc-4836-b682-d1d667d2e30f"
Sep 16 19:39:55.931479 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:55.931345    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-drgl8" podUID="b4ea9fa1-48c7-45be-bfcf-4594ee751086"
Sep 16 19:39:55.931479 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:55.931369    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qpp69"
Sep 16 19:39:55.931479 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:55.931455    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qpp69" podUID="3428bc6c-ab5c-47f1-8f1d-f5923fdacfce"
Sep 16 19:39:56.068833 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:56.068792    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7cdd" event={"ID":"0e9a620d-8ac5-4f5d-8e0f-f172664bc35e","Type":"ContainerStarted","Data":"388c3336dc88c6bbb34727296fd7954e4c7bda3624c599fca5f190603409ca07"}
Sep 16 19:39:56.073095 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:56.073068    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k" event={"ID":"98bd8e7f-1aa8-418b-ae78-208cbfb13392","Type":"ContainerStarted","Data":"97c38af35cd161296b947ce1ae640833d3799df1915e953861ce33f861434db7"}
Sep 16 19:39:56.088923 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:56.088870    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7cdd" podStartSLOduration=3.886121515 podStartE2EDuration="24.088854189s" totalImagesPullingTime="20.202732674s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:39:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:39:56.088606143 +0000 UTC m=+24.613994541" watchObservedRunningTime="2026-09-16 19:39:56.088854189 +0000 UTC m=+24.614242565"
Sep 16 19:39:56.253730 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:56.253688    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-v57p4"
Sep 16 19:39:56.254368 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:56.254348    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-v57p4"
Sep 16 19:39:57.075376 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:57.075345    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-v57p4"
Sep 16 19:39:57.075880 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:57.075861    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-v57p4"
Sep 16 19:39:57.931700 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:57.931668    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qpp69"
Sep 16 19:39:57.931700 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:57.931703    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-drgl8"
Sep 16 19:39:57.931927 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:57.931797    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qpp69" podUID="3428bc6c-ab5c-47f1-8f1d-f5923fdacfce"
Sep 16 19:39:57.931927 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:57.931833    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-drgl8" podUID="b4ea9fa1-48c7-45be-bfcf-4594ee751086"
Sep 16 19:39:57.931927 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:57.931879    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2cjmm"
Sep 16 19:39:57.932033 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:57.931934    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2cjmm" podUID="b9840a22-6dcc-4836-b682-d1d667d2e30f"
Sep 16 19:39:59.083240 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:59.083203    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k" event={"ID":"98bd8e7f-1aa8-418b-ae78-208cbfb13392","Type":"ContainerStarted","Data":"db79c7a1af2e6f212a861280d44e242d33b90214f0b1695a942bcd99b3aa241d"}
Sep 16 19:39:59.084113 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:59.083421    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:59.084113 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:59.083504    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:59.085357 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:59.085333    2557 generic.go:356] "Generic (PLEG): container finished" podID="fb2ab49a-ef77-4425-91b4-2b6a6ca89630" containerID="a2d3ce880e080a896b20d88495d90bea6805fdbda6129ba0bfdcdccf25b005ca" exitCode=0
Sep 16 19:39:59.085459 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:59.085413    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xv4vd" event={"ID":"fb2ab49a-ef77-4425-91b4-2b6a6ca89630","Type":"ContainerDied","Data":"a2d3ce880e080a896b20d88495d90bea6805fdbda6129ba0bfdcdccf25b005ca"}
Sep 16 19:39:59.098908 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:59.098891    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:39:59.116533 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:59.116487    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k" podStartSLOduration=8.956370621 podStartE2EDuration="27.11647512s" totalImagesPullingTime="18.160104499s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:39:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:39:59.115154839 +0000 UTC m=+27.640543236" watchObservedRunningTime="2026-09-16 19:39:59.11647512 +0000 UTC m=+27.641863505"
Sep 16 19:39:59.931256 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:59.931060    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-drgl8"
Sep 16 19:39:59.931510 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:59.931060    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2cjmm"
Sep 16 19:39:59.931510 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:59.931344    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-drgl8" podUID="b4ea9fa1-48c7-45be-bfcf-4594ee751086"
Sep 16 19:39:59.931510 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:59.931433    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2cjmm" podUID="b9840a22-6dcc-4836-b682-d1d667d2e30f"
Sep 16 19:39:59.931510 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:39:59.931132    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qpp69"
Sep 16 19:39:59.931704 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:39:59.931548    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qpp69" podUID="3428bc6c-ab5c-47f1-8f1d-f5923fdacfce"
Sep 16 19:40:00.006238 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:00.006128    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qpp69"]
Sep 16 19:40:00.011433 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:00.011393    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2cjmm"]
Sep 16 19:40:00.012287 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:00.012264    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-drgl8"]
Sep 16 19:40:00.087465 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:00.087435    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2cjmm"
Sep 16 19:40:00.087861 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:00.087476    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-drgl8"
Sep 16 19:40:00.087861 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:00.087429    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qpp69"
Sep 16 19:40:00.087861 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:40:00.087559    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2cjmm" podUID="b9840a22-6dcc-4836-b682-d1d667d2e30f"
Sep 16 19:40:00.087861 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:40:00.087652    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qpp69" podUID="3428bc6c-ab5c-47f1-8f1d-f5923fdacfce"
Sep 16 19:40:00.087861 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:40:00.087802    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-drgl8" podUID="b4ea9fa1-48c7-45be-bfcf-4594ee751086"
Sep 16 19:40:00.088338 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:00.088315    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:40:00.106185 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:00.106165    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:40:01.092301 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:01.092268    2557 generic.go:356] "Generic (PLEG): container finished" podID="fb2ab49a-ef77-4425-91b4-2b6a6ca89630" containerID="5f3d399641b6fd0d7e88781ab2d265bde0e4cd5faf2659c68d72d27cab551ebb" exitCode=0
Sep 16 19:40:01.092830 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:01.092360    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xv4vd" event={"ID":"fb2ab49a-ef77-4425-91b4-2b6a6ca89630","Type":"ContainerDied","Data":"5f3d399641b6fd0d7e88781ab2d265bde0e4cd5faf2659c68d72d27cab551ebb"}
Sep 16 19:40:01.931878 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:01.931827    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-drgl8"
Sep 16 19:40:01.932055 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:40:01.931981    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-drgl8" podUID="b4ea9fa1-48c7-45be-bfcf-4594ee751086"
Sep 16 19:40:01.932055 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:01.931833    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2cjmm"
Sep 16 19:40:01.932184 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:01.932111    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qpp69"
Sep 16 19:40:01.932184 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:40:01.932136    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2cjmm" podUID="b9840a22-6dcc-4836-b682-d1d667d2e30f"
Sep 16 19:40:01.932290 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:40:01.932223    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qpp69" podUID="3428bc6c-ab5c-47f1-8f1d-f5923fdacfce"
Sep 16 19:40:03.100179 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:03.099940    2557 generic.go:356] "Generic (PLEG): container finished" podID="fb2ab49a-ef77-4425-91b4-2b6a6ca89630" containerID="bbcec6924cfcf25b4d28826dcd1e9968d892ee3018378524ed59f4f0a26d33a4" exitCode=0
Sep 16 19:40:03.100179 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:03.100026    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xv4vd" event={"ID":"fb2ab49a-ef77-4425-91b4-2b6a6ca89630","Type":"ContainerDied","Data":"bbcec6924cfcf25b4d28826dcd1e9968d892ee3018378524ed59f4f0a26d33a4"}
Sep 16 19:40:03.931615 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:03.931582    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2cjmm"
Sep 16 19:40:03.931800 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:03.931582    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-drgl8"
Sep 16 19:40:03.931800 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:03.931731    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qpp69"
Sep 16 19:40:03.931800 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:40:03.931713    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2cjmm" podUID="b9840a22-6dcc-4836-b682-d1d667d2e30f"
Sep 16 19:40:03.931966 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:40:03.931833    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-drgl8" podUID="b4ea9fa1-48c7-45be-bfcf-4594ee751086"
Sep 16 19:40:03.931966 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:40:03.931943    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qpp69" podUID="3428bc6c-ab5c-47f1-8f1d-f5923fdacfce"
Sep 16 19:40:05.672695 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:05.672653    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3428bc6c-ab5c-47f1-8f1d-f5923fdacfce-original-pull-secret\") pod \"global-pull-secret-syncer-qpp69\" (UID: \"3428bc6c-ab5c-47f1-8f1d-f5923fdacfce\") " pod="kube-system/global-pull-secret-syncer-qpp69"
Sep 16 19:40:05.672695 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:05.672709    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4ea9fa1-48c7-45be-bfcf-4594ee751086-metrics-certs\") pod \"network-metrics-daemon-drgl8\" (UID: \"b4ea9fa1-48c7-45be-bfcf-4594ee751086\") " pod="openshift-multus/network-metrics-daemon-drgl8"
Sep 16 19:40:05.673213 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:40:05.672821    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 19:40:05.673213 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:40:05.672833    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 19:40:05.673213 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:40:05.672916    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4ea9fa1-48c7-45be-bfcf-4594ee751086-metrics-certs podName:b4ea9fa1-48c7-45be-bfcf-4594ee751086 nodeName:}" failed. No retries permitted until 2026-09-16 19:40:37.672896584 +0000 UTC m=+66.198284947 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b4ea9fa1-48c7-45be-bfcf-4594ee751086-metrics-certs") pod "network-metrics-daemon-drgl8" (UID: "b4ea9fa1-48c7-45be-bfcf-4594ee751086") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 19:40:05.673213 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:40:05.672935    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3428bc6c-ab5c-47f1-8f1d-f5923fdacfce-original-pull-secret podName:3428bc6c-ab5c-47f1-8f1d-f5923fdacfce nodeName:}" failed. No retries permitted until 2026-09-16 19:40:37.672925608 +0000 UTC m=+66.198313970 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3428bc6c-ab5c-47f1-8f1d-f5923fdacfce-original-pull-secret") pod "global-pull-secret-syncer-qpp69" (UID: "3428bc6c-ab5c-47f1-8f1d-f5923fdacfce") : object "kube-system"/"original-pull-secret" not registered
Sep 16 19:40:05.773975 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:05.773939    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb28t\" (UniqueName: \"kubernetes.io/projected/b9840a22-6dcc-4836-b682-d1d667d2e30f-kube-api-access-xb28t\") pod \"network-check-target-2cjmm\" (UID: \"b9840a22-6dcc-4836-b682-d1d667d2e30f\") " pod="openshift-network-diagnostics/network-check-target-2cjmm"
Sep 16 19:40:05.774174 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:40:05.774114    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 19:40:05.774174 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:40:05.774134    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 19:40:05.774174 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:40:05.774148    2557 projected.go:196] Error preparing data for projected volume kube-api-access-xb28t for pod openshift-network-diagnostics/network-check-target-2cjmm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 19:40:05.774360 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:40:05.774210    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b9840a22-6dcc-4836-b682-d1d667d2e30f-kube-api-access-xb28t podName:b9840a22-6dcc-4836-b682-d1d667d2e30f nodeName:}" failed. No retries permitted until 2026-09-16 19:40:37.77419215 +0000 UTC m=+66.299580518 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-xb28t" (UniqueName: "kubernetes.io/projected/b9840a22-6dcc-4836-b682-d1d667d2e30f-kube-api-access-xb28t") pod "network-check-target-2cjmm" (UID: "b9840a22-6dcc-4836-b682-d1d667d2e30f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 19:40:05.931681 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:05.931607    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2cjmm"
Sep 16 19:40:05.931862 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:05.931607    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-drgl8"
Sep 16 19:40:05.931862 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:40:05.931726    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2cjmm" podUID="b9840a22-6dcc-4836-b682-d1d667d2e30f"
Sep 16 19:40:05.931862 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:05.931751    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qpp69"
Sep 16 19:40:05.932005 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:40:05.931865    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-drgl8" podUID="b4ea9fa1-48c7-45be-bfcf-4594ee751086"
Sep 16 19:40:05.932005 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:40:05.931948    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qpp69" podUID="3428bc6c-ab5c-47f1-8f1d-f5923fdacfce"
Sep 16 19:40:06.741665 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:06.741628    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-81.ec2.internal" event="NodeReady"
Sep 16 19:40:06.742112 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:06.741780    2557 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 19:40:06.829538 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:06.829458    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-r2rtx"]
Sep 16 19:40:06.832220 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:06.832191    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-r2rtx"
Sep 16 19:40:06.838356 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:06.838328    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-d8hqp"]
Sep 16 19:40:06.840989 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:06.840601    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 19:40:06.840989 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:06.840657    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d8hqp"
Sep 16 19:40:06.840989 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:06.840737    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 19:40:06.840989 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:06.840787    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-tf6pp\""
Sep 16 19:40:06.840989 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:06.840902    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 19:40:06.847156 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:06.847129    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 19:40:06.847280 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:06.847242    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wdb5m\""
Sep 16 19:40:06.847388 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:06.847373    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 19:40:06.853330 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:06.853293    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-r2rtx"]
Sep 16 19:40:06.876225 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:06.876146    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-d8hqp"]
Sep 16 19:40:06.984134 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:06.984095    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/955ae2fa-b138-46de-b15d-a15850b54bb0-config-volume\") pod \"dns-default-d8hqp\" (UID: \"955ae2fa-b138-46de-b15d-a15850b54bb0\") " pod="openshift-dns/dns-default-d8hqp"
Sep 16 19:40:06.984134 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:06.984136    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/955ae2fa-b138-46de-b15d-a15850b54bb0-tmp-dir\") pod \"dns-default-d8hqp\" (UID: \"955ae2fa-b138-46de-b15d-a15850b54bb0\") " pod="openshift-dns/dns-default-d8hqp"
Sep 16 19:40:06.984380 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:06.984159    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mvcj\" (UniqueName: \"kubernetes.io/projected/955ae2fa-b138-46de-b15d-a15850b54bb0-kube-api-access-6mvcj\") pod \"dns-default-d8hqp\" (UID: \"955ae2fa-b138-46de-b15d-a15850b54bb0\") " pod="openshift-dns/dns-default-d8hqp"
Sep 16 19:40:06.984380 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:06.984322    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/955ae2fa-b138-46de-b15d-a15850b54bb0-metrics-tls\") pod \"dns-default-d8hqp\" (UID: \"955ae2fa-b138-46de-b15d-a15850b54bb0\") " pod="openshift-dns/dns-default-d8hqp"
Sep 16 19:40:06.984453 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:06.984392    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjj6t\" (UniqueName: \"kubernetes.io/projected/af904ddd-40b6-48f8-9917-8ed2eb4b256a-kube-api-access-gjj6t\") pod \"ingress-canary-r2rtx\" (UID: \"af904ddd-40b6-48f8-9917-8ed2eb4b256a\") " pod="openshift-ingress-canary/ingress-canary-r2rtx"
Sep 16 19:40:06.984453 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:06.984431    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/af904ddd-40b6-48f8-9917-8ed2eb4b256a-cert\") pod \"ingress-canary-r2rtx\" (UID: \"af904ddd-40b6-48f8-9917-8ed2eb4b256a\") " pod="openshift-ingress-canary/ingress-canary-r2rtx"
Sep 16 19:40:07.085082 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:07.085004    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/955ae2fa-b138-46de-b15d-a15850b54bb0-config-volume\") pod \"dns-default-d8hqp\" (UID: \"955ae2fa-b138-46de-b15d-a15850b54bb0\") " pod="openshift-dns/dns-default-d8hqp"
Sep 16 19:40:07.085082 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:07.085041    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/955ae2fa-b138-46de-b15d-a15850b54bb0-tmp-dir\") pod \"dns-default-d8hqp\" (UID: \"955ae2fa-b138-46de-b15d-a15850b54bb0\") " pod="openshift-dns/dns-default-d8hqp"
Sep 16 19:40:07.085082 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:07.085059    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mvcj\" (UniqueName: \"kubernetes.io/projected/955ae2fa-b138-46de-b15d-a15850b54bb0-kube-api-access-6mvcj\") pod \"dns-default-d8hqp\" (UID: \"955ae2fa-b138-46de-b15d-a15850b54bb0\") " pod="openshift-dns/dns-default-d8hqp"
Sep 16 19:40:07.085331 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:07.085290    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/955ae2fa-b138-46de-b15d-a15850b54bb0-metrics-tls\") pod \"dns-default-d8hqp\" (UID: \"955ae2fa-b138-46de-b15d-a15850b54bb0\") " pod="openshift-dns/dns-default-d8hqp"
Sep 16 19:40:07.085386 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:07.085353    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjj6t\" (UniqueName: \"kubernetes.io/projected/af904ddd-40b6-48f8-9917-8ed2eb4b256a-kube-api-access-gjj6t\") pod \"ingress-canary-r2rtx\" (UID: \"af904ddd-40b6-48f8-9917-8ed2eb4b256a\") " pod="openshift-ingress-canary/ingress-canary-r2rtx"
Sep 16 19:40:07.085441 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:07.085406    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/af904ddd-40b6-48f8-9917-8ed2eb4b256a-cert\") pod \"ingress-canary-r2rtx\" (UID: \"af904ddd-40b6-48f8-9917-8ed2eb4b256a\") " pod="openshift-ingress-canary/ingress-canary-r2rtx"
Sep 16 19:40:07.085441 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:40:07.085423    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 19:40:07.085519 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:07.085423    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/955ae2fa-b138-46de-b15d-a15850b54bb0-tmp-dir\") pod \"dns-default-d8hqp\" (UID: \"955ae2fa-b138-46de-b15d-a15850b54bb0\") " pod="openshift-dns/dns-default-d8hqp"
Sep 16 19:40:07.085519 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:40:07.085479    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/955ae2fa-b138-46de-b15d-a15850b54bb0-metrics-tls podName:955ae2fa-b138-46de-b15d-a15850b54bb0 nodeName:}" failed. No retries permitted until 2026-09-16 19:40:07.585457447 +0000 UTC m=+36.110845815 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/955ae2fa-b138-46de-b15d-a15850b54bb0-metrics-tls") pod "dns-default-d8hqp" (UID: "955ae2fa-b138-46de-b15d-a15850b54bb0") : secret "dns-default-metrics-tls" not found
Sep 16 19:40:07.085705 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:07.085680    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/955ae2fa-b138-46de-b15d-a15850b54bb0-config-volume\") pod \"dns-default-d8hqp\" (UID: \"955ae2fa-b138-46de-b15d-a15850b54bb0\") " pod="openshift-dns/dns-default-d8hqp"
Sep 16 19:40:07.089731 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:07.089699    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/af904ddd-40b6-48f8-9917-8ed2eb4b256a-cert\") pod \"ingress-canary-r2rtx\" (UID: \"af904ddd-40b6-48f8-9917-8ed2eb4b256a\") " pod="openshift-ingress-canary/ingress-canary-r2rtx"
Sep 16 19:40:07.094924 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:07.094891    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjj6t\" (UniqueName: \"kubernetes.io/projected/af904ddd-40b6-48f8-9917-8ed2eb4b256a-kube-api-access-gjj6t\") pod \"ingress-canary-r2rtx\" (UID: \"af904ddd-40b6-48f8-9917-8ed2eb4b256a\") " pod="openshift-ingress-canary/ingress-canary-r2rtx"
Sep 16 19:40:07.095048 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:07.094891    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mvcj\" (UniqueName: \"kubernetes.io/projected/955ae2fa-b138-46de-b15d-a15850b54bb0-kube-api-access-6mvcj\") pod \"dns-default-d8hqp\" (UID: \"955ae2fa-b138-46de-b15d-a15850b54bb0\") " pod="openshift-dns/dns-default-d8hqp"
Sep 16 19:40:07.145143 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:07.145100    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-r2rtx"
Sep 16 19:40:07.428700 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:07.428396    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-r2rtx"]
Sep 16 19:40:07.431501 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:40:07.431474    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf904ddd_40b6_48f8_9917_8ed2eb4b256a.slice/crio-4bd759ea5d2f5479e14a4292d38603a148d94eb642f36cd0929e6db3d1dee9f0 WatchSource:0}: Error finding container 4bd759ea5d2f5479e14a4292d38603a148d94eb642f36cd0929e6db3d1dee9f0: Status 404 returned error can't find the container with id 4bd759ea5d2f5479e14a4292d38603a148d94eb642f36cd0929e6db3d1dee9f0
Sep 16 19:40:07.589433 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:07.589396    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/955ae2fa-b138-46de-b15d-a15850b54bb0-metrics-tls\") pod \"dns-default-d8hqp\" (UID: \"955ae2fa-b138-46de-b15d-a15850b54bb0\") " pod="openshift-dns/dns-default-d8hqp"
Sep 16 19:40:07.589552 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:40:07.589523    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 19:40:07.589588 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:40:07.589579    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/955ae2fa-b138-46de-b15d-a15850b54bb0-metrics-tls podName:955ae2fa-b138-46de-b15d-a15850b54bb0 nodeName:}" failed. No retries permitted until 2026-09-16 19:40:08.589566182 +0000 UTC m=+37.114954544 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/955ae2fa-b138-46de-b15d-a15850b54bb0-metrics-tls") pod "dns-default-d8hqp" (UID: "955ae2fa-b138-46de-b15d-a15850b54bb0") : secret "dns-default-metrics-tls" not found
Sep 16 19:40:07.931422 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:07.931386    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qpp69"
Sep 16 19:40:07.932365 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:07.931386    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2cjmm"
Sep 16 19:40:07.932365 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:07.931532    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-drgl8"
Sep 16 19:40:07.934748 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:07.934728    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qmvpx\""
Sep 16 19:40:07.936039 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:07.936016    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 19:40:07.936188 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:07.936174    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 19:40:07.936269 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:07.936185    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 19:40:07.936269 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:07.936219    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 19:40:07.936571 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:07.936552    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2djhj\""
Sep 16 19:40:08.115768 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:08.115714    2557 generic.go:356] "Generic (PLEG): container finished" podID="fb2ab49a-ef77-4425-91b4-2b6a6ca89630" containerID="c6a250bd03410e1f217c2bd457e3185fe50a6148344d0c32c5e9b6dd23e5c643" exitCode=0
Sep 16 19:40:08.115938 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:08.115857    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xv4vd" event={"ID":"fb2ab49a-ef77-4425-91b4-2b6a6ca89630","Type":"ContainerDied","Data":"c6a250bd03410e1f217c2bd457e3185fe50a6148344d0c32c5e9b6dd23e5c643"}
Sep 16 19:40:08.117223 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:08.117180    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-r2rtx" event={"ID":"af904ddd-40b6-48f8-9917-8ed2eb4b256a","Type":"ContainerStarted","Data":"4bd759ea5d2f5479e14a4292d38603a148d94eb642f36cd0929e6db3d1dee9f0"}
Sep 16 19:40:08.598126 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:08.598079    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/955ae2fa-b138-46de-b15d-a15850b54bb0-metrics-tls\") pod \"dns-default-d8hqp\" (UID: \"955ae2fa-b138-46de-b15d-a15850b54bb0\") " pod="openshift-dns/dns-default-d8hqp"
Sep 16 19:40:08.598304 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:40:08.598246    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 19:40:08.598387 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:40:08.598314    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/955ae2fa-b138-46de-b15d-a15850b54bb0-metrics-tls podName:955ae2fa-b138-46de-b15d-a15850b54bb0 nodeName:}" failed. No retries permitted until 2026-09-16 19:40:10.598294284 +0000 UTC m=+39.123682646 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/955ae2fa-b138-46de-b15d-a15850b54bb0-metrics-tls") pod "dns-default-d8hqp" (UID: "955ae2fa-b138-46de-b15d-a15850b54bb0") : secret "dns-default-metrics-tls" not found
Sep 16 19:40:09.123691 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:09.123606    2557 generic.go:356] "Generic (PLEG): container finished" podID="fb2ab49a-ef77-4425-91b4-2b6a6ca89630" containerID="4a73e8179d13961e1891679b2700466d953399673465f01855fdd7451d30665a" exitCode=0
Sep 16 19:40:09.123691 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:09.123670    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xv4vd" event={"ID":"fb2ab49a-ef77-4425-91b4-2b6a6ca89630","Type":"ContainerDied","Data":"4a73e8179d13961e1891679b2700466d953399673465f01855fdd7451d30665a"}
Sep 16 19:40:10.131178 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:10.131139    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xv4vd" event={"ID":"fb2ab49a-ef77-4425-91b4-2b6a6ca89630","Type":"ContainerStarted","Data":"c0b6208d1b83ba7711b69c7400adbfca7e10b3de0a509f4e48173e66ba46e306"}
Sep 16 19:40:10.132684 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:10.132660    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-r2rtx" event={"ID":"af904ddd-40b6-48f8-9917-8ed2eb4b256a","Type":"ContainerStarted","Data":"d52596a1e401491c568368f80d44bc7b1d4e8b13161c5071821d87b1cbf0cad1"}
Sep 16 19:40:10.158259 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:10.156474    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xv4vd" podStartSLOduration=9.53838593 podStartE2EDuration="38.156455641s" totalImagesPullingTime="28.618069711s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:39:32 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:40:10.155981076 +0000 UTC m=+38.681369477" watchObservedRunningTime="2026-09-16 19:40:10.156455641 +0000 UTC m=+38.681844026"
Sep 16 19:40:10.172647 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:10.172591    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-r2rtx" podStartSLOduration=1.671580946 podStartE2EDuration="4.172576274s" totalImagesPullingTime="2.500995328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:40:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:40:10.171740203 +0000 UTC m=+38.697128587" watchObservedRunningTime="2026-09-16 19:40:10.172576274 +0000 UTC m=+38.697964683"
Sep 16 19:40:10.612524 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:10.612325    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/955ae2fa-b138-46de-b15d-a15850b54bb0-metrics-tls\") pod \"dns-default-d8hqp\" (UID: \"955ae2fa-b138-46de-b15d-a15850b54bb0\") " pod="openshift-dns/dns-default-d8hqp"
Sep 16 19:40:10.612709 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:40:10.612466    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 19:40:10.612709 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:40:10.612623    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/955ae2fa-b138-46de-b15d-a15850b54bb0-metrics-tls podName:955ae2fa-b138-46de-b15d-a15850b54bb0 nodeName:}" failed. No retries permitted until 2026-09-16 19:40:14.612608625 +0000 UTC m=+43.137996988 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/955ae2fa-b138-46de-b15d-a15850b54bb0-metrics-tls") pod "dns-default-d8hqp" (UID: "955ae2fa-b138-46de-b15d-a15850b54bb0") : secret "dns-default-metrics-tls" not found
Sep 16 19:40:14.642714 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:14.642674    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/955ae2fa-b138-46de-b15d-a15850b54bb0-metrics-tls\") pod \"dns-default-d8hqp\" (UID: \"955ae2fa-b138-46de-b15d-a15850b54bb0\") " pod="openshift-dns/dns-default-d8hqp"
Sep 16 19:40:14.643137 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:40:14.642805    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 19:40:14.643137 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:40:14.642881    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/955ae2fa-b138-46de-b15d-a15850b54bb0-metrics-tls podName:955ae2fa-b138-46de-b15d-a15850b54bb0 nodeName:}" failed. No retries permitted until 2026-09-16 19:40:22.642862508 +0000 UTC m=+51.168250875 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/955ae2fa-b138-46de-b15d-a15850b54bb0-metrics-tls") pod "dns-default-d8hqp" (UID: "955ae2fa-b138-46de-b15d-a15850b54bb0") : secret "dns-default-metrics-tls" not found
Sep 16 19:40:22.702532 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:22.702484    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/955ae2fa-b138-46de-b15d-a15850b54bb0-metrics-tls\") pod \"dns-default-d8hqp\" (UID: \"955ae2fa-b138-46de-b15d-a15850b54bb0\") " pod="openshift-dns/dns-default-d8hqp"
Sep 16 19:40:22.702993 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:40:22.702615    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 19:40:22.702993 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:40:22.702671    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/955ae2fa-b138-46de-b15d-a15850b54bb0-metrics-tls podName:955ae2fa-b138-46de-b15d-a15850b54bb0 nodeName:}" failed. No retries permitted until 2026-09-16 19:40:38.702657338 +0000 UTC m=+67.228045701 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/955ae2fa-b138-46de-b15d-a15850b54bb0-metrics-tls") pod "dns-default-d8hqp" (UID: "955ae2fa-b138-46de-b15d-a15850b54bb0") : secret "dns-default-metrics-tls" not found
Sep 16 19:40:32.112728 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:32.112699    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qbv6k"
Sep 16 19:40:37.693059 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:37.693016    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3428bc6c-ab5c-47f1-8f1d-f5923fdacfce-original-pull-secret\") pod \"global-pull-secret-syncer-qpp69\" (UID: \"3428bc6c-ab5c-47f1-8f1d-f5923fdacfce\") " pod="kube-system/global-pull-secret-syncer-qpp69"
Sep 16 19:40:37.693527 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:37.693067    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4ea9fa1-48c7-45be-bfcf-4594ee751086-metrics-certs\") pod \"network-metrics-daemon-drgl8\" (UID: \"b4ea9fa1-48c7-45be-bfcf-4594ee751086\") " pod="openshift-multus/network-metrics-daemon-drgl8"
Sep 16 19:40:37.696002 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:37.695981    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 19:40:37.696002 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:37.695997    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 19:40:37.703358 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:40:37.703342    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 19:40:37.703416 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:40:37.703408    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4ea9fa1-48c7-45be-bfcf-4594ee751086-metrics-certs podName:b4ea9fa1-48c7-45be-bfcf-4594ee751086 nodeName:}" failed. No retries permitted until 2026-09-16 19:41:41.703392061 +0000 UTC m=+130.228780424 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b4ea9fa1-48c7-45be-bfcf-4594ee751086-metrics-certs") pod "network-metrics-daemon-drgl8" (UID: "b4ea9fa1-48c7-45be-bfcf-4594ee751086") : secret "metrics-daemon-secret" not found
Sep 16 19:40:37.706706 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:37.706687    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3428bc6c-ab5c-47f1-8f1d-f5923fdacfce-original-pull-secret\") pod \"global-pull-secret-syncer-qpp69\" (UID: \"3428bc6c-ab5c-47f1-8f1d-f5923fdacfce\") " pod="kube-system/global-pull-secret-syncer-qpp69"
Sep 16 19:40:37.793619 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:37.793564    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb28t\" (UniqueName: \"kubernetes.io/projected/b9840a22-6dcc-4836-b682-d1d667d2e30f-kube-api-access-xb28t\") pod \"network-check-target-2cjmm\" (UID: \"b9840a22-6dcc-4836-b682-d1d667d2e30f\") " pod="openshift-network-diagnostics/network-check-target-2cjmm"
Sep 16 19:40:37.796954 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:37.796930    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 19:40:37.807818 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:37.807790    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 19:40:37.817873 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:37.817827    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb28t\" (UniqueName: \"kubernetes.io/projected/b9840a22-6dcc-4836-b682-d1d667d2e30f-kube-api-access-xb28t\") pod \"network-check-target-2cjmm\" (UID: \"b9840a22-6dcc-4836-b682-d1d667d2e30f\") " pod="openshift-network-diagnostics/network-check-target-2cjmm"
Sep 16 19:40:37.944698 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:37.944628    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qpp69"
Sep 16 19:40:37.956801 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:37.956777    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2djhj\""
Sep 16 19:40:37.963547 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:37.963528    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2cjmm"
Sep 16 19:40:38.237258 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:38.237226    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qpp69"]
Sep 16 19:40:38.240619 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:40:38.240593    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3428bc6c_ab5c_47f1_8f1d_f5923fdacfce.slice/crio-3aea7484d55f0022471cf5742b866c7cbfe342872799ed8d4de1e5e7c54e5870 WatchSource:0}: Error finding container 3aea7484d55f0022471cf5742b866c7cbfe342872799ed8d4de1e5e7c54e5870: Status 404 returned error can't find the container with id 3aea7484d55f0022471cf5742b866c7cbfe342872799ed8d4de1e5e7c54e5870
Sep 16 19:40:38.264257 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:38.264227    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2cjmm"]
Sep 16 19:40:38.268657 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:40:38.268630    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9840a22_6dcc_4836_b682_d1d667d2e30f.slice/crio-9c4baa21146e595f7d9ecdaa1bfda79621589f8d8a27c512d7b5adaa3fbc8c51 WatchSource:0}: Error finding container 9c4baa21146e595f7d9ecdaa1bfda79621589f8d8a27c512d7b5adaa3fbc8c51: Status 404 returned error can't find the container with id 9c4baa21146e595f7d9ecdaa1bfda79621589f8d8a27c512d7b5adaa3fbc8c51
Sep 16 19:40:38.800897 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:38.800835    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/955ae2fa-b138-46de-b15d-a15850b54bb0-metrics-tls\") pod \"dns-default-d8hqp\" (UID: \"955ae2fa-b138-46de-b15d-a15850b54bb0\") " pod="openshift-dns/dns-default-d8hqp"
Sep 16 19:40:38.801319 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:40:38.800993    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 19:40:38.801319 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:40:38.801058    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/955ae2fa-b138-46de-b15d-a15850b54bb0-metrics-tls podName:955ae2fa-b138-46de-b15d-a15850b54bb0 nodeName:}" failed. No retries permitted until 2026-09-16 19:41:10.801041282 +0000 UTC m=+99.326429666 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/955ae2fa-b138-46de-b15d-a15850b54bb0-metrics-tls") pod "dns-default-d8hqp" (UID: "955ae2fa-b138-46de-b15d-a15850b54bb0") : secret "dns-default-metrics-tls" not found
Sep 16 19:40:39.205795 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:39.205711    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2cjmm" event={"ID":"b9840a22-6dcc-4836-b682-d1d667d2e30f","Type":"ContainerStarted","Data":"9c4baa21146e595f7d9ecdaa1bfda79621589f8d8a27c512d7b5adaa3fbc8c51"}
Sep 16 19:40:39.207308 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:39.207254    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qpp69" event={"ID":"3428bc6c-ab5c-47f1-8f1d-f5923fdacfce","Type":"ContainerStarted","Data":"3aea7484d55f0022471cf5742b866c7cbfe342872799ed8d4de1e5e7c54e5870"}
Sep 16 19:40:43.219587 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:43.219538    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2cjmm" event={"ID":"b9840a22-6dcc-4836-b682-d1d667d2e30f","Type":"ContainerStarted","Data":"24d6ec6b5c4fee57763e4004969146aae24734d7a6bde3285e98267c55f71914"}
Sep 16 19:40:43.220158 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:43.219651    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2cjmm"
Sep 16 19:40:43.221140 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:43.221109    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qpp69" event={"ID":"3428bc6c-ab5c-47f1-8f1d-f5923fdacfce","Type":"ContainerStarted","Data":"49fecb3ffe718c149f8d055ab30937ab68d7327cd792fa373c2be1caec5d068f"}
Sep 16 19:40:43.238049 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:40:43.238000    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2cjmm" podStartSLOduration=66.805890391 podStartE2EDuration="1m11.237985811s" totalImagesPullingTime="4.43209542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:39:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:40:43.23722872 +0000 UTC m=+71.762617108" watchObservedRunningTime="2026-09-16 19:40:43.237985811 +0000 UTC m=+71.763374196"
Sep 16 19:41:10.816253 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:10.816199    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/955ae2fa-b138-46de-b15d-a15850b54bb0-metrics-tls\") pod \"dns-default-d8hqp\" (UID: \"955ae2fa-b138-46de-b15d-a15850b54bb0\") " pod="openshift-dns/dns-default-d8hqp"
Sep 16 19:41:10.816684 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:41:10.816359    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 19:41:10.816684 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:41:10.816434    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/955ae2fa-b138-46de-b15d-a15850b54bb0-metrics-tls podName:955ae2fa-b138-46de-b15d-a15850b54bb0 nodeName:}" failed. No retries permitted until 2026-09-16 19:42:14.816417442 +0000 UTC m=+163.341805808 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/955ae2fa-b138-46de-b15d-a15850b54bb0-metrics-tls") pod "dns-default-d8hqp" (UID: "955ae2fa-b138-46de-b15d-a15850b54bb0") : secret "dns-default-metrics-tls" not found
Sep 16 19:41:14.226141 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:14.226108    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2cjmm"
Sep 16 19:41:14.246675 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:14.246623    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-qpp69" podStartSLOduration=97.782938212 podStartE2EDuration="1m42.246606961s" totalImagesPullingTime="4.463668749s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:39:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:40:43.252978919 +0000 UTC m=+71.778367295" watchObservedRunningTime="2026-09-16 19:41:14.246606961 +0000 UTC m=+102.771995393"
Sep 16 19:41:37.800967 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:37.800921    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-fbswk"]
Sep 16 19:41:37.803165 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:37.803143    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fbswk"
Sep 16 19:41:37.809610 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:37.809587    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 16 19:41:37.809723 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:37.809610    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-lmzbx\""
Sep 16 19:41:37.809723 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:37.809589    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 19:41:37.809723 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:37.809612    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 16 19:41:37.809723 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:37.809588    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 19:41:37.815767 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:37.815742    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-fbswk"]
Sep 16 19:41:37.895692 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:37.895653    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d6e55a03-bcc6-42f0-9b0c-b3c62b89d3ad-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-fbswk\" (UID: \"d6e55a03-bcc6-42f0-9b0c-b3c62b89d3ad\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fbswk"
Sep 16 19:41:37.895926 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:37.895729    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm4j7\" (UniqueName: \"kubernetes.io/projected/d6e55a03-bcc6-42f0-9b0c-b3c62b89d3ad-kube-api-access-jm4j7\") pod \"cluster-monitoring-operator-6cd954b897-fbswk\" (UID: \"d6e55a03-bcc6-42f0-9b0c-b3c62b89d3ad\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fbswk"
Sep 16 19:41:37.895926 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:37.895772    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d6e55a03-bcc6-42f0-9b0c-b3c62b89d3ad-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-fbswk\" (UID: \"d6e55a03-bcc6-42f0-9b0c-b3c62b89d3ad\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fbswk"
Sep 16 19:41:37.919451 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:37.919416    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-b4b87b796-l78vs"]
Sep 16 19:41:37.922051 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:37.921747    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-b4b87b796-l78vs"
Sep 16 19:41:37.922425 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:37.922396    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-9tm7s"]
Sep 16 19:41:37.924483 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:37.924462    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-9tm7s"
Sep 16 19:41:37.928519 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:37.928498    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 16 19:41:37.928633 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:37.928568    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 19:41:37.928778 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:37.928765    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 16 19:41:37.929311 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:37.929292    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-4h9s8\""
Sep 16 19:41:37.929405 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:37.929330    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 19:41:37.934772 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:37.934750    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 16 19:41:37.934895 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:37.934775    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-bxh4r\""
Sep 16 19:41:37.936388 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:37.936373    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 16 19:41:37.936936 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:37.936923    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 16 19:41:37.937769 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:37.937757    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 19:41:37.942540 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:37.942519    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 16 19:41:37.942755 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:37.942740    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 16 19:41:37.946805 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:37.946789    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 19:41:37.975355 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:37.975320    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-9tm7s"]
Sep 16 19:41:37.996803 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:37.996776    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/25b2d6d3-b32d-4e8f-96c1-7701d5609d6f-default-certificate\") pod \"router-default-b4b87b796-l78vs\" (UID: \"25b2d6d3-b32d-4e8f-96c1-7701d5609d6f\") " pod="openshift-ingress/router-default-b4b87b796-l78vs"
Sep 16 19:41:37.996932 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:37.996807    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25b2d6d3-b32d-4e8f-96c1-7701d5609d6f-metrics-certs\") pod \"router-default-b4b87b796-l78vs\" (UID: \"25b2d6d3-b32d-4e8f-96c1-7701d5609d6f\") " pod="openshift-ingress/router-default-b4b87b796-l78vs"
Sep 16 19:41:37.996932 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:37.996872    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm4j7\" (UniqueName: \"kubernetes.io/projected/d6e55a03-bcc6-42f0-9b0c-b3c62b89d3ad-kube-api-access-jm4j7\") pod \"cluster-monitoring-operator-6cd954b897-fbswk\" (UID: \"d6e55a03-bcc6-42f0-9b0c-b3c62b89d3ad\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fbswk"
Sep 16 19:41:37.996932 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:37.996898    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8ebbee05-192c-4db3-ba7a-f0de3c33983c-serving-cert\") pod \"insights-operator-5844b856db-9tm7s\" (UID: \"8ebbee05-192c-4db3-ba7a-f0de3c33983c\") " pod="openshift-insights/insights-operator-5844b856db-9tm7s"
Sep 16 19:41:37.997038 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:37.996931    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8ebbee05-192c-4db3-ba7a-f0de3c33983c-snapshots\") pod \"insights-operator-5844b856db-9tm7s\" (UID: \"8ebbee05-192c-4db3-ba7a-f0de3c33983c\") " pod="openshift-insights/insights-operator-5844b856db-9tm7s"
Sep 16 19:41:37.997038 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:37.996948    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/25b2d6d3-b32d-4e8f-96c1-7701d5609d6f-stats-auth\") pod \"router-default-b4b87b796-l78vs\" (UID: \"25b2d6d3-b32d-4e8f-96c1-7701d5609d6f\") " pod="openshift-ingress/router-default-b4b87b796-l78vs"
Sep 16 19:41:37.997038 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:37.996987    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d6e55a03-bcc6-42f0-9b0c-b3c62b89d3ad-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-fbswk\" (UID: \"d6e55a03-bcc6-42f0-9b0c-b3c62b89d3ad\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fbswk"
Sep 16 19:41:37.997144 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:37.997034    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ebbee05-192c-4db3-ba7a-f0de3c33983c-trusted-ca-bundle\") pod \"insights-operator-5844b856db-9tm7s\" (UID: \"8ebbee05-192c-4db3-ba7a-f0de3c33983c\") " pod="openshift-insights/insights-operator-5844b856db-9tm7s"
Sep 16 19:41:37.997144 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:37.997054    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ebbee05-192c-4db3-ba7a-f0de3c33983c-service-ca-bundle\") pod \"insights-operator-5844b856db-9tm7s\" (UID: \"8ebbee05-192c-4db3-ba7a-f0de3c33983c\") " pod="openshift-insights/insights-operator-5844b856db-9tm7s"
Sep 16 19:41:37.997144 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:37.997070    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25b2d6d3-b32d-4e8f-96c1-7701d5609d6f-service-ca-bundle\") pod \"router-default-b4b87b796-l78vs\" (UID: \"25b2d6d3-b32d-4e8f-96c1-7701d5609d6f\") " pod="openshift-ingress/router-default-b4b87b796-l78vs"
Sep 16 19:41:37.997144 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:37.997093    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx77b\" (UniqueName: \"kubernetes.io/projected/25b2d6d3-b32d-4e8f-96c1-7701d5609d6f-kube-api-access-sx77b\") pod \"router-default-b4b87b796-l78vs\" (UID: \"25b2d6d3-b32d-4e8f-96c1-7701d5609d6f\") " pod="openshift-ingress/router-default-b4b87b796-l78vs"
Sep 16 19:41:37.997144 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:37.997119    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ebbee05-192c-4db3-ba7a-f0de3c33983c-tmp\") pod \"insights-operator-5844b856db-9tm7s\" (UID: \"8ebbee05-192c-4db3-ba7a-f0de3c33983c\") " pod="openshift-insights/insights-operator-5844b856db-9tm7s"
Sep 16 19:41:37.997366 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:37.997210    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d6e55a03-bcc6-42f0-9b0c-b3c62b89d3ad-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-fbswk\" (UID: \"d6e55a03-bcc6-42f0-9b0c-b3c62b89d3ad\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fbswk"
Sep 16 19:41:37.997366 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:37.997251    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plj7j\" (UniqueName: \"kubernetes.io/projected/8ebbee05-192c-4db3-ba7a-f0de3c33983c-kube-api-access-plj7j\") pod \"insights-operator-5844b856db-9tm7s\" (UID: \"8ebbee05-192c-4db3-ba7a-f0de3c33983c\") " pod="openshift-insights/insights-operator-5844b856db-9tm7s"
Sep 16 19:41:37.997366 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:41:37.997319    2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 19:41:37.997474 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:41:37.997373    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6e55a03-bcc6-42f0-9b0c-b3c62b89d3ad-cluster-monitoring-operator-tls podName:d6e55a03-bcc6-42f0-9b0c-b3c62b89d3ad nodeName:}" failed. No retries permitted until 2026-09-16 19:41:38.497358117 +0000 UTC m=+127.022746480 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d6e55a03-bcc6-42f0-9b0c-b3c62b89d3ad-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-fbswk" (UID: "d6e55a03-bcc6-42f0-9b0c-b3c62b89d3ad") : secret "cluster-monitoring-operator-tls" not found
Sep 16 19:41:37.998365 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:37.998345    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d6e55a03-bcc6-42f0-9b0c-b3c62b89d3ad-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-fbswk\" (UID: \"d6e55a03-bcc6-42f0-9b0c-b3c62b89d3ad\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fbswk"
Sep 16 19:41:38.006779 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:38.006755    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-b4b87b796-l78vs"]
Sep 16 19:41:38.016346 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:38.016324    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm4j7\" (UniqueName: \"kubernetes.io/projected/d6e55a03-bcc6-42f0-9b0c-b3c62b89d3ad-kube-api-access-jm4j7\") pod \"cluster-monitoring-operator-6cd954b897-fbswk\" (UID: \"d6e55a03-bcc6-42f0-9b0c-b3c62b89d3ad\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fbswk"
Sep 16 19:41:38.017148 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:38.017128    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-qfdnr"]
Sep 16 19:41:38.019126 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:38.019108    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qfdnr"
Sep 16 19:41:38.022957 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:38.022940    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 16 19:41:38.023057 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:38.022976    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 16 19:41:38.023283 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:38.023267    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-fmq7t\""
Sep 16 19:41:38.023383 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:38.023355    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 16 19:41:38.023479 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:38.023464    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 16 19:41:38.024474 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:38.024451    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-hz77x"]
Sep 16 19:41:38.026722 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:38.026706    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-hz77x"
Sep 16 19:41:38.030116 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:38.030099    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-nljrb\""
Sep 16 19:41:38.030915 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:38.030894    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-qfdnr"]
Sep 16 19:41:38.038787 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:38.038765    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-hz77x"]
Sep 16 19:41:38.097835 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:38.097744    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v97ds\" (UniqueName: \"kubernetes.io/projected/8c006c38-92a7-44d9-8791-6d4c194111ca-kube-api-access-v97ds\") pod \"network-check-source-6f8b768455-hz77x\" (UID: \"8c006c38-92a7-44d9-8791-6d4c194111ca\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-hz77x"
Sep 16 19:41:38.097835 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:38.097806    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plj7j\" (UniqueName: \"kubernetes.io/projected/8ebbee05-192c-4db3-ba7a-f0de3c33983c-kube-api-access-plj7j\") pod \"insights-operator-5844b856db-9tm7s\" (UID: \"8ebbee05-192c-4db3-ba7a-f0de3c33983c\") " pod="openshift-insights/insights-operator-5844b856db-9tm7s"
Sep 16 19:41:38.098089 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:38.097904    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/25b2d6d3-b32d-4e8f-96c1-7701d5609d6f-default-certificate\") pod \"router-default-b4b87b796-l78vs\" (UID: \"25b2d6d3-b32d-4e8f-96c1-7701d5609d6f\") " pod="openshift-ingress/router-default-b4b87b796-l78vs"
Sep 16 19:41:38.098089 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:38.097937    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25b2d6d3-b32d-4e8f-96c1-7701d5609d6f-metrics-certs\") pod \"router-default-b4b87b796-l78vs\" (UID: \"25b2d6d3-b32d-4e8f-96c1-7701d5609d6f\") " pod="openshift-ingress/router-default-b4b87b796-l78vs"
Sep 16 19:41:38.098089 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:38.097965    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcdll\" (UniqueName: \"kubernetes.io/projected/0f022ef3-16e6-4890-ae68-3644f69ee234-kube-api-access-gcdll\") pod \"service-ca-operator-cf7c776b7-qfdnr\" (UID: \"0f022ef3-16e6-4890-ae68-3644f69ee234\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qfdnr"
Sep 16 19:41:38.098089 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:38.098007    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8ebbee05-192c-4db3-ba7a-f0de3c33983c-serving-cert\") pod \"insights-operator-5844b856db-9tm7s\" (UID: \"8ebbee05-192c-4db3-ba7a-f0de3c33983c\") " pod="openshift-insights/insights-operator-5844b856db-9tm7s"
Sep 16 19:41:38.098089 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:38.098051    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8ebbee05-192c-4db3-ba7a-f0de3c33983c-snapshots\") pod \"insights-operator-5844b856db-9tm7s\" (UID: \"8ebbee05-192c-4db3-ba7a-f0de3c33983c\") " pod="openshift-insights/insights-operator-5844b856db-9tm7s"
Sep 16 19:41:38.098089 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:41:38.098073    2557 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 19:41:38.098385 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:38.098077    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/25b2d6d3-b32d-4e8f-96c1-7701d5609d6f-stats-auth\") pod \"router-default-b4b87b796-l78vs\" (UID: \"25b2d6d3-b32d-4e8f-96c1-7701d5609d6f\") " pod="openshift-ingress/router-default-b4b87b796-l78vs"
Sep 16 19:41:38.098385 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:41:38.098188    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25b2d6d3-b32d-4e8f-96c1-7701d5609d6f-metrics-certs podName:25b2d6d3-b32d-4e8f-96c1-7701d5609d6f nodeName:}" failed. No retries permitted until 2026-09-16 19:41:38.598165615 +0000 UTC m=+127.123553994 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/25b2d6d3-b32d-4e8f-96c1-7701d5609d6f-metrics-certs") pod "router-default-b4b87b796-l78vs" (UID: "25b2d6d3-b32d-4e8f-96c1-7701d5609d6f") : secret "router-metrics-certs-default" not found
Sep 16 19:41:38.098385 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:38.098214    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0f022ef3-16e6-4890-ae68-3644f69ee234-config\") pod \"service-ca-operator-cf7c776b7-qfdnr\" (UID: \"0f022ef3-16e6-4890-ae68-3644f69ee234\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qfdnr"
Sep 16 19:41:38.098385 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:38.098244    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0f022ef3-16e6-4890-ae68-3644f69ee234-serving-cert\") pod \"service-ca-operator-cf7c776b7-qfdnr\" (UID: \"0f022ef3-16e6-4890-ae68-3644f69ee234\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qfdnr"
Sep 16 19:41:38.098385 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:38.098296    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ebbee05-192c-4db3-ba7a-f0de3c33983c-trusted-ca-bundle\") pod \"insights-operator-5844b856db-9tm7s\" (UID: \"8ebbee05-192c-4db3-ba7a-f0de3c33983c\") " pod="openshift-insights/insights-operator-5844b856db-9tm7s"
Sep 16 19:41:38.098385 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:38.098328    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ebbee05-192c-4db3-ba7a-f0de3c33983c-service-ca-bundle\") pod \"insights-operator-5844b856db-9tm7s\" (UID: \"8ebbee05-192c-4db3-ba7a-f0de3c33983c\") " pod="openshift-insights/insights-operator-5844b856db-9tm7s"
Sep 16 19:41:38.098385 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:38.098357    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25b2d6d3-b32d-4e8f-96c1-7701d5609d6f-service-ca-bundle\") pod \"router-default-b4b87b796-l78vs\" (UID: \"25b2d6d3-b32d-4e8f-96c1-7701d5609d6f\") " pod="openshift-ingress/router-default-b4b87b796-l78vs"
Sep 16 19:41:38.098385 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:38.098382    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx77b\" (UniqueName: \"kubernetes.io/projected/25b2d6d3-b32d-4e8f-96c1-7701d5609d6f-kube-api-access-sx77b\") pod \"router-default-b4b87b796-l78vs\" (UID: \"25b2d6d3-b32d-4e8f-96c1-7701d5609d6f\") " pod="openshift-ingress/router-default-b4b87b796-l78vs"
Sep 16 19:41:38.098770 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:38.098415    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ebbee05-192c-4db3-ba7a-f0de3c33983c-tmp\") pod \"insights-operator-5844b856db-9tm7s\" (UID: \"8ebbee05-192c-4db3-ba7a-f0de3c33983c\") " pod="openshift-insights/insights-operator-5844b856db-9tm7s"
Sep 16 19:41:38.098770 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:41:38.098544    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/25b2d6d3-b32d-4e8f-96c1-7701d5609d6f-service-ca-bundle podName:25b2d6d3-b32d-4e8f-96c1-7701d5609d6f nodeName:}" failed. No retries permitted until 2026-09-16 19:41:38.598525704 +0000 UTC m=+127.123914069 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/25b2d6d3-b32d-4e8f-96c1-7701d5609d6f-service-ca-bundle") pod "router-default-b4b87b796-l78vs" (UID: "25b2d6d3-b32d-4e8f-96c1-7701d5609d6f") : configmap references non-existent config key: service-ca.crt
Sep 16 19:41:38.098770 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:38.098752    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ebbee05-192c-4db3-ba7a-f0de3c33983c-tmp\") pod \"insights-operator-5844b856db-9tm7s\" (UID: \"8ebbee05-192c-4db3-ba7a-f0de3c33983c\") " pod="openshift-insights/insights-operator-5844b856db-9tm7s"
Sep 16 19:41:38.098951 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:38.098777    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8ebbee05-192c-4db3-ba7a-f0de3c33983c-snapshots\") pod \"insights-operator-5844b856db-9tm7s\" (UID: \"8ebbee05-192c-4db3-ba7a-f0de3c33983c\") " pod="openshift-insights/insights-operator-5844b856db-9tm7s"
Sep 16 19:41:38.099077 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:38.099053    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ebbee05-192c-4db3-ba7a-f0de3c33983c-service-ca-bundle\") pod \"insights-operator-5844b856db-9tm7s\" (UID: \"8ebbee05-192c-4db3-ba7a-f0de3c33983c\") " pod="openshift-insights/insights-operator-5844b856db-9tm7s"
Sep 16 19:41:38.099181 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:38.099163    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ebbee05-192c-4db3-ba7a-f0de3c33983c-trusted-ca-bundle\") pod \"insights-operator-5844b856db-9tm7s\" (UID: \"8ebbee05-192c-4db3-ba7a-f0de3c33983c\") " pod="openshift-insights/insights-operator-5844b856db-9tm7s"
Sep 16 19:41:38.100536 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:38.100508    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8ebbee05-192c-4db3-ba7a-f0de3c33983c-serving-cert\") pod \"insights-operator-5844b856db-9tm7s\" (UID: \"8ebbee05-192c-4db3-ba7a-f0de3c33983c\") " pod="openshift-insights/insights-operator-5844b856db-9tm7s"
Sep 16 19:41:38.100659 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:38.100573    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/25b2d6d3-b32d-4e8f-96c1-7701d5609d6f-default-certificate\") pod \"router-default-b4b87b796-l78vs\" (UID: \"25b2d6d3-b32d-4e8f-96c1-7701d5609d6f\") " pod="openshift-ingress/router-default-b4b87b796-l78vs"
Sep 16 19:41:38.100659 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:38.100593    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/25b2d6d3-b32d-4e8f-96c1-7701d5609d6f-stats-auth\") pod \"router-default-b4b87b796-l78vs\" (UID: \"25b2d6d3-b32d-4e8f-96c1-7701d5609d6f\") " pod="openshift-ingress/router-default-b4b87b796-l78vs"
Sep 16 19:41:38.117515 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:38.117495    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plj7j\" (UniqueName: \"kubernetes.io/projected/8ebbee05-192c-4db3-ba7a-f0de3c33983c-kube-api-access-plj7j\") pod \"insights-operator-5844b856db-9tm7s\" (UID: \"8ebbee05-192c-4db3-ba7a-f0de3c33983c\") " pod="openshift-insights/insights-operator-5844b856db-9tm7s"
Sep 16 19:41:38.124913 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:38.124893    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx77b\" (UniqueName: \"kubernetes.io/projected/25b2d6d3-b32d-4e8f-96c1-7701d5609d6f-kube-api-access-sx77b\") pod \"router-default-b4b87b796-l78vs\" (UID: \"25b2d6d3-b32d-4e8f-96c1-7701d5609d6f\") " pod="openshift-ingress/router-default-b4b87b796-l78vs"
Sep 16 19:41:38.199100 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:38.199063    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcdll\" (UniqueName: \"kubernetes.io/projected/0f022ef3-16e6-4890-ae68-3644f69ee234-kube-api-access-gcdll\") pod \"service-ca-operator-cf7c776b7-qfdnr\" (UID: \"0f022ef3-16e6-4890-ae68-3644f69ee234\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qfdnr"
Sep 16 19:41:38.199300 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:38.199123    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0f022ef3-16e6-4890-ae68-3644f69ee234-config\") pod \"service-ca-operator-cf7c776b7-qfdnr\" (UID: \"0f022ef3-16e6-4890-ae68-3644f69ee234\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qfdnr"
Sep 16 19:41:38.199300 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:38.199245    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0f022ef3-16e6-4890-ae68-3644f69ee234-serving-cert\") pod \"service-ca-operator-cf7c776b7-qfdnr\" (UID: \"0f022ef3-16e6-4890-ae68-3644f69ee234\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qfdnr"
Sep 16 19:41:38.199418 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:38.199319    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v97ds\" (UniqueName: \"kubernetes.io/projected/8c006c38-92a7-44d9-8791-6d4c194111ca-kube-api-access-v97ds\") pod \"network-check-source-6f8b768455-hz77x\" (UID: \"8c006c38-92a7-44d9-8791-6d4c194111ca\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-hz77x"
Sep 16 19:41:38.199641 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:38.199613    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0f022ef3-16e6-4890-ae68-3644f69ee234-config\") pod \"service-ca-operator-cf7c776b7-qfdnr\" (UID: \"0f022ef3-16e6-4890-ae68-3644f69ee234\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qfdnr"
Sep 16 19:41:38.201511 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:38.201492    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0f022ef3-16e6-4890-ae68-3644f69ee234-serving-cert\") pod \"service-ca-operator-cf7c776b7-qfdnr\" (UID: \"0f022ef3-16e6-4890-ae68-3644f69ee234\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qfdnr"
Sep 16 19:41:38.208481 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:38.208457    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcdll\" (UniqueName: \"kubernetes.io/projected/0f022ef3-16e6-4890-ae68-3644f69ee234-kube-api-access-gcdll\") pod \"service-ca-operator-cf7c776b7-qfdnr\" (UID: \"0f022ef3-16e6-4890-ae68-3644f69ee234\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qfdnr"
Sep 16 19:41:38.208584 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:38.208525    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v97ds\" (UniqueName: \"kubernetes.io/projected/8c006c38-92a7-44d9-8791-6d4c194111ca-kube-api-access-v97ds\") pod \"network-check-source-6f8b768455-hz77x\" (UID: \"8c006c38-92a7-44d9-8791-6d4c194111ca\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-hz77x"
Sep 16 19:41:38.236561 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:38.236536    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-9tm7s"
Sep 16 19:41:38.328433 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:38.328376    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qfdnr"
Sep 16 19:41:38.335545 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:38.335525    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-hz77x"
Sep 16 19:41:38.502443 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:38.502401    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d6e55a03-bcc6-42f0-9b0c-b3c62b89d3ad-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-fbswk\" (UID: \"d6e55a03-bcc6-42f0-9b0c-b3c62b89d3ad\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fbswk"
Sep 16 19:41:38.502606 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:41:38.502548    2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 19:41:38.502646 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:41:38.502616    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6e55a03-bcc6-42f0-9b0c-b3c62b89d3ad-cluster-monitoring-operator-tls podName:d6e55a03-bcc6-42f0-9b0c-b3c62b89d3ad nodeName:}" failed. No retries permitted until 2026-09-16 19:41:39.502600775 +0000 UTC m=+128.027989138 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d6e55a03-bcc6-42f0-9b0c-b3c62b89d3ad-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-fbswk" (UID: "d6e55a03-bcc6-42f0-9b0c-b3c62b89d3ad") : secret "cluster-monitoring-operator-tls" not found
Sep 16 19:41:38.509260 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:38.509233    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-9tm7s"]
Sep 16 19:41:38.513619 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:41:38.512203    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ebbee05_192c_4db3_ba7a_f0de3c33983c.slice/crio-5e064b53876b9f4fbbadf9ebf12b9685d04a55f5f87ca8fffbf581fc8b2c7f94 WatchSource:0}: Error finding container 5e064b53876b9f4fbbadf9ebf12b9685d04a55f5f87ca8fffbf581fc8b2c7f94: Status 404 returned error can't find the container with id 5e064b53876b9f4fbbadf9ebf12b9685d04a55f5f87ca8fffbf581fc8b2c7f94
Sep 16 19:41:38.599419 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:38.599392    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-qfdnr"]
Sep 16 19:41:38.601493 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:41:38.601462    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f022ef3_16e6_4890_ae68_3644f69ee234.slice/crio-aac5a892af8eb99c980eb3a3c2e9e21168f96b56247dbc48e8fc24608d69c8c2 WatchSource:0}: Error finding container aac5a892af8eb99c980eb3a3c2e9e21168f96b56247dbc48e8fc24608d69c8c2: Status 404 returned error can't find the container with id aac5a892af8eb99c980eb3a3c2e9e21168f96b56247dbc48e8fc24608d69c8c2
Sep 16 19:41:38.602854 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:38.602817    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25b2d6d3-b32d-4e8f-96c1-7701d5609d6f-service-ca-bundle\") pod \"router-default-b4b87b796-l78vs\" (UID: \"25b2d6d3-b32d-4e8f-96c1-7701d5609d6f\") " pod="openshift-ingress/router-default-b4b87b796-l78vs"
Sep 16 19:41:38.602951 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:38.602918    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25b2d6d3-b32d-4e8f-96c1-7701d5609d6f-metrics-certs\") pod \"router-default-b4b87b796-l78vs\" (UID: \"25b2d6d3-b32d-4e8f-96c1-7701d5609d6f\") " pod="openshift-ingress/router-default-b4b87b796-l78vs"
Sep 16 19:41:38.603120 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:41:38.603033    2557 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 19:41:38.603120 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:41:38.603062    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/25b2d6d3-b32d-4e8f-96c1-7701d5609d6f-service-ca-bundle podName:25b2d6d3-b32d-4e8f-96c1-7701d5609d6f nodeName:}" failed. No retries permitted until 2026-09-16 19:41:39.603037608 +0000 UTC m=+128.128425978 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/25b2d6d3-b32d-4e8f-96c1-7701d5609d6f-service-ca-bundle") pod "router-default-b4b87b796-l78vs" (UID: "25b2d6d3-b32d-4e8f-96c1-7701d5609d6f") : configmap references non-existent config key: service-ca.crt
Sep 16 19:41:38.603120 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:41:38.603091    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25b2d6d3-b32d-4e8f-96c1-7701d5609d6f-metrics-certs podName:25b2d6d3-b32d-4e8f-96c1-7701d5609d6f nodeName:}" failed. No retries permitted until 2026-09-16 19:41:39.603079259 +0000 UTC m=+128.128467628 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/25b2d6d3-b32d-4e8f-96c1-7701d5609d6f-metrics-certs") pod "router-default-b4b87b796-l78vs" (UID: "25b2d6d3-b32d-4e8f-96c1-7701d5609d6f") : secret "router-metrics-certs-default" not found
Sep 16 19:41:38.616000 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:38.615974    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-hz77x"]
Sep 16 19:41:38.618494 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:41:38.618468    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c006c38_92a7_44d9_8791_6d4c194111ca.slice/crio-9f3a9b69ade1fc02397ed5228fe95f5d91b0f3f446d92ff38f737c3fe5554098 WatchSource:0}: Error finding container 9f3a9b69ade1fc02397ed5228fe95f5d91b0f3f446d92ff38f737c3fe5554098: Status 404 returned error can't find the container with id 9f3a9b69ade1fc02397ed5228fe95f5d91b0f3f446d92ff38f737c3fe5554098
Sep 16 19:41:39.364949 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:39.364906    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-hz77x" event={"ID":"8c006c38-92a7-44d9-8791-6d4c194111ca","Type":"ContainerStarted","Data":"98af4bae32d59403aabbf0be262093f3dcbd0df1577fee127734b9a0525aeb6e"}
Sep 16 19:41:39.365429 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:39.365011    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-hz77x" event={"ID":"8c006c38-92a7-44d9-8791-6d4c194111ca","Type":"ContainerStarted","Data":"9f3a9b69ade1fc02397ed5228fe95f5d91b0f3f446d92ff38f737c3fe5554098"}
Sep 16 19:41:39.366956 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:39.366918    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-9tm7s" event={"ID":"8ebbee05-192c-4db3-ba7a-f0de3c33983c","Type":"ContainerStarted","Data":"5e064b53876b9f4fbbadf9ebf12b9685d04a55f5f87ca8fffbf581fc8b2c7f94"}
Sep 16 19:41:39.368470 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:39.368443    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qfdnr" event={"ID":"0f022ef3-16e6-4890-ae68-3644f69ee234","Type":"ContainerStarted","Data":"aac5a892af8eb99c980eb3a3c2e9e21168f96b56247dbc48e8fc24608d69c8c2"}
Sep 16 19:41:39.512552 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:39.512506    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d6e55a03-bcc6-42f0-9b0c-b3c62b89d3ad-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-fbswk\" (UID: \"d6e55a03-bcc6-42f0-9b0c-b3c62b89d3ad\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fbswk"
Sep 16 19:41:39.512736 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:41:39.512680    2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 19:41:39.512795 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:41:39.512764    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6e55a03-bcc6-42f0-9b0c-b3c62b89d3ad-cluster-monitoring-operator-tls podName:d6e55a03-bcc6-42f0-9b0c-b3c62b89d3ad nodeName:}" failed. No retries permitted until 2026-09-16 19:41:41.512741404 +0000 UTC m=+130.038129772 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d6e55a03-bcc6-42f0-9b0c-b3c62b89d3ad-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-fbswk" (UID: "d6e55a03-bcc6-42f0-9b0c-b3c62b89d3ad") : secret "cluster-monitoring-operator-tls" not found
Sep 16 19:41:39.613563 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:39.613519    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25b2d6d3-b32d-4e8f-96c1-7701d5609d6f-service-ca-bundle\") pod \"router-default-b4b87b796-l78vs\" (UID: \"25b2d6d3-b32d-4e8f-96c1-7701d5609d6f\") " pod="openshift-ingress/router-default-b4b87b796-l78vs"
Sep 16 19:41:39.613792 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:39.613621    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25b2d6d3-b32d-4e8f-96c1-7701d5609d6f-metrics-certs\") pod \"router-default-b4b87b796-l78vs\" (UID: \"25b2d6d3-b32d-4e8f-96c1-7701d5609d6f\") " pod="openshift-ingress/router-default-b4b87b796-l78vs"
Sep 16 19:41:39.613792 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:41:39.613726    2557 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 19:41:39.613792 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:41:39.613726    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/25b2d6d3-b32d-4e8f-96c1-7701d5609d6f-service-ca-bundle podName:25b2d6d3-b32d-4e8f-96c1-7701d5609d6f nodeName:}" failed. No retries permitted until 2026-09-16 19:41:41.613701373 +0000 UTC m=+130.139089751 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/25b2d6d3-b32d-4e8f-96c1-7701d5609d6f-service-ca-bundle") pod "router-default-b4b87b796-l78vs" (UID: "25b2d6d3-b32d-4e8f-96c1-7701d5609d6f") : configmap references non-existent config key: service-ca.crt
Sep 16 19:41:39.613792 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:41:39.613788    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25b2d6d3-b32d-4e8f-96c1-7701d5609d6f-metrics-certs podName:25b2d6d3-b32d-4e8f-96c1-7701d5609d6f nodeName:}" failed. No retries permitted until 2026-09-16 19:41:41.613772596 +0000 UTC m=+130.139160965 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/25b2d6d3-b32d-4e8f-96c1-7701d5609d6f-metrics-certs") pod "router-default-b4b87b796-l78vs" (UID: "25b2d6d3-b32d-4e8f-96c1-7701d5609d6f") : secret "router-metrics-certs-default" not found
Sep 16 19:41:41.376008 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:41.375964    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-9tm7s" event={"ID":"8ebbee05-192c-4db3-ba7a-f0de3c33983c","Type":"ContainerStarted","Data":"ed722bbaf6011cb3633fa26b18396928036155cacb52ba6e5021128078f42ec6"}
Sep 16 19:41:41.377451 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:41.377420    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qfdnr" event={"ID":"0f022ef3-16e6-4890-ae68-3644f69ee234","Type":"ContainerStarted","Data":"f71c54f15b23a38e372a4980a54960e13dc5e41b98c4503ea8c3b087515313fa"}
Sep 16 19:41:41.399140 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:41.399089    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-9tm7s" podStartSLOduration=2.021253169 podStartE2EDuration="4.399071963s" totalImagesPullingTime="2.377818794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:41:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:41:41.398392416 +0000 UTC m=+129.923780801" watchObservedRunningTime="2026-09-16 19:41:41.399071963 +0000 UTC m=+129.924460349"
Sep 16 19:41:41.399319 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:41.399202    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-hz77x" podStartSLOduration=4.3991944 podStartE2EDuration="4.3991944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:41:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:41:39.383656945 +0000 UTC m=+127.909045330" watchObservedRunningTime="2026-09-16 19:41:41.3991944 +0000 UTC m=+129.924582782"
Sep 16 19:41:41.420829 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:41.420763    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qfdnr" podStartSLOduration=2.131385651 podStartE2EDuration="4.420741542s" totalImagesPullingTime="2.289355891s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:41:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:41:41.419559122 +0000 UTC m=+129.944947510" watchObservedRunningTime="2026-09-16 19:41:41.420741542 +0000 UTC m=+129.946129928"
Sep 16 19:41:41.532085 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:41.532037    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d6e55a03-bcc6-42f0-9b0c-b3c62b89d3ad-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-fbswk\" (UID: \"d6e55a03-bcc6-42f0-9b0c-b3c62b89d3ad\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fbswk"
Sep 16 19:41:41.532288 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:41:41.532178    2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 19:41:41.532288 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:41:41.532260    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6e55a03-bcc6-42f0-9b0c-b3c62b89d3ad-cluster-monitoring-operator-tls podName:d6e55a03-bcc6-42f0-9b0c-b3c62b89d3ad nodeName:}" failed. No retries permitted until 2026-09-16 19:41:45.532238575 +0000 UTC m=+134.057626953 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d6e55a03-bcc6-42f0-9b0c-b3c62b89d3ad-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-fbswk" (UID: "d6e55a03-bcc6-42f0-9b0c-b3c62b89d3ad") : secret "cluster-monitoring-operator-tls" not found
Sep 16 19:41:41.634369 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:41.634264    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25b2d6d3-b32d-4e8f-96c1-7701d5609d6f-metrics-certs\") pod \"router-default-b4b87b796-l78vs\" (UID: \"25b2d6d3-b32d-4e8f-96c1-7701d5609d6f\") " pod="openshift-ingress/router-default-b4b87b796-l78vs"
Sep 16 19:41:41.634369 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:41.634352    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25b2d6d3-b32d-4e8f-96c1-7701d5609d6f-service-ca-bundle\") pod \"router-default-b4b87b796-l78vs\" (UID: \"25b2d6d3-b32d-4e8f-96c1-7701d5609d6f\") " pod="openshift-ingress/router-default-b4b87b796-l78vs"
Sep 16 19:41:41.634556 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:41:41.634427    2557 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 19:41:41.634556 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:41:41.634473    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/25b2d6d3-b32d-4e8f-96c1-7701d5609d6f-service-ca-bundle podName:25b2d6d3-b32d-4e8f-96c1-7701d5609d6f nodeName:}" failed. No retries permitted until 2026-09-16 19:41:45.634457153 +0000 UTC m=+134.159845518 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/25b2d6d3-b32d-4e8f-96c1-7701d5609d6f-service-ca-bundle") pod "router-default-b4b87b796-l78vs" (UID: "25b2d6d3-b32d-4e8f-96c1-7701d5609d6f") : configmap references non-existent config key: service-ca.crt
Sep 16 19:41:41.634556 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:41:41.634490    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25b2d6d3-b32d-4e8f-96c1-7701d5609d6f-metrics-certs podName:25b2d6d3-b32d-4e8f-96c1-7701d5609d6f nodeName:}" failed. No retries permitted until 2026-09-16 19:41:45.634482252 +0000 UTC m=+134.159870614 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/25b2d6d3-b32d-4e8f-96c1-7701d5609d6f-metrics-certs") pod "router-default-b4b87b796-l78vs" (UID: "25b2d6d3-b32d-4e8f-96c1-7701d5609d6f") : secret "router-metrics-certs-default" not found
Sep 16 19:41:41.734853 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:41.734806    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4ea9fa1-48c7-45be-bfcf-4594ee751086-metrics-certs\") pod \"network-metrics-daemon-drgl8\" (UID: \"b4ea9fa1-48c7-45be-bfcf-4594ee751086\") " pod="openshift-multus/network-metrics-daemon-drgl8"
Sep 16 19:41:41.735000 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:41:41.734949    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 19:41:41.735061 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:41:41.735011    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4ea9fa1-48c7-45be-bfcf-4594ee751086-metrics-certs podName:b4ea9fa1-48c7-45be-bfcf-4594ee751086 nodeName:}" failed. No retries permitted until 2026-09-16 19:43:43.734996488 +0000 UTC m=+252.260384855 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b4ea9fa1-48c7-45be-bfcf-4594ee751086-metrics-certs") pod "network-metrics-daemon-drgl8" (UID: "b4ea9fa1-48c7-45be-bfcf-4594ee751086") : secret "metrics-daemon-secret" not found
Sep 16 19:41:43.269305 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:43.269274    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-v5rhc"]
Sep 16 19:41:43.272159 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:43.272137    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-v5rhc"
Sep 16 19:41:43.275704 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:43.275683    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 19:41:43.275932 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:43.275917    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 19:41:43.276009 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:43.275957    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bsdbz\""
Sep 16 19:41:43.304755 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:43.304729    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-v5rhc"]
Sep 16 19:41:43.323415 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:43.323385    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-wlg99"]
Sep 16 19:41:43.325579 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:43.325562    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-wlg99"
Sep 16 19:41:43.329073 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:43.329053    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-sn77d\""
Sep 16 19:41:43.329185 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:43.329085    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 16 19:41:43.334472 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:43.334456    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 16 19:41:43.339185 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:43.339165    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-wlg99"]
Sep 16 19:41:43.350780 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:43.350754    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9489a4e3-e69c-4f07-a65a-9debfc38d24c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v5rhc\" (UID: \"9489a4e3-e69c-4f07-a65a-9debfc38d24c\") " pod="openshift-insights/insights-runtime-extractor-v5rhc"
Sep 16 19:41:43.350898 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:43.350807    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9489a4e3-e69c-4f07-a65a-9debfc38d24c-crio-socket\") pod \"insights-runtime-extractor-v5rhc\" (UID: \"9489a4e3-e69c-4f07-a65a-9debfc38d24c\") " pod="openshift-insights/insights-runtime-extractor-v5rhc"
Sep 16 19:41:43.350898 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:43.350878    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9489a4e3-e69c-4f07-a65a-9debfc38d24c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v5rhc\" (UID: \"9489a4e3-e69c-4f07-a65a-9debfc38d24c\") " pod="openshift-insights/insights-runtime-extractor-v5rhc"
Sep 16 19:41:43.350974 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:43.350910    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwm84\" (UniqueName: \"kubernetes.io/projected/9489a4e3-e69c-4f07-a65a-9debfc38d24c-kube-api-access-zwm84\") pod \"insights-runtime-extractor-v5rhc\" (UID: \"9489a4e3-e69c-4f07-a65a-9debfc38d24c\") " pod="openshift-insights/insights-runtime-extractor-v5rhc"
Sep 16 19:41:43.350974 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:43.350927    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9489a4e3-e69c-4f07-a65a-9debfc38d24c-data-volume\") pod \"insights-runtime-extractor-v5rhc\" (UID: \"9489a4e3-e69c-4f07-a65a-9debfc38d24c\") " pod="openshift-insights/insights-runtime-extractor-v5rhc"
Sep 16 19:41:43.451754 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:43.451709    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9489a4e3-e69c-4f07-a65a-9debfc38d24c-crio-socket\") pod \"insights-runtime-extractor-v5rhc\" (UID: \"9489a4e3-e69c-4f07-a65a-9debfc38d24c\") " pod="openshift-insights/insights-runtime-extractor-v5rhc"
Sep 16 19:41:43.451974 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:43.451769    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9489a4e3-e69c-4f07-a65a-9debfc38d24c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v5rhc\" (UID: \"9489a4e3-e69c-4f07-a65a-9debfc38d24c\") " pod="openshift-insights/insights-runtime-extractor-v5rhc"
Sep 16 19:41:43.451974 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:43.451798    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwm84\" (UniqueName: \"kubernetes.io/projected/9489a4e3-e69c-4f07-a65a-9debfc38d24c-kube-api-access-zwm84\") pod \"insights-runtime-extractor-v5rhc\" (UID: \"9489a4e3-e69c-4f07-a65a-9debfc38d24c\") " pod="openshift-insights/insights-runtime-extractor-v5rhc"
Sep 16 19:41:43.451974 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:43.451818    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9489a4e3-e69c-4f07-a65a-9debfc38d24c-data-volume\") pod \"insights-runtime-extractor-v5rhc\" (UID: \"9489a4e3-e69c-4f07-a65a-9debfc38d24c\") " pod="openshift-insights/insights-runtime-extractor-v5rhc"
Sep 16 19:41:43.451974 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:43.451870    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2rb2\" (UniqueName: \"kubernetes.io/projected/d8ce792d-c238-411a-9b23-80b06844837d-kube-api-access-x2rb2\") pod \"migrator-8b4d697c5-wlg99\" (UID: \"d8ce792d-c238-411a-9b23-80b06844837d\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-wlg99"
Sep 16 19:41:43.451974 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:43.451880    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9489a4e3-e69c-4f07-a65a-9debfc38d24c-crio-socket\") pod \"insights-runtime-extractor-v5rhc\" (UID: \"9489a4e3-e69c-4f07-a65a-9debfc38d24c\") " pod="openshift-insights/insights-runtime-extractor-v5rhc"
Sep 16 19:41:43.451974 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:43.451954    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9489a4e3-e69c-4f07-a65a-9debfc38d24c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v5rhc\" (UID: \"9489a4e3-e69c-4f07-a65a-9debfc38d24c\") " pod="openshift-insights/insights-runtime-extractor-v5rhc"
Sep 16 19:41:43.452244 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:41:43.452030    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 19:41:43.452244 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:41:43.452062    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 19:41:43.452244 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:41:43.452105    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9489a4e3-e69c-4f07-a65a-9debfc38d24c-kube-rbac-proxy-cm podName:9489a4e3-e69c-4f07-a65a-9debfc38d24c nodeName:}" failed. No retries permitted until 2026-09-16 19:41:43.952077201 +0000 UTC m=+132.477465582 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9489a4e3-e69c-4f07-a65a-9debfc38d24c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v5rhc" (UID: "9489a4e3-e69c-4f07-a65a-9debfc38d24c") : configmap "kube-rbac-proxy" not found
Sep 16 19:41:43.452244 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:41:43.452125    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9489a4e3-e69c-4f07-a65a-9debfc38d24c-insights-runtime-extractor-tls podName:9489a4e3-e69c-4f07-a65a-9debfc38d24c nodeName:}" failed. No retries permitted until 2026-09-16 19:41:43.95211533 +0000 UTC m=+132.477503699 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9489a4e3-e69c-4f07-a65a-9debfc38d24c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v5rhc" (UID: "9489a4e3-e69c-4f07-a65a-9debfc38d24c") : secret "insights-runtime-extractor-tls" not found
Sep 16 19:41:43.452244 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:43.452174    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9489a4e3-e69c-4f07-a65a-9debfc38d24c-data-volume\") pod \"insights-runtime-extractor-v5rhc\" (UID: \"9489a4e3-e69c-4f07-a65a-9debfc38d24c\") " pod="openshift-insights/insights-runtime-extractor-v5rhc"
Sep 16 19:41:43.469898 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:43.469867    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwm84\" (UniqueName: \"kubernetes.io/projected/9489a4e3-e69c-4f07-a65a-9debfc38d24c-kube-api-access-zwm84\") pod \"insights-runtime-extractor-v5rhc\" (UID: \"9489a4e3-e69c-4f07-a65a-9debfc38d24c\") " pod="openshift-insights/insights-runtime-extractor-v5rhc"
Sep 16 19:41:43.553166 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:43.553080    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2rb2\" (UniqueName: \"kubernetes.io/projected/d8ce792d-c238-411a-9b23-80b06844837d-kube-api-access-x2rb2\") pod \"migrator-8b4d697c5-wlg99\" (UID: \"d8ce792d-c238-411a-9b23-80b06844837d\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-wlg99"
Sep 16 19:41:43.567567 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:43.567532    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2rb2\" (UniqueName: \"kubernetes.io/projected/d8ce792d-c238-411a-9b23-80b06844837d-kube-api-access-x2rb2\") pod \"migrator-8b4d697c5-wlg99\" (UID: \"d8ce792d-c238-411a-9b23-80b06844837d\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-wlg99"
Sep 16 19:41:43.634891 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:43.634835    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-wlg99"
Sep 16 19:41:43.909931 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:43.909829    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-wlg99"]
Sep 16 19:41:43.913677 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:41:43.913650    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8ce792d_c238_411a_9b23_80b06844837d.slice/crio-57b8d004425c0368f46e1cc26952cc1cb71b4614edd4b2f6135d71662441f83d WatchSource:0}: Error finding container 57b8d004425c0368f46e1cc26952cc1cb71b4614edd4b2f6135d71662441f83d: Status 404 returned error can't find the container with id 57b8d004425c0368f46e1cc26952cc1cb71b4614edd4b2f6135d71662441f83d
Sep 16 19:41:43.958253 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:43.958205    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9489a4e3-e69c-4f07-a65a-9debfc38d24c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v5rhc\" (UID: \"9489a4e3-e69c-4f07-a65a-9debfc38d24c\") " pod="openshift-insights/insights-runtime-extractor-v5rhc"
Sep 16 19:41:43.958423 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:43.958295    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9489a4e3-e69c-4f07-a65a-9debfc38d24c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v5rhc\" (UID: \"9489a4e3-e69c-4f07-a65a-9debfc38d24c\") " pod="openshift-insights/insights-runtime-extractor-v5rhc"
Sep 16 19:41:43.958423 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:41:43.958328    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 19:41:43.958423 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:41:43.958392    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9489a4e3-e69c-4f07-a65a-9debfc38d24c-insights-runtime-extractor-tls podName:9489a4e3-e69c-4f07-a65a-9debfc38d24c nodeName:}" failed. No retries permitted until 2026-09-16 19:41:44.958374701 +0000 UTC m=+133.483763085 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9489a4e3-e69c-4f07-a65a-9debfc38d24c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v5rhc" (UID: "9489a4e3-e69c-4f07-a65a-9debfc38d24c") : secret "insights-runtime-extractor-tls" not found
Sep 16 19:41:43.958423 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:41:43.958408    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 19:41:43.958574 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:41:43.958455    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9489a4e3-e69c-4f07-a65a-9debfc38d24c-kube-rbac-proxy-cm podName:9489a4e3-e69c-4f07-a65a-9debfc38d24c nodeName:}" failed. No retries permitted until 2026-09-16 19:41:44.958443793 +0000 UTC m=+133.483832160 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9489a4e3-e69c-4f07-a65a-9debfc38d24c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v5rhc" (UID: "9489a4e3-e69c-4f07-a65a-9debfc38d24c") : configmap "kube-rbac-proxy" not found
Sep 16 19:41:44.388747 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:44.388703    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-wlg99" event={"ID":"d8ce792d-c238-411a-9b23-80b06844837d","Type":"ContainerStarted","Data":"57b8d004425c0368f46e1cc26952cc1cb71b4614edd4b2f6135d71662441f83d"}
Sep 16 19:41:44.965903 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:44.965869    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9489a4e3-e69c-4f07-a65a-9debfc38d24c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v5rhc\" (UID: \"9489a4e3-e69c-4f07-a65a-9debfc38d24c\") " pod="openshift-insights/insights-runtime-extractor-v5rhc"
Sep 16 19:41:44.966041 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:41:44.965964    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 19:41:44.966041 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:44.965988    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9489a4e3-e69c-4f07-a65a-9debfc38d24c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v5rhc\" (UID: \"9489a4e3-e69c-4f07-a65a-9debfc38d24c\") " pod="openshift-insights/insights-runtime-extractor-v5rhc"
Sep 16 19:41:44.966041 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:41:44.966029    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9489a4e3-e69c-4f07-a65a-9debfc38d24c-kube-rbac-proxy-cm podName:9489a4e3-e69c-4f07-a65a-9debfc38d24c nodeName:}" failed. No retries permitted until 2026-09-16 19:41:46.966008551 +0000 UTC m=+135.491396937 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9489a4e3-e69c-4f07-a65a-9debfc38d24c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v5rhc" (UID: "9489a4e3-e69c-4f07-a65a-9debfc38d24c") : configmap "kube-rbac-proxy" not found
Sep 16 19:41:44.966218 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:41:44.966074    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 19:41:44.966218 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:41:44.966125    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9489a4e3-e69c-4f07-a65a-9debfc38d24c-insights-runtime-extractor-tls podName:9489a4e3-e69c-4f07-a65a-9debfc38d24c nodeName:}" failed. No retries permitted until 2026-09-16 19:41:46.966110131 +0000 UTC m=+135.491498494 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9489a4e3-e69c-4f07-a65a-9debfc38d24c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v5rhc" (UID: "9489a4e3-e69c-4f07-a65a-9debfc38d24c") : secret "insights-runtime-extractor-tls" not found
Sep 16 19:41:44.975722 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:44.975703    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-clfb8_142b9c47-4826-4fa8-b228-5500d98fee67/dns-node-resolver/0.log"
Sep 16 19:41:45.393763 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:45.393722    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-wlg99" event={"ID":"d8ce792d-c238-411a-9b23-80b06844837d","Type":"ContainerStarted","Data":"23143de48e6434c9bf9a1630d3f141f34028be3a9454b31863444d104662b6e1"}
Sep 16 19:41:45.394169 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:45.393785    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-wlg99" event={"ID":"d8ce792d-c238-411a-9b23-80b06844837d","Type":"ContainerStarted","Data":"48a3116144649fcdd9a835b7fb45379577e0c1a25538630c7c9ab22f43fe8f30"}
Sep 16 19:41:45.436525 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:45.436471    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-wlg99" podStartSLOduration=1.474818595 podStartE2EDuration="2.436455859s" totalImagesPullingTime="961.637264ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:41:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:41:45.432705715 +0000 UTC m=+133.958094100" watchObservedRunningTime="2026-09-16 19:41:45.436455859 +0000 UTC m=+133.961844278"
Sep 16 19:41:45.570431 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:45.570397    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d6e55a03-bcc6-42f0-9b0c-b3c62b89d3ad-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-fbswk\" (UID: \"d6e55a03-bcc6-42f0-9b0c-b3c62b89d3ad\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fbswk"
Sep 16 19:41:45.570573 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:41:45.570553    2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 19:41:45.570636 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:41:45.570625    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6e55a03-bcc6-42f0-9b0c-b3c62b89d3ad-cluster-monitoring-operator-tls podName:d6e55a03-bcc6-42f0-9b0c-b3c62b89d3ad nodeName:}" failed. No retries permitted until 2026-09-16 19:41:53.570607667 +0000 UTC m=+142.095996044 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d6e55a03-bcc6-42f0-9b0c-b3c62b89d3ad-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-fbswk" (UID: "d6e55a03-bcc6-42f0-9b0c-b3c62b89d3ad") : secret "cluster-monitoring-operator-tls" not found
Sep 16 19:41:45.671794 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:45.671699    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25b2d6d3-b32d-4e8f-96c1-7701d5609d6f-metrics-certs\") pod \"router-default-b4b87b796-l78vs\" (UID: \"25b2d6d3-b32d-4e8f-96c1-7701d5609d6f\") " pod="openshift-ingress/router-default-b4b87b796-l78vs"
Sep 16 19:41:45.671964 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:45.671799    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25b2d6d3-b32d-4e8f-96c1-7701d5609d6f-service-ca-bundle\") pod \"router-default-b4b87b796-l78vs\" (UID: \"25b2d6d3-b32d-4e8f-96c1-7701d5609d6f\") " pod="openshift-ingress/router-default-b4b87b796-l78vs"
Sep 16 19:41:45.671964 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:41:45.671873    2557 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 19:41:45.671964 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:41:45.671941    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25b2d6d3-b32d-4e8f-96c1-7701d5609d6f-metrics-certs podName:25b2d6d3-b32d-4e8f-96c1-7701d5609d6f nodeName:}" failed. No retries permitted until 2026-09-16 19:41:53.671925498 +0000 UTC m=+142.197313861 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/25b2d6d3-b32d-4e8f-96c1-7701d5609d6f-metrics-certs") pod "router-default-b4b87b796-l78vs" (UID: "25b2d6d3-b32d-4e8f-96c1-7701d5609d6f") : secret "router-metrics-certs-default" not found
Sep 16 19:41:45.671964 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:41:45.671960    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/25b2d6d3-b32d-4e8f-96c1-7701d5609d6f-service-ca-bundle podName:25b2d6d3-b32d-4e8f-96c1-7701d5609d6f nodeName:}" failed. No retries permitted until 2026-09-16 19:41:53.671951001 +0000 UTC m=+142.197339363 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/25b2d6d3-b32d-4e8f-96c1-7701d5609d6f-service-ca-bundle") pod "router-default-b4b87b796-l78vs" (UID: "25b2d6d3-b32d-4e8f-96c1-7701d5609d6f") : configmap references non-existent config key: service-ca.crt
Sep 16 19:41:46.177116 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:46.177088    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hhvcc_bb2436af-e07f-464b-8973-1b0370888a65/node-ca/0.log"
Sep 16 19:41:46.969398 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:46.969372    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-r2rtx_af904ddd-40b6-48f8-9917-8ed2eb4b256a/serve-healthcheck-canary/0.log"
Sep 16 19:41:46.982587 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:46.982543    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9489a4e3-e69c-4f07-a65a-9debfc38d24c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v5rhc\" (UID: \"9489a4e3-e69c-4f07-a65a-9debfc38d24c\") " pod="openshift-insights/insights-runtime-extractor-v5rhc"
Sep 16 19:41:46.982748 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:46.982636    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9489a4e3-e69c-4f07-a65a-9debfc38d24c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v5rhc\" (UID: \"9489a4e3-e69c-4f07-a65a-9debfc38d24c\") " pod="openshift-insights/insights-runtime-extractor-v5rhc"
Sep 16 19:41:46.982748 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:41:46.982709    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 19:41:46.982833 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:41:46.982750    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 19:41:46.982833 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:41:46.982805    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9489a4e3-e69c-4f07-a65a-9debfc38d24c-insights-runtime-extractor-tls podName:9489a4e3-e69c-4f07-a65a-9debfc38d24c nodeName:}" failed. No retries permitted until 2026-09-16 19:41:50.98278521 +0000 UTC m=+139.508173576 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9489a4e3-e69c-4f07-a65a-9debfc38d24c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v5rhc" (UID: "9489a4e3-e69c-4f07-a65a-9debfc38d24c") : secret "insights-runtime-extractor-tls" not found
Sep 16 19:41:46.982833 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:41:46.982820    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9489a4e3-e69c-4f07-a65a-9debfc38d24c-kube-rbac-proxy-cm podName:9489a4e3-e69c-4f07-a65a-9debfc38d24c nodeName:}" failed. No retries permitted until 2026-09-16 19:41:50.982814217 +0000 UTC m=+139.508202580 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9489a4e3-e69c-4f07-a65a-9debfc38d24c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v5rhc" (UID: "9489a4e3-e69c-4f07-a65a-9debfc38d24c") : configmap "kube-rbac-proxy" not found
Sep 16 19:41:51.013652 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:51.013596    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9489a4e3-e69c-4f07-a65a-9debfc38d24c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v5rhc\" (UID: \"9489a4e3-e69c-4f07-a65a-9debfc38d24c\") " pod="openshift-insights/insights-runtime-extractor-v5rhc"
Sep 16 19:41:51.014094 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:51.013709    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9489a4e3-e69c-4f07-a65a-9debfc38d24c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v5rhc\" (UID: \"9489a4e3-e69c-4f07-a65a-9debfc38d24c\") " pod="openshift-insights/insights-runtime-extractor-v5rhc"
Sep 16 19:41:51.014094 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:41:51.013760    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 19:41:51.014094 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:41:51.013824    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 19:41:51.014094 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:41:51.013859    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9489a4e3-e69c-4f07-a65a-9debfc38d24c-insights-runtime-extractor-tls podName:9489a4e3-e69c-4f07-a65a-9debfc38d24c nodeName:}" failed. No retries permitted until 2026-09-16 19:41:59.013825257 +0000 UTC m=+147.539213634 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9489a4e3-e69c-4f07-a65a-9debfc38d24c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v5rhc" (UID: "9489a4e3-e69c-4f07-a65a-9debfc38d24c") : secret "insights-runtime-extractor-tls" not found
Sep 16 19:41:51.014094 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:41:51.013904    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9489a4e3-e69c-4f07-a65a-9debfc38d24c-kube-rbac-proxy-cm podName:9489a4e3-e69c-4f07-a65a-9debfc38d24c nodeName:}" failed. No retries permitted until 2026-09-16 19:41:59.013890237 +0000 UTC m=+147.539278600 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9489a4e3-e69c-4f07-a65a-9debfc38d24c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v5rhc" (UID: "9489a4e3-e69c-4f07-a65a-9debfc38d24c") : configmap "kube-rbac-proxy" not found
Sep 16 19:41:53.635995 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:53.635953    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d6e55a03-bcc6-42f0-9b0c-b3c62b89d3ad-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-fbswk\" (UID: \"d6e55a03-bcc6-42f0-9b0c-b3c62b89d3ad\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fbswk"
Sep 16 19:41:53.636367 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:41:53.636112    2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 19:41:53.636367 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:41:53.636184    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6e55a03-bcc6-42f0-9b0c-b3c62b89d3ad-cluster-monitoring-operator-tls podName:d6e55a03-bcc6-42f0-9b0c-b3c62b89d3ad nodeName:}" failed. No retries permitted until 2026-09-16 19:42:09.636167749 +0000 UTC m=+158.161556112 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d6e55a03-bcc6-42f0-9b0c-b3c62b89d3ad-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-fbswk" (UID: "d6e55a03-bcc6-42f0-9b0c-b3c62b89d3ad") : secret "cluster-monitoring-operator-tls" not found
Sep 16 19:41:53.736462 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:53.736411    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25b2d6d3-b32d-4e8f-96c1-7701d5609d6f-metrics-certs\") pod \"router-default-b4b87b796-l78vs\" (UID: \"25b2d6d3-b32d-4e8f-96c1-7701d5609d6f\") " pod="openshift-ingress/router-default-b4b87b796-l78vs"
Sep 16 19:41:53.736619 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:53.736522    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25b2d6d3-b32d-4e8f-96c1-7701d5609d6f-service-ca-bundle\") pod \"router-default-b4b87b796-l78vs\" (UID: \"25b2d6d3-b32d-4e8f-96c1-7701d5609d6f\") " pod="openshift-ingress/router-default-b4b87b796-l78vs"
Sep 16 19:41:53.736674 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:41:53.736662    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/25b2d6d3-b32d-4e8f-96c1-7701d5609d6f-service-ca-bundle podName:25b2d6d3-b32d-4e8f-96c1-7701d5609d6f nodeName:}" failed. No retries permitted until 2026-09-16 19:42:09.736648061 +0000 UTC m=+158.262036425 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/25b2d6d3-b32d-4e8f-96c1-7701d5609d6f-service-ca-bundle") pod "router-default-b4b87b796-l78vs" (UID: "25b2d6d3-b32d-4e8f-96c1-7701d5609d6f") : configmap references non-existent config key: service-ca.crt
Sep 16 19:41:53.738901 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:53.738867    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25b2d6d3-b32d-4e8f-96c1-7701d5609d6f-metrics-certs\") pod \"router-default-b4b87b796-l78vs\" (UID: \"25b2d6d3-b32d-4e8f-96c1-7701d5609d6f\") " pod="openshift-ingress/router-default-b4b87b796-l78vs"
Sep 16 19:41:59.082533 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:59.082489    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9489a4e3-e69c-4f07-a65a-9debfc38d24c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v5rhc\" (UID: \"9489a4e3-e69c-4f07-a65a-9debfc38d24c\") " pod="openshift-insights/insights-runtime-extractor-v5rhc"
Sep 16 19:41:59.083143 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:59.082572    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9489a4e3-e69c-4f07-a65a-9debfc38d24c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v5rhc\" (UID: \"9489a4e3-e69c-4f07-a65a-9debfc38d24c\") " pod="openshift-insights/insights-runtime-extractor-v5rhc"
Sep 16 19:41:59.083256 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:59.083233    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9489a4e3-e69c-4f07-a65a-9debfc38d24c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v5rhc\" (UID: \"9489a4e3-e69c-4f07-a65a-9debfc38d24c\") " pod="openshift-insights/insights-runtime-extractor-v5rhc"
Sep 16 19:41:59.085003 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:59.084976    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9489a4e3-e69c-4f07-a65a-9debfc38d24c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v5rhc\" (UID: \"9489a4e3-e69c-4f07-a65a-9debfc38d24c\") " pod="openshift-insights/insights-runtime-extractor-v5rhc"
Sep 16 19:41:59.180900 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:59.180864    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-v5rhc"
Sep 16 19:41:59.468702 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:41:59.468619    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-v5rhc"]
Sep 16 19:41:59.471607 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:41:59.471578    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9489a4e3_e69c_4f07_a65a_9debfc38d24c.slice/crio-f72034d72589b528fa59aa69dc2af69a25f8c245d9dd15c1c33b3d2cb1964e94 WatchSource:0}: Error finding container f72034d72589b528fa59aa69dc2af69a25f8c245d9dd15c1c33b3d2cb1964e94: Status 404 returned error can't find the container with id f72034d72589b528fa59aa69dc2af69a25f8c245d9dd15c1c33b3d2cb1964e94
Sep 16 19:42:00.437314 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:00.437228    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v5rhc" event={"ID":"9489a4e3-e69c-4f07-a65a-9debfc38d24c","Type":"ContainerStarted","Data":"a7ce1ead428cc4555e7a5ef7dd00ae752a6df2db9af49c51de569935459610b5"}
Sep 16 19:42:00.437314 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:00.437284    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v5rhc" event={"ID":"9489a4e3-e69c-4f07-a65a-9debfc38d24c","Type":"ContainerStarted","Data":"36d186fdb0d23c374e10dd142358a5411ce8b8842b24dbbb1acdc5eab3650b0d"}
Sep 16 19:42:00.437314 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:00.437301    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v5rhc" event={"ID":"9489a4e3-e69c-4f07-a65a-9debfc38d24c","Type":"ContainerStarted","Data":"f72034d72589b528fa59aa69dc2af69a25f8c245d9dd15c1c33b3d2cb1964e94"}
Sep 16 19:42:02.445341 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:02.445299    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v5rhc" event={"ID":"9489a4e3-e69c-4f07-a65a-9debfc38d24c","Type":"ContainerStarted","Data":"3aa7da105e13e54b0b29daa431d2bd0c61831efe5f03a20fc25bb9debaecad94"}
Sep 16 19:42:02.474419 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:02.474368    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-v5rhc" podStartSLOduration=17.585686717 podStartE2EDuration="19.474351793s" totalImagesPullingTime="1.888665076s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:41:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:42:02.472773659 +0000 UTC m=+150.998162086" watchObservedRunningTime="2026-09-16 19:42:02.474351793 +0000 UTC m=+150.999740178"
Sep 16 19:42:06.128367 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:06.128333    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-2kj2q"]
Sep 16 19:42:06.133011 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:06.132981    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-2kj2q"
Sep 16 19:42:06.141599 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:06.141578    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 16 19:42:06.141722 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:06.141627    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 16 19:42:06.141905 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:06.141890    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-k7cv2\""
Sep 16 19:42:06.150199 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:06.150171    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-2kj2q"]
Sep 16 19:42:06.231396 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:06.231353    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e60fe99d-8dc5-46bb-bd98-0003a5aca1e8-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-2kj2q\" (UID: \"e60fe99d-8dc5-46bb-bd98-0003a5aca1e8\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-2kj2q"
Sep 16 19:42:06.231570 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:06.231486    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e60fe99d-8dc5-46bb-bd98-0003a5aca1e8-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-2kj2q\" (UID: \"e60fe99d-8dc5-46bb-bd98-0003a5aca1e8\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-2kj2q"
Sep 16 19:42:06.332005 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:06.331966    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e60fe99d-8dc5-46bb-bd98-0003a5aca1e8-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-2kj2q\" (UID: \"e60fe99d-8dc5-46bb-bd98-0003a5aca1e8\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-2kj2q"
Sep 16 19:42:06.332199 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:06.332114    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e60fe99d-8dc5-46bb-bd98-0003a5aca1e8-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-2kj2q\" (UID: \"e60fe99d-8dc5-46bb-bd98-0003a5aca1e8\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-2kj2q"
Sep 16 19:42:06.332795 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:06.332768    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e60fe99d-8dc5-46bb-bd98-0003a5aca1e8-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-2kj2q\" (UID: \"e60fe99d-8dc5-46bb-bd98-0003a5aca1e8\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-2kj2q"
Sep 16 19:42:06.334409 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:06.334389    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e60fe99d-8dc5-46bb-bd98-0003a5aca1e8-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-2kj2q\" (UID: \"e60fe99d-8dc5-46bb-bd98-0003a5aca1e8\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-2kj2q"
Sep 16 19:42:06.441495 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:06.441381    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-2kj2q"
Sep 16 19:42:06.710430 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:06.710342    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-2kj2q"]
Sep 16 19:42:06.713654 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:42:06.713624    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode60fe99d_8dc5_46bb_bd98_0003a5aca1e8.slice/crio-98af9d88b4751cef648f36d9bda4123d10c760f9320b29bf8bc756a102582a8f WatchSource:0}: Error finding container 98af9d88b4751cef648f36d9bda4123d10c760f9320b29bf8bc756a102582a8f: Status 404 returned error can't find the container with id 98af9d88b4751cef648f36d9bda4123d10c760f9320b29bf8bc756a102582a8f
Sep 16 19:42:07.462976 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:07.462933    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-2kj2q" event={"ID":"e60fe99d-8dc5-46bb-bd98-0003a5aca1e8","Type":"ContainerStarted","Data":"98af9d88b4751cef648f36d9bda4123d10c760f9320b29bf8bc756a102582a8f"}
Sep 16 19:42:08.467180 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:08.467141    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-2kj2q" event={"ID":"e60fe99d-8dc5-46bb-bd98-0003a5aca1e8","Type":"ContainerStarted","Data":"b6c13ec8bee53b980109d4ad85eb587222731ec3a311dbd370fefda45eec2aec"}
Sep 16 19:42:08.486749 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:08.486694    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-2kj2q" podStartSLOduration=1.459332934 podStartE2EDuration="2.486677026s" totalImagesPullingTime="1.027344092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:42:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:42:08.486521342 +0000 UTC m=+157.011909730" watchObservedRunningTime="2026-09-16 19:42:08.486677026 +0000 UTC m=+157.012065402"
Sep 16 19:42:09.658194 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:09.658138    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d6e55a03-bcc6-42f0-9b0c-b3c62b89d3ad-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-fbswk\" (UID: \"d6e55a03-bcc6-42f0-9b0c-b3c62b89d3ad\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fbswk"
Sep 16 19:42:09.660575 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:09.660547    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d6e55a03-bcc6-42f0-9b0c-b3c62b89d3ad-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-fbswk\" (UID: \"d6e55a03-bcc6-42f0-9b0c-b3c62b89d3ad\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fbswk"
Sep 16 19:42:09.758720 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:09.758680    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25b2d6d3-b32d-4e8f-96c1-7701d5609d6f-service-ca-bundle\") pod \"router-default-b4b87b796-l78vs\" (UID: \"25b2d6d3-b32d-4e8f-96c1-7701d5609d6f\") " pod="openshift-ingress/router-default-b4b87b796-l78vs"
Sep 16 19:42:09.759340 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:09.759313    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25b2d6d3-b32d-4e8f-96c1-7701d5609d6f-service-ca-bundle\") pod \"router-default-b4b87b796-l78vs\" (UID: \"25b2d6d3-b32d-4e8f-96c1-7701d5609d6f\") " pod="openshift-ingress/router-default-b4b87b796-l78vs"
Sep 16 19:42:09.852235 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:42:09.852132    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-d8hqp" podUID="955ae2fa-b138-46de-b15d-a15850b54bb0"
Sep 16 19:42:09.911365 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:09.911331    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fbswk"
Sep 16 19:42:10.031511 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:10.031467    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-b4b87b796-l78vs"
Sep 16 19:42:10.181903 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:10.181814    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-fbswk"]
Sep 16 19:42:10.184751 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:42:10.184724    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6e55a03_bcc6_42f0_9b0c_b3c62b89d3ad.slice/crio-aab6942dc8824a708bf65486bfc6b767a3c623f273fc1c40b07ab67046e8a6ed WatchSource:0}: Error finding container aab6942dc8824a708bf65486bfc6b767a3c623f273fc1c40b07ab67046e8a6ed: Status 404 returned error can't find the container with id aab6942dc8824a708bf65486bfc6b767a3c623f273fc1c40b07ab67046e8a6ed
Sep 16 19:42:10.302956 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:10.302808    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-b4b87b796-l78vs"]
Sep 16 19:42:10.305341 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:42:10.305318    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25b2d6d3_b32d_4e8f_96c1_7701d5609d6f.slice/crio-4131556f8e724e049b152ea2e8e15f72dfba77f5a573ae7817bb29b33acdc558 WatchSource:0}: Error finding container 4131556f8e724e049b152ea2e8e15f72dfba77f5a573ae7817bb29b33acdc558: Status 404 returned error can't find the container with id 4131556f8e724e049b152ea2e8e15f72dfba77f5a573ae7817bb29b33acdc558
Sep 16 19:42:10.475232 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:10.475134    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-b4b87b796-l78vs" event={"ID":"25b2d6d3-b32d-4e8f-96c1-7701d5609d6f","Type":"ContainerStarted","Data":"67801363dc844dbf0f1374d61be18ce423c4af19243bdeeeda3ec6092219a1d9"}
Sep 16 19:42:10.475232 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:10.475200    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-b4b87b796-l78vs" event={"ID":"25b2d6d3-b32d-4e8f-96c1-7701d5609d6f","Type":"ContainerStarted","Data":"4131556f8e724e049b152ea2e8e15f72dfba77f5a573ae7817bb29b33acdc558"}
Sep 16 19:42:10.476337 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:10.476318    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d8hqp"
Sep 16 19:42:10.476425 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:10.476330    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fbswk" event={"ID":"d6e55a03-bcc6-42f0-9b0c-b3c62b89d3ad","Type":"ContainerStarted","Data":"aab6942dc8824a708bf65486bfc6b767a3c623f273fc1c40b07ab67046e8a6ed"}
Sep 16 19:42:10.495721 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:10.495667    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-b4b87b796-l78vs" podStartSLOduration=33.495646974 podStartE2EDuration="33.495646974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:41:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:42:10.493874792 +0000 UTC m=+159.019263178" watchObservedRunningTime="2026-09-16 19:42:10.495646974 +0000 UTC m=+159.021035358"
Sep 16 19:42:10.958902 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:42:10.958821    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-drgl8" podUID="b4ea9fa1-48c7-45be-bfcf-4594ee751086"
Sep 16 19:42:11.032221 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:11.032186    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-b4b87b796-l78vs"
Sep 16 19:42:11.035284 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:11.035258    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-b4b87b796-l78vs"
Sep 16 19:42:11.480304 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:11.480270    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-b4b87b796-l78vs"
Sep 16 19:42:11.481756 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:11.481731    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-b4b87b796-l78vs"
Sep 16 19:42:12.413145 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:12.413109    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-9wz5v"]
Sep 16 19:42:12.415326 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:12.415309    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-9wz5v"
Sep 16 19:42:12.419059 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:12.419034    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-vjrz5\""
Sep 16 19:42:12.419209 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:12.419191    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 16 19:42:12.426083 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:12.426055    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-9wz5v"]
Sep 16 19:42:12.481528 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:12.481491    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/09c71130-048a-4881-b61a-5fb73b2d007d-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-9wz5v\" (UID: \"09c71130-048a-4881-b61a-5fb73b2d007d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-9wz5v"
Sep 16 19:42:12.485581 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:12.485552    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fbswk" event={"ID":"d6e55a03-bcc6-42f0-9b0c-b3c62b89d3ad","Type":"ContainerStarted","Data":"bf400be9bdc0def02ac433014586a56d37f7009c3635048da66cdb049ef9a8ab"}
Sep 16 19:42:12.505849 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:12.505797    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fbswk" podStartSLOduration=33.829758258 podStartE2EDuration="35.50578154s" totalImagesPullingTime="1.676023282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:41:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:42:12.505273422 +0000 UTC m=+161.030661811" watchObservedRunningTime="2026-09-16 19:42:12.50578154 +0000 UTC m=+161.031169925"
Sep 16 19:42:12.582537 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:12.582500    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/09c71130-048a-4881-b61a-5fb73b2d007d-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-9wz5v\" (UID: \"09c71130-048a-4881-b61a-5fb73b2d007d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-9wz5v"
Sep 16 19:42:12.582734 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:42:12.582707    2557 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 16 19:42:12.582808 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:42:12.582780    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09c71130-048a-4881-b61a-5fb73b2d007d-tls-certificates podName:09c71130-048a-4881-b61a-5fb73b2d007d nodeName:}" failed. No retries permitted until 2026-09-16 19:42:13.082758769 +0000 UTC m=+161.608147144 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/09c71130-048a-4881-b61a-5fb73b2d007d-tls-certificates") pod "prometheus-operator-admission-webhook-5864cd99d7-9wz5v" (UID: "09c71130-048a-4881-b61a-5fb73b2d007d") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 16 19:42:13.086609 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:13.086570    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/09c71130-048a-4881-b61a-5fb73b2d007d-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-9wz5v\" (UID: \"09c71130-048a-4881-b61a-5fb73b2d007d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-9wz5v"
Sep 16 19:42:13.088930 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:13.088906    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/09c71130-048a-4881-b61a-5fb73b2d007d-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-9wz5v\" (UID: \"09c71130-048a-4881-b61a-5fb73b2d007d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-9wz5v"
Sep 16 19:42:13.324041 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:13.324008    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-9wz5v"
Sep 16 19:42:13.592686 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:13.592654    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-9wz5v"]
Sep 16 19:42:13.596245 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:42:13.596221    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09c71130_048a_4881_b61a_5fb73b2d007d.slice/crio-0545f4ae373a8f54b3ceb088f1b08e507c7bbf08a47a11c049fd0ad8ac8709ef WatchSource:0}: Error finding container 0545f4ae373a8f54b3ceb088f1b08e507c7bbf08a47a11c049fd0ad8ac8709ef: Status 404 returned error can't find the container with id 0545f4ae373a8f54b3ceb088f1b08e507c7bbf08a47a11c049fd0ad8ac8709ef
Sep 16 19:42:14.496314 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:14.496276    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-9wz5v" event={"ID":"09c71130-048a-4881-b61a-5fb73b2d007d","Type":"ContainerStarted","Data":"0545f4ae373a8f54b3ceb088f1b08e507c7bbf08a47a11c049fd0ad8ac8709ef"}
Sep 16 19:42:14.899735 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:14.899649    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/955ae2fa-b138-46de-b15d-a15850b54bb0-metrics-tls\") pod \"dns-default-d8hqp\" (UID: \"955ae2fa-b138-46de-b15d-a15850b54bb0\") " pod="openshift-dns/dns-default-d8hqp"
Sep 16 19:42:14.901924 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:14.901901    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/955ae2fa-b138-46de-b15d-a15850b54bb0-metrics-tls\") pod \"dns-default-d8hqp\" (UID: \"955ae2fa-b138-46de-b15d-a15850b54bb0\") " pod="openshift-dns/dns-default-d8hqp"
Sep 16 19:42:14.986333 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:14.986306    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wdb5m\""
Sep 16 19:42:14.987340 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:14.987322    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d8hqp"
Sep 16 19:42:15.441901 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:15.441764    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-d8hqp"]
Sep 16 19:42:15.444446 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:42:15.444418    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod955ae2fa_b138_46de_b15d_a15850b54bb0.slice/crio-7654f6f4949fefb2ed162acd16a95770bfada695092932d3edfdc23f79c2d3be WatchSource:0}: Error finding container 7654f6f4949fefb2ed162acd16a95770bfada695092932d3edfdc23f79c2d3be: Status 404 returned error can't find the container with id 7654f6f4949fefb2ed162acd16a95770bfada695092932d3edfdc23f79c2d3be
Sep 16 19:42:15.500647 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:15.500610    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-9wz5v" event={"ID":"09c71130-048a-4881-b61a-5fb73b2d007d","Type":"ContainerStarted","Data":"99954280e411876c445e9a6ff6bc9c637f62c4eaaefe46f473b991eb9b047358"}
Sep 16 19:42:15.500804 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:15.500743    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-9wz5v"
Sep 16 19:42:15.501820 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:15.501797    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d8hqp" event={"ID":"955ae2fa-b138-46de-b15d-a15850b54bb0","Type":"ContainerStarted","Data":"7654f6f4949fefb2ed162acd16a95770bfada695092932d3edfdc23f79c2d3be"}
Sep 16 19:42:15.505742 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:15.505725    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-9wz5v"
Sep 16 19:42:15.519895 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:15.519829    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-9wz5v" podStartSLOduration=1.941822806 podStartE2EDuration="3.519815696s" totalImagesPullingTime="1.57799289s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:42:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:42:15.518128569 +0000 UTC m=+164.043516953" watchObservedRunningTime="2026-09-16 19:42:15.519815696 +0000 UTC m=+164.045204081"
Sep 16 19:42:17.510922 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:17.510880    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d8hqp" event={"ID":"955ae2fa-b138-46de-b15d-a15850b54bb0","Type":"ContainerStarted","Data":"3d4fd49fd7b75948cd7b9c146aac5aac5480910d9a074f0d7f21b893b2695219"}
Sep 16 19:42:17.511358 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:17.510942    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d8hqp" event={"ID":"955ae2fa-b138-46de-b15d-a15850b54bb0","Type":"ContainerStarted","Data":"48049cf161afe2db271cbdf27c272075d09d2cb0ac372a3b8602b1114bf003c7"}
Sep 16 19:42:17.539811 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:17.539761    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-d8hqp" podStartSLOduration=130.116543967 podStartE2EDuration="2m11.539746045s" totalImagesPullingTime="1.423202078s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:40:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:42:17.539626928 +0000 UTC m=+166.065015315" watchObservedRunningTime="2026-09-16 19:42:17.539746045 +0000 UTC m=+166.065134468"
Sep 16 19:42:18.513939 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:18.513912    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-d8hqp"
Sep 16 19:42:20.978749 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:20.978713    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rsdqh"]
Sep 16 19:42:20.981489 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:20.981462    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-267mk"]
Sep 16 19:42:20.981645 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:20.981593    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rsdqh"
Sep 16 19:42:20.984103 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:20.984078    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-267mk"
Sep 16 19:42:20.986021 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:20.985998    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 19:42:20.986134 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:20.986031    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-sl9cc\""
Sep 16 19:42:20.987297 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:20.987205    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 19:42:20.987409 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:20.987345    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 19:42:20.987482 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:20.987466    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 19:42:20.992483 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:20.992465    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-9m82c\""
Sep 16 19:42:20.994063 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:20.994044    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 16 19:42:20.994152 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:20.994117    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 16 19:42:21.016395 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.016361    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-267mk"]
Sep 16 19:42:21.048827 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.048793    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/215db735-5f5f-4194-ba50-d75b6ca11eaa-node-exporter-wtmp\") pod \"node-exporter-rsdqh\" (UID: \"215db735-5f5f-4194-ba50-d75b6ca11eaa\") " pod="openshift-monitoring/node-exporter-rsdqh"
Sep 16 19:42:21.049011 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.048834    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/215db735-5f5f-4194-ba50-d75b6ca11eaa-node-exporter-textfile\") pod \"node-exporter-rsdqh\" (UID: \"215db735-5f5f-4194-ba50-d75b6ca11eaa\") " pod="openshift-monitoring/node-exporter-rsdqh"
Sep 16 19:42:21.049011 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.048921    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/215db735-5f5f-4194-ba50-d75b6ca11eaa-root\") pod \"node-exporter-rsdqh\" (UID: \"215db735-5f5f-4194-ba50-d75b6ca11eaa\") " pod="openshift-monitoring/node-exporter-rsdqh"
Sep 16 19:42:21.049011 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.048952    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad1e0f37-fcd1-4038-8d64-8b2aeeac356c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-267mk\" (UID: \"ad1e0f37-fcd1-4038-8d64-8b2aeeac356c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-267mk"
Sep 16 19:42:21.049011 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.048972    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbr28\" (UniqueName: \"kubernetes.io/projected/ad1e0f37-fcd1-4038-8d64-8b2aeeac356c-kube-api-access-hbr28\") pod \"openshift-state-metrics-67dcf8c979-267mk\" (UID: \"ad1e0f37-fcd1-4038-8d64-8b2aeeac356c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-267mk"
Sep 16 19:42:21.049011 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.049008    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/215db735-5f5f-4194-ba50-d75b6ca11eaa-metrics-client-ca\") pod \"node-exporter-rsdqh\" (UID: \"215db735-5f5f-4194-ba50-d75b6ca11eaa\") " pod="openshift-monitoring/node-exporter-rsdqh"
Sep 16 19:42:21.049223 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.049039    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/215db735-5f5f-4194-ba50-d75b6ca11eaa-node-exporter-tls\") pod \"node-exporter-rsdqh\" (UID: \"215db735-5f5f-4194-ba50-d75b6ca11eaa\") " pod="openshift-monitoring/node-exporter-rsdqh"
Sep 16 19:42:21.049223 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.049065    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpgm7\" (UniqueName: \"kubernetes.io/projected/215db735-5f5f-4194-ba50-d75b6ca11eaa-kube-api-access-mpgm7\") pod \"node-exporter-rsdqh\" (UID: \"215db735-5f5f-4194-ba50-d75b6ca11eaa\") " pod="openshift-monitoring/node-exporter-rsdqh"
Sep 16 19:42:21.049223 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.049089    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/215db735-5f5f-4194-ba50-d75b6ca11eaa-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rsdqh\" (UID: \"215db735-5f5f-4194-ba50-d75b6ca11eaa\") " pod="openshift-monitoring/node-exporter-rsdqh"
Sep 16 19:42:21.049223 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.049111    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/215db735-5f5f-4194-ba50-d75b6ca11eaa-node-exporter-accelerators-collector-config\") pod \"node-exporter-rsdqh\" (UID: \"215db735-5f5f-4194-ba50-d75b6ca11eaa\") " pod="openshift-monitoring/node-exporter-rsdqh"
Sep 16 19:42:21.049223 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.049129    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ad1e0f37-fcd1-4038-8d64-8b2aeeac356c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-267mk\" (UID: \"ad1e0f37-fcd1-4038-8d64-8b2aeeac356c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-267mk"
Sep 16 19:42:21.049223 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.049166    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad1e0f37-fcd1-4038-8d64-8b2aeeac356c-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-267mk\" (UID: \"ad1e0f37-fcd1-4038-8d64-8b2aeeac356c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-267mk"
Sep 16 19:42:21.049223 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.049203    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/215db735-5f5f-4194-ba50-d75b6ca11eaa-sys\") pod \"node-exporter-rsdqh\" (UID: \"215db735-5f5f-4194-ba50-d75b6ca11eaa\") " pod="openshift-monitoring/node-exporter-rsdqh"
Sep 16 19:42:21.149652 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.149612    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/215db735-5f5f-4194-ba50-d75b6ca11eaa-root\") pod \"node-exporter-rsdqh\" (UID: \"215db735-5f5f-4194-ba50-d75b6ca11eaa\") " pod="openshift-monitoring/node-exporter-rsdqh"
Sep 16 19:42:21.149817 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.149659    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad1e0f37-fcd1-4038-8d64-8b2aeeac356c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-267mk\" (UID: \"ad1e0f37-fcd1-4038-8d64-8b2aeeac356c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-267mk"
Sep 16 19:42:21.149817 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.149690    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbr28\" (UniqueName: \"kubernetes.io/projected/ad1e0f37-fcd1-4038-8d64-8b2aeeac356c-kube-api-access-hbr28\") pod \"openshift-state-metrics-67dcf8c979-267mk\" (UID: \"ad1e0f37-fcd1-4038-8d64-8b2aeeac356c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-267mk"
Sep 16 19:42:21.149817 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.149720    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/215db735-5f5f-4194-ba50-d75b6ca11eaa-root\") pod \"node-exporter-rsdqh\" (UID: \"215db735-5f5f-4194-ba50-d75b6ca11eaa\") " pod="openshift-monitoring/node-exporter-rsdqh"
Sep 16 19:42:21.149817 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.149737    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/215db735-5f5f-4194-ba50-d75b6ca11eaa-metrics-client-ca\") pod \"node-exporter-rsdqh\" (UID: \"215db735-5f5f-4194-ba50-d75b6ca11eaa\") " pod="openshift-monitoring/node-exporter-rsdqh"
Sep 16 19:42:21.149817 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.149769    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/215db735-5f5f-4194-ba50-d75b6ca11eaa-node-exporter-tls\") pod \"node-exporter-rsdqh\" (UID: \"215db735-5f5f-4194-ba50-d75b6ca11eaa\") " pod="openshift-monitoring/node-exporter-rsdqh"
Sep 16 19:42:21.149817 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.149789    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpgm7\" (UniqueName: \"kubernetes.io/projected/215db735-5f5f-4194-ba50-d75b6ca11eaa-kube-api-access-mpgm7\") pod \"node-exporter-rsdqh\" (UID: \"215db735-5f5f-4194-ba50-d75b6ca11eaa\") " pod="openshift-monitoring/node-exporter-rsdqh"
Sep 16 19:42:21.150115 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.149817    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/215db735-5f5f-4194-ba50-d75b6ca11eaa-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rsdqh\" (UID: \"215db735-5f5f-4194-ba50-d75b6ca11eaa\") " pod="openshift-monitoring/node-exporter-rsdqh"
Sep 16 19:42:21.150115 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:42:21.150009    2557 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 16 19:42:21.150115 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.150045    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/215db735-5f5f-4194-ba50-d75b6ca11eaa-node-exporter-accelerators-collector-config\") pod \"node-exporter-rsdqh\" (UID: \"215db735-5f5f-4194-ba50-d75b6ca11eaa\") " pod="openshift-monitoring/node-exporter-rsdqh"
Sep 16 19:42:21.150115 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:42:21.150093    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/215db735-5f5f-4194-ba50-d75b6ca11eaa-node-exporter-tls podName:215db735-5f5f-4194-ba50-d75b6ca11eaa nodeName:}" failed. No retries permitted until 2026-09-16 19:42:21.650072353 +0000 UTC m=+170.175460729 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/215db735-5f5f-4194-ba50-d75b6ca11eaa-node-exporter-tls") pod "node-exporter-rsdqh" (UID: "215db735-5f5f-4194-ba50-d75b6ca11eaa") : secret "node-exporter-tls" not found
Sep 16 19:42:21.150332 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.150114    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ad1e0f37-fcd1-4038-8d64-8b2aeeac356c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-267mk\" (UID: \"ad1e0f37-fcd1-4038-8d64-8b2aeeac356c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-267mk"
Sep 16 19:42:21.150332 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.150164    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad1e0f37-fcd1-4038-8d64-8b2aeeac356c-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-267mk\" (UID: \"ad1e0f37-fcd1-4038-8d64-8b2aeeac356c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-267mk"
Sep 16 19:42:21.150332 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.150199    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/215db735-5f5f-4194-ba50-d75b6ca11eaa-sys\") pod \"node-exporter-rsdqh\" (UID: \"215db735-5f5f-4194-ba50-d75b6ca11eaa\") " pod="openshift-monitoring/node-exporter-rsdqh"
Sep 16 19:42:21.150332 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.150241    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/215db735-5f5f-4194-ba50-d75b6ca11eaa-node-exporter-wtmp\") pod \"node-exporter-rsdqh\" (UID: \"215db735-5f5f-4194-ba50-d75b6ca11eaa\") " pod="openshift-monitoring/node-exporter-rsdqh"
Sep 16 19:42:21.150332 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.150266    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/215db735-5f5f-4194-ba50-d75b6ca11eaa-node-exporter-textfile\") pod \"node-exporter-rsdqh\" (UID: \"215db735-5f5f-4194-ba50-d75b6ca11eaa\") " pod="openshift-monitoring/node-exporter-rsdqh"
Sep 16 19:42:21.150332 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.150318    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/215db735-5f5f-4194-ba50-d75b6ca11eaa-sys\") pod \"node-exporter-rsdqh\" (UID: \"215db735-5f5f-4194-ba50-d75b6ca11eaa\") " pod="openshift-monitoring/node-exporter-rsdqh"
Sep 16 19:42:21.150614 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.150464    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/215db735-5f5f-4194-ba50-d75b6ca11eaa-node-exporter-wtmp\") pod \"node-exporter-rsdqh\" (UID: \"215db735-5f5f-4194-ba50-d75b6ca11eaa\") " pod="openshift-monitoring/node-exporter-rsdqh"
Sep 16 19:42:21.150614 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.150576    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/215db735-5f5f-4194-ba50-d75b6ca11eaa-metrics-client-ca\") pod \"node-exporter-rsdqh\" (UID: \"215db735-5f5f-4194-ba50-d75b6ca11eaa\") " pod="openshift-monitoring/node-exporter-rsdqh"
Sep 16 19:42:21.150614 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.150591    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/215db735-5f5f-4194-ba50-d75b6ca11eaa-node-exporter-accelerators-collector-config\") pod \"node-exporter-rsdqh\" (UID: \"215db735-5f5f-4194-ba50-d75b6ca11eaa\") " pod="openshift-monitoring/node-exporter-rsdqh"
Sep 16 19:42:21.150767 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.150744    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/215db735-5f5f-4194-ba50-d75b6ca11eaa-node-exporter-textfile\") pod \"node-exporter-rsdqh\" (UID: \"215db735-5f5f-4194-ba50-d75b6ca11eaa\") " pod="openshift-monitoring/node-exporter-rsdqh"
Sep 16 19:42:21.150998 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.150982    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad1e0f37-fcd1-4038-8d64-8b2aeeac356c-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-267mk\" (UID: \"ad1e0f37-fcd1-4038-8d64-8b2aeeac356c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-267mk"
Sep 16 19:42:21.152457 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.152436    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad1e0f37-fcd1-4038-8d64-8b2aeeac356c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-267mk\" (UID: \"ad1e0f37-fcd1-4038-8d64-8b2aeeac356c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-267mk"
Sep 16 19:42:21.152694 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.152674    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/215db735-5f5f-4194-ba50-d75b6ca11eaa-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rsdqh\" (UID: \"215db735-5f5f-4194-ba50-d75b6ca11eaa\") " pod="openshift-monitoring/node-exporter-rsdqh"
Sep 16 19:42:21.152757 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.152741    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ad1e0f37-fcd1-4038-8d64-8b2aeeac356c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-267mk\" (UID: \"ad1e0f37-fcd1-4038-8d64-8b2aeeac356c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-267mk"
Sep 16 19:42:21.158596 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.158574    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbr28\" (UniqueName: \"kubernetes.io/projected/ad1e0f37-fcd1-4038-8d64-8b2aeeac356c-kube-api-access-hbr28\") pod \"openshift-state-metrics-67dcf8c979-267mk\" (UID: \"ad1e0f37-fcd1-4038-8d64-8b2aeeac356c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-267mk"
Sep 16 19:42:21.160127 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.160108    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpgm7\" (UniqueName: \"kubernetes.io/projected/215db735-5f5f-4194-ba50-d75b6ca11eaa-kube-api-access-mpgm7\") pod \"node-exporter-rsdqh\" (UID: \"215db735-5f5f-4194-ba50-d75b6ca11eaa\") " pod="openshift-monitoring/node-exporter-rsdqh"
Sep 16 19:42:21.297738 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.297692    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-267mk"
Sep 16 19:42:21.568954 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.568867    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-267mk"]
Sep 16 19:42:21.571878 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:42:21.571824    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad1e0f37_fcd1_4038_8d64_8b2aeeac356c.slice/crio-3b229cbd15c41f50e573d9ac2dc003fe2d5ac271effa15c33587a9b9acb74d32 WatchSource:0}: Error finding container 3b229cbd15c41f50e573d9ac2dc003fe2d5ac271effa15c33587a9b9acb74d32: Status 404 returned error can't find the container with id 3b229cbd15c41f50e573d9ac2dc003fe2d5ac271effa15c33587a9b9acb74d32
Sep 16 19:42:21.653670 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.653637    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/215db735-5f5f-4194-ba50-d75b6ca11eaa-node-exporter-tls\") pod \"node-exporter-rsdqh\" (UID: \"215db735-5f5f-4194-ba50-d75b6ca11eaa\") " pod="openshift-monitoring/node-exporter-rsdqh"
Sep 16 19:42:21.655925 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.655899    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/215db735-5f5f-4194-ba50-d75b6ca11eaa-node-exporter-tls\") pod \"node-exporter-rsdqh\" (UID: \"215db735-5f5f-4194-ba50-d75b6ca11eaa\") " pod="openshift-monitoring/node-exporter-rsdqh"
Sep 16 19:42:21.892536 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.892440    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rsdqh"
Sep 16 19:42:21.902305 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:42:21.902276    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod215db735_5f5f_4194_ba50_d75b6ca11eaa.slice/crio-adcebfc4b54041fb7807ae377840483739ef7ae98cc358ae0c86f2e320ce25e1 WatchSource:0}: Error finding container adcebfc4b54041fb7807ae377840483739ef7ae98cc358ae0c86f2e320ce25e1: Status 404 returned error can't find the container with id adcebfc4b54041fb7807ae377840483739ef7ae98cc358ae0c86f2e320ce25e1
Sep 16 19:42:21.917956 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.917906    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 19:42:21.921438 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.921418    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:42:21.925949 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.925912    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 16 19:42:21.926122 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.925952    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 16 19:42:21.926122 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.925968    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 16 19:42:21.926122 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.926062    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 16 19:42:21.926122 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.926061    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 16 19:42:21.926330 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.926151    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 16 19:42:21.926376 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.926354    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 16 19:42:21.926446 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.926428    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-qd72j\""
Sep 16 19:42:21.926497 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.926358    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 16 19:42:21.931064 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.931040    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-drgl8"
Sep 16 19:42:21.933328 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.933307    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 16 19:42:21.939262 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.939236    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 19:42:21.955943 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.955903    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/03bbdade-309c-4e37-978b-06fb3ef595f3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"03bbdade-309c-4e37-978b-06fb3ef595f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:42:21.955943 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.955944    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/03bbdade-309c-4e37-978b-06fb3ef595f3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"03bbdade-309c-4e37-978b-06fb3ef595f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:42:21.956165 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.955966    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/03bbdade-309c-4e37-978b-06fb3ef595f3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"03bbdade-309c-4e37-978b-06fb3ef595f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:42:21.956165 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.956051    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/03bbdade-309c-4e37-978b-06fb3ef595f3-config-out\") pod \"alertmanager-main-0\" (UID: \"03bbdade-309c-4e37-978b-06fb3ef595f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:42:21.956165 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.956090    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/03bbdade-309c-4e37-978b-06fb3ef595f3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"03bbdade-309c-4e37-978b-06fb3ef595f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:42:21.956165 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.956117    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03bbdade-309c-4e37-978b-06fb3ef595f3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"03bbdade-309c-4e37-978b-06fb3ef595f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:42:21.956165 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.956133    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/03bbdade-309c-4e37-978b-06fb3ef595f3-config-volume\") pod \"alertmanager-main-0\" (UID: \"03bbdade-309c-4e37-978b-06fb3ef595f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:42:21.956165 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.956154    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/03bbdade-309c-4e37-978b-06fb3ef595f3-web-config\") pod \"alertmanager-main-0\" (UID: \"03bbdade-309c-4e37-978b-06fb3ef595f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:42:21.956453 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.956248    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqkdc\" (UniqueName: \"kubernetes.io/projected/03bbdade-309c-4e37-978b-06fb3ef595f3-kube-api-access-hqkdc\") pod \"alertmanager-main-0\" (UID: \"03bbdade-309c-4e37-978b-06fb3ef595f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:42:21.956453 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.956298    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/03bbdade-309c-4e37-978b-06fb3ef595f3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"03bbdade-309c-4e37-978b-06fb3ef595f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:42:21.956453 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.956346    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03bbdade-309c-4e37-978b-06fb3ef595f3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"03bbdade-309c-4e37-978b-06fb3ef595f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:42:21.956453 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.956391    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/03bbdade-309c-4e37-978b-06fb3ef595f3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"03bbdade-309c-4e37-978b-06fb3ef595f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:42:21.956453 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:21.956415    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/03bbdade-309c-4e37-978b-06fb3ef595f3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"03bbdade-309c-4e37-978b-06fb3ef595f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:42:22.057412 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:22.057378    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/03bbdade-309c-4e37-978b-06fb3ef595f3-config-out\") pod \"alertmanager-main-0\" (UID: \"03bbdade-309c-4e37-978b-06fb3ef595f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:42:22.057939 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:22.057424    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/03bbdade-309c-4e37-978b-06fb3ef595f3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"03bbdade-309c-4e37-978b-06fb3ef595f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:42:22.057939 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:22.057454    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03bbdade-309c-4e37-978b-06fb3ef595f3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"03bbdade-309c-4e37-978b-06fb3ef595f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:42:22.057939 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:22.057476    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/03bbdade-309c-4e37-978b-06fb3ef595f3-config-volume\") pod \"alertmanager-main-0\" (UID: \"03bbdade-309c-4e37-978b-06fb3ef595f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:42:22.057939 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:22.057500    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/03bbdade-309c-4e37-978b-06fb3ef595f3-web-config\") pod \"alertmanager-main-0\" (UID: \"03bbdade-309c-4e37-978b-06fb3ef595f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:42:22.057939 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:22.057548    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqkdc\" (UniqueName: \"kubernetes.io/projected/03bbdade-309c-4e37-978b-06fb3ef595f3-kube-api-access-hqkdc\") pod \"alertmanager-main-0\" (UID: \"03bbdade-309c-4e37-978b-06fb3ef595f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:42:22.057939 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:22.057592    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/03bbdade-309c-4e37-978b-06fb3ef595f3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"03bbdade-309c-4e37-978b-06fb3ef595f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:42:22.057939 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:22.057650    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03bbdade-309c-4e37-978b-06fb3ef595f3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"03bbdade-309c-4e37-978b-06fb3ef595f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:42:22.057939 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:22.057824    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/03bbdade-309c-4e37-978b-06fb3ef595f3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"03bbdade-309c-4e37-978b-06fb3ef595f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:42:22.057939 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:22.057874    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/03bbdade-309c-4e37-978b-06fb3ef595f3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"03bbdade-309c-4e37-978b-06fb3ef595f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:42:22.057939 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:22.057898    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/03bbdade-309c-4e37-978b-06fb3ef595f3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"03bbdade-309c-4e37-978b-06fb3ef595f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:42:22.057939 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:22.057922    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/03bbdade-309c-4e37-978b-06fb3ef595f3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"03bbdade-309c-4e37-978b-06fb3ef595f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:42:22.058485 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:22.057951    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/03bbdade-309c-4e37-978b-06fb3ef595f3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"03bbdade-309c-4e37-978b-06fb3ef595f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:42:22.058682 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:22.058655    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03bbdade-309c-4e37-978b-06fb3ef595f3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"03bbdade-309c-4e37-978b-06fb3ef595f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:42:22.059654 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:22.059243    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/03bbdade-309c-4e37-978b-06fb3ef595f3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"03bbdade-309c-4e37-978b-06fb3ef595f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:42:22.059654 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:42:22.059353    2557 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 16 19:42:22.059654 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:42:22.059417    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03bbdade-309c-4e37-978b-06fb3ef595f3-secret-alertmanager-main-tls podName:03bbdade-309c-4e37-978b-06fb3ef595f3 nodeName:}" failed. No retries permitted until 2026-09-16 19:42:22.559396232 +0000 UTC m=+171.084784600 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/03bbdade-309c-4e37-978b-06fb3ef595f3-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "03bbdade-309c-4e37-978b-06fb3ef595f3") : secret "alertmanager-main-tls" not found
Sep 16 19:42:22.059654 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:22.059515    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03bbdade-309c-4e37-978b-06fb3ef595f3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"03bbdade-309c-4e37-978b-06fb3ef595f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:42:22.060412 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:22.060386    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/03bbdade-309c-4e37-978b-06fb3ef595f3-config-out\") pod \"alertmanager-main-0\" (UID: \"03bbdade-309c-4e37-978b-06fb3ef595f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:42:22.060676 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:22.060654    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/03bbdade-309c-4e37-978b-06fb3ef595f3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"03bbdade-309c-4e37-978b-06fb3ef595f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:42:22.060979 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:22.060951    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/03bbdade-309c-4e37-978b-06fb3ef595f3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"03bbdade-309c-4e37-978b-06fb3ef595f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:42:22.061088 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:22.060987    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/03bbdade-309c-4e37-978b-06fb3ef595f3-web-config\") pod \"alertmanager-main-0\" (UID: \"03bbdade-309c-4e37-978b-06fb3ef595f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:42:22.061635 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:22.061605    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/03bbdade-309c-4e37-978b-06fb3ef595f3-config-volume\") pod \"alertmanager-main-0\" (UID: \"03bbdade-309c-4e37-978b-06fb3ef595f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:42:22.061992 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:22.061972    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/03bbdade-309c-4e37-978b-06fb3ef595f3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"03bbdade-309c-4e37-978b-06fb3ef595f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:42:22.062293 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:22.062277    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/03bbdade-309c-4e37-978b-06fb3ef595f3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"03bbdade-309c-4e37-978b-06fb3ef595f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:42:22.062436 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:22.062421    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/03bbdade-309c-4e37-978b-06fb3ef595f3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"03bbdade-309c-4e37-978b-06fb3ef595f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:42:22.067311 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:22.067292    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqkdc\" (UniqueName: \"kubernetes.io/projected/03bbdade-309c-4e37-978b-06fb3ef595f3-kube-api-access-hqkdc\") pod \"alertmanager-main-0\" (UID: \"03bbdade-309c-4e37-978b-06fb3ef595f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:42:22.254824 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:22.254788    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-7kt88"]
Sep 16 19:42:22.260788 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:22.260751    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-7kt88"
Sep 16 19:42:22.263870 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:22.263828    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-gjs8f\""
Sep 16 19:42:22.264452 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:22.264417    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 19:42:22.264633 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:22.264608    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 19:42:22.280165 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:22.280133    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-7kt88"]
Sep 16 19:42:22.364023 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:22.363982    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf7pv\" (UniqueName: \"kubernetes.io/projected/f4c1d81c-50de-47c5-a944-cf1571f49abb-kube-api-access-vf7pv\") pod \"downloads-76f99584f4-7kt88\" (UID: \"f4c1d81c-50de-47c5-a944-cf1571f49abb\") " pod="openshift-console/downloads-76f99584f4-7kt88"
Sep 16 19:42:22.465305 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:22.465270    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf7pv\" (UniqueName: \"kubernetes.io/projected/f4c1d81c-50de-47c5-a944-cf1571f49abb-kube-api-access-vf7pv\") pod \"downloads-76f99584f4-7kt88\" (UID: \"f4c1d81c-50de-47c5-a944-cf1571f49abb\") " pod="openshift-console/downloads-76f99584f4-7kt88"
Sep 16 19:42:22.491616 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:22.491551    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf7pv\" (UniqueName: \"kubernetes.io/projected/f4c1d81c-50de-47c5-a944-cf1571f49abb-kube-api-access-vf7pv\") pod \"downloads-76f99584f4-7kt88\" (UID: \"f4c1d81c-50de-47c5-a944-cf1571f49abb\") " pod="openshift-console/downloads-76f99584f4-7kt88"
Sep 16 19:42:22.533294 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:22.533190    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-267mk" event={"ID":"ad1e0f37-fcd1-4038-8d64-8b2aeeac356c","Type":"ContainerStarted","Data":"5142975ee3724d1ee7c8ed398a4308d7b2ac944b31a3e5c69005a2ff28029c81"}
Sep 16 19:42:22.533294 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:22.533270    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-267mk" event={"ID":"ad1e0f37-fcd1-4038-8d64-8b2aeeac356c","Type":"ContainerStarted","Data":"61b6709b1bb633707658241eab45363f712dfade859d48ce0e002bfb37a76c62"}
Sep 16 19:42:22.533519 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:22.533298    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-267mk" event={"ID":"ad1e0f37-fcd1-4038-8d64-8b2aeeac356c","Type":"ContainerStarted","Data":"3b229cbd15c41f50e573d9ac2dc003fe2d5ac271effa15c33587a9b9acb74d32"}
Sep 16 19:42:22.534823 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:22.534791    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rsdqh" event={"ID":"215db735-5f5f-4194-ba50-d75b6ca11eaa","Type":"ContainerStarted","Data":"adcebfc4b54041fb7807ae377840483739ef7ae98cc358ae0c86f2e320ce25e1"}
Sep 16 19:42:22.566435 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:22.566330    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/03bbdade-309c-4e37-978b-06fb3ef595f3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"03bbdade-309c-4e37-978b-06fb3ef595f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:42:22.569492 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:22.569436    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/03bbdade-309c-4e37-978b-06fb3ef595f3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"03bbdade-309c-4e37-978b-06fb3ef595f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:42:22.574276 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:22.574157    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-7kt88"
Sep 16 19:42:22.832352 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:22.832261    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:42:23.235452 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:23.235421    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-7kt88"]
Sep 16 19:42:23.237692 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:42:23.237659    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4c1d81c_50de_47c5_a944_cf1571f49abb.slice/crio-a05df935e998b9e580559f0a31514e4f92ceefdc690c53a6a7db80373c03c026 WatchSource:0}: Error finding container a05df935e998b9e580559f0a31514e4f92ceefdc690c53a6a7db80373c03c026: Status 404 returned error can't find the container with id a05df935e998b9e580559f0a31514e4f92ceefdc690c53a6a7db80373c03c026
Sep 16 19:42:23.252656 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:23.252615    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 19:42:23.255223 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:42:23.255198    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03bbdade_309c_4e37_978b_06fb3ef595f3.slice/crio-86801deb2e556a42fb18bf23dde46a3a90a3404092b27cf9cebd4ad1a863017f WatchSource:0}: Error finding container 86801deb2e556a42fb18bf23dde46a3a90a3404092b27cf9cebd4ad1a863017f: Status 404 returned error can't find the container with id 86801deb2e556a42fb18bf23dde46a3a90a3404092b27cf9cebd4ad1a863017f
Sep 16 19:42:23.541088 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:23.541050    2557 generic.go:356] "Generic (PLEG): container finished" podID="215db735-5f5f-4194-ba50-d75b6ca11eaa" containerID="868273c6d7e6ab790eb40e9bdd55816f0602ee1fa68ed9892bfd99bf4374d2aa" exitCode=0
Sep 16 19:42:23.541274 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:23.541129    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rsdqh" event={"ID":"215db735-5f5f-4194-ba50-d75b6ca11eaa","Type":"ContainerDied","Data":"868273c6d7e6ab790eb40e9bdd55816f0602ee1fa68ed9892bfd99bf4374d2aa"}
Sep 16 19:42:23.542314 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:23.542286    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-7kt88" event={"ID":"f4c1d81c-50de-47c5-a944-cf1571f49abb","Type":"ContainerStarted","Data":"a05df935e998b9e580559f0a31514e4f92ceefdc690c53a6a7db80373c03c026"}
Sep 16 19:42:23.544723 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:23.544697    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-267mk" event={"ID":"ad1e0f37-fcd1-4038-8d64-8b2aeeac356c","Type":"ContainerStarted","Data":"264297825d06930317ef92b4de04cca21dcd08bbdcd045a19fed4fe189f1f0f9"}
Sep 16 19:42:23.545951 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:23.545927    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03bbdade-309c-4e37-978b-06fb3ef595f3","Type":"ContainerStarted","Data":"86801deb2e556a42fb18bf23dde46a3a90a3404092b27cf9cebd4ad1a863017f"}
Sep 16 19:42:23.630008 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:23.629958    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-267mk" podStartSLOduration=2.392055182 podStartE2EDuration="3.629938192s" totalImagesPullingTime="1.23788301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:42:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:42:23.628653384 +0000 UTC m=+172.154041769" watchObservedRunningTime="2026-09-16 19:42:23.629938192 +0000 UTC m=+172.155326576"
Sep 16 19:42:24.045618 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:24.045571    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-84dbb595c9-fgcq7"]
Sep 16 19:42:24.049145 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:24.049121    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-84dbb595c9-fgcq7"
Sep 16 19:42:24.052962 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:24.052645    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 16 19:42:24.052962 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:24.052645    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 16 19:42:24.052962 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:24.052782    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-9v8h4\""
Sep 16 19:42:24.053593 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:24.053428    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 16 19:42:24.054033 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:24.053865    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-etjvmlj8b3vop\""
Sep 16 19:42:24.054033 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:24.053924    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 16 19:42:24.054033 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:24.053995    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 16 19:42:24.064544 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:24.064461    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-84dbb595c9-fgcq7"]
Sep 16 19:42:24.181471 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:24.181379    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3a10832d-3f24-4464-9291-a8b8c78adfbb-secret-grpc-tls\") pod \"thanos-querier-84dbb595c9-fgcq7\" (UID: \"3a10832d-3f24-4464-9291-a8b8c78adfbb\") " pod="openshift-monitoring/thanos-querier-84dbb595c9-fgcq7"
Sep 16 19:42:24.181679 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:24.181448    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3a10832d-3f24-4464-9291-a8b8c78adfbb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-84dbb595c9-fgcq7\" (UID: \"3a10832d-3f24-4464-9291-a8b8c78adfbb\") " pod="openshift-monitoring/thanos-querier-84dbb595c9-fgcq7"
Sep 16 19:42:24.181679 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:24.181617    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3a10832d-3f24-4464-9291-a8b8c78adfbb-secret-thanos-querier-tls\") pod \"thanos-querier-84dbb595c9-fgcq7\" (UID: \"3a10832d-3f24-4464-9291-a8b8c78adfbb\") " pod="openshift-monitoring/thanos-querier-84dbb595c9-fgcq7"
Sep 16 19:42:24.181819 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:24.181697    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3a10832d-3f24-4464-9291-a8b8c78adfbb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-84dbb595c9-fgcq7\" (UID: \"3a10832d-3f24-4464-9291-a8b8c78adfbb\") " pod="openshift-monitoring/thanos-querier-84dbb595c9-fgcq7"
Sep 16 19:42:24.181819 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:24.181768    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a10832d-3f24-4464-9291-a8b8c78adfbb-metrics-client-ca\") pod \"thanos-querier-84dbb595c9-fgcq7\" (UID: \"3a10832d-3f24-4464-9291-a8b8c78adfbb\") " pod="openshift-monitoring/thanos-querier-84dbb595c9-fgcq7"
Sep 16 19:42:24.181819 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:24.181797    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8zz9\" (UniqueName: \"kubernetes.io/projected/3a10832d-3f24-4464-9291-a8b8c78adfbb-kube-api-access-g8zz9\") pod \"thanos-querier-84dbb595c9-fgcq7\" (UID: \"3a10832d-3f24-4464-9291-a8b8c78adfbb\") " pod="openshift-monitoring/thanos-querier-84dbb595c9-fgcq7"
Sep 16 19:42:24.182021 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:24.181822    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3a10832d-3f24-4464-9291-a8b8c78adfbb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-84dbb595c9-fgcq7\" (UID: \"3a10832d-3f24-4464-9291-a8b8c78adfbb\") " pod="openshift-monitoring/thanos-querier-84dbb595c9-fgcq7"
Sep 16 19:42:24.182021 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:24.181861    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3a10832d-3f24-4464-9291-a8b8c78adfbb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-84dbb595c9-fgcq7\" (UID: \"3a10832d-3f24-4464-9291-a8b8c78adfbb\") " pod="openshift-monitoring/thanos-querier-84dbb595c9-fgcq7"
Sep 16 19:42:24.283088 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:24.282992    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3a10832d-3f24-4464-9291-a8b8c78adfbb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-84dbb595c9-fgcq7\" (UID: \"3a10832d-3f24-4464-9291-a8b8c78adfbb\") " pod="openshift-monitoring/thanos-querier-84dbb595c9-fgcq7"
Sep 16 19:42:24.283088 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:24.283046    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3a10832d-3f24-4464-9291-a8b8c78adfbb-secret-thanos-querier-tls\") pod \"thanos-querier-84dbb595c9-fgcq7\" (UID: \"3a10832d-3f24-4464-9291-a8b8c78adfbb\") " pod="openshift-monitoring/thanos-querier-84dbb595c9-fgcq7"
Sep 16 19:42:24.283593 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:24.283113    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3a10832d-3f24-4464-9291-a8b8c78adfbb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-84dbb595c9-fgcq7\" (UID: \"3a10832d-3f24-4464-9291-a8b8c78adfbb\") " pod="openshift-monitoring/thanos-querier-84dbb595c9-fgcq7"
Sep 16 19:42:24.283593 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:24.283184    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a10832d-3f24-4464-9291-a8b8c78adfbb-metrics-client-ca\") pod \"thanos-querier-84dbb595c9-fgcq7\" (UID: \"3a10832d-3f24-4464-9291-a8b8c78adfbb\") " pod="openshift-monitoring/thanos-querier-84dbb595c9-fgcq7"
Sep 16 19:42:24.283593 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:24.283211    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8zz9\" (UniqueName: \"kubernetes.io/projected/3a10832d-3f24-4464-9291-a8b8c78adfbb-kube-api-access-g8zz9\") pod \"thanos-querier-84dbb595c9-fgcq7\" (UID: \"3a10832d-3f24-4464-9291-a8b8c78adfbb\") " pod="openshift-monitoring/thanos-querier-84dbb595c9-fgcq7"
Sep 16 19:42:24.283593 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:24.283237    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3a10832d-3f24-4464-9291-a8b8c78adfbb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-84dbb595c9-fgcq7\" (UID: \"3a10832d-3f24-4464-9291-a8b8c78adfbb\") " pod="openshift-monitoring/thanos-querier-84dbb595c9-fgcq7"
Sep 16 19:42:24.283593 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:24.283263    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3a10832d-3f24-4464-9291-a8b8c78adfbb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-84dbb595c9-fgcq7\" (UID: \"3a10832d-3f24-4464-9291-a8b8c78adfbb\") " pod="openshift-monitoring/thanos-querier-84dbb595c9-fgcq7"
Sep 16 19:42:24.283593 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:24.283307    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3a10832d-3f24-4464-9291-a8b8c78adfbb-secret-grpc-tls\") pod \"thanos-querier-84dbb595c9-fgcq7\" (UID: \"3a10832d-3f24-4464-9291-a8b8c78adfbb\") " pod="openshift-monitoring/thanos-querier-84dbb595c9-fgcq7"
Sep 16 19:42:24.284335 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:24.284283    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a10832d-3f24-4464-9291-a8b8c78adfbb-metrics-client-ca\") pod \"thanos-querier-84dbb595c9-fgcq7\" (UID: \"3a10832d-3f24-4464-9291-a8b8c78adfbb\") " pod="openshift-monitoring/thanos-querier-84dbb595c9-fgcq7"
Sep 16 19:42:24.286608 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:24.286559    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3a10832d-3f24-4464-9291-a8b8c78adfbb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-84dbb595c9-fgcq7\" (UID: \"3a10832d-3f24-4464-9291-a8b8c78adfbb\") " pod="openshift-monitoring/thanos-querier-84dbb595c9-fgcq7"
Sep 16 19:42:24.286749 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:24.286675    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3a10832d-3f24-4464-9291-a8b8c78adfbb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-84dbb595c9-fgcq7\" (UID: \"3a10832d-3f24-4464-9291-a8b8c78adfbb\") " pod="openshift-monitoring/thanos-querier-84dbb595c9-fgcq7"
Sep 16 19:42:24.286821 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:24.286757    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3a10832d-3f24-4464-9291-a8b8c78adfbb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-84dbb595c9-fgcq7\" (UID: \"3a10832d-3f24-4464-9291-a8b8c78adfbb\") " pod="openshift-monitoring/thanos-querier-84dbb595c9-fgcq7"
Sep 16 19:42:24.286953 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:24.286913    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3a10832d-3f24-4464-9291-a8b8c78adfbb-secret-thanos-querier-tls\") pod \"thanos-querier-84dbb595c9-fgcq7\" (UID: \"3a10832d-3f24-4464-9291-a8b8c78adfbb\") " pod="openshift-monitoring/thanos-querier-84dbb595c9-fgcq7"
Sep 16 19:42:24.287502 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:24.287474    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3a10832d-3f24-4464-9291-a8b8c78adfbb-secret-grpc-tls\") pod \"thanos-querier-84dbb595c9-fgcq7\" (UID: \"3a10832d-3f24-4464-9291-a8b8c78adfbb\") " pod="openshift-monitoring/thanos-querier-84dbb595c9-fgcq7"
Sep 16 19:42:24.288282 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:24.288259    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3a10832d-3f24-4464-9291-a8b8c78adfbb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-84dbb595c9-fgcq7\" (UID: \"3a10832d-3f24-4464-9291-a8b8c78adfbb\") " pod="openshift-monitoring/thanos-querier-84dbb595c9-fgcq7"
Sep 16 19:42:24.295907 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:24.295850    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8zz9\" (UniqueName: \"kubernetes.io/projected/3a10832d-3f24-4464-9291-a8b8c78adfbb-kube-api-access-g8zz9\") pod \"thanos-querier-84dbb595c9-fgcq7\" (UID: \"3a10832d-3f24-4464-9291-a8b8c78adfbb\") " pod="openshift-monitoring/thanos-querier-84dbb595c9-fgcq7"
Sep 16 19:42:24.362616 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:24.362559    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-84dbb595c9-fgcq7"
Sep 16 19:42:24.553066 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:24.552982    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03bbdade-309c-4e37-978b-06fb3ef595f3","Type":"ContainerStarted","Data":"a648292ee86fab1f53e79ce1a67daf5c7e6a0702dc26aa7272fe8bcdf63f6a76"}
Sep 16 19:42:24.556006 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:24.555975    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rsdqh" event={"ID":"215db735-5f5f-4194-ba50-d75b6ca11eaa","Type":"ContainerStarted","Data":"e870ab6d1bdbdf7076a8b84c2640d9331d32f1403c713016f5d0cfba3c06c5a8"}
Sep 16 19:42:24.556099 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:24.556035    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rsdqh" event={"ID":"215db735-5f5f-4194-ba50-d75b6ca11eaa","Type":"ContainerStarted","Data":"8af7107204f77357efcca427b0d99289c12ad52eaf13e86ac73cb355c770911c"}
Sep 16 19:42:24.610297 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:24.610238    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rsdqh" podStartSLOduration=3.503162272 podStartE2EDuration="4.61022156s" totalImagesPullingTime="1.037420421s" totalInitContainerRuntime="69.638867ms" isStatefulPod=true podCreationTimestamp="2026-09-16 19:42:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:42:24.609135014 +0000 UTC m=+173.134523400" watchObservedRunningTime="2026-09-16 19:42:24.61022156 +0000 UTC m=+173.135609969"
Sep 16 19:42:24.693868 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:24.693815    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-84dbb595c9-fgcq7"]
Sep 16 19:42:24.698688 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:42:24.698655    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a10832d_3f24_4464_9291_a8b8c78adfbb.slice/crio-853fb6a6ffefa7f711747b5b5753e0399a2d62778950b16d1b49c505a652b470 WatchSource:0}: Error finding container 853fb6a6ffefa7f711747b5b5753e0399a2d62778950b16d1b49c505a652b470: Status 404 returned error can't find the container with id 853fb6a6ffefa7f711747b5b5753e0399a2d62778950b16d1b49c505a652b470
Sep 16 19:42:25.562784 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:25.562745    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84dbb595c9-fgcq7" event={"ID":"3a10832d-3f24-4464-9291-a8b8c78adfbb","Type":"ContainerStarted","Data":"853fb6a6ffefa7f711747b5b5753e0399a2d62778950b16d1b49c505a652b470"}
Sep 16 19:42:25.564884 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:25.564625    2557 generic.go:356] "Generic (PLEG): container finished" podID="03bbdade-309c-4e37-978b-06fb3ef595f3" containerID="a648292ee86fab1f53e79ce1a67daf5c7e6a0702dc26aa7272fe8bcdf63f6a76" exitCode=0
Sep 16 19:42:25.565080 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:25.565055    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03bbdade-309c-4e37-978b-06fb3ef595f3","Type":"ContainerDied","Data":"a648292ee86fab1f53e79ce1a67daf5c7e6a0702dc26aa7272fe8bcdf63f6a76"}
Sep 16 19:42:25.643715 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:25.643677    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-zpbj5"]
Sep 16 19:42:25.646655 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:25.646617    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-zpbj5"
Sep 16 19:42:25.649561 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:25.649494    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 16 19:42:25.650184 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:25.649955    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-mr69j\""
Sep 16 19:42:25.657352 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:25.657318    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-zpbj5"]
Sep 16 19:42:25.700458 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:25.700419    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4b06b88d-9541-4e30-9b3d-24d6121f17ab-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-zpbj5\" (UID: \"4b06b88d-9541-4e30-9b3d-24d6121f17ab\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-zpbj5"
Sep 16 19:42:25.700702 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:25.700477    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rfj2\" (UniqueName: \"kubernetes.io/projected/4b06b88d-9541-4e30-9b3d-24d6121f17ab-kube-api-access-2rfj2\") pod \"monitoring-plugin-5d85778b9d-zpbj5\" (UID: \"4b06b88d-9541-4e30-9b3d-24d6121f17ab\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-zpbj5"
Sep 16 19:42:25.802188 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:25.801907    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4b06b88d-9541-4e30-9b3d-24d6121f17ab-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-zpbj5\" (UID: \"4b06b88d-9541-4e30-9b3d-24d6121f17ab\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-zpbj5"
Sep 16 19:42:25.802188 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:25.801962    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rfj2\" (UniqueName: \"kubernetes.io/projected/4b06b88d-9541-4e30-9b3d-24d6121f17ab-kube-api-access-2rfj2\") pod \"monitoring-plugin-5d85778b9d-zpbj5\" (UID: \"4b06b88d-9541-4e30-9b3d-24d6121f17ab\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-zpbj5"
Sep 16 19:42:25.802188 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:42:25.802020    2557 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 16 19:42:25.802188 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:42:25.802086    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b06b88d-9541-4e30-9b3d-24d6121f17ab-monitoring-plugin-cert podName:4b06b88d-9541-4e30-9b3d-24d6121f17ab nodeName:}" failed. No retries permitted until 2026-09-16 19:42:26.302067178 +0000 UTC m=+174.827455558 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/4b06b88d-9541-4e30-9b3d-24d6121f17ab-monitoring-plugin-cert") pod "monitoring-plugin-5d85778b9d-zpbj5" (UID: "4b06b88d-9541-4e30-9b3d-24d6121f17ab") : secret "monitoring-plugin-cert" not found
Sep 16 19:42:25.814405 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:25.814325    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rfj2\" (UniqueName: \"kubernetes.io/projected/4b06b88d-9541-4e30-9b3d-24d6121f17ab-kube-api-access-2rfj2\") pod \"monitoring-plugin-5d85778b9d-zpbj5\" (UID: \"4b06b88d-9541-4e30-9b3d-24d6121f17ab\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-zpbj5"
Sep 16 19:42:26.170169 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:26.170127    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-68bf5dcbc6-k6gkh"]
Sep 16 19:42:26.172997 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:26.172948    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-68bf5dcbc6-k6gkh"
Sep 16 19:42:26.180210 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:26.175956    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 16 19:42:26.180210 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:26.176206    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 16 19:42:26.180210 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:26.177007    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-kmd5j\""
Sep 16 19:42:26.180210 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:26.177295    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 16 19:42:26.180210 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:26.177582    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 16 19:42:26.180210 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:26.177805    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 16 19:42:26.183774 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:26.183753    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 16 19:42:26.188439 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:26.188368    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-68bf5dcbc6-k6gkh"]
Sep 16 19:42:26.308672 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:26.308629    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d4ad5c1-bf25-43ca-af03-bafda1baddfa-serving-certs-ca-bundle\") pod \"telemeter-client-68bf5dcbc6-k6gkh\" (UID: \"7d4ad5c1-bf25-43ca-af03-bafda1baddfa\") " pod="openshift-monitoring/telemeter-client-68bf5dcbc6-k6gkh"
Sep 16 19:42:26.308893 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:26.308684    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7d4ad5c1-bf25-43ca-af03-bafda1baddfa-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-68bf5dcbc6-k6gkh\" (UID: \"7d4ad5c1-bf25-43ca-af03-bafda1baddfa\") " pod="openshift-monitoring/telemeter-client-68bf5dcbc6-k6gkh"
Sep 16 19:42:26.308893 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:26.308717    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7d4ad5c1-bf25-43ca-af03-bafda1baddfa-federate-client-tls\") pod \"telemeter-client-68bf5dcbc6-k6gkh\" (UID: \"7d4ad5c1-bf25-43ca-af03-bafda1baddfa\") " pod="openshift-monitoring/telemeter-client-68bf5dcbc6-k6gkh"
Sep 16 19:42:26.308893 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:26.308810    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d4ad5c1-bf25-43ca-af03-bafda1baddfa-metrics-client-ca\") pod \"telemeter-client-68bf5dcbc6-k6gkh\" (UID: \"7d4ad5c1-bf25-43ca-af03-bafda1baddfa\") " pod="openshift-monitoring/telemeter-client-68bf5dcbc6-k6gkh"
Sep 16 19:42:26.308893 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:26.308872    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4b06b88d-9541-4e30-9b3d-24d6121f17ab-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-zpbj5\" (UID: \"4b06b88d-9541-4e30-9b3d-24d6121f17ab\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-zpbj5"
Sep 16 19:42:26.309097 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:26.308904    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7d4ad5c1-bf25-43ca-af03-bafda1baddfa-telemeter-client-tls\") pod \"telemeter-client-68bf5dcbc6-k6gkh\" (UID: \"7d4ad5c1-bf25-43ca-af03-bafda1baddfa\") " pod="openshift-monitoring/telemeter-client-68bf5dcbc6-k6gkh"
Sep 16 19:42:26.309097 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:26.308960    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d4ad5c1-bf25-43ca-af03-bafda1baddfa-telemeter-trusted-ca-bundle\") pod \"telemeter-client-68bf5dcbc6-k6gkh\" (UID: \"7d4ad5c1-bf25-43ca-af03-bafda1baddfa\") " pod="openshift-monitoring/telemeter-client-68bf5dcbc6-k6gkh"
Sep 16 19:42:26.309097 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:26.308998    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg5sv\" (UniqueName: \"kubernetes.io/projected/7d4ad5c1-bf25-43ca-af03-bafda1baddfa-kube-api-access-fg5sv\") pod \"telemeter-client-68bf5dcbc6-k6gkh\" (UID: \"7d4ad5c1-bf25-43ca-af03-bafda1baddfa\") " pod="openshift-monitoring/telemeter-client-68bf5dcbc6-k6gkh"
Sep 16 19:42:26.309237 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:26.309103    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7d4ad5c1-bf25-43ca-af03-bafda1baddfa-secret-telemeter-client\") pod \"telemeter-client-68bf5dcbc6-k6gkh\" (UID: \"7d4ad5c1-bf25-43ca-af03-bafda1baddfa\") " pod="openshift-monitoring/telemeter-client-68bf5dcbc6-k6gkh"
Sep 16 19:42:26.311901 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:26.311831    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4b06b88d-9541-4e30-9b3d-24d6121f17ab-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-zpbj5\" (UID: \"4b06b88d-9541-4e30-9b3d-24d6121f17ab\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-zpbj5"
Sep 16 19:42:26.410146 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:26.410053    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7d4ad5c1-bf25-43ca-af03-bafda1baddfa-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-68bf5dcbc6-k6gkh\" (UID: \"7d4ad5c1-bf25-43ca-af03-bafda1baddfa\") " pod="openshift-monitoring/telemeter-client-68bf5dcbc6-k6gkh"
Sep 16 19:42:26.410146 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:26.410106    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7d4ad5c1-bf25-43ca-af03-bafda1baddfa-federate-client-tls\") pod \"telemeter-client-68bf5dcbc6-k6gkh\" (UID: \"7d4ad5c1-bf25-43ca-af03-bafda1baddfa\") " pod="openshift-monitoring/telemeter-client-68bf5dcbc6-k6gkh"
Sep 16 19:42:26.410363 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:26.410181    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d4ad5c1-bf25-43ca-af03-bafda1baddfa-metrics-client-ca\") pod \"telemeter-client-68bf5dcbc6-k6gkh\" (UID: \"7d4ad5c1-bf25-43ca-af03-bafda1baddfa\") " pod="openshift-monitoring/telemeter-client-68bf5dcbc6-k6gkh"
Sep 16 19:42:26.410363 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:26.410214    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7d4ad5c1-bf25-43ca-af03-bafda1baddfa-telemeter-client-tls\") pod \"telemeter-client-68bf5dcbc6-k6gkh\" (UID: \"7d4ad5c1-bf25-43ca-af03-bafda1baddfa\") " pod="openshift-monitoring/telemeter-client-68bf5dcbc6-k6gkh"
Sep 16 19:42:26.410363 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:26.410252    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d4ad5c1-bf25-43ca-af03-bafda1baddfa-telemeter-trusted-ca-bundle\") pod \"telemeter-client-68bf5dcbc6-k6gkh\" (UID: \"7d4ad5c1-bf25-43ca-af03-bafda1baddfa\") " pod="openshift-monitoring/telemeter-client-68bf5dcbc6-k6gkh"
Sep 16 19:42:26.410363 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:26.410288    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg5sv\" (UniqueName: \"kubernetes.io/projected/7d4ad5c1-bf25-43ca-af03-bafda1baddfa-kube-api-access-fg5sv\") pod \"telemeter-client-68bf5dcbc6-k6gkh\" (UID: \"7d4ad5c1-bf25-43ca-af03-bafda1baddfa\") " pod="openshift-monitoring/telemeter-client-68bf5dcbc6-k6gkh"
Sep 16 19:42:26.410363 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:26.410322    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7d4ad5c1-bf25-43ca-af03-bafda1baddfa-secret-telemeter-client\") pod \"telemeter-client-68bf5dcbc6-k6gkh\" (UID: \"7d4ad5c1-bf25-43ca-af03-bafda1baddfa\") " pod="openshift-monitoring/telemeter-client-68bf5dcbc6-k6gkh"
Sep 16 19:42:26.410363 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:26.410356    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d4ad5c1-bf25-43ca-af03-bafda1baddfa-serving-certs-ca-bundle\") pod \"telemeter-client-68bf5dcbc6-k6gkh\" (UID: \"7d4ad5c1-bf25-43ca-af03-bafda1baddfa\") " pod="openshift-monitoring/telemeter-client-68bf5dcbc6-k6gkh"
Sep 16 19:42:26.411174 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:26.411138    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d4ad5c1-bf25-43ca-af03-bafda1baddfa-serving-certs-ca-bundle\") pod \"telemeter-client-68bf5dcbc6-k6gkh\" (UID: \"7d4ad5c1-bf25-43ca-af03-bafda1baddfa\") " pod="openshift-monitoring/telemeter-client-68bf5dcbc6-k6gkh"
Sep 16 19:42:26.411315 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:26.411288    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d4ad5c1-bf25-43ca-af03-bafda1baddfa-metrics-client-ca\") pod \"telemeter-client-68bf5dcbc6-k6gkh\" (UID: \"7d4ad5c1-bf25-43ca-af03-bafda1baddfa\") " pod="openshift-monitoring/telemeter-client-68bf5dcbc6-k6gkh"
Sep 16 19:42:26.411512 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:26.411432    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d4ad5c1-bf25-43ca-af03-bafda1baddfa-telemeter-trusted-ca-bundle\") pod \"telemeter-client-68bf5dcbc6-k6gkh\" (UID: \"7d4ad5c1-bf25-43ca-af03-bafda1baddfa\") " pod="openshift-monitoring/telemeter-client-68bf5dcbc6-k6gkh"
Sep 16 19:42:26.413592 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:26.413547    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7d4ad5c1-bf25-43ca-af03-bafda1baddfa-federate-client-tls\") pod \"telemeter-client-68bf5dcbc6-k6gkh\" (UID: \"7d4ad5c1-bf25-43ca-af03-bafda1baddfa\") " pod="openshift-monitoring/telemeter-client-68bf5dcbc6-k6gkh"
Sep 16 19:42:26.414069 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:26.414044    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7d4ad5c1-bf25-43ca-af03-bafda1baddfa-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-68bf5dcbc6-k6gkh\" (UID: \"7d4ad5c1-bf25-43ca-af03-bafda1baddfa\") " pod="openshift-monitoring/telemeter-client-68bf5dcbc6-k6gkh"
Sep 16 19:42:26.414202 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:26.414159    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7d4ad5c1-bf25-43ca-af03-bafda1baddfa-telemeter-client-tls\") pod \"telemeter-client-68bf5dcbc6-k6gkh\" (UID: \"7d4ad5c1-bf25-43ca-af03-bafda1baddfa\") " pod="openshift-monitoring/telemeter-client-68bf5dcbc6-k6gkh"
Sep 16 19:42:26.414997 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:26.414955    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7d4ad5c1-bf25-43ca-af03-bafda1baddfa-secret-telemeter-client\") pod \"telemeter-client-68bf5dcbc6-k6gkh\" (UID: \"7d4ad5c1-bf25-43ca-af03-bafda1baddfa\") " pod="openshift-monitoring/telemeter-client-68bf5dcbc6-k6gkh"
Sep 16 19:42:26.420351 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:26.420326    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg5sv\" (UniqueName: \"kubernetes.io/projected/7d4ad5c1-bf25-43ca-af03-bafda1baddfa-kube-api-access-fg5sv\") pod \"telemeter-client-68bf5dcbc6-k6gkh\" (UID: \"7d4ad5c1-bf25-43ca-af03-bafda1baddfa\") " pod="openshift-monitoring/telemeter-client-68bf5dcbc6-k6gkh"
Sep 16 19:42:26.486602 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:26.486531    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-68bf5dcbc6-k6gkh"
Sep 16 19:42:26.560495 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:26.560433    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-zpbj5"
Sep 16 19:42:26.572755 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:26.572712    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03bbdade-309c-4e37-978b-06fb3ef595f3","Type":"ContainerStarted","Data":"f6327c3a31c11d3fc9abb35655537fd1a3888984c56f0d9a8d467bbb0997c2cb"}
Sep 16 19:42:27.050765 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.050505    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-zpbj5"]
Sep 16 19:42:27.057512 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:42:27.057458    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b06b88d_9541_4e30_9b3d_24d6121f17ab.slice/crio-7f064aee675992506b7a7d43b10340356e31d876bab085a7f8bfe56baf1b6168 WatchSource:0}: Error finding container 7f064aee675992506b7a7d43b10340356e31d876bab085a7f8bfe56baf1b6168: Status 404 returned error can't find the container with id 7f064aee675992506b7a7d43b10340356e31d876bab085a7f8bfe56baf1b6168
Sep 16 19:42:27.059537 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.059513    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-68bf5dcbc6-k6gkh"]
Sep 16 19:42:27.064307 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:42:27.064254    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d4ad5c1_bf25_43ca_af03_bafda1baddfa.slice/crio-a056189ffd22609b8376bc8258584bd6ceea42c8dedb5554941cf18aa438c773 WatchSource:0}: Error finding container a056189ffd22609b8376bc8258584bd6ceea42c8dedb5554941cf18aa438c773: Status 404 returned error can't find the container with id a056189ffd22609b8376bc8258584bd6ceea42c8dedb5554941cf18aa438c773
Sep 16 19:42:27.378529 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.378450    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 19:42:27.381861 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.381824    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.385018 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.384988    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 19:42:27.385340 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.385315    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 19:42:27.385457 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.385344    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 19:42:27.385457 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.385315    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-2m7pc243auj7n\""
Sep 16 19:42:27.385868 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.385695    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 19:42:27.385868 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.385761    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 19:42:27.385868 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.385775    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 19:42:27.386231 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.386206    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 19:42:27.386231 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.386226    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 19:42:27.386412 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.386285    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 19:42:27.386412 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.386389    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 19:42:27.386510 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.386497    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 16 19:42:27.386718 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.386699    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-7rsf2\""
Sep 16 19:42:27.387029 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.387007    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 19:42:27.387130 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.386956    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 19:42:27.387253 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.387238    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 19:42:27.389345 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.389323    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 19:42:27.405979 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.405952    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 19:42:27.521986 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.521942    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/87ef17b7-b3ef-467c-a198-93fe1d18b0db-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.522189 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.522017    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/87ef17b7-b3ef-467c-a198-93fe1d18b0db-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.522189 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.522066    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/87ef17b7-b3ef-467c-a198-93fe1d18b0db-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.522189 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.522096    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/87ef17b7-b3ef-467c-a198-93fe1d18b0db-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.522189 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.522147    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/87ef17b7-b3ef-467c-a198-93fe1d18b0db-config-out\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.522189 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.522188    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/87ef17b7-b3ef-467c-a198-93fe1d18b0db-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.522453 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.522286    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/87ef17b7-b3ef-467c-a198-93fe1d18b0db-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.522453 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.522327    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/87ef17b7-b3ef-467c-a198-93fe1d18b0db-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.522453 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.522363    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/87ef17b7-b3ef-467c-a198-93fe1d18b0db-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.522453 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.522393    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/87ef17b7-b3ef-467c-a198-93fe1d18b0db-web-config\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.522453 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.522422    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87ef17b7-b3ef-467c-a198-93fe1d18b0db-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.522687 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.522481    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qws2\" (UniqueName: \"kubernetes.io/projected/87ef17b7-b3ef-467c-a198-93fe1d18b0db-kube-api-access-4qws2\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.522687 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.522522    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/87ef17b7-b3ef-467c-a198-93fe1d18b0db-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.522687 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.522583    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/87ef17b7-b3ef-467c-a198-93fe1d18b0db-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.522687 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.522612    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87ef17b7-b3ef-467c-a198-93fe1d18b0db-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.522687 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.522636    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/87ef17b7-b3ef-467c-a198-93fe1d18b0db-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.522687 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.522685    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/87ef17b7-b3ef-467c-a198-93fe1d18b0db-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.523004 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.522710    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/87ef17b7-b3ef-467c-a198-93fe1d18b0db-config\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.523004 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.522734    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/87ef17b7-b3ef-467c-a198-93fe1d18b0db-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.523004 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.522759    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87ef17b7-b3ef-467c-a198-93fe1d18b0db-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.584708 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.584503    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84dbb595c9-fgcq7" event={"ID":"3a10832d-3f24-4464-9291-a8b8c78adfbb","Type":"ContainerStarted","Data":"21ef298541bc38b5d062ab327f7931c785730e1fec72feee6fb4271bbd4f75c7"}
Sep 16 19:42:27.584708 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.584599    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84dbb595c9-fgcq7" event={"ID":"3a10832d-3f24-4464-9291-a8b8c78adfbb","Type":"ContainerStarted","Data":"70565a31fd9789bca1ba32852c4df2f0c36f23ab2ab12d562940650cafec0535"}
Sep 16 19:42:27.584708 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.584641    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84dbb595c9-fgcq7" event={"ID":"3a10832d-3f24-4464-9291-a8b8c78adfbb","Type":"ContainerStarted","Data":"ab2a2678efa481870a9c7044c0a026157f333c5a21f67f90747c4a9459796033"}
Sep 16 19:42:27.587429 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.587346    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-68bf5dcbc6-k6gkh" event={"ID":"7d4ad5c1-bf25-43ca-af03-bafda1baddfa","Type":"ContainerStarted","Data":"a056189ffd22609b8376bc8258584bd6ceea42c8dedb5554941cf18aa438c773"}
Sep 16 19:42:27.592365 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.592111    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03bbdade-309c-4e37-978b-06fb3ef595f3","Type":"ContainerStarted","Data":"71324b144c43d6c597d5912f4e860a8909d6c3b3b8eb428b06331c06720879db"}
Sep 16 19:42:27.592365 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.592234    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03bbdade-309c-4e37-978b-06fb3ef595f3","Type":"ContainerStarted","Data":"506294b1222b6a2bc634f98c8120e33431291556a6e3f83569cb98593175b65b"}
Sep 16 19:42:27.592365 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.592275    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03bbdade-309c-4e37-978b-06fb3ef595f3","Type":"ContainerStarted","Data":"67f60b9639d4e0e3046b03c4b25fee98079dfc6d0d984c1d30ff2df32f924cc4"}
Sep 16 19:42:27.592365 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.592318    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03bbdade-309c-4e37-978b-06fb3ef595f3","Type":"ContainerStarted","Data":"14cd63f2c2f6a5035079bae5ceed4dfe734c157272076036b925ca36145a6bc2"}
Sep 16 19:42:27.593750 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.593691    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-zpbj5" event={"ID":"4b06b88d-9541-4e30-9b3d-24d6121f17ab","Type":"ContainerStarted","Data":"7f064aee675992506b7a7d43b10340356e31d876bab085a7f8bfe56baf1b6168"}
Sep 16 19:42:27.623860 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.623780    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/87ef17b7-b3ef-467c-a198-93fe1d18b0db-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.623860 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.623850    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/87ef17b7-b3ef-467c-a198-93fe1d18b0db-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.624099 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.623880    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87ef17b7-b3ef-467c-a198-93fe1d18b0db-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.624099 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.623909    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/87ef17b7-b3ef-467c-a198-93fe1d18b0db-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.624099 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.623961    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/87ef17b7-b3ef-467c-a198-93fe1d18b0db-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.624099 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.623990    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/87ef17b7-b3ef-467c-a198-93fe1d18b0db-config\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.624099 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.624014    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/87ef17b7-b3ef-467c-a198-93fe1d18b0db-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.624099 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.624038    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87ef17b7-b3ef-467c-a198-93fe1d18b0db-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.624099 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.624075    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/87ef17b7-b3ef-467c-a198-93fe1d18b0db-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.624430 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.624109    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/87ef17b7-b3ef-467c-a198-93fe1d18b0db-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.624430 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.624166    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/87ef17b7-b3ef-467c-a198-93fe1d18b0db-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.624430 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.624190    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/87ef17b7-b3ef-467c-a198-93fe1d18b0db-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.624430 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.624220    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/87ef17b7-b3ef-467c-a198-93fe1d18b0db-config-out\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.624430 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.624257    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/87ef17b7-b3ef-467c-a198-93fe1d18b0db-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.624430 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.624316    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/87ef17b7-b3ef-467c-a198-93fe1d18b0db-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.624430 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.624353    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/87ef17b7-b3ef-467c-a198-93fe1d18b0db-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.624430 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.624395    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/87ef17b7-b3ef-467c-a198-93fe1d18b0db-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.624430 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.624424    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/87ef17b7-b3ef-467c-a198-93fe1d18b0db-web-config\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.624882 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.624448    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87ef17b7-b3ef-467c-a198-93fe1d18b0db-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.624882 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.624479    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qws2\" (UniqueName: \"kubernetes.io/projected/87ef17b7-b3ef-467c-a198-93fe1d18b0db-kube-api-access-4qws2\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.625646 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.625616    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87ef17b7-b3ef-467c-a198-93fe1d18b0db-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.626731 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.626699    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/87ef17b7-b3ef-467c-a198-93fe1d18b0db-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.627734 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.627234    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/87ef17b7-b3ef-467c-a198-93fe1d18b0db-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.628063 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.628035    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/87ef17b7-b3ef-467c-a198-93fe1d18b0db-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.629037 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.628629    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/87ef17b7-b3ef-467c-a198-93fe1d18b0db-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.629828 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.629798    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87ef17b7-b3ef-467c-a198-93fe1d18b0db-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.630464 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.630443    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87ef17b7-b3ef-467c-a198-93fe1d18b0db-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.633504 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.630449    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/87ef17b7-b3ef-467c-a198-93fe1d18b0db-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.634598 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.632215    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/87ef17b7-b3ef-467c-a198-93fe1d18b0db-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.634756 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.632318    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/87ef17b7-b3ef-467c-a198-93fe1d18b0db-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.634905 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.633116    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/87ef17b7-b3ef-467c-a198-93fe1d18b0db-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.635016 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.633344    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/87ef17b7-b3ef-467c-a198-93fe1d18b0db-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.635282 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.633579    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/87ef17b7-b3ef-467c-a198-93fe1d18b0db-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.635415 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.634027    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/87ef17b7-b3ef-467c-a198-93fe1d18b0db-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.635524 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.634153    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/87ef17b7-b3ef-467c-a198-93fe1d18b0db-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.635634 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.634534    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/87ef17b7-b3ef-467c-a198-93fe1d18b0db-config\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.636391 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.635631    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/87ef17b7-b3ef-467c-a198-93fe1d18b0db-web-config\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.637104 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.637064    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/87ef17b7-b3ef-467c-a198-93fe1d18b0db-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.638447 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.638402    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/87ef17b7-b3ef-467c-a198-93fe1d18b0db-config-out\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.644478 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.644427    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qws2\" (UniqueName: \"kubernetes.io/projected/87ef17b7-b3ef-467c-a198-93fe1d18b0db-kube-api-access-4qws2\") pod \"prometheus-k8s-0\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:27.696518 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:27.696081    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:28.033364 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:28.033338    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 19:42:28.039097 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:42:28.039056    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87ef17b7_b3ef_467c_a198_93fe1d18b0db.slice/crio-e1e451ba1496a45902675d5e47f29c1befc61fbe39623e849e75d3dfb53f5849 WatchSource:0}: Error finding container e1e451ba1496a45902675d5e47f29c1befc61fbe39623e849e75d3dfb53f5849: Status 404 returned error can't find the container with id e1e451ba1496a45902675d5e47f29c1befc61fbe39623e849e75d3dfb53f5849
Sep 16 19:42:28.521886 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:28.521851    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-d8hqp"
Sep 16 19:42:28.603702 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:28.603525    2557 generic.go:356] "Generic (PLEG): container finished" podID="87ef17b7-b3ef-467c-a198-93fe1d18b0db" containerID="be8f941313221ef84455ab9d15f78fe375dadf240ff3c8f767f60eeb002edf44" exitCode=0
Sep 16 19:42:28.604193 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:28.603731    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"87ef17b7-b3ef-467c-a198-93fe1d18b0db","Type":"ContainerDied","Data":"be8f941313221ef84455ab9d15f78fe375dadf240ff3c8f767f60eeb002edf44"}
Sep 16 19:42:28.604193 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:28.603856    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"87ef17b7-b3ef-467c-a198-93fe1d18b0db","Type":"ContainerStarted","Data":"e1e451ba1496a45902675d5e47f29c1befc61fbe39623e849e75d3dfb53f5849"}
Sep 16 19:42:28.608160 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:28.607998    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84dbb595c9-fgcq7" event={"ID":"3a10832d-3f24-4464-9291-a8b8c78adfbb","Type":"ContainerStarted","Data":"9b254623b782be7df7836b67d9396b001dedcdfeb4555c9b2d3cf7b0ba103f62"}
Sep 16 19:42:28.608160 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:28.608073    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84dbb595c9-fgcq7" event={"ID":"3a10832d-3f24-4464-9291-a8b8c78adfbb","Type":"ContainerStarted","Data":"418e6732561f4067021105d575c52253e58bb4d60ed759f2f302cc84e66e83bb"}
Sep 16 19:42:28.614874 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:28.614827    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03bbdade-309c-4e37-978b-06fb3ef595f3","Type":"ContainerStarted","Data":"329b5fb452b9e443e193aa6012484118d837f37493b7cf4c0a188096ccad3e76"}
Sep 16 19:42:28.668916 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:28.668825    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.069557878 podStartE2EDuration="7.668800421s" totalImagesPullingTime="3.577234499s" totalInitContainerRuntime="22.008044ms" isStatefulPod=false podCreationTimestamp="2026-09-16 19:42:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:42:28.666201908 +0000 UTC m=+177.191590294" watchObservedRunningTime="2026-09-16 19:42:28.668800421 +0000 UTC m=+177.194188810"
Sep 16 19:42:29.625876 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:29.625395    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84dbb595c9-fgcq7" event={"ID":"3a10832d-3f24-4464-9291-a8b8c78adfbb","Type":"ContainerStarted","Data":"0cd3fad808548fe0ebdd29ccfbae8ef9e931447b1008b619d4fabdf32997bbd8"}
Sep 16 19:42:29.625876 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:29.625550    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-84dbb595c9-fgcq7"
Sep 16 19:42:29.628875 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:29.628746    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-68bf5dcbc6-k6gkh" event={"ID":"7d4ad5c1-bf25-43ca-af03-bafda1baddfa","Type":"ContainerStarted","Data":"0018e5585f3dbd76e125feac414f73ae50b481f1a299cd309f9690f342e98283"}
Sep 16 19:42:29.628875 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:29.628807    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-68bf5dcbc6-k6gkh" event={"ID":"7d4ad5c1-bf25-43ca-af03-bafda1baddfa","Type":"ContainerStarted","Data":"dc7aa9fe8c880a2706754b5750475a26c7c4ea9c3921c5369cdabd7ba7b36181"}
Sep 16 19:42:29.629107 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:29.628861    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-68bf5dcbc6-k6gkh" event={"ID":"7d4ad5c1-bf25-43ca-af03-bafda1baddfa","Type":"ContainerStarted","Data":"3f07fc9424c5ebf4c6792c1f228a3f0a72ad3bb496efebdc4acc9fc51af66366"}
Sep 16 19:42:29.630926 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:29.630834    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-zpbj5" event={"ID":"4b06b88d-9541-4e30-9b3d-24d6121f17ab","Type":"ContainerStarted","Data":"c5dc2bd1392866f1ab09ade994c02ffc2ca2b22fd39f40c015e7b05b42764987"}
Sep 16 19:42:29.630926 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:29.630900    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-zpbj5"
Sep 16 19:42:29.638053 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:29.638031    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-zpbj5"
Sep 16 19:42:29.656418 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:29.656222    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-84dbb595c9-fgcq7" podStartSLOduration=2.576383449 podStartE2EDuration="5.656199453s" totalImagesPullingTime="3.079816004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:42:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:42:29.653406159 +0000 UTC m=+178.178794548" watchObservedRunningTime="2026-09-16 19:42:29.656199453 +0000 UTC m=+178.181587839"
Sep 16 19:42:29.673634 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:29.673569    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-zpbj5" podStartSLOduration=2.509078248 podStartE2EDuration="4.673548394s" totalImagesPullingTime="2.164470146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:42:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:42:29.671972961 +0000 UTC m=+178.197361357" watchObservedRunningTime="2026-09-16 19:42:29.673548394 +0000 UTC m=+178.198936780"
Sep 16 19:42:29.701572 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:29.701507    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-68bf5dcbc6-k6gkh" podStartSLOduration=1.54131267 podStartE2EDuration="3.701482093s" totalImagesPullingTime="2.160169423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:42:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:42:29.699891915 +0000 UTC m=+178.225280300" watchObservedRunningTime="2026-09-16 19:42:29.701482093 +0000 UTC m=+178.226870482"
Sep 16 19:42:33.658321 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:33.658280    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"87ef17b7-b3ef-467c-a198-93fe1d18b0db","Type":"ContainerStarted","Data":"f8c717d5be338f41b2bc1d025fd141722a0da8bb02b795ad96655ede69edb294"}
Sep 16 19:42:33.658772 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:33.658408    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"87ef17b7-b3ef-467c-a198-93fe1d18b0db","Type":"ContainerStarted","Data":"ee46327ad99558c3f50bdbe3227beb5c82e136c191ed7946147500842467a15e"}
Sep 16 19:42:33.658772 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:33.658454    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"87ef17b7-b3ef-467c-a198-93fe1d18b0db","Type":"ContainerStarted","Data":"5c24f42cc048bc921e61888787f56b7486a0307d4ed6c3cf38f844f8f841ce27"}
Sep 16 19:42:33.658772 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:33.658500    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"87ef17b7-b3ef-467c-a198-93fe1d18b0db","Type":"ContainerStarted","Data":"c16042680722e3582b05025107d6c0fccd24cf12f33aac57090700fe3ced7dc5"}
Sep 16 19:42:33.658772 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:33.658539    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"87ef17b7-b3ef-467c-a198-93fe1d18b0db","Type":"ContainerStarted","Data":"f3b2dae5f1f9a744707aa1174c43799067703047d79496f82b3f20d84ec1f420"}
Sep 16 19:42:33.658772 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:33.658576    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"87ef17b7-b3ef-467c-a198-93fe1d18b0db","Type":"ContainerStarted","Data":"b1ac4e810c919eae85f68c5e4840180177825a332c386104898d51f5a2e1abde"}
Sep 16 19:42:33.690472 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:33.690406    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.402386627 podStartE2EDuration="6.690382487s" totalImagesPullingTime="4.259522186s" totalInitContainerRuntime="28.473674ms" isStatefulPod=false podCreationTimestamp="2026-09-16 19:42:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:42:33.688574524 +0000 UTC m=+182.213962934" watchObservedRunningTime="2026-09-16 19:42:33.690382487 +0000 UTC m=+182.215770873"
Sep 16 19:42:35.644264 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:35.644228    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-84dbb595c9-fgcq7"
Sep 16 19:42:37.697221 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:37.697176    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:42:41.700903 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:41.700834    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-7kt88" event={"ID":"f4c1d81c-50de-47c5-a944-cf1571f49abb","Type":"ContainerStarted","Data":"37d5a1a5a32264f6f03da37ab7258d13d9481cb3deccb14c74ad875a88fb89b2"}
Sep 16 19:42:41.701396 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:41.701083    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-7kt88"
Sep 16 19:42:41.722005 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:41.721940    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-7kt88" podStartSLOduration=2.312456505 podStartE2EDuration="19.721921446s" totalImagesPullingTime="17.409464941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:42:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:42:41.720358236 +0000 UTC m=+190.245746633" watchObservedRunningTime="2026-09-16 19:42:41.721921446 +0000 UTC m=+190.247309833"
Sep 16 19:42:41.722588 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:41.722561    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-7kt88"
Sep 16 19:42:52.756681 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:52.756645    2557 generic.go:356] "Generic (PLEG): container finished" podID="8ebbee05-192c-4db3-ba7a-f0de3c33983c" containerID="ed722bbaf6011cb3633fa26b18396928036155cacb52ba6e5021128078f42ec6" exitCode=0
Sep 16 19:42:52.757107 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:52.756716    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-9tm7s" event={"ID":"8ebbee05-192c-4db3-ba7a-f0de3c33983c","Type":"ContainerDied","Data":"ed722bbaf6011cb3633fa26b18396928036155cacb52ba6e5021128078f42ec6"}
Sep 16 19:42:52.757107 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:52.757103    2557 scope.go:117] "RemoveContainer" containerID="ed722bbaf6011cb3633fa26b18396928036155cacb52ba6e5021128078f42ec6"
Sep 16 19:42:53.762995 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:53.762959    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-9tm7s" event={"ID":"8ebbee05-192c-4db3-ba7a-f0de3c33983c","Type":"ContainerStarted","Data":"3d08f17b1ff0fdb81d8f8b4abd88f249f3348062e72a8deb1aa96958b231b056"}
Sep 16 19:42:57.781009 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:57.780972    2557 generic.go:356] "Generic (PLEG): container finished" podID="0f022ef3-16e6-4890-ae68-3644f69ee234" containerID="f71c54f15b23a38e372a4980a54960e13dc5e41b98c4503ea8c3b087515313fa" exitCode=0
Sep 16 19:42:57.781451 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:57.781035    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qfdnr" event={"ID":"0f022ef3-16e6-4890-ae68-3644f69ee234","Type":"ContainerDied","Data":"f71c54f15b23a38e372a4980a54960e13dc5e41b98c4503ea8c3b087515313fa"}
Sep 16 19:42:57.781451 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:57.781327    2557 scope.go:117] "RemoveContainer" containerID="f71c54f15b23a38e372a4980a54960e13dc5e41b98c4503ea8c3b087515313fa"
Sep 16 19:42:58.789475 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:42:58.789441    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qfdnr" event={"ID":"0f022ef3-16e6-4890-ae68-3644f69ee234","Type":"ContainerStarted","Data":"7f4b894b7efbbd66fca210b711f6635259153cd360a0fa32b7296e81949a7b7b"}
Sep 16 19:43:27.696966 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:27.696918    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:27.713207 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:27.713179    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:27.936819 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:27.936793    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:41.552425 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:41.552388    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 19:43:41.552931 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:41.552673    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="03bbdade-309c-4e37-978b-06fb3ef595f3" containerName="alertmanager" containerID="cri-o://f6327c3a31c11d3fc9abb35655537fd1a3888984c56f0d9a8d467bbb0997c2cb" gracePeriod=120
Sep 16 19:43:41.552931 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:41.552754    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="03bbdade-309c-4e37-978b-06fb3ef595f3" containerName="kube-rbac-proxy-metric" containerID="cri-o://71324b144c43d6c597d5912f4e860a8909d6c3b3b8eb428b06331c06720879db" gracePeriod=120
Sep 16 19:43:41.552931 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:41.552788    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="03bbdade-309c-4e37-978b-06fb3ef595f3" containerName="kube-rbac-proxy" containerID="cri-o://506294b1222b6a2bc634f98c8120e33431291556a6e3f83569cb98593175b65b" gracePeriod=120
Sep 16 19:43:41.552931 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:41.552786    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="03bbdade-309c-4e37-978b-06fb3ef595f3" containerName="kube-rbac-proxy-web" containerID="cri-o://67f60b9639d4e0e3046b03c4b25fee98079dfc6d0d984c1d30ff2df32f924cc4" gracePeriod=120
Sep 16 19:43:41.552931 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:41.552811    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="03bbdade-309c-4e37-978b-06fb3ef595f3" containerName="config-reloader" containerID="cri-o://14cd63f2c2f6a5035079bae5ceed4dfe734c157272076036b925ca36145a6bc2" gracePeriod=120
Sep 16 19:43:41.552931 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:41.552881    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="03bbdade-309c-4e37-978b-06fb3ef595f3" containerName="prom-label-proxy" containerID="cri-o://329b5fb452b9e443e193aa6012484118d837f37493b7cf4c0a188096ccad3e76" gracePeriod=120
Sep 16 19:43:41.985091 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:41.985061    2557 generic.go:356] "Generic (PLEG): container finished" podID="03bbdade-309c-4e37-978b-06fb3ef595f3" containerID="329b5fb452b9e443e193aa6012484118d837f37493b7cf4c0a188096ccad3e76" exitCode=0
Sep 16 19:43:41.985091 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:41.985086    2557 generic.go:356] "Generic (PLEG): container finished" podID="03bbdade-309c-4e37-978b-06fb3ef595f3" containerID="71324b144c43d6c597d5912f4e860a8909d6c3b3b8eb428b06331c06720879db" exitCode=0
Sep 16 19:43:41.985091 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:41.985093    2557 generic.go:356] "Generic (PLEG): container finished" podID="03bbdade-309c-4e37-978b-06fb3ef595f3" containerID="506294b1222b6a2bc634f98c8120e33431291556a6e3f83569cb98593175b65b" exitCode=0
Sep 16 19:43:41.985091 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:41.985099    2557 generic.go:356] "Generic (PLEG): container finished" podID="03bbdade-309c-4e37-978b-06fb3ef595f3" containerID="14cd63f2c2f6a5035079bae5ceed4dfe734c157272076036b925ca36145a6bc2" exitCode=0
Sep 16 19:43:41.985326 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:41.985106    2557 generic.go:356] "Generic (PLEG): container finished" podID="03bbdade-309c-4e37-978b-06fb3ef595f3" containerID="f6327c3a31c11d3fc9abb35655537fd1a3888984c56f0d9a8d467bbb0997c2cb" exitCode=0
Sep 16 19:43:41.985326 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:41.985128    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03bbdade-309c-4e37-978b-06fb3ef595f3","Type":"ContainerDied","Data":"329b5fb452b9e443e193aa6012484118d837f37493b7cf4c0a188096ccad3e76"}
Sep 16 19:43:41.985326 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:41.985203    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03bbdade-309c-4e37-978b-06fb3ef595f3","Type":"ContainerDied","Data":"71324b144c43d6c597d5912f4e860a8909d6c3b3b8eb428b06331c06720879db"}
Sep 16 19:43:41.985326 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:41.985233    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03bbdade-309c-4e37-978b-06fb3ef595f3","Type":"ContainerDied","Data":"506294b1222b6a2bc634f98c8120e33431291556a6e3f83569cb98593175b65b"}
Sep 16 19:43:41.985326 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:41.985258    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03bbdade-309c-4e37-978b-06fb3ef595f3","Type":"ContainerDied","Data":"14cd63f2c2f6a5035079bae5ceed4dfe734c157272076036b925ca36145a6bc2"}
Sep 16 19:43:41.985326 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:41.985280    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03bbdade-309c-4e37-978b-06fb3ef595f3","Type":"ContainerDied","Data":"f6327c3a31c11d3fc9abb35655537fd1a3888984c56f0d9a8d467bbb0997c2cb"}
Sep 16 19:43:42.800807 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:42.800782    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:43:42.943648 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:42.943567    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/03bbdade-309c-4e37-978b-06fb3ef595f3-cluster-tls-config\") pod \"03bbdade-309c-4e37-978b-06fb3ef595f3\" (UID: \"03bbdade-309c-4e37-978b-06fb3ef595f3\") "
Sep 16 19:43:42.943648 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:42.943616    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03bbdade-309c-4e37-978b-06fb3ef595f3-alertmanager-trusted-ca-bundle\") pod \"03bbdade-309c-4e37-978b-06fb3ef595f3\" (UID: \"03bbdade-309c-4e37-978b-06fb3ef595f3\") "
Sep 16 19:43:42.943648 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:42.943637    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/03bbdade-309c-4e37-978b-06fb3ef595f3-secret-alertmanager-kube-rbac-proxy-web\") pod \"03bbdade-309c-4e37-978b-06fb3ef595f3\" (UID: \"03bbdade-309c-4e37-978b-06fb3ef595f3\") "
Sep 16 19:43:42.943922 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:42.943656    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hqkdc\" (UniqueName: \"kubernetes.io/projected/03bbdade-309c-4e37-978b-06fb3ef595f3-kube-api-access-hqkdc\") pod \"03bbdade-309c-4e37-978b-06fb3ef595f3\" (UID: \"03bbdade-309c-4e37-978b-06fb3ef595f3\") "
Sep 16 19:43:42.943922 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:42.943672    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/03bbdade-309c-4e37-978b-06fb3ef595f3-secret-alertmanager-kube-rbac-proxy\") pod \"03bbdade-309c-4e37-978b-06fb3ef595f3\" (UID: \"03bbdade-309c-4e37-978b-06fb3ef595f3\") "
Sep 16 19:43:42.943922 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:42.943724    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/03bbdade-309c-4e37-978b-06fb3ef595f3-secret-alertmanager-main-tls\") pod \"03bbdade-309c-4e37-978b-06fb3ef595f3\" (UID: \"03bbdade-309c-4e37-978b-06fb3ef595f3\") "
Sep 16 19:43:42.943922 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:42.943747    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/03bbdade-309c-4e37-978b-06fb3ef595f3-tls-assets\") pod \"03bbdade-309c-4e37-978b-06fb3ef595f3\" (UID: \"03bbdade-309c-4e37-978b-06fb3ef595f3\") "
Sep 16 19:43:42.943922 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:42.943785    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/03bbdade-309c-4e37-978b-06fb3ef595f3-web-config\") pod \"03bbdade-309c-4e37-978b-06fb3ef595f3\" (UID: \"03bbdade-309c-4e37-978b-06fb3ef595f3\") "
Sep 16 19:43:42.943922 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:42.943816    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/03bbdade-309c-4e37-978b-06fb3ef595f3-alertmanager-main-db\") pod \"03bbdade-309c-4e37-978b-06fb3ef595f3\" (UID: \"03bbdade-309c-4e37-978b-06fb3ef595f3\") "
Sep 16 19:43:42.943922 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:42.943879    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/03bbdade-309c-4e37-978b-06fb3ef595f3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"03bbdade-309c-4e37-978b-06fb3ef595f3\" (UID: \"03bbdade-309c-4e37-978b-06fb3ef595f3\") "
Sep 16 19:43:42.943922 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:42.943906    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/03bbdade-309c-4e37-978b-06fb3ef595f3-config-volume\") pod \"03bbdade-309c-4e37-978b-06fb3ef595f3\" (UID: \"03bbdade-309c-4e37-978b-06fb3ef595f3\") "
Sep 16 19:43:42.944369 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:42.943934    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03bbdade-309c-4e37-978b-06fb3ef595f3-metrics-client-ca\") pod \"03bbdade-309c-4e37-978b-06fb3ef595f3\" (UID: \"03bbdade-309c-4e37-978b-06fb3ef595f3\") "
Sep 16 19:43:42.944369 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:42.943975    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/03bbdade-309c-4e37-978b-06fb3ef595f3-config-out\") pod \"03bbdade-309c-4e37-978b-06fb3ef595f3\" (UID: \"03bbdade-309c-4e37-978b-06fb3ef595f3\") "
Sep 16 19:43:42.944369 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:42.944217    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03bbdade-309c-4e37-978b-06fb3ef595f3-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "03bbdade-309c-4e37-978b-06fb3ef595f3" (UID: "03bbdade-309c-4e37-978b-06fb3ef595f3"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:43:42.944369 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:42.944237    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03bbdade-309c-4e37-978b-06fb3ef595f3-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "03bbdade-309c-4e37-978b-06fb3ef595f3" (UID: "03bbdade-309c-4e37-978b-06fb3ef595f3"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:43:42.944570 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:42.944391    2557 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03bbdade-309c-4e37-978b-06fb3ef595f3-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 19:43:42.944570 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:42.944413    2557 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/03bbdade-309c-4e37-978b-06fb3ef595f3-alertmanager-main-db\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 19:43:42.945084 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:42.945046    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03bbdade-309c-4e37-978b-06fb3ef595f3-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "03bbdade-309c-4e37-978b-06fb3ef595f3" (UID: "03bbdade-309c-4e37-978b-06fb3ef595f3"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:43:42.946599 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:42.946550    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03bbdade-309c-4e37-978b-06fb3ef595f3-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "03bbdade-309c-4e37-978b-06fb3ef595f3" (UID: "03bbdade-309c-4e37-978b-06fb3ef595f3"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:43:42.946953 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:42.946926    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03bbdade-309c-4e37-978b-06fb3ef595f3-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "03bbdade-309c-4e37-978b-06fb3ef595f3" (UID: "03bbdade-309c-4e37-978b-06fb3ef595f3"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:43:42.947281 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:42.947241    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03bbdade-309c-4e37-978b-06fb3ef595f3-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "03bbdade-309c-4e37-978b-06fb3ef595f3" (UID: "03bbdade-309c-4e37-978b-06fb3ef595f3"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:43:42.947379 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:42.947330    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03bbdade-309c-4e37-978b-06fb3ef595f3-config-volume" (OuterVolumeSpecName: "config-volume") pod "03bbdade-309c-4e37-978b-06fb3ef595f3" (UID: "03bbdade-309c-4e37-978b-06fb3ef595f3"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:43:42.947604 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:42.947577    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03bbdade-309c-4e37-978b-06fb3ef595f3-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "03bbdade-309c-4e37-978b-06fb3ef595f3" (UID: "03bbdade-309c-4e37-978b-06fb3ef595f3"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:43:42.947822 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:42.947792    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03bbdade-309c-4e37-978b-06fb3ef595f3-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "03bbdade-309c-4e37-978b-06fb3ef595f3" (UID: "03bbdade-309c-4e37-978b-06fb3ef595f3"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:43:42.948186 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:42.948162    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03bbdade-309c-4e37-978b-06fb3ef595f3-kube-api-access-hqkdc" (OuterVolumeSpecName: "kube-api-access-hqkdc") pod "03bbdade-309c-4e37-978b-06fb3ef595f3" (UID: "03bbdade-309c-4e37-978b-06fb3ef595f3"). InnerVolumeSpecName "kube-api-access-hqkdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:43:42.948265 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:42.948247    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03bbdade-309c-4e37-978b-06fb3ef595f3-config-out" (OuterVolumeSpecName: "config-out") pod "03bbdade-309c-4e37-978b-06fb3ef595f3" (UID: "03bbdade-309c-4e37-978b-06fb3ef595f3"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:43:42.951460 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:42.951429    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03bbdade-309c-4e37-978b-06fb3ef595f3-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "03bbdade-309c-4e37-978b-06fb3ef595f3" (UID: "03bbdade-309c-4e37-978b-06fb3ef595f3"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:43:42.958698 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:42.958671    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03bbdade-309c-4e37-978b-06fb3ef595f3-web-config" (OuterVolumeSpecName: "web-config") pod "03bbdade-309c-4e37-978b-06fb3ef595f3" (UID: "03bbdade-309c-4e37-978b-06fb3ef595f3"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:43:42.992262 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:42.992227    2557 generic.go:356] "Generic (PLEG): container finished" podID="03bbdade-309c-4e37-978b-06fb3ef595f3" containerID="67f60b9639d4e0e3046b03c4b25fee98079dfc6d0d984c1d30ff2df32f924cc4" exitCode=0
Sep 16 19:43:42.992425 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:42.992306    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:43:42.992425 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:42.992300    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03bbdade-309c-4e37-978b-06fb3ef595f3","Type":"ContainerDied","Data":"67f60b9639d4e0e3046b03c4b25fee98079dfc6d0d984c1d30ff2df32f924cc4"}
Sep 16 19:43:42.992496 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:42.992439    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03bbdade-309c-4e37-978b-06fb3ef595f3","Type":"ContainerDied","Data":"86801deb2e556a42fb18bf23dde46a3a90a3404092b27cf9cebd4ad1a863017f"}
Sep 16 19:43:42.992496 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:42.992483    2557 scope.go:117] "RemoveContainer" containerID="329b5fb452b9e443e193aa6012484118d837f37493b7cf4c0a188096ccad3e76"
Sep 16 19:43:43.000803 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.000785    2557 scope.go:117] "RemoveContainer" containerID="71324b144c43d6c597d5912f4e860a8909d6c3b3b8eb428b06331c06720879db"
Sep 16 19:43:43.009225 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.009205    2557 scope.go:117] "RemoveContainer" containerID="506294b1222b6a2bc634f98c8120e33431291556a6e3f83569cb98593175b65b"
Sep 16 19:43:43.019655 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.019635    2557 scope.go:117] "RemoveContainer" containerID="67f60b9639d4e0e3046b03c4b25fee98079dfc6d0d984c1d30ff2df32f924cc4"
Sep 16 19:43:43.027431 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.027375    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 19:43:43.027502 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.027483    2557 scope.go:117] "RemoveContainer" containerID="14cd63f2c2f6a5035079bae5ceed4dfe734c157272076036b925ca36145a6bc2"
Sep 16 19:43:43.033274 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.033252    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 19:43:43.035189 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.035168    2557 scope.go:117] "RemoveContainer" containerID="f6327c3a31c11d3fc9abb35655537fd1a3888984c56f0d9a8d467bbb0997c2cb"
Sep 16 19:43:43.043530 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.043511    2557 scope.go:117] "RemoveContainer" containerID="a648292ee86fab1f53e79ce1a67daf5c7e6a0702dc26aa7272fe8bcdf63f6a76"
Sep 16 19:43:43.044885 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.044859    2557 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/03bbdade-309c-4e37-978b-06fb3ef595f3-secret-alertmanager-main-tls\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 19:43:43.044984 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.044886    2557 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/03bbdade-309c-4e37-978b-06fb3ef595f3-tls-assets\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 19:43:43.044984 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.044897    2557 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/03bbdade-309c-4e37-978b-06fb3ef595f3-web-config\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 19:43:43.044984 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.044905    2557 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/03bbdade-309c-4e37-978b-06fb3ef595f3-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 19:43:43.044984 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.044916    2557 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/03bbdade-309c-4e37-978b-06fb3ef595f3-config-volume\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 19:43:43.044984 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.044925    2557 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03bbdade-309c-4e37-978b-06fb3ef595f3-metrics-client-ca\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 19:43:43.044984 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.044933    2557 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/03bbdade-309c-4e37-978b-06fb3ef595f3-config-out\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 19:43:43.044984 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.044942    2557 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/03bbdade-309c-4e37-978b-06fb3ef595f3-cluster-tls-config\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 19:43:43.044984 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.044950    2557 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/03bbdade-309c-4e37-978b-06fb3ef595f3-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 19:43:43.044984 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.044960    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqkdc\" (UniqueName: \"kubernetes.io/projected/03bbdade-309c-4e37-978b-06fb3ef595f3-kube-api-access-hqkdc\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 19:43:43.044984 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.044968    2557 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/03bbdade-309c-4e37-978b-06fb3ef595f3-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 19:43:43.053117 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.053086    2557 scope.go:117] "RemoveContainer" containerID="329b5fb452b9e443e193aa6012484118d837f37493b7cf4c0a188096ccad3e76"
Sep 16 19:43:43.053649 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:43:43.053510    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"329b5fb452b9e443e193aa6012484118d837f37493b7cf4c0a188096ccad3e76\": container with ID starting with 329b5fb452b9e443e193aa6012484118d837f37493b7cf4c0a188096ccad3e76 not found: ID does not exist" containerID="329b5fb452b9e443e193aa6012484118d837f37493b7cf4c0a188096ccad3e76"
Sep 16 19:43:43.053649 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.053551    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"329b5fb452b9e443e193aa6012484118d837f37493b7cf4c0a188096ccad3e76"} err="failed to get container status \"329b5fb452b9e443e193aa6012484118d837f37493b7cf4c0a188096ccad3e76\": rpc error: code = NotFound desc = could not find container \"329b5fb452b9e443e193aa6012484118d837f37493b7cf4c0a188096ccad3e76\": container with ID starting with 329b5fb452b9e443e193aa6012484118d837f37493b7cf4c0a188096ccad3e76 not found: ID does not exist"
Sep 16 19:43:43.053649 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.053597    2557 scope.go:117] "RemoveContainer" containerID="71324b144c43d6c597d5912f4e860a8909d6c3b3b8eb428b06331c06720879db"
Sep 16 19:43:43.054018 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:43:43.053995    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71324b144c43d6c597d5912f4e860a8909d6c3b3b8eb428b06331c06720879db\": container with ID starting with 71324b144c43d6c597d5912f4e860a8909d6c3b3b8eb428b06331c06720879db not found: ID does not exist" containerID="71324b144c43d6c597d5912f4e860a8909d6c3b3b8eb428b06331c06720879db"
Sep 16 19:43:43.054072 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.054025    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71324b144c43d6c597d5912f4e860a8909d6c3b3b8eb428b06331c06720879db"} err="failed to get container status \"71324b144c43d6c597d5912f4e860a8909d6c3b3b8eb428b06331c06720879db\": rpc error: code = NotFound desc = could not find container \"71324b144c43d6c597d5912f4e860a8909d6c3b3b8eb428b06331c06720879db\": container with ID starting with 71324b144c43d6c597d5912f4e860a8909d6c3b3b8eb428b06331c06720879db not found: ID does not exist"
Sep 16 19:43:43.054072 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.054041    2557 scope.go:117] "RemoveContainer" containerID="506294b1222b6a2bc634f98c8120e33431291556a6e3f83569cb98593175b65b"
Sep 16 19:43:43.054319 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:43:43.054299    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"506294b1222b6a2bc634f98c8120e33431291556a6e3f83569cb98593175b65b\": container with ID starting with 506294b1222b6a2bc634f98c8120e33431291556a6e3f83569cb98593175b65b not found: ID does not exist" containerID="506294b1222b6a2bc634f98c8120e33431291556a6e3f83569cb98593175b65b"
Sep 16 19:43:43.054379 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.054327    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"506294b1222b6a2bc634f98c8120e33431291556a6e3f83569cb98593175b65b"} err="failed to get container status \"506294b1222b6a2bc634f98c8120e33431291556a6e3f83569cb98593175b65b\": rpc error: code = NotFound desc = could not find container \"506294b1222b6a2bc634f98c8120e33431291556a6e3f83569cb98593175b65b\": container with ID starting with 506294b1222b6a2bc634f98c8120e33431291556a6e3f83569cb98593175b65b not found: ID does not exist"
Sep 16 19:43:43.054379 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.054351    2557 scope.go:117] "RemoveContainer" containerID="67f60b9639d4e0e3046b03c4b25fee98079dfc6d0d984c1d30ff2df32f924cc4"
Sep 16 19:43:43.054619 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:43:43.054601    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67f60b9639d4e0e3046b03c4b25fee98079dfc6d0d984c1d30ff2df32f924cc4\": container with ID starting with 67f60b9639d4e0e3046b03c4b25fee98079dfc6d0d984c1d30ff2df32f924cc4 not found: ID does not exist" containerID="67f60b9639d4e0e3046b03c4b25fee98079dfc6d0d984c1d30ff2df32f924cc4"
Sep 16 19:43:43.054691 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.054625    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67f60b9639d4e0e3046b03c4b25fee98079dfc6d0d984c1d30ff2df32f924cc4"} err="failed to get container status \"67f60b9639d4e0e3046b03c4b25fee98079dfc6d0d984c1d30ff2df32f924cc4\": rpc error: code = NotFound desc = could not find container \"67f60b9639d4e0e3046b03c4b25fee98079dfc6d0d984c1d30ff2df32f924cc4\": container with ID starting with 67f60b9639d4e0e3046b03c4b25fee98079dfc6d0d984c1d30ff2df32f924cc4 not found: ID does not exist"
Sep 16 19:43:43.054691 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.054640    2557 scope.go:117] "RemoveContainer" containerID="14cd63f2c2f6a5035079bae5ceed4dfe734c157272076036b925ca36145a6bc2"
Sep 16 19:43:43.054900 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:43:43.054880    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14cd63f2c2f6a5035079bae5ceed4dfe734c157272076036b925ca36145a6bc2\": container with ID starting with 14cd63f2c2f6a5035079bae5ceed4dfe734c157272076036b925ca36145a6bc2 not found: ID does not exist" containerID="14cd63f2c2f6a5035079bae5ceed4dfe734c157272076036b925ca36145a6bc2"
Sep 16 19:43:43.054969 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.054908    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14cd63f2c2f6a5035079bae5ceed4dfe734c157272076036b925ca36145a6bc2"} err="failed to get container status \"14cd63f2c2f6a5035079bae5ceed4dfe734c157272076036b925ca36145a6bc2\": rpc error: code = NotFound desc = could not find container \"14cd63f2c2f6a5035079bae5ceed4dfe734c157272076036b925ca36145a6bc2\": container with ID starting with 14cd63f2c2f6a5035079bae5ceed4dfe734c157272076036b925ca36145a6bc2 not found: ID does not exist"
Sep 16 19:43:43.054969 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.054929    2557 scope.go:117] "RemoveContainer" containerID="f6327c3a31c11d3fc9abb35655537fd1a3888984c56f0d9a8d467bbb0997c2cb"
Sep 16 19:43:43.055193 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:43:43.055176    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6327c3a31c11d3fc9abb35655537fd1a3888984c56f0d9a8d467bbb0997c2cb\": container with ID starting with f6327c3a31c11d3fc9abb35655537fd1a3888984c56f0d9a8d467bbb0997c2cb not found: ID does not exist" containerID="f6327c3a31c11d3fc9abb35655537fd1a3888984c56f0d9a8d467bbb0997c2cb"
Sep 16 19:43:43.055235 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.055198    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6327c3a31c11d3fc9abb35655537fd1a3888984c56f0d9a8d467bbb0997c2cb"} err="failed to get container status \"f6327c3a31c11d3fc9abb35655537fd1a3888984c56f0d9a8d467bbb0997c2cb\": rpc error: code = NotFound desc = could not find container \"f6327c3a31c11d3fc9abb35655537fd1a3888984c56f0d9a8d467bbb0997c2cb\": container with ID starting with f6327c3a31c11d3fc9abb35655537fd1a3888984c56f0d9a8d467bbb0997c2cb not found: ID does not exist"
Sep 16 19:43:43.055235 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.055212    2557 scope.go:117] "RemoveContainer" containerID="a648292ee86fab1f53e79ce1a67daf5c7e6a0702dc26aa7272fe8bcdf63f6a76"
Sep 16 19:43:43.055410 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:43:43.055393    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a648292ee86fab1f53e79ce1a67daf5c7e6a0702dc26aa7272fe8bcdf63f6a76\": container with ID starting with a648292ee86fab1f53e79ce1a67daf5c7e6a0702dc26aa7272fe8bcdf63f6a76 not found: ID does not exist" containerID="a648292ee86fab1f53e79ce1a67daf5c7e6a0702dc26aa7272fe8bcdf63f6a76"
Sep 16 19:43:43.055451 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.055415    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a648292ee86fab1f53e79ce1a67daf5c7e6a0702dc26aa7272fe8bcdf63f6a76"} err="failed to get container status \"a648292ee86fab1f53e79ce1a67daf5c7e6a0702dc26aa7272fe8bcdf63f6a76\": rpc error: code = NotFound desc = could not find container \"a648292ee86fab1f53e79ce1a67daf5c7e6a0702dc26aa7272fe8bcdf63f6a76\": container with ID starting with a648292ee86fab1f53e79ce1a67daf5c7e6a0702dc26aa7272fe8bcdf63f6a76 not found: ID does not exist"
Sep 16 19:43:43.082746 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.082705    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 19:43:43.083097 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.083082    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03bbdade-309c-4e37-978b-06fb3ef595f3" containerName="init-config-reloader"
Sep 16 19:43:43.083097 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.083097    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="03bbdade-309c-4e37-978b-06fb3ef595f3" containerName="init-config-reloader"
Sep 16 19:43:43.083196 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.083107    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03bbdade-309c-4e37-978b-06fb3ef595f3" containerName="kube-rbac-proxy-metric"
Sep 16 19:43:43.083196 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.083113    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="03bbdade-309c-4e37-978b-06fb3ef595f3" containerName="kube-rbac-proxy-metric"
Sep 16 19:43:43.083196 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.083129    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03bbdade-309c-4e37-978b-06fb3ef595f3" containerName="alertmanager"
Sep 16 19:43:43.083196 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.083135    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="03bbdade-309c-4e37-978b-06fb3ef595f3" containerName="alertmanager"
Sep 16 19:43:43.083196 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.083143    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03bbdade-309c-4e37-978b-06fb3ef595f3" containerName="prom-label-proxy"
Sep 16 19:43:43.083196 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.083149    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="03bbdade-309c-4e37-978b-06fb3ef595f3" containerName="prom-label-proxy"
Sep 16 19:43:43.083196 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.083156    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03bbdade-309c-4e37-978b-06fb3ef595f3" containerName="config-reloader"
Sep 16 19:43:43.083196 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.083161    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="03bbdade-309c-4e37-978b-06fb3ef595f3" containerName="config-reloader"
Sep 16 19:43:43.083196 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.083166    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03bbdade-309c-4e37-978b-06fb3ef595f3" containerName="kube-rbac-proxy"
Sep 16 19:43:43.083196 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.083171    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="03bbdade-309c-4e37-978b-06fb3ef595f3" containerName="kube-rbac-proxy"
Sep 16 19:43:43.083196 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.083178    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03bbdade-309c-4e37-978b-06fb3ef595f3" containerName="kube-rbac-proxy-web"
Sep 16 19:43:43.083196 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.083184    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="03bbdade-309c-4e37-978b-06fb3ef595f3" containerName="kube-rbac-proxy-web"
Sep 16 19:43:43.083587 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.083237    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="03bbdade-309c-4e37-978b-06fb3ef595f3" containerName="kube-rbac-proxy-metric"
Sep 16 19:43:43.083587 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.083246    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="03bbdade-309c-4e37-978b-06fb3ef595f3" containerName="config-reloader"
Sep 16 19:43:43.083587 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.083252    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="03bbdade-309c-4e37-978b-06fb3ef595f3" containerName="prom-label-proxy"
Sep 16 19:43:43.083587 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.083259    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="03bbdade-309c-4e37-978b-06fb3ef595f3" containerName="kube-rbac-proxy"
Sep 16 19:43:43.083587 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.083266    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="03bbdade-309c-4e37-978b-06fb3ef595f3" containerName="kube-rbac-proxy-web"
Sep 16 19:43:43.083587 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.083272    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="03bbdade-309c-4e37-978b-06fb3ef595f3" containerName="alertmanager"
Sep 16 19:43:43.087244 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.087222    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:43:43.090684 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.090654    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 16 19:43:43.093062 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.093034    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 16 19:43:43.093238 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.093219    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 16 19:43:43.093383 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.093361    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 16 19:43:43.093474 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.093456    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 16 19:43:43.093628 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.093576    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 16 19:43:43.093628 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.093620    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 16 19:43:43.094037 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.094019    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-qd72j\""
Sep 16 19:43:43.094466 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.094447    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 16 19:43:43.102394 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.102358    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 16 19:43:43.105001 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.104974    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 19:43:43.145768 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.145724    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9213f9dd-3f19-4700-9586-3b9cb8616647-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9213f9dd-3f19-4700-9586-3b9cb8616647\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:43:43.145768 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.145777    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9213f9dd-3f19-4700-9586-3b9cb8616647-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9213f9dd-3f19-4700-9586-3b9cb8616647\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:43:43.146076 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.145796    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9213f9dd-3f19-4700-9586-3b9cb8616647-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9213f9dd-3f19-4700-9586-3b9cb8616647\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:43:43.146076 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.145948    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9213f9dd-3f19-4700-9586-3b9cb8616647-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9213f9dd-3f19-4700-9586-3b9cb8616647\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:43:43.146076 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.145975    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9213f9dd-3f19-4700-9586-3b9cb8616647-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9213f9dd-3f19-4700-9586-3b9cb8616647\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:43:43.146076 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.146003    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9213f9dd-3f19-4700-9586-3b9cb8616647-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9213f9dd-3f19-4700-9586-3b9cb8616647\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:43:43.146076 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.146026    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9213f9dd-3f19-4700-9586-3b9cb8616647-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9213f9dd-3f19-4700-9586-3b9cb8616647\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:43:43.146076 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.146058    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9213f9dd-3f19-4700-9586-3b9cb8616647-config-out\") pod \"alertmanager-main-0\" (UID: \"9213f9dd-3f19-4700-9586-3b9cb8616647\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:43:43.146290 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.146087    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9213f9dd-3f19-4700-9586-3b9cb8616647-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9213f9dd-3f19-4700-9586-3b9cb8616647\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:43:43.146290 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.146119    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9213f9dd-3f19-4700-9586-3b9cb8616647-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9213f9dd-3f19-4700-9586-3b9cb8616647\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:43:43.146290 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.146133    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxtfn\" (UniqueName: \"kubernetes.io/projected/9213f9dd-3f19-4700-9586-3b9cb8616647-kube-api-access-jxtfn\") pod \"alertmanager-main-0\" (UID: \"9213f9dd-3f19-4700-9586-3b9cb8616647\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:43:43.146290 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.146159    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9213f9dd-3f19-4700-9586-3b9cb8616647-config-volume\") pod \"alertmanager-main-0\" (UID: \"9213f9dd-3f19-4700-9586-3b9cb8616647\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:43:43.146290 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.146202    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9213f9dd-3f19-4700-9586-3b9cb8616647-web-config\") pod \"alertmanager-main-0\" (UID: \"9213f9dd-3f19-4700-9586-3b9cb8616647\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:43:43.246793 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.246759    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9213f9dd-3f19-4700-9586-3b9cb8616647-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9213f9dd-3f19-4700-9586-3b9cb8616647\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:43:43.246992 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.246798    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxtfn\" (UniqueName: \"kubernetes.io/projected/9213f9dd-3f19-4700-9586-3b9cb8616647-kube-api-access-jxtfn\") pod \"alertmanager-main-0\" (UID: \"9213f9dd-3f19-4700-9586-3b9cb8616647\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:43:43.246992 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.246820    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9213f9dd-3f19-4700-9586-3b9cb8616647-config-volume\") pod \"alertmanager-main-0\" (UID: \"9213f9dd-3f19-4700-9586-3b9cb8616647\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:43:43.246992 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.246857    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9213f9dd-3f19-4700-9586-3b9cb8616647-web-config\") pod \"alertmanager-main-0\" (UID: \"9213f9dd-3f19-4700-9586-3b9cb8616647\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:43:43.246992 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.246910    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9213f9dd-3f19-4700-9586-3b9cb8616647-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9213f9dd-3f19-4700-9586-3b9cb8616647\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:43:43.246992 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.246963    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9213f9dd-3f19-4700-9586-3b9cb8616647-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9213f9dd-3f19-4700-9586-3b9cb8616647\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:43:43.246992 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.246981    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9213f9dd-3f19-4700-9586-3b9cb8616647-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9213f9dd-3f19-4700-9586-3b9cb8616647\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:43:43.247313 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.247022    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9213f9dd-3f19-4700-9586-3b9cb8616647-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9213f9dd-3f19-4700-9586-3b9cb8616647\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:43:43.247313 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.247039    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9213f9dd-3f19-4700-9586-3b9cb8616647-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9213f9dd-3f19-4700-9586-3b9cb8616647\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:43:43.247313 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.247162    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9213f9dd-3f19-4700-9586-3b9cb8616647-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9213f9dd-3f19-4700-9586-3b9cb8616647\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:43:43.247313 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.247206    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9213f9dd-3f19-4700-9586-3b9cb8616647-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9213f9dd-3f19-4700-9586-3b9cb8616647\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:43:43.247313 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.247241    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9213f9dd-3f19-4700-9586-3b9cb8616647-config-out\") pod \"alertmanager-main-0\" (UID: \"9213f9dd-3f19-4700-9586-3b9cb8616647\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:43:43.247313 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.247277    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9213f9dd-3f19-4700-9586-3b9cb8616647-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9213f9dd-3f19-4700-9586-3b9cb8616647\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:43:43.247818 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.247796    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9213f9dd-3f19-4700-9586-3b9cb8616647-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9213f9dd-3f19-4700-9586-3b9cb8616647\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:43:43.247979 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.247956    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9213f9dd-3f19-4700-9586-3b9cb8616647-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9213f9dd-3f19-4700-9586-3b9cb8616647\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:43:43.248567 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.248258    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9213f9dd-3f19-4700-9586-3b9cb8616647-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9213f9dd-3f19-4700-9586-3b9cb8616647\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:43:43.250263 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.250014    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9213f9dd-3f19-4700-9586-3b9cb8616647-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9213f9dd-3f19-4700-9586-3b9cb8616647\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:43:43.250263 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.250039    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9213f9dd-3f19-4700-9586-3b9cb8616647-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9213f9dd-3f19-4700-9586-3b9cb8616647\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:43:43.250263 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.250143    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9213f9dd-3f19-4700-9586-3b9cb8616647-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9213f9dd-3f19-4700-9586-3b9cb8616647\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:43:43.250263 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.250194    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9213f9dd-3f19-4700-9586-3b9cb8616647-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9213f9dd-3f19-4700-9586-3b9cb8616647\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:43:43.250464 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.250269    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9213f9dd-3f19-4700-9586-3b9cb8616647-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9213f9dd-3f19-4700-9586-3b9cb8616647\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:43:43.250464 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.250447    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9213f9dd-3f19-4700-9586-3b9cb8616647-web-config\") pod \"alertmanager-main-0\" (UID: \"9213f9dd-3f19-4700-9586-3b9cb8616647\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:43:43.250592 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.250576    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9213f9dd-3f19-4700-9586-3b9cb8616647-config-volume\") pod \"alertmanager-main-0\" (UID: \"9213f9dd-3f19-4700-9586-3b9cb8616647\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:43:43.250810 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.250794    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9213f9dd-3f19-4700-9586-3b9cb8616647-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9213f9dd-3f19-4700-9586-3b9cb8616647\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:43:43.251743 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.251728    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9213f9dd-3f19-4700-9586-3b9cb8616647-config-out\") pod \"alertmanager-main-0\" (UID: \"9213f9dd-3f19-4700-9586-3b9cb8616647\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:43:43.262925 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.262832    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxtfn\" (UniqueName: \"kubernetes.io/projected/9213f9dd-3f19-4700-9586-3b9cb8616647-kube-api-access-jxtfn\") pod \"alertmanager-main-0\" (UID: \"9213f9dd-3f19-4700-9586-3b9cb8616647\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:43:43.397602 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.397565    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:43:43.688478 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:43:43.688435    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9213f9dd_3f19_4700_9586_3b9cb8616647.slice/crio-d63f6852eea0c7390f7776dbc3d92f49a2043bfdcd2033e1b658add999c6b097 WatchSource:0}: Error finding container d63f6852eea0c7390f7776dbc3d92f49a2043bfdcd2033e1b658add999c6b097: Status 404 returned error can't find the container with id d63f6852eea0c7390f7776dbc3d92f49a2043bfdcd2033e1b658add999c6b097
Sep 16 19:43:43.689780 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.689748    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 19:43:43.752132 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.752097    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4ea9fa1-48c7-45be-bfcf-4594ee751086-metrics-certs\") pod \"network-metrics-daemon-drgl8\" (UID: \"b4ea9fa1-48c7-45be-bfcf-4594ee751086\") " pod="openshift-multus/network-metrics-daemon-drgl8"
Sep 16 19:43:43.754374 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.754346    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4ea9fa1-48c7-45be-bfcf-4594ee751086-metrics-certs\") pod \"network-metrics-daemon-drgl8\" (UID: \"b4ea9fa1-48c7-45be-bfcf-4594ee751086\") " pod="openshift-multus/network-metrics-daemon-drgl8"
Sep 16 19:43:43.834991 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.834956    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qmvpx\""
Sep 16 19:43:43.842509 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.842486    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-drgl8"
Sep 16 19:43:43.936792 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.936757    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03bbdade-309c-4e37-978b-06fb3ef595f3" path="/var/lib/kubelet/pods/03bbdade-309c-4e37-978b-06fb3ef595f3/volumes"
Sep 16 19:43:43.998762 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.998725    2557 generic.go:356] "Generic (PLEG): container finished" podID="9213f9dd-3f19-4700-9586-3b9cb8616647" containerID="dc1851cd7bcc0306dc5190e8e71cf18d13cdbcde6784683769f04b6f06238923" exitCode=0
Sep 16 19:43:43.998978 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.998803    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9213f9dd-3f19-4700-9586-3b9cb8616647","Type":"ContainerDied","Data":"dc1851cd7bcc0306dc5190e8e71cf18d13cdbcde6784683769f04b6f06238923"}
Sep 16 19:43:43.998978 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:43.998899    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9213f9dd-3f19-4700-9586-3b9cb8616647","Type":"ContainerStarted","Data":"d63f6852eea0c7390f7776dbc3d92f49a2043bfdcd2033e1b658add999c6b097"}
Sep 16 19:43:44.113646 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:44.113528    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-drgl8"]
Sep 16 19:43:44.116016 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:43:44.115988    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4ea9fa1_48c7_45be_bfcf_4594ee751086.slice/crio-0b9c5f94d317fe6275c0b373fdbf402036699920aafe5b355104d50cce862e0b WatchSource:0}: Error finding container 0b9c5f94d317fe6275c0b373fdbf402036699920aafe5b355104d50cce862e0b: Status 404 returned error can't find the container with id 0b9c5f94d317fe6275c0b373fdbf402036699920aafe5b355104d50cce862e0b
Sep 16 19:43:45.004099 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:45.004074    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-drgl8" event={"ID":"b4ea9fa1-48c7-45be-bfcf-4594ee751086","Type":"ContainerStarted","Data":"0b9c5f94d317fe6275c0b373fdbf402036699920aafe5b355104d50cce862e0b"}
Sep 16 19:43:45.007500 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:45.007474    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9213f9dd-3f19-4700-9586-3b9cb8616647","Type":"ContainerStarted","Data":"b08e2e7076b07e1c9028fc3bae8c486a3ebb3f63800439216ec4cb8762342e18"}
Sep 16 19:43:45.007603 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:45.007536    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9213f9dd-3f19-4700-9586-3b9cb8616647","Type":"ContainerStarted","Data":"ab2b45050a8d152423fa34be399474a2423bee68ee4937c3dae9ff57b11f4861"}
Sep 16 19:43:45.007603 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:45.007562    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9213f9dd-3f19-4700-9586-3b9cb8616647","Type":"ContainerStarted","Data":"916f19565f1be04c264cbd60373ba77bd77b7896a619a46f0f30af06f2dfb524"}
Sep 16 19:43:45.007603 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:45.007587    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9213f9dd-3f19-4700-9586-3b9cb8616647","Type":"ContainerStarted","Data":"b7f603d61d39afe76a39af786e347cefcaf4b5f55f6fe55e3b59413a0b41a2e9"}
Sep 16 19:43:45.007701 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:45.007611    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9213f9dd-3f19-4700-9586-3b9cb8616647","Type":"ContainerStarted","Data":"8531a0fdb3f368bc38367f491ac8764e70de0995f074390ea845a2951a7112a8"}
Sep 16 19:43:45.007701 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:45.007634    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9213f9dd-3f19-4700-9586-3b9cb8616647","Type":"ContainerStarted","Data":"16658902f986c6daced8fbbd280d3db289112e323bfc80e897e2582d4fee65b1"}
Sep 16 19:43:45.044309 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:45.044243    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.029236146 podStartE2EDuration="2.044220951s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.984805ms" isStatefulPod=false podCreationTimestamp="2026-09-16 19:43:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:43:45.04264303 +0000 UTC m=+253.568031417" watchObservedRunningTime="2026-09-16 19:43:45.044220951 +0000 UTC m=+253.569609337"
Sep 16 19:43:45.812046 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:45.812006    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 19:43:45.812362 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:45.812324    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="87ef17b7-b3ef-467c-a198-93fe1d18b0db" containerName="prometheus" containerID="cri-o://b1ac4e810c919eae85f68c5e4840180177825a332c386104898d51f5a2e1abde" gracePeriod=600
Sep 16 19:43:45.812362 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:45.812348    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="87ef17b7-b3ef-467c-a198-93fe1d18b0db" containerName="thanos-sidecar" containerID="cri-o://c16042680722e3582b05025107d6c0fccd24cf12f33aac57090700fe3ced7dc5" gracePeriod=600
Sep 16 19:43:45.812568 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:45.812379    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="87ef17b7-b3ef-467c-a198-93fe1d18b0db" containerName="kube-rbac-proxy-thanos" containerID="cri-o://f8c717d5be338f41b2bc1d025fd141722a0da8bb02b795ad96655ede69edb294" gracePeriod=600
Sep 16 19:43:45.812568 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:45.812401    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="87ef17b7-b3ef-467c-a198-93fe1d18b0db" containerName="config-reloader" containerID="cri-o://f3b2dae5f1f9a744707aa1174c43799067703047d79496f82b3f20d84ec1f420" gracePeriod=600
Sep 16 19:43:45.812568 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:45.812348    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="87ef17b7-b3ef-467c-a198-93fe1d18b0db" containerName="kube-rbac-proxy" containerID="cri-o://ee46327ad99558c3f50bdbe3227beb5c82e136c191ed7946147500842467a15e" gracePeriod=600
Sep 16 19:43:45.812568 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:45.812373    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="87ef17b7-b3ef-467c-a198-93fe1d18b0db" containerName="kube-rbac-proxy-web" containerID="cri-o://5c24f42cc048bc921e61888787f56b7486a0307d4ed6c3cf38f844f8f841ce27" gracePeriod=600
Sep 16 19:43:46.019810 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.019776    2557 generic.go:356] "Generic (PLEG): container finished" podID="87ef17b7-b3ef-467c-a198-93fe1d18b0db" containerID="f8c717d5be338f41b2bc1d025fd141722a0da8bb02b795ad96655ede69edb294" exitCode=0
Sep 16 19:43:46.019810 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.019802    2557 generic.go:356] "Generic (PLEG): container finished" podID="87ef17b7-b3ef-467c-a198-93fe1d18b0db" containerID="ee46327ad99558c3f50bdbe3227beb5c82e136c191ed7946147500842467a15e" exitCode=0
Sep 16 19:43:46.019810 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.019810    2557 generic.go:356] "Generic (PLEG): container finished" podID="87ef17b7-b3ef-467c-a198-93fe1d18b0db" containerID="5c24f42cc048bc921e61888787f56b7486a0307d4ed6c3cf38f844f8f841ce27" exitCode=0
Sep 16 19:43:46.019810 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.019816    2557 generic.go:356] "Generic (PLEG): container finished" podID="87ef17b7-b3ef-467c-a198-93fe1d18b0db" containerID="c16042680722e3582b05025107d6c0fccd24cf12f33aac57090700fe3ced7dc5" exitCode=0
Sep 16 19:43:46.019810 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.019822    2557 generic.go:356] "Generic (PLEG): container finished" podID="87ef17b7-b3ef-467c-a198-93fe1d18b0db" containerID="f3b2dae5f1f9a744707aa1174c43799067703047d79496f82b3f20d84ec1f420" exitCode=0
Sep 16 19:43:46.020546 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.019828    2557 generic.go:356] "Generic (PLEG): container finished" podID="87ef17b7-b3ef-467c-a198-93fe1d18b0db" containerID="b1ac4e810c919eae85f68c5e4840180177825a332c386104898d51f5a2e1abde" exitCode=0
Sep 16 19:43:46.020546 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.019861    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"87ef17b7-b3ef-467c-a198-93fe1d18b0db","Type":"ContainerDied","Data":"f8c717d5be338f41b2bc1d025fd141722a0da8bb02b795ad96655ede69edb294"}
Sep 16 19:43:46.020546 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.019923    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"87ef17b7-b3ef-467c-a198-93fe1d18b0db","Type":"ContainerDied","Data":"ee46327ad99558c3f50bdbe3227beb5c82e136c191ed7946147500842467a15e"}
Sep 16 19:43:46.020546 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.019949    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"87ef17b7-b3ef-467c-a198-93fe1d18b0db","Type":"ContainerDied","Data":"5c24f42cc048bc921e61888787f56b7486a0307d4ed6c3cf38f844f8f841ce27"}
Sep 16 19:43:46.020546 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.019977    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"87ef17b7-b3ef-467c-a198-93fe1d18b0db","Type":"ContainerDied","Data":"c16042680722e3582b05025107d6c0fccd24cf12f33aac57090700fe3ced7dc5"}
Sep 16 19:43:46.020546 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.020011    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"87ef17b7-b3ef-467c-a198-93fe1d18b0db","Type":"ContainerDied","Data":"f3b2dae5f1f9a744707aa1174c43799067703047d79496f82b3f20d84ec1f420"}
Sep 16 19:43:46.020546 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.020052    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"87ef17b7-b3ef-467c-a198-93fe1d18b0db","Type":"ContainerDied","Data":"b1ac4e810c919eae85f68c5e4840180177825a332c386104898d51f5a2e1abde"}
Sep 16 19:43:46.021784 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.021761    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-drgl8" event={"ID":"b4ea9fa1-48c7-45be-bfcf-4594ee751086","Type":"ContainerStarted","Data":"997f31d9a61104daf2baa9acc6fc9ae2594912eadbbb73d0175ae07102f25d70"}
Sep 16 19:43:46.021938 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.021802    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-drgl8" event={"ID":"b4ea9fa1-48c7-45be-bfcf-4594ee751086","Type":"ContainerStarted","Data":"adf83dd3d56518e1934c64e3c19f5d4b08f18ad034eff5afb646b3c20626218f"}
Sep 16 19:43:46.042674 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.042629    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-drgl8" podStartSLOduration=253.162526195 podStartE2EDuration="4m14.042610537s" totalImagesPullingTime="880.084342ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:39:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:43:46.040726441 +0000 UTC m=+254.566114848" watchObservedRunningTime="2026-09-16 19:43:46.042610537 +0000 UTC m=+254.567998921"
Sep 16 19:43:46.051075 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.051050    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:46.174352 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.174247    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/87ef17b7-b3ef-467c-a198-93fe1d18b0db-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") "
Sep 16 19:43:46.174352 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.174297    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/87ef17b7-b3ef-467c-a198-93fe1d18b0db-secret-prometheus-k8s-tls\") pod \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") "
Sep 16 19:43:46.174352 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.174319    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87ef17b7-b3ef-467c-a198-93fe1d18b0db-prometheus-trusted-ca-bundle\") pod \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") "
Sep 16 19:43:46.174632 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.174359    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87ef17b7-b3ef-467c-a198-93fe1d18b0db-configmap-kubelet-serving-ca-bundle\") pod \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") "
Sep 16 19:43:46.174632 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.174382    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/87ef17b7-b3ef-467c-a198-93fe1d18b0db-thanos-prometheus-http-client-file\") pod \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") "
Sep 16 19:43:46.174632 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.174428    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/87ef17b7-b3ef-467c-a198-93fe1d18b0db-secret-kube-rbac-proxy\") pod \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") "
Sep 16 19:43:46.174632 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.174457    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4qws2\" (UniqueName: \"kubernetes.io/projected/87ef17b7-b3ef-467c-a198-93fe1d18b0db-kube-api-access-4qws2\") pod \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") "
Sep 16 19:43:46.174632 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.174484    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/87ef17b7-b3ef-467c-a198-93fe1d18b0db-prometheus-k8s-db\") pod \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") "
Sep 16 19:43:46.174632 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.174521    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87ef17b7-b3ef-467c-a198-93fe1d18b0db-configmap-serving-certs-ca-bundle\") pod \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") "
Sep 16 19:43:46.174632 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.174558    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/87ef17b7-b3ef-467c-a198-93fe1d18b0db-web-config\") pod \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") "
Sep 16 19:43:46.174632 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.174590    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/87ef17b7-b3ef-467c-a198-93fe1d18b0db-secret-metrics-client-certs\") pod \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") "
Sep 16 19:43:46.174632 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.174619    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/87ef17b7-b3ef-467c-a198-93fe1d18b0db-prometheus-k8s-rulefiles-2\") pod \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") "
Sep 16 19:43:46.175091 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.174641    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/87ef17b7-b3ef-467c-a198-93fe1d18b0db-secret-grpc-tls\") pod \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") "
Sep 16 19:43:46.175091 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.174675    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/87ef17b7-b3ef-467c-a198-93fe1d18b0db-config-out\") pod \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") "
Sep 16 19:43:46.175091 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.174698    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/87ef17b7-b3ef-467c-a198-93fe1d18b0db-prometheus-k8s-rulefiles-0\") pod \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") "
Sep 16 19:43:46.175091 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.174722    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/87ef17b7-b3ef-467c-a198-93fe1d18b0db-tls-assets\") pod \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") "
Sep 16 19:43:46.175091 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.174750    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/87ef17b7-b3ef-467c-a198-93fe1d18b0db-configmap-metrics-client-ca\") pod \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") "
Sep 16 19:43:46.175091 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.174784    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/87ef17b7-b3ef-467c-a198-93fe1d18b0db-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") "
Sep 16 19:43:46.175091 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.174818    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/87ef17b7-b3ef-467c-a198-93fe1d18b0db-config\") pod \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") "
Sep 16 19:43:46.175091 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.174882    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87ef17b7-b3ef-467c-a198-93fe1d18b0db-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "87ef17b7-b3ef-467c-a198-93fe1d18b0db" (UID: "87ef17b7-b3ef-467c-a198-93fe1d18b0db"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:43:46.175091 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.174916    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/87ef17b7-b3ef-467c-a198-93fe1d18b0db-prometheus-k8s-rulefiles-1\") pod \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\" (UID: \"87ef17b7-b3ef-467c-a198-93fe1d18b0db\") "
Sep 16 19:43:46.175091 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.175001    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87ef17b7-b3ef-467c-a198-93fe1d18b0db-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "87ef17b7-b3ef-467c-a198-93fe1d18b0db" (UID: "87ef17b7-b3ef-467c-a198-93fe1d18b0db"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:43:46.175683 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.175658    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87ef17b7-b3ef-467c-a198-93fe1d18b0db-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "87ef17b7-b3ef-467c-a198-93fe1d18b0db" (UID: "87ef17b7-b3ef-467c-a198-93fe1d18b0db"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:43:46.175683 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.175663    2557 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87ef17b7-b3ef-467c-a198-93fe1d18b0db-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 19:43:46.176648 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.176615    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87ef17b7-b3ef-467c-a198-93fe1d18b0db-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "87ef17b7-b3ef-467c-a198-93fe1d18b0db" (UID: "87ef17b7-b3ef-467c-a198-93fe1d18b0db"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:43:46.177242 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.177216    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87ef17b7-b3ef-467c-a198-93fe1d18b0db-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "87ef17b7-b3ef-467c-a198-93fe1d18b0db" (UID: "87ef17b7-b3ef-467c-a198-93fe1d18b0db"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:43:46.177361 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.177243    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87ef17b7-b3ef-467c-a198-93fe1d18b0db-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "87ef17b7-b3ef-467c-a198-93fe1d18b0db" (UID: "87ef17b7-b3ef-467c-a198-93fe1d18b0db"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:43:46.177361 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.177285    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87ef17b7-b3ef-467c-a198-93fe1d18b0db-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "87ef17b7-b3ef-467c-a198-93fe1d18b0db" (UID: "87ef17b7-b3ef-467c-a198-93fe1d18b0db"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:43:46.177607 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.177584    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87ef17b7-b3ef-467c-a198-93fe1d18b0db-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "87ef17b7-b3ef-467c-a198-93fe1d18b0db" (UID: "87ef17b7-b3ef-467c-a198-93fe1d18b0db"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:43:46.177908 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.177879    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87ef17b7-b3ef-467c-a198-93fe1d18b0db-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "87ef17b7-b3ef-467c-a198-93fe1d18b0db" (UID: "87ef17b7-b3ef-467c-a198-93fe1d18b0db"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:43:46.178128 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.178001    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87ef17b7-b3ef-467c-a198-93fe1d18b0db-config-out" (OuterVolumeSpecName: "config-out") pod "87ef17b7-b3ef-467c-a198-93fe1d18b0db" (UID: "87ef17b7-b3ef-467c-a198-93fe1d18b0db"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:43:46.178207 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.178162    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87ef17b7-b3ef-467c-a198-93fe1d18b0db-kube-api-access-4qws2" (OuterVolumeSpecName: "kube-api-access-4qws2") pod "87ef17b7-b3ef-467c-a198-93fe1d18b0db" (UID: "87ef17b7-b3ef-467c-a198-93fe1d18b0db"). InnerVolumeSpecName "kube-api-access-4qws2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:43:46.178476 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.178446    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87ef17b7-b3ef-467c-a198-93fe1d18b0db-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "87ef17b7-b3ef-467c-a198-93fe1d18b0db" (UID: "87ef17b7-b3ef-467c-a198-93fe1d18b0db"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:43:46.178476 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.178460    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87ef17b7-b3ef-467c-a198-93fe1d18b0db-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "87ef17b7-b3ef-467c-a198-93fe1d18b0db" (UID: "87ef17b7-b3ef-467c-a198-93fe1d18b0db"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:43:46.178629 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.178515    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87ef17b7-b3ef-467c-a198-93fe1d18b0db-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "87ef17b7-b3ef-467c-a198-93fe1d18b0db" (UID: "87ef17b7-b3ef-467c-a198-93fe1d18b0db"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:43:46.178791 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.178765    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87ef17b7-b3ef-467c-a198-93fe1d18b0db-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "87ef17b7-b3ef-467c-a198-93fe1d18b0db" (UID: "87ef17b7-b3ef-467c-a198-93fe1d18b0db"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:43:46.178905 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.178879    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87ef17b7-b3ef-467c-a198-93fe1d18b0db-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "87ef17b7-b3ef-467c-a198-93fe1d18b0db" (UID: "87ef17b7-b3ef-467c-a198-93fe1d18b0db"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:43:46.179111 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.179083    2557 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87ef17b7-b3ef-467c-a198-93fe1d18b0db-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 19:43:46.180279 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.180258    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87ef17b7-b3ef-467c-a198-93fe1d18b0db-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "87ef17b7-b3ef-467c-a198-93fe1d18b0db" (UID: "87ef17b7-b3ef-467c-a198-93fe1d18b0db"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:43:46.180382 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.180307    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87ef17b7-b3ef-467c-a198-93fe1d18b0db-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "87ef17b7-b3ef-467c-a198-93fe1d18b0db" (UID: "87ef17b7-b3ef-467c-a198-93fe1d18b0db"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:43:46.180693 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.180679    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87ef17b7-b3ef-467c-a198-93fe1d18b0db-config" (OuterVolumeSpecName: "config") pod "87ef17b7-b3ef-467c-a198-93fe1d18b0db" (UID: "87ef17b7-b3ef-467c-a198-93fe1d18b0db"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:43:46.190597 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.190566    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87ef17b7-b3ef-467c-a198-93fe1d18b0db-web-config" (OuterVolumeSpecName: "web-config") pod "87ef17b7-b3ef-467c-a198-93fe1d18b0db" (UID: "87ef17b7-b3ef-467c-a198-93fe1d18b0db"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:43:46.280273 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.280236    2557 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/87ef17b7-b3ef-467c-a198-93fe1d18b0db-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 19:43:46.280273 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.280268    2557 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/87ef17b7-b3ef-467c-a198-93fe1d18b0db-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 19:43:46.280273 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.280278    2557 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/87ef17b7-b3ef-467c-a198-93fe1d18b0db-secret-prometheus-k8s-tls\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 19:43:46.280503 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.280289    2557 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87ef17b7-b3ef-467c-a198-93fe1d18b0db-prometheus-trusted-ca-bundle\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 19:43:46.280503 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.280298    2557 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/87ef17b7-b3ef-467c-a198-93fe1d18b0db-thanos-prometheus-http-client-file\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 19:43:46.280503 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.280307    2557 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/87ef17b7-b3ef-467c-a198-93fe1d18b0db-secret-kube-rbac-proxy\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 19:43:46.280503 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.280315    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qws2\" (UniqueName: \"kubernetes.io/projected/87ef17b7-b3ef-467c-a198-93fe1d18b0db-kube-api-access-4qws2\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 19:43:46.280503 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.280323    2557 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/87ef17b7-b3ef-467c-a198-93fe1d18b0db-prometheus-k8s-db\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 19:43:46.280503 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.280332    2557 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/87ef17b7-b3ef-467c-a198-93fe1d18b0db-web-config\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 19:43:46.280503 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.280340    2557 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/87ef17b7-b3ef-467c-a198-93fe1d18b0db-secret-metrics-client-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 19:43:46.280503 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.280348    2557 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/87ef17b7-b3ef-467c-a198-93fe1d18b0db-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 19:43:46.280503 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.280357    2557 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/87ef17b7-b3ef-467c-a198-93fe1d18b0db-secret-grpc-tls\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 19:43:46.280503 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.280364    2557 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/87ef17b7-b3ef-467c-a198-93fe1d18b0db-config-out\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 19:43:46.280503 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.280375    2557 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/87ef17b7-b3ef-467c-a198-93fe1d18b0db-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 19:43:46.280503 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.280382    2557 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/87ef17b7-b3ef-467c-a198-93fe1d18b0db-tls-assets\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 19:43:46.280503 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.280391    2557 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/87ef17b7-b3ef-467c-a198-93fe1d18b0db-configmap-metrics-client-ca\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 19:43:46.280503 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.280399    2557 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/87ef17b7-b3ef-467c-a198-93fe1d18b0db-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 19:43:46.280503 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:46.280407    2557 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/87ef17b7-b3ef-467c-a198-93fe1d18b0db-config\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 19:43:47.029529 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.029495    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.029529 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.029497    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"87ef17b7-b3ef-467c-a198-93fe1d18b0db","Type":"ContainerDied","Data":"e1e451ba1496a45902675d5e47f29c1befc61fbe39623e849e75d3dfb53f5849"}
Sep 16 19:43:47.030075 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.029577    2557 scope.go:117] "RemoveContainer" containerID="f8c717d5be338f41b2bc1d025fd141722a0da8bb02b795ad96655ede69edb294"
Sep 16 19:43:47.038648 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.038627    2557 scope.go:117] "RemoveContainer" containerID="ee46327ad99558c3f50bdbe3227beb5c82e136c191ed7946147500842467a15e"
Sep 16 19:43:47.046315 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.046176    2557 scope.go:117] "RemoveContainer" containerID="5c24f42cc048bc921e61888787f56b7486a0307d4ed6c3cf38f844f8f841ce27"
Sep 16 19:43:47.053589 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.053568    2557 scope.go:117] "RemoveContainer" containerID="c16042680722e3582b05025107d6c0fccd24cf12f33aac57090700fe3ced7dc5"
Sep 16 19:43:47.062451 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.062428    2557 scope.go:117] "RemoveContainer" containerID="f3b2dae5f1f9a744707aa1174c43799067703047d79496f82b3f20d84ec1f420"
Sep 16 19:43:47.063082 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.063060    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 19:43:47.070235 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.070212    2557 scope.go:117] "RemoveContainer" containerID="b1ac4e810c919eae85f68c5e4840180177825a332c386104898d51f5a2e1abde"
Sep 16 19:43:47.072976 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.072954    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 19:43:47.078500 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.078483    2557 scope.go:117] "RemoveContainer" containerID="be8f941313221ef84455ab9d15f78fe375dadf240ff3c8f767f60eeb002edf44"
Sep 16 19:43:47.121999 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.121962    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 19:43:47.122359 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.122342    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87ef17b7-b3ef-467c-a198-93fe1d18b0db" containerName="kube-rbac-proxy"
Sep 16 19:43:47.122431 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.122362    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="87ef17b7-b3ef-467c-a198-93fe1d18b0db" containerName="kube-rbac-proxy"
Sep 16 19:43:47.122431 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.122387    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87ef17b7-b3ef-467c-a198-93fe1d18b0db" containerName="kube-rbac-proxy-web"
Sep 16 19:43:47.122431 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.122396    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="87ef17b7-b3ef-467c-a198-93fe1d18b0db" containerName="kube-rbac-proxy-web"
Sep 16 19:43:47.122431 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.122413    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87ef17b7-b3ef-467c-a198-93fe1d18b0db" containerName="init-config-reloader"
Sep 16 19:43:47.122431 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.122421    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="87ef17b7-b3ef-467c-a198-93fe1d18b0db" containerName="init-config-reloader"
Sep 16 19:43:47.122431 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.122430    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87ef17b7-b3ef-467c-a198-93fe1d18b0db" containerName="config-reloader"
Sep 16 19:43:47.122719 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.122438    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="87ef17b7-b3ef-467c-a198-93fe1d18b0db" containerName="config-reloader"
Sep 16 19:43:47.122719 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.122452    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87ef17b7-b3ef-467c-a198-93fe1d18b0db" containerName="prometheus"
Sep 16 19:43:47.122719 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.122460    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="87ef17b7-b3ef-467c-a198-93fe1d18b0db" containerName="prometheus"
Sep 16 19:43:47.122719 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.122483    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87ef17b7-b3ef-467c-a198-93fe1d18b0db" containerName="thanos-sidecar"
Sep 16 19:43:47.122719 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.122493    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="87ef17b7-b3ef-467c-a198-93fe1d18b0db" containerName="thanos-sidecar"
Sep 16 19:43:47.122719 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.122503    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87ef17b7-b3ef-467c-a198-93fe1d18b0db" containerName="kube-rbac-proxy-thanos"
Sep 16 19:43:47.122719 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.122512    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="87ef17b7-b3ef-467c-a198-93fe1d18b0db" containerName="kube-rbac-proxy-thanos"
Sep 16 19:43:47.122719 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.122592    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="87ef17b7-b3ef-467c-a198-93fe1d18b0db" containerName="thanos-sidecar"
Sep 16 19:43:47.122719 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.122605    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="87ef17b7-b3ef-467c-a198-93fe1d18b0db" containerName="kube-rbac-proxy-thanos"
Sep 16 19:43:47.122719 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.122617    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="87ef17b7-b3ef-467c-a198-93fe1d18b0db" containerName="prometheus"
Sep 16 19:43:47.122719 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.122626    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="87ef17b7-b3ef-467c-a198-93fe1d18b0db" containerName="kube-rbac-proxy-web"
Sep 16 19:43:47.122719 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.122635    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="87ef17b7-b3ef-467c-a198-93fe1d18b0db" containerName="config-reloader"
Sep 16 19:43:47.122719 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.122646    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="87ef17b7-b3ef-467c-a198-93fe1d18b0db" containerName="kube-rbac-proxy"
Sep 16 19:43:47.125507 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.125484    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.136419 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.136393    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 19:43:47.136556 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.136420    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-2m7pc243auj7n\""
Sep 16 19:43:47.159929 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.159897    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 19:43:47.160734 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.160685    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 19:43:47.160734 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.160686    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 19:43:47.160954 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.160688    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 19:43:47.160954 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.160804    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 19:43:47.161039 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.161010    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 19:43:47.161086 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.161059    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 19:43:47.161137 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.161081    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 19:43:47.161137 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.161126    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 16 19:43:47.161223 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.161138    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-7rsf2\""
Sep 16 19:43:47.161440 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.161422    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 19:43:47.161520 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.161507    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 19:43:47.164742 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.164723    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 19:43:47.164991 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.164975    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 19:43:47.190054 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.190018    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f99c26bc-67a2-411b-8fff-e09d444fdd1c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.190165 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.190058    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f99c26bc-67a2-411b-8fff-e09d444fdd1c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.190165 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.190085    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f99c26bc-67a2-411b-8fff-e09d444fdd1c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.190238 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.190156    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f99c26bc-67a2-411b-8fff-e09d444fdd1c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.190238 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.190213    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f99c26bc-67a2-411b-8fff-e09d444fdd1c-web-config\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.190308 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.190238    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f99c26bc-67a2-411b-8fff-e09d444fdd1c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.190308 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.190300    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f99c26bc-67a2-411b-8fff-e09d444fdd1c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.190384 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.190334    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f99c26bc-67a2-411b-8fff-e09d444fdd1c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.190384 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.190355    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f99c26bc-67a2-411b-8fff-e09d444fdd1c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.190453 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.190381    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f99c26bc-67a2-411b-8fff-e09d444fdd1c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.190453 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.190411    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f99c26bc-67a2-411b-8fff-e09d444fdd1c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.190453 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.190436    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f99c26bc-67a2-411b-8fff-e09d444fdd1c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.190551 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.190474    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f99c26bc-67a2-411b-8fff-e09d444fdd1c-config\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.190551 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.190491    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f99c26bc-67a2-411b-8fff-e09d444fdd1c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.190551 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.190510    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f99c26bc-67a2-411b-8fff-e09d444fdd1c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.190649 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.190561    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f99c26bc-67a2-411b-8fff-e09d444fdd1c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.190649 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.190586    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f99c26bc-67a2-411b-8fff-e09d444fdd1c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.190649 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.190622    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsm8t\" (UniqueName: \"kubernetes.io/projected/f99c26bc-67a2-411b-8fff-e09d444fdd1c-kube-api-access-xsm8t\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.190649 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.190645    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f99c26bc-67a2-411b-8fff-e09d444fdd1c-config-out\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.190775 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.190679    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f99c26bc-67a2-411b-8fff-e09d444fdd1c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.195850 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.195821    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 19:43:47.206573 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.206534    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 19:43:47.291404 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.291362    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsm8t\" (UniqueName: \"kubernetes.io/projected/f99c26bc-67a2-411b-8fff-e09d444fdd1c-kube-api-access-xsm8t\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.291404 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.291404    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f99c26bc-67a2-411b-8fff-e09d444fdd1c-config-out\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.291670 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.291437    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f99c26bc-67a2-411b-8fff-e09d444fdd1c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.291670 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.291472    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f99c26bc-67a2-411b-8fff-e09d444fdd1c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.291670 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.291498    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f99c26bc-67a2-411b-8fff-e09d444fdd1c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.291670 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.291525    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f99c26bc-67a2-411b-8fff-e09d444fdd1c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.292548 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.292086    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f99c26bc-67a2-411b-8fff-e09d444fdd1c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.292548 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.292201    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f99c26bc-67a2-411b-8fff-e09d444fdd1c-web-config\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.292548 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.292317    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f99c26bc-67a2-411b-8fff-e09d444fdd1c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.292548 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.292237    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f99c26bc-67a2-411b-8fff-e09d444fdd1c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.292548 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.292423    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f99c26bc-67a2-411b-8fff-e09d444fdd1c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.292835 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.292633    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f99c26bc-67a2-411b-8fff-e09d444fdd1c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.292835 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.292666    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f99c26bc-67a2-411b-8fff-e09d444fdd1c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.292835 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.292821    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f99c26bc-67a2-411b-8fff-e09d444fdd1c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.292999 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.292875    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f99c26bc-67a2-411b-8fff-e09d444fdd1c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.292999 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.292907    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f99c26bc-67a2-411b-8fff-e09d444fdd1c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.293090 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.292948    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f99c26bc-67a2-411b-8fff-e09d444fdd1c-config\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.293090 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.293046    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f99c26bc-67a2-411b-8fff-e09d444fdd1c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.293192 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.293178    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f99c26bc-67a2-411b-8fff-e09d444fdd1c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.293356 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.293215    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f99c26bc-67a2-411b-8fff-e09d444fdd1c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.293409 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.293372    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f99c26bc-67a2-411b-8fff-e09d444fdd1c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.294780 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.294240    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f99c26bc-67a2-411b-8fff-e09d444fdd1c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.296212 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.294944    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f99c26bc-67a2-411b-8fff-e09d444fdd1c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.296212 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.295574    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f99c26bc-67a2-411b-8fff-e09d444fdd1c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.296212 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.295999    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f99c26bc-67a2-411b-8fff-e09d444fdd1c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.296212 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.296022    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f99c26bc-67a2-411b-8fff-e09d444fdd1c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.297229 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.296755    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f99c26bc-67a2-411b-8fff-e09d444fdd1c-web-config\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.299035 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.299011    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f99c26bc-67a2-411b-8fff-e09d444fdd1c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.299783 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.299754    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f99c26bc-67a2-411b-8fff-e09d444fdd1c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.300205 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.300179    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f99c26bc-67a2-411b-8fff-e09d444fdd1c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.300664 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.300642    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f99c26bc-67a2-411b-8fff-e09d444fdd1c-config-out\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.303327 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.302037    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f99c26bc-67a2-411b-8fff-e09d444fdd1c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.303327 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.302424    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f99c26bc-67a2-411b-8fff-e09d444fdd1c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.304207 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.304164    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f99c26bc-67a2-411b-8fff-e09d444fdd1c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.304207 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.304175    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f99c26bc-67a2-411b-8fff-e09d444fdd1c-config\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.304369 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.304217    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f99c26bc-67a2-411b-8fff-e09d444fdd1c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.304369 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.304285    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f99c26bc-67a2-411b-8fff-e09d444fdd1c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.304807 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.304787    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f99c26bc-67a2-411b-8fff-e09d444fdd1c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.305472 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.305450    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f99c26bc-67a2-411b-8fff-e09d444fdd1c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.306597 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.306578    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsm8t\" (UniqueName: \"kubernetes.io/projected/f99c26bc-67a2-411b-8fff-e09d444fdd1c-kube-api-access-xsm8t\") pod \"prometheus-k8s-0\" (UID: \"f99c26bc-67a2-411b-8fff-e09d444fdd1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.435613 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.435507    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:43:47.719124 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.719096    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 19:43:47.720244 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:43:47.720218    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf99c26bc_67a2_411b_8fff_e09d444fdd1c.slice/crio-fc948368ea8cbca796e8af9c8424488bac3b40792633495dbc36a2dc572229ca WatchSource:0}: Error finding container fc948368ea8cbca796e8af9c8424488bac3b40792633495dbc36a2dc572229ca: Status 404 returned error can't find the container with id fc948368ea8cbca796e8af9c8424488bac3b40792633495dbc36a2dc572229ca
Sep 16 19:43:47.936317 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:47.936283    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87ef17b7-b3ef-467c-a198-93fe1d18b0db" path="/var/lib/kubelet/pods/87ef17b7-b3ef-467c-a198-93fe1d18b0db/volumes"
Sep 16 19:43:48.035104 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:48.035072    2557 generic.go:356] "Generic (PLEG): container finished" podID="f99c26bc-67a2-411b-8fff-e09d444fdd1c" containerID="15c1a96eb16b84c9b3b7576ca9bb70c156e0b5e6afd2be4440c963ff8f9a79af" exitCode=0
Sep 16 19:43:48.035531 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:48.035163    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f99c26bc-67a2-411b-8fff-e09d444fdd1c","Type":"ContainerDied","Data":"15c1a96eb16b84c9b3b7576ca9bb70c156e0b5e6afd2be4440c963ff8f9a79af"}
Sep 16 19:43:48.035531 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:48.035240    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f99c26bc-67a2-411b-8fff-e09d444fdd1c","Type":"ContainerStarted","Data":"fc948368ea8cbca796e8af9c8424488bac3b40792633495dbc36a2dc572229ca"}
Sep 16 19:43:49.045068 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:49.045032    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f99c26bc-67a2-411b-8fff-e09d444fdd1c","Type":"ContainerStarted","Data":"d86ff92da10db929f22e2b6b3bf829d0fb6896e7dd275e25a494a0498c8b7c8d"}
Sep 16 19:43:49.045492 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:49.045100    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f99c26bc-67a2-411b-8fff-e09d444fdd1c","Type":"ContainerStarted","Data":"d4647c4c28c52f194d59572dc0a487917b2862af17052cae6b8f70665a87402c"}
Sep 16 19:43:49.045492 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:49.045125    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f99c26bc-67a2-411b-8fff-e09d444fdd1c","Type":"ContainerStarted","Data":"944ec06cee54e0cdbab55e9d13f082313d6eca5015c58178f796877b253dd89e"}
Sep 16 19:43:49.045492 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:49.045152    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f99c26bc-67a2-411b-8fff-e09d444fdd1c","Type":"ContainerStarted","Data":"25f0e257ea1c41f527c2a482c0737ae0c3deecc4b6ba9cb742093ab4ae86aeec"}
Sep 16 19:43:49.045492 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:49.045175    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f99c26bc-67a2-411b-8fff-e09d444fdd1c","Type":"ContainerStarted","Data":"47093d1b81bfbc207b804562d5d3e5cf8c13c345cb499a22b15799b2ffda4ac6"}
Sep 16 19:43:49.045492 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:49.045202    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f99c26bc-67a2-411b-8fff-e09d444fdd1c","Type":"ContainerStarted","Data":"aacd7e144660c57b85455eec2dc657223bbb33fa807163a532153dddf835a8ad"}
Sep 16 19:43:49.076821 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:49.076762    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.053540001 podStartE2EDuration="2.076734657s" totalImagesPullingTime="0s" totalInitContainerRuntime="23.194656ms" isStatefulPod=false podCreationTimestamp="2026-09-16 19:43:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:43:49.075067498 +0000 UTC m=+257.600455920" watchObservedRunningTime="2026-09-16 19:43:49.076734657 +0000 UTC m=+257.602123041"
Sep 16 19:43:52.436108 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:43:52.436064    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:44:31.836254 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:44:31.836228    2557 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 19:44:47.436509 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:44:47.436459    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:44:47.452577 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:44:47.452544    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:44:48.325329 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:44:48.325292    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:46:33.792652 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:46:33.792609    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-pz7cc"]
Sep 16 19:46:33.796884 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:46:33.796858    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-pz7cc"
Sep 16 19:46:33.800738 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:46:33.800709    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-k4qnv\""
Sep 16 19:46:33.800979 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:46:33.800957    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 16 19:46:33.801968 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:46:33.801951    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 16 19:46:33.805894 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:46:33.805868    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-pz7cc"]
Sep 16 19:46:33.857227 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:46:33.857179    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlblt\" (UniqueName: \"kubernetes.io/projected/738780fd-69d2-44aa-853c-12acaa3dd4a4-kube-api-access-xlblt\") pod \"cert-manager-webhook-6486ffd997-pz7cc\" (UID: \"738780fd-69d2-44aa-853c-12acaa3dd4a4\") " pod="cert-manager/cert-manager-webhook-6486ffd997-pz7cc"
Sep 16 19:46:33.857425 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:46:33.857248    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/738780fd-69d2-44aa-853c-12acaa3dd4a4-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-pz7cc\" (UID: \"738780fd-69d2-44aa-853c-12acaa3dd4a4\") " pod="cert-manager/cert-manager-webhook-6486ffd997-pz7cc"
Sep 16 19:46:33.957726 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:46:33.957687    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/738780fd-69d2-44aa-853c-12acaa3dd4a4-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-pz7cc\" (UID: \"738780fd-69d2-44aa-853c-12acaa3dd4a4\") " pod="cert-manager/cert-manager-webhook-6486ffd997-pz7cc"
Sep 16 19:46:33.957940 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:46:33.957774    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlblt\" (UniqueName: \"kubernetes.io/projected/738780fd-69d2-44aa-853c-12acaa3dd4a4-kube-api-access-xlblt\") pod \"cert-manager-webhook-6486ffd997-pz7cc\" (UID: \"738780fd-69d2-44aa-853c-12acaa3dd4a4\") " pod="cert-manager/cert-manager-webhook-6486ffd997-pz7cc"
Sep 16 19:46:33.967492 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:46:33.967457    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/738780fd-69d2-44aa-853c-12acaa3dd4a4-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-pz7cc\" (UID: \"738780fd-69d2-44aa-853c-12acaa3dd4a4\") " pod="cert-manager/cert-manager-webhook-6486ffd997-pz7cc"
Sep 16 19:46:33.967652 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:46:33.967631    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlblt\" (UniqueName: \"kubernetes.io/projected/738780fd-69d2-44aa-853c-12acaa3dd4a4-kube-api-access-xlblt\") pod \"cert-manager-webhook-6486ffd997-pz7cc\" (UID: \"738780fd-69d2-44aa-853c-12acaa3dd4a4\") " pod="cert-manager/cert-manager-webhook-6486ffd997-pz7cc"
Sep 16 19:46:34.114658 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:46:34.114554    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-pz7cc"
Sep 16 19:46:34.389802 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:46:34.389726    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-pz7cc"]
Sep 16 19:46:34.392250 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:46:34.392217    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod738780fd_69d2_44aa_853c_12acaa3dd4a4.slice/crio-b2c09a88290db97ea2b01bcba9c8b68c53f57d50e512d0eedb91cda8340bd6e6 WatchSource:0}: Error finding container b2c09a88290db97ea2b01bcba9c8b68c53f57d50e512d0eedb91cda8340bd6e6: Status 404 returned error can't find the container with id b2c09a88290db97ea2b01bcba9c8b68c53f57d50e512d0eedb91cda8340bd6e6
Sep 16 19:46:34.394254 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:46:34.394235    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:46:34.750458 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:46:34.750418    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-pz7cc" event={"ID":"738780fd-69d2-44aa-853c-12acaa3dd4a4","Type":"ContainerStarted","Data":"b2c09a88290db97ea2b01bcba9c8b68c53f57d50e512d0eedb91cda8340bd6e6"}
Sep 16 19:46:37.768094 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:46:37.768059    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-pz7cc" event={"ID":"738780fd-69d2-44aa-853c-12acaa3dd4a4","Type":"ContainerStarted","Data":"5dadc78c9f2a941a37c07da68aa43b3edbe82bb297882070570ffd23d8704035"}
Sep 16 19:46:37.768487 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:46:37.768278    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-pz7cc"
Sep 16 19:46:37.788691 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:46:37.788638    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-pz7cc" podStartSLOduration=1.874524866 podStartE2EDuration="4.788622655s" totalImagesPullingTime="2.914097789s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:46:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:46:37.786285712 +0000 UTC m=+426.311674098" watchObservedRunningTime="2026-09-16 19:46:37.788622655 +0000 UTC m=+426.314011041"
Sep 16 19:46:43.775792 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:46:43.775755    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-pz7cc"
Sep 16 19:46:46.104516 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:46:46.104472    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cgl9l"]
Sep 16 19:46:46.107111 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:46:46.107094    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cgl9l"
Sep 16 19:46:46.109811 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:46:46.109782    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 16 19:46:46.111097 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:46:46.111045    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-grngw\""
Sep 16 19:46:46.111097 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:46:46.111066    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 16 19:46:46.117362 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:46:46.117334    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cgl9l"]
Sep 16 19:46:46.156164 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:46:46.156116    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtvbj\" (UniqueName: \"kubernetes.io/projected/9af5e15a-b673-47ab-aa06-a9902c7bf091-kube-api-access-qtvbj\") pod \"openshift-lws-operator-fd8ccff4c-cgl9l\" (UID: \"9af5e15a-b673-47ab-aa06-a9902c7bf091\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cgl9l"
Sep 16 19:46:46.156164 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:46:46.156168    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9af5e15a-b673-47ab-aa06-a9902c7bf091-tmp\") pod \"openshift-lws-operator-fd8ccff4c-cgl9l\" (UID: \"9af5e15a-b673-47ab-aa06-a9902c7bf091\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cgl9l"
Sep 16 19:46:46.257549 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:46:46.257507    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtvbj\" (UniqueName: \"kubernetes.io/projected/9af5e15a-b673-47ab-aa06-a9902c7bf091-kube-api-access-qtvbj\") pod \"openshift-lws-operator-fd8ccff4c-cgl9l\" (UID: \"9af5e15a-b673-47ab-aa06-a9902c7bf091\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cgl9l"
Sep 16 19:46:46.257549 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:46:46.257554    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9af5e15a-b673-47ab-aa06-a9902c7bf091-tmp\") pod \"openshift-lws-operator-fd8ccff4c-cgl9l\" (UID: \"9af5e15a-b673-47ab-aa06-a9902c7bf091\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cgl9l"
Sep 16 19:46:46.258033 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:46:46.258013    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9af5e15a-b673-47ab-aa06-a9902c7bf091-tmp\") pod \"openshift-lws-operator-fd8ccff4c-cgl9l\" (UID: \"9af5e15a-b673-47ab-aa06-a9902c7bf091\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cgl9l"
Sep 16 19:46:46.267750 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:46:46.267717    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtvbj\" (UniqueName: \"kubernetes.io/projected/9af5e15a-b673-47ab-aa06-a9902c7bf091-kube-api-access-qtvbj\") pod \"openshift-lws-operator-fd8ccff4c-cgl9l\" (UID: \"9af5e15a-b673-47ab-aa06-a9902c7bf091\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cgl9l"
Sep 16 19:46:46.416901 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:46:46.416788    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cgl9l"
Sep 16 19:46:46.693012 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:46:46.692983    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cgl9l"]
Sep 16 19:46:46.696135 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:46:46.696108    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9af5e15a_b673_47ab_aa06_a9902c7bf091.slice/crio-f64232abd49e1720908403bda32668549ffca0bec89968d54dee683c2d967c26 WatchSource:0}: Error finding container f64232abd49e1720908403bda32668549ffca0bec89968d54dee683c2d967c26: Status 404 returned error can't find the container with id f64232abd49e1720908403bda32668549ffca0bec89968d54dee683c2d967c26
Sep 16 19:46:46.807718 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:46:46.807684    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cgl9l" event={"ID":"9af5e15a-b673-47ab-aa06-a9902c7bf091","Type":"ContainerStarted","Data":"f64232abd49e1720908403bda32668549ffca0bec89968d54dee683c2d967c26"}
Sep 16 19:46:49.823914 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:46:49.823873    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cgl9l" event={"ID":"9af5e15a-b673-47ab-aa06-a9902c7bf091","Type":"ContainerStarted","Data":"a0419c9016141b708361bc4c08dd924c9fe95c878f12e8d92aa782b9243476f6"}
Sep 16 19:46:49.842571 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:46:49.842511    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cgl9l" podStartSLOduration=1.6770448839999998 podStartE2EDuration="3.842495091s" totalImagesPullingTime="2.165450207s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:46:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:46:49.840788692 +0000 UTC m=+438.366177078" watchObservedRunningTime="2026-09-16 19:46:49.842495091 +0000 UTC m=+438.367883476"
Sep 16 19:47:09.419375 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:47:09.419285    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-b4645d7c8-qpgls"]
Sep 16 19:47:09.425922 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:47:09.425899    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-b4645d7c8-qpgls"
Sep 16 19:47:09.435331 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:47:09.435304    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 16 19:47:09.435331 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:47:09.435319    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 16 19:47:09.435528 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:47:09.435319    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-rzklv\""
Sep 16 19:47:09.435528 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:47:09.435324    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 16 19:47:09.478946 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:47:09.478902    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-b4645d7c8-qpgls"]
Sep 16 19:47:09.561123 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:47:09.561079    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d88a58d2-fcbc-4de1-8402-aa3c03d9dcbf-cert\") pod \"lws-controller-manager-b4645d7c8-qpgls\" (UID: \"d88a58d2-fcbc-4de1-8402-aa3c03d9dcbf\") " pod="openshift-lws-operator/lws-controller-manager-b4645d7c8-qpgls"
Sep 16 19:47:09.561280 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:47:09.561227    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mg87\" (UniqueName: \"kubernetes.io/projected/d88a58d2-fcbc-4de1-8402-aa3c03d9dcbf-kube-api-access-4mg87\") pod \"lws-controller-manager-b4645d7c8-qpgls\" (UID: \"d88a58d2-fcbc-4de1-8402-aa3c03d9dcbf\") " pod="openshift-lws-operator/lws-controller-manager-b4645d7c8-qpgls"
Sep 16 19:47:09.561322 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:47:09.561277    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d88a58d2-fcbc-4de1-8402-aa3c03d9dcbf-manager-config\") pod \"lws-controller-manager-b4645d7c8-qpgls\" (UID: \"d88a58d2-fcbc-4de1-8402-aa3c03d9dcbf\") " pod="openshift-lws-operator/lws-controller-manager-b4645d7c8-qpgls"
Sep 16 19:47:09.561322 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:47:09.561310    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d88a58d2-fcbc-4de1-8402-aa3c03d9dcbf-metrics-cert\") pod \"lws-controller-manager-b4645d7c8-qpgls\" (UID: \"d88a58d2-fcbc-4de1-8402-aa3c03d9dcbf\") " pod="openshift-lws-operator/lws-controller-manager-b4645d7c8-qpgls"
Sep 16 19:47:09.662211 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:47:09.662165    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mg87\" (UniqueName: \"kubernetes.io/projected/d88a58d2-fcbc-4de1-8402-aa3c03d9dcbf-kube-api-access-4mg87\") pod \"lws-controller-manager-b4645d7c8-qpgls\" (UID: \"d88a58d2-fcbc-4de1-8402-aa3c03d9dcbf\") " pod="openshift-lws-operator/lws-controller-manager-b4645d7c8-qpgls"
Sep 16 19:47:09.662211 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:47:09.662221    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d88a58d2-fcbc-4de1-8402-aa3c03d9dcbf-manager-config\") pod \"lws-controller-manager-b4645d7c8-qpgls\" (UID: \"d88a58d2-fcbc-4de1-8402-aa3c03d9dcbf\") " pod="openshift-lws-operator/lws-controller-manager-b4645d7c8-qpgls"
Sep 16 19:47:09.662430 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:47:09.662248    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d88a58d2-fcbc-4de1-8402-aa3c03d9dcbf-metrics-cert\") pod \"lws-controller-manager-b4645d7c8-qpgls\" (UID: \"d88a58d2-fcbc-4de1-8402-aa3c03d9dcbf\") " pod="openshift-lws-operator/lws-controller-manager-b4645d7c8-qpgls"
Sep 16 19:47:09.662430 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:47:09.662269    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d88a58d2-fcbc-4de1-8402-aa3c03d9dcbf-cert\") pod \"lws-controller-manager-b4645d7c8-qpgls\" (UID: \"d88a58d2-fcbc-4de1-8402-aa3c03d9dcbf\") " pod="openshift-lws-operator/lws-controller-manager-b4645d7c8-qpgls"
Sep 16 19:47:09.663124 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:47:09.663095    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d88a58d2-fcbc-4de1-8402-aa3c03d9dcbf-manager-config\") pod \"lws-controller-manager-b4645d7c8-qpgls\" (UID: \"d88a58d2-fcbc-4de1-8402-aa3c03d9dcbf\") " pod="openshift-lws-operator/lws-controller-manager-b4645d7c8-qpgls"
Sep 16 19:47:09.664820 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:47:09.664796    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d88a58d2-fcbc-4de1-8402-aa3c03d9dcbf-cert\") pod \"lws-controller-manager-b4645d7c8-qpgls\" (UID: \"d88a58d2-fcbc-4de1-8402-aa3c03d9dcbf\") " pod="openshift-lws-operator/lws-controller-manager-b4645d7c8-qpgls"
Sep 16 19:47:09.664983 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:47:09.664962    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d88a58d2-fcbc-4de1-8402-aa3c03d9dcbf-metrics-cert\") pod \"lws-controller-manager-b4645d7c8-qpgls\" (UID: \"d88a58d2-fcbc-4de1-8402-aa3c03d9dcbf\") " pod="openshift-lws-operator/lws-controller-manager-b4645d7c8-qpgls"
Sep 16 19:47:09.688733 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:47:09.688657    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mg87\" (UniqueName: \"kubernetes.io/projected/d88a58d2-fcbc-4de1-8402-aa3c03d9dcbf-kube-api-access-4mg87\") pod \"lws-controller-manager-b4645d7c8-qpgls\" (UID: \"d88a58d2-fcbc-4de1-8402-aa3c03d9dcbf\") " pod="openshift-lws-operator/lws-controller-manager-b4645d7c8-qpgls"
Sep 16 19:47:09.735478 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:47:09.735443    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-b4645d7c8-qpgls"
Sep 16 19:47:10.044204 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:47:10.044177    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-b4645d7c8-qpgls"]
Sep 16 19:47:10.046725 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:47:10.046693    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd88a58d2_fcbc_4de1_8402_aa3c03d9dcbf.slice/crio-21f4131a0d27271c7329621e52f2acf88683a4655c5c63cb5002abc650ee06fb WatchSource:0}: Error finding container 21f4131a0d27271c7329621e52f2acf88683a4655c5c63cb5002abc650ee06fb: Status 404 returned error can't find the container with id 21f4131a0d27271c7329621e52f2acf88683a4655c5c63cb5002abc650ee06fb
Sep 16 19:47:10.924200 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:47:10.924159    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-b4645d7c8-qpgls" event={"ID":"d88a58d2-fcbc-4de1-8402-aa3c03d9dcbf","Type":"ContainerStarted","Data":"21f4131a0d27271c7329621e52f2acf88683a4655c5c63cb5002abc650ee06fb"}
Sep 16 19:47:11.937340 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:47:11.937260    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-b4645d7c8-qpgls"
Sep 16 19:47:11.937340 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:47:11.937307    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-b4645d7c8-qpgls" event={"ID":"d88a58d2-fcbc-4de1-8402-aa3c03d9dcbf","Type":"ContainerStarted","Data":"36308bd82caf0c047f24f294e7a96c91bc8b5d407c9571678b0befdb7e32cd23"}
Sep 16 19:47:11.963548 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:47:11.963493    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-b4645d7c8-qpgls" podStartSLOduration=1.519061739 podStartE2EDuration="2.963476379s" totalImagesPullingTime="1.44441464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:47:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:47:11.963205372 +0000 UTC m=+460.488593759" watchObservedRunningTime="2026-09-16 19:47:11.963476379 +0000 UTC m=+460.488864764"
Sep 16 19:47:22.940608 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:47:22.940576    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-b4645d7c8-qpgls"
Sep 16 19:47:42.483636 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:47:42.483601    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-49qfs"]
Sep 16 19:47:42.485989 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:47:42.485972    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-49qfs"
Sep 16 19:47:42.489767 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:47:42.489744    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-lchgp\""
Sep 16 19:47:42.490459 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:47:42.490433    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 16 19:47:42.490578 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:47:42.490525    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 16 19:47:42.508565 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:47:42.508530    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-49qfs"]
Sep 16 19:47:42.565221 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:47:42.565183    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2647\" (UniqueName: \"kubernetes.io/projected/c65b34e0-35f4-4d8a-8ddb-558df53723de-kube-api-access-k2647\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-49qfs\" (UID: \"c65b34e0-35f4-4d8a-8ddb-558df53723de\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-49qfs"
Sep 16 19:47:42.666253 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:47:42.666191    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2647\" (UniqueName: \"kubernetes.io/projected/c65b34e0-35f4-4d8a-8ddb-558df53723de-kube-api-access-k2647\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-49qfs\" (UID: \"c65b34e0-35f4-4d8a-8ddb-558df53723de\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-49qfs"
Sep 16 19:47:42.680693 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:47:42.680658    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2647\" (UniqueName: \"kubernetes.io/projected/c65b34e0-35f4-4d8a-8ddb-558df53723de-kube-api-access-k2647\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-49qfs\" (UID: \"c65b34e0-35f4-4d8a-8ddb-558df53723de\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-49qfs"
Sep 16 19:47:42.797176 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:47:42.797077    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-49qfs"
Sep 16 19:47:43.080482 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:47:43.080449    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-49qfs"]
Sep 16 19:47:43.082586 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:47:43.082555    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc65b34e0_35f4_4d8a_8ddb_558df53723de.slice/crio-a0478b5d939f766785ecfa0562173e6bd46348226d667961843e6c8a985532bf WatchSource:0}: Error finding container a0478b5d939f766785ecfa0562173e6bd46348226d667961843e6c8a985532bf: Status 404 returned error can't find the container with id a0478b5d939f766785ecfa0562173e6bd46348226d667961843e6c8a985532bf
Sep 16 19:47:44.089426 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:47:44.089377    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-49qfs" event={"ID":"c65b34e0-35f4-4d8a-8ddb-558df53723de","Type":"ContainerStarted","Data":"a0478b5d939f766785ecfa0562173e6bd46348226d667961843e6c8a985532bf"}
Sep 16 19:47:46.101040 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:47:46.101002    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-49qfs" event={"ID":"c65b34e0-35f4-4d8a-8ddb-558df53723de","Type":"ContainerStarted","Data":"25dc649825bcee8a1da55f1eb426a30f1d0971364986e36cc06bd3726a229a01"}
Sep 16 19:47:46.101474 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:47:46.101063    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-49qfs"
Sep 16 19:47:57.108458 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:47:57.108424    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-49qfs"
Sep 16 19:47:57.133433 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:47:57.133372    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-49qfs" podStartSLOduration=12.776975997 podStartE2EDuration="15.133356656s" totalImagesPullingTime="2.356380659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:47:46.126961093 +0000 UTC m=+494.652349478" watchObservedRunningTime="2026-09-16 19:47:57.133356656 +0000 UTC m=+505.658745042"
Sep 16 19:48:25.628539 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:48:25.628452    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-zgpc9"]
Sep 16 19:48:25.636061 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:48:25.636033    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-zgpc9"
Sep 16 19:48:25.639335 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:48:25.639302    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-zbh5x\""
Sep 16 19:48:25.639902 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:48:25.639878    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-zgpc9"]
Sep 16 19:48:25.758171 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:48:25.758128    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz8ql\" (UniqueName: \"kubernetes.io/projected/4ff5e35b-6681-4d19-9930-ae62b533b438-kube-api-access-dz8ql\") pod \"authorino-7bcfd8f9cd-zgpc9\" (UID: \"4ff5e35b-6681-4d19-9930-ae62b533b438\") " pod="kuadrant-system/authorino-7bcfd8f9cd-zgpc9"
Sep 16 19:48:25.858983 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:48:25.858940    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz8ql\" (UniqueName: \"kubernetes.io/projected/4ff5e35b-6681-4d19-9930-ae62b533b438-kube-api-access-dz8ql\") pod \"authorino-7bcfd8f9cd-zgpc9\" (UID: \"4ff5e35b-6681-4d19-9930-ae62b533b438\") " pod="kuadrant-system/authorino-7bcfd8f9cd-zgpc9"
Sep 16 19:48:25.868183 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:48:25.868153    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz8ql\" (UniqueName: \"kubernetes.io/projected/4ff5e35b-6681-4d19-9930-ae62b533b438-kube-api-access-dz8ql\") pod \"authorino-7bcfd8f9cd-zgpc9\" (UID: \"4ff5e35b-6681-4d19-9930-ae62b533b438\") " pod="kuadrant-system/authorino-7bcfd8f9cd-zgpc9"
Sep 16 19:48:25.947010 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:48:25.946928    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-zgpc9"
Sep 16 19:48:26.217936 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:48:26.217812    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-zgpc9"]
Sep 16 19:48:26.224427 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:48:26.224398    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ff5e35b_6681_4d19_9930_ae62b533b438.slice/crio-79814c2b7aa06a6d5add52dbf7fb4389c5bb65020ef8198f111600c32ad6b894 WatchSource:0}: Error finding container 79814c2b7aa06a6d5add52dbf7fb4389c5bb65020ef8198f111600c32ad6b894: Status 404 returned error can't find the container with id 79814c2b7aa06a6d5add52dbf7fb4389c5bb65020ef8198f111600c32ad6b894
Sep 16 19:48:26.293832 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:48:26.293792    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-zgpc9" event={"ID":"4ff5e35b-6681-4d19-9930-ae62b533b438","Type":"ContainerStarted","Data":"79814c2b7aa06a6d5add52dbf7fb4389c5bb65020ef8198f111600c32ad6b894"}
Sep 16 19:48:29.754818 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:48:29.754779    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-zgpc9"]
Sep 16 19:48:30.316376 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:48:30.316339    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-zgpc9" event={"ID":"4ff5e35b-6681-4d19-9930-ae62b533b438","Type":"ContainerStarted","Data":"81ccaeaa4d88aa9ed2f3294dba323c7ce6ec04732d99ac17a6df887c15c85a1a"}
Sep 16 19:48:30.333127 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:48:30.333071    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-zgpc9" podStartSLOduration=2.107009111 podStartE2EDuration="5.333054108s" totalImagesPullingTime="3.226044997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:48:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:48:30.332023744 +0000 UTC m=+538.857412130" watchObservedRunningTime="2026-09-16 19:48:30.333054108 +0000 UTC m=+538.858442501"
Sep 16 19:48:31.320828 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:48:31.320767    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-zgpc9" podUID="4ff5e35b-6681-4d19-9930-ae62b533b438" containerName="authorino" containerID="cri-o://81ccaeaa4d88aa9ed2f3294dba323c7ce6ec04732d99ac17a6df887c15c85a1a" gracePeriod=30
Sep 16 19:48:31.570682 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:48:31.570656    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-zgpc9"
Sep 16 19:48:31.613107 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:48:31.613009    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dz8ql\" (UniqueName: \"kubernetes.io/projected/4ff5e35b-6681-4d19-9930-ae62b533b438-kube-api-access-dz8ql\") pod \"4ff5e35b-6681-4d19-9930-ae62b533b438\" (UID: \"4ff5e35b-6681-4d19-9930-ae62b533b438\") "
Sep 16 19:48:31.615415 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:48:31.615376    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ff5e35b-6681-4d19-9930-ae62b533b438-kube-api-access-dz8ql" (OuterVolumeSpecName: "kube-api-access-dz8ql") pod "4ff5e35b-6681-4d19-9930-ae62b533b438" (UID: "4ff5e35b-6681-4d19-9930-ae62b533b438"). InnerVolumeSpecName "kube-api-access-dz8ql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:48:31.713912 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:48:31.713869    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dz8ql\" (UniqueName: \"kubernetes.io/projected/4ff5e35b-6681-4d19-9930-ae62b533b438-kube-api-access-dz8ql\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 19:48:32.327329 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:48:32.326207    2557 generic.go:356] "Generic (PLEG): container finished" podID="4ff5e35b-6681-4d19-9930-ae62b533b438" containerID="81ccaeaa4d88aa9ed2f3294dba323c7ce6ec04732d99ac17a6df887c15c85a1a" exitCode=0
Sep 16 19:48:32.327329 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:48:32.326352    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-zgpc9" event={"ID":"4ff5e35b-6681-4d19-9930-ae62b533b438","Type":"ContainerDied","Data":"81ccaeaa4d88aa9ed2f3294dba323c7ce6ec04732d99ac17a6df887c15c85a1a"}
Sep 16 19:48:32.327329 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:48:32.326388    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-zgpc9" event={"ID":"4ff5e35b-6681-4d19-9930-ae62b533b438","Type":"ContainerDied","Data":"79814c2b7aa06a6d5add52dbf7fb4389c5bb65020ef8198f111600c32ad6b894"}
Sep 16 19:48:32.327329 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:48:32.326409    2557 scope.go:117] "RemoveContainer" containerID="81ccaeaa4d88aa9ed2f3294dba323c7ce6ec04732d99ac17a6df887c15c85a1a"
Sep 16 19:48:32.327329 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:48:32.326573    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-zgpc9"
Sep 16 19:48:32.337318 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:48:32.337295    2557 scope.go:117] "RemoveContainer" containerID="81ccaeaa4d88aa9ed2f3294dba323c7ce6ec04732d99ac17a6df887c15c85a1a"
Sep 16 19:48:32.337612 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:48:32.337588    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81ccaeaa4d88aa9ed2f3294dba323c7ce6ec04732d99ac17a6df887c15c85a1a\": container with ID starting with 81ccaeaa4d88aa9ed2f3294dba323c7ce6ec04732d99ac17a6df887c15c85a1a not found: ID does not exist" containerID="81ccaeaa4d88aa9ed2f3294dba323c7ce6ec04732d99ac17a6df887c15c85a1a"
Sep 16 19:48:32.337678 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:48:32.337626    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81ccaeaa4d88aa9ed2f3294dba323c7ce6ec04732d99ac17a6df887c15c85a1a"} err="failed to get container status \"81ccaeaa4d88aa9ed2f3294dba323c7ce6ec04732d99ac17a6df887c15c85a1a\": rpc error: code = NotFound desc = could not find container \"81ccaeaa4d88aa9ed2f3294dba323c7ce6ec04732d99ac17a6df887c15c85a1a\": container with ID starting with 81ccaeaa4d88aa9ed2f3294dba323c7ce6ec04732d99ac17a6df887c15c85a1a not found: ID does not exist"
Sep 16 19:48:32.346149 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:48:32.346111    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-zgpc9"]
Sep 16 19:48:32.349801 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:48:32.349777    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-zgpc9"]
Sep 16 19:48:33.937505 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:48:33.937456    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ff5e35b-6681-4d19-9930-ae62b533b438" path="/var/lib/kubelet/pods/4ff5e35b-6681-4d19-9930-ae62b533b438/volumes"
Sep 16 19:48:47.388318 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:48:47.388274    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d9898bd99-gwnhm"]
Sep 16 19:48:47.388778 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:48:47.388651    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ff5e35b-6681-4d19-9930-ae62b533b438" containerName="authorino"
Sep 16 19:48:47.388778 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:48:47.388663    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ff5e35b-6681-4d19-9930-ae62b533b438" containerName="authorino"
Sep 16 19:48:47.388778 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:48:47.388731    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ff5e35b-6681-4d19-9930-ae62b533b438" containerName="authorino"
Sep 16 19:48:47.393026 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:48:47.393001    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9898bd99-gwnhm"
Sep 16 19:48:47.398002 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:48:47.397970    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 16 19:48:47.398228 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:48:47.398185    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-zbh5x\""
Sep 16 19:48:47.400157 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:48:47.400131    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9898bd99-gwnhm"]
Sep 16 19:48:47.554483 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:48:47.554441    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brpwf\" (UniqueName: \"kubernetes.io/projected/de06a41d-3376-4092-b1f6-d5e588104656-kube-api-access-brpwf\") pod \"authorino-6d9898bd99-gwnhm\" (UID: \"de06a41d-3376-4092-b1f6-d5e588104656\") " pod="kuadrant-system/authorino-6d9898bd99-gwnhm"
Sep 16 19:48:47.554650 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:48:47.554550    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/de06a41d-3376-4092-b1f6-d5e588104656-tls-cert\") pod \"authorino-6d9898bd99-gwnhm\" (UID: \"de06a41d-3376-4092-b1f6-d5e588104656\") " pod="kuadrant-system/authorino-6d9898bd99-gwnhm"
Sep 16 19:48:47.656026 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:48:47.655932    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brpwf\" (UniqueName: \"kubernetes.io/projected/de06a41d-3376-4092-b1f6-d5e588104656-kube-api-access-brpwf\") pod \"authorino-6d9898bd99-gwnhm\" (UID: \"de06a41d-3376-4092-b1f6-d5e588104656\") " pod="kuadrant-system/authorino-6d9898bd99-gwnhm"
Sep 16 19:48:47.656026 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:48:47.656023    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/de06a41d-3376-4092-b1f6-d5e588104656-tls-cert\") pod \"authorino-6d9898bd99-gwnhm\" (UID: \"de06a41d-3376-4092-b1f6-d5e588104656\") " pod="kuadrant-system/authorino-6d9898bd99-gwnhm"
Sep 16 19:48:47.658601 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:48:47.658567    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/de06a41d-3376-4092-b1f6-d5e588104656-tls-cert\") pod \"authorino-6d9898bd99-gwnhm\" (UID: \"de06a41d-3376-4092-b1f6-d5e588104656\") " pod="kuadrant-system/authorino-6d9898bd99-gwnhm"
Sep 16 19:48:47.665982 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:48:47.665958    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brpwf\" (UniqueName: \"kubernetes.io/projected/de06a41d-3376-4092-b1f6-d5e588104656-kube-api-access-brpwf\") pod \"authorino-6d9898bd99-gwnhm\" (UID: \"de06a41d-3376-4092-b1f6-d5e588104656\") " pod="kuadrant-system/authorino-6d9898bd99-gwnhm"
Sep 16 19:48:47.703625 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:48:47.703595    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9898bd99-gwnhm"
Sep 16 19:48:47.972757 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:48:47.972723    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9898bd99-gwnhm"]
Sep 16 19:48:47.975791 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:48:47.975765    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde06a41d_3376_4092_b1f6_d5e588104656.slice/crio-aedad7c3b2b6bb34ca5d2ff5993b29d59b2bf89e2888395b23aff657d62c32bf WatchSource:0}: Error finding container aedad7c3b2b6bb34ca5d2ff5993b29d59b2bf89e2888395b23aff657d62c32bf: Status 404 returned error can't find the container with id aedad7c3b2b6bb34ca5d2ff5993b29d59b2bf89e2888395b23aff657d62c32bf
Sep 16 19:48:48.406632 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:48:48.406591    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9898bd99-gwnhm" event={"ID":"de06a41d-3376-4092-b1f6-d5e588104656","Type":"ContainerStarted","Data":"8575efae8dc41b3534b5f1d2c972e98e8e28dd8d857579f1c75ca8087595bcca"}
Sep 16 19:48:48.406632 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:48:48.406643    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9898bd99-gwnhm" event={"ID":"de06a41d-3376-4092-b1f6-d5e588104656","Type":"ContainerStarted","Data":"aedad7c3b2b6bb34ca5d2ff5993b29d59b2bf89e2888395b23aff657d62c32bf"}
Sep 16 19:48:48.423251 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:48:48.423186    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6d9898bd99-gwnhm" podStartSLOduration=1.114060586 podStartE2EDuration="1.423165846s" totalImagesPullingTime="309.10526ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:48:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:48:48.422313865 +0000 UTC m=+556.947702254" watchObservedRunningTime="2026-09-16 19:48:48.423165846 +0000 UTC m=+556.948554230"
Sep 16 19:49:24.634006 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:49:24.633968    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-8q7vx"]
Sep 16 19:49:24.637739 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:49:24.637719    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-8q7vx"
Sep 16 19:49:24.640443 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:49:24.640415    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 16 19:49:24.640564 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:49:24.640415    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 19:49:24.640564 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:49:24.640426    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 19:49:24.641580 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:49:24.641566    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-h8z9s\""
Sep 16 19:49:24.647719 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:49:24.647689    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-8q7vx"]
Sep 16 19:49:24.817334 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:49:24.817288    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4d120a48-88a6-49f3-82b2-af6cdd4a0642-data\") pod \"seaweedfs-64568bcd49-8q7vx\" (UID: \"4d120a48-88a6-49f3-82b2-af6cdd4a0642\") " pod="kserve/seaweedfs-64568bcd49-8q7vx"
Sep 16 19:49:24.817334 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:49:24.817334    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrjns\" (UniqueName: \"kubernetes.io/projected/4d120a48-88a6-49f3-82b2-af6cdd4a0642-kube-api-access-wrjns\") pod \"seaweedfs-64568bcd49-8q7vx\" (UID: \"4d120a48-88a6-49f3-82b2-af6cdd4a0642\") " pod="kserve/seaweedfs-64568bcd49-8q7vx"
Sep 16 19:49:24.918655 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:49:24.918571    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4d120a48-88a6-49f3-82b2-af6cdd4a0642-data\") pod \"seaweedfs-64568bcd49-8q7vx\" (UID: \"4d120a48-88a6-49f3-82b2-af6cdd4a0642\") " pod="kserve/seaweedfs-64568bcd49-8q7vx"
Sep 16 19:49:24.918655 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:49:24.918618    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrjns\" (UniqueName: \"kubernetes.io/projected/4d120a48-88a6-49f3-82b2-af6cdd4a0642-kube-api-access-wrjns\") pod \"seaweedfs-64568bcd49-8q7vx\" (UID: \"4d120a48-88a6-49f3-82b2-af6cdd4a0642\") " pod="kserve/seaweedfs-64568bcd49-8q7vx"
Sep 16 19:49:24.919070 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:49:24.919050    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4d120a48-88a6-49f3-82b2-af6cdd4a0642-data\") pod \"seaweedfs-64568bcd49-8q7vx\" (UID: \"4d120a48-88a6-49f3-82b2-af6cdd4a0642\") " pod="kserve/seaweedfs-64568bcd49-8q7vx"
Sep 16 19:49:24.928720 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:49:24.928682    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrjns\" (UniqueName: \"kubernetes.io/projected/4d120a48-88a6-49f3-82b2-af6cdd4a0642-kube-api-access-wrjns\") pod \"seaweedfs-64568bcd49-8q7vx\" (UID: \"4d120a48-88a6-49f3-82b2-af6cdd4a0642\") " pod="kserve/seaweedfs-64568bcd49-8q7vx"
Sep 16 19:49:24.948265 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:49:24.948232    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-8q7vx"
Sep 16 19:49:25.233344 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:49:25.233316    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-8q7vx"]
Sep 16 19:49:25.235377 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:49:25.235351    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d120a48_88a6_49f3_82b2_af6cdd4a0642.slice/crio-bfc74a9df89c731cbaa63166cfeb6b952deed4b2eb8e7d24943c58ecc024694b WatchSource:0}: Error finding container bfc74a9df89c731cbaa63166cfeb6b952deed4b2eb8e7d24943c58ecc024694b: Status 404 returned error can't find the container with id bfc74a9df89c731cbaa63166cfeb6b952deed4b2eb8e7d24943c58ecc024694b
Sep 16 19:49:25.591049 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:49:25.591017    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-8q7vx" event={"ID":"4d120a48-88a6-49f3-82b2-af6cdd4a0642","Type":"ContainerStarted","Data":"bfc74a9df89c731cbaa63166cfeb6b952deed4b2eb8e7d24943c58ecc024694b"}
Sep 16 19:49:28.609960 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:49:28.609917    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-8q7vx" event={"ID":"4d120a48-88a6-49f3-82b2-af6cdd4a0642","Type":"ContainerStarted","Data":"b5f17c07bd6c9e44b4b258d63ef7e5f7a5cd77b29504c51600944c2697239e20"}
Sep 16 19:49:28.610355 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:49:28.610010    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-8q7vx"
Sep 16 19:49:28.637716 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:49:28.637657    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-8q7vx" podStartSLOduration=1.781429695 podStartE2EDuration="4.637638118s" totalImagesPullingTime="2.856208423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:49:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:49:28.633994973 +0000 UTC m=+597.159383358" watchObservedRunningTime="2026-09-16 19:49:28.637638118 +0000 UTC m=+597.163026503"
Sep 16 19:49:34.617932 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:49:34.617897    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-8q7vx"
Sep 16 19:50:43.399336 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:50:43.399297    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-2hb7c"]
Sep 16 19:50:43.403169 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:50:43.403142    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-2hb7c"
Sep 16 19:50:43.406129 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:50:43.406106    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 16 19:50:43.406273 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:50:43.406145    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-4b474\""
Sep 16 19:50:43.414235 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:50:43.414208    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-2hb7c"]
Sep 16 19:50:43.418103 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:50:43.418078    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-ps5ff"]
Sep 16 19:50:43.421423 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:50:43.421406    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-ps5ff"
Sep 16 19:50:43.424239 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:50:43.424214    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 16 19:50:43.424360 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:50:43.424236    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-s7dn4\""
Sep 16 19:50:43.424360 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:50:43.424316    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 16 19:50:43.430752 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:50:43.430725    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-ps5ff"]
Sep 16 19:50:43.513497 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:50:43.513450    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tft8f\" (UniqueName: \"kubernetes.io/projected/d729dd68-1fe0-420f-9cf4-a049f927044b-kube-api-access-tft8f\") pod \"odh-model-controller-7b966d7984-ps5ff\" (UID: \"d729dd68-1fe0-420f-9cf4-a049f927044b\") " pod="kserve/odh-model-controller-7b966d7984-ps5ff"
Sep 16 19:50:43.513694 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:50:43.513567    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d729dd68-1fe0-420f-9cf4-a049f927044b-metrics-certs\") pod \"odh-model-controller-7b966d7984-ps5ff\" (UID: \"d729dd68-1fe0-420f-9cf4-a049f927044b\") " pod="kserve/odh-model-controller-7b966d7984-ps5ff"
Sep 16 19:50:43.513694 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:50:43.513623    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f606e249-5ba6-4f99-9c67-13efa9033d09-tls-certs\") pod \"model-serving-api-7b954ffc95-2hb7c\" (UID: \"f606e249-5ba6-4f99-9c67-13efa9033d09\") " pod="kserve/model-serving-api-7b954ffc95-2hb7c"
Sep 16 19:50:43.513694 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:50:43.513640    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsvdd\" (UniqueName: \"kubernetes.io/projected/f606e249-5ba6-4f99-9c67-13efa9033d09-kube-api-access-lsvdd\") pod \"model-serving-api-7b954ffc95-2hb7c\" (UID: \"f606e249-5ba6-4f99-9c67-13efa9033d09\") " pod="kserve/model-serving-api-7b954ffc95-2hb7c"
Sep 16 19:50:43.513694 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:50:43.513661    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d729dd68-1fe0-420f-9cf4-a049f927044b-cert\") pod \"odh-model-controller-7b966d7984-ps5ff\" (UID: \"d729dd68-1fe0-420f-9cf4-a049f927044b\") " pod="kserve/odh-model-controller-7b966d7984-ps5ff"
Sep 16 19:50:43.614432 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:50:43.614399    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d729dd68-1fe0-420f-9cf4-a049f927044b-metrics-certs\") pod \"odh-model-controller-7b966d7984-ps5ff\" (UID: \"d729dd68-1fe0-420f-9cf4-a049f927044b\") " pod="kserve/odh-model-controller-7b966d7984-ps5ff"
Sep 16 19:50:43.614631 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:50:43.614455    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f606e249-5ba6-4f99-9c67-13efa9033d09-tls-certs\") pod \"model-serving-api-7b954ffc95-2hb7c\" (UID: \"f606e249-5ba6-4f99-9c67-13efa9033d09\") " pod="kserve/model-serving-api-7b954ffc95-2hb7c"
Sep 16 19:50:43.614631 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:50:43.614474    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsvdd\" (UniqueName: \"kubernetes.io/projected/f606e249-5ba6-4f99-9c67-13efa9033d09-kube-api-access-lsvdd\") pod \"model-serving-api-7b954ffc95-2hb7c\" (UID: \"f606e249-5ba6-4f99-9c67-13efa9033d09\") " pod="kserve/model-serving-api-7b954ffc95-2hb7c"
Sep 16 19:50:43.614631 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:50:43.614503    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d729dd68-1fe0-420f-9cf4-a049f927044b-cert\") pod \"odh-model-controller-7b966d7984-ps5ff\" (UID: \"d729dd68-1fe0-420f-9cf4-a049f927044b\") " pod="kserve/odh-model-controller-7b966d7984-ps5ff"
Sep 16 19:50:43.614631 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:50:43.614554    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tft8f\" (UniqueName: \"kubernetes.io/projected/d729dd68-1fe0-420f-9cf4-a049f927044b-kube-api-access-tft8f\") pod \"odh-model-controller-7b966d7984-ps5ff\" (UID: \"d729dd68-1fe0-420f-9cf4-a049f927044b\") " pod="kserve/odh-model-controller-7b966d7984-ps5ff"
Sep 16 19:50:43.614631 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:50:43.614576    2557 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 16 19:50:43.614808 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:50:43.614656    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f606e249-5ba6-4f99-9c67-13efa9033d09-tls-certs podName:f606e249-5ba6-4f99-9c67-13efa9033d09 nodeName:}" failed. No retries permitted until 2026-09-16 19:50:44.11463552 +0000 UTC m=+672.640023897 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/f606e249-5ba6-4f99-9c67-13efa9033d09-tls-certs") pod "model-serving-api-7b954ffc95-2hb7c" (UID: "f606e249-5ba6-4f99-9c67-13efa9033d09") : secret "model-serving-api-tls" not found
Sep 16 19:50:43.617092 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:50:43.617053    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d729dd68-1fe0-420f-9cf4-a049f927044b-cert\") pod \"odh-model-controller-7b966d7984-ps5ff\" (UID: \"d729dd68-1fe0-420f-9cf4-a049f927044b\") " pod="kserve/odh-model-controller-7b966d7984-ps5ff"
Sep 16 19:50:43.617205 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:50:43.617093    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d729dd68-1fe0-420f-9cf4-a049f927044b-metrics-certs\") pod \"odh-model-controller-7b966d7984-ps5ff\" (UID: \"d729dd68-1fe0-420f-9cf4-a049f927044b\") " pod="kserve/odh-model-controller-7b966d7984-ps5ff"
Sep 16 19:50:43.628910 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:50:43.626227    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tft8f\" (UniqueName: \"kubernetes.io/projected/d729dd68-1fe0-420f-9cf4-a049f927044b-kube-api-access-tft8f\") pod \"odh-model-controller-7b966d7984-ps5ff\" (UID: \"d729dd68-1fe0-420f-9cf4-a049f927044b\") " pod="kserve/odh-model-controller-7b966d7984-ps5ff"
Sep 16 19:50:43.632724 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:50:43.632701    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsvdd\" (UniqueName: \"kubernetes.io/projected/f606e249-5ba6-4f99-9c67-13efa9033d09-kube-api-access-lsvdd\") pod \"model-serving-api-7b954ffc95-2hb7c\" (UID: \"f606e249-5ba6-4f99-9c67-13efa9033d09\") " pod="kserve/model-serving-api-7b954ffc95-2hb7c"
Sep 16 19:50:43.733099 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:50:43.733062    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-ps5ff"
Sep 16 19:50:44.006347 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:50:44.006319    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-ps5ff"]
Sep 16 19:50:44.008280 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:50:44.008252    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd729dd68_1fe0_420f_9cf4_a049f927044b.slice/crio-384b29482e4f5d5ec08fdc8308a890c463cd31a87fc85c53b5e26e61a4d2eb50 WatchSource:0}: Error finding container 384b29482e4f5d5ec08fdc8308a890c463cd31a87fc85c53b5e26e61a4d2eb50: Status 404 returned error can't find the container with id 384b29482e4f5d5ec08fdc8308a890c463cd31a87fc85c53b5e26e61a4d2eb50
Sep 16 19:50:44.119298 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:50:44.119261    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f606e249-5ba6-4f99-9c67-13efa9033d09-tls-certs\") pod \"model-serving-api-7b954ffc95-2hb7c\" (UID: \"f606e249-5ba6-4f99-9c67-13efa9033d09\") " pod="kserve/model-serving-api-7b954ffc95-2hb7c"
Sep 16 19:50:44.121644 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:50:44.121620    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f606e249-5ba6-4f99-9c67-13efa9033d09-tls-certs\") pod \"model-serving-api-7b954ffc95-2hb7c\" (UID: \"f606e249-5ba6-4f99-9c67-13efa9033d09\") " pod="kserve/model-serving-api-7b954ffc95-2hb7c"
Sep 16 19:50:44.315031 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:50:44.314943    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-2hb7c"
Sep 16 19:50:44.586316 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:50:44.586289    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-2hb7c"]
Sep 16 19:50:44.588069 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:50:44.588043    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf606e249_5ba6_4f99_9c67_13efa9033d09.slice/crio-1314bd847c5818d382512ed797f7c11581147cc5535321c1e64fa72694c2409a WatchSource:0}: Error finding container 1314bd847c5818d382512ed797f7c11581147cc5535321c1e64fa72694c2409a: Status 404 returned error can't find the container with id 1314bd847c5818d382512ed797f7c11581147cc5535321c1e64fa72694c2409a
Sep 16 19:50:44.986227 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:50:44.986185    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-2hb7c" event={"ID":"f606e249-5ba6-4f99-9c67-13efa9033d09","Type":"ContainerStarted","Data":"1314bd847c5818d382512ed797f7c11581147cc5535321c1e64fa72694c2409a"}
Sep 16 19:50:44.987901 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:50:44.987868    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-ps5ff" event={"ID":"d729dd68-1fe0-420f-9cf4-a049f927044b","Type":"ContainerStarted","Data":"384b29482e4f5d5ec08fdc8308a890c463cd31a87fc85c53b5e26e61a4d2eb50"}
Sep 16 19:50:49.014110 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:50:49.014063    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-2hb7c" event={"ID":"f606e249-5ba6-4f99-9c67-13efa9033d09","Type":"ContainerStarted","Data":"c406054a334c515d4abbe78e16bdb344dab97129afedad7fac3aca087a34e62d"}
Sep 16 19:50:49.014596 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:50:49.014260    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-2hb7c"
Sep 16 19:50:49.015751 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:50:49.015727    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-ps5ff" event={"ID":"d729dd68-1fe0-420f-9cf4-a049f927044b","Type":"ContainerStarted","Data":"8ecbac27e4f213a958a48bc96bc36f4504dbf0fbc050c4e2090805d8db9192a5"}
Sep 16 19:50:49.015834 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:50:49.015787    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-ps5ff"
Sep 16 19:50:49.043925 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:50:49.043854    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-2hb7c" podStartSLOduration=2.587439188 podStartE2EDuration="6.043820416s" totalImagesPullingTime="3.456381228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:50:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:50:49.042405225 +0000 UTC m=+677.567793612" watchObservedRunningTime="2026-09-16 19:50:49.043820416 +0000 UTC m=+677.569208802"
Sep 16 19:50:49.068012 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:50:49.067951    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-ps5ff" podStartSLOduration=2.071735431 podStartE2EDuration="6.067932872s" totalImagesPullingTime="3.996197441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:50:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:50:49.066680915 +0000 UTC m=+677.592069314" watchObservedRunningTime="2026-09-16 19:50:49.067932872 +0000 UTC m=+677.593321295"
Sep 16 19:51:00.023622 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:51:00.023589    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-ps5ff"
Sep 16 19:51:00.025390 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:51:00.025374    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-2hb7c"
Sep 16 19:52:39.565115 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:52:39.565082    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schekt25v"]
Sep 16 19:52:39.568582 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:52:39.568560    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schekt25v"
Sep 16 19:52:39.572829 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:52:39.572804    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-t8zdj\""
Sep 16 19:52:39.572988 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:52:39.572888    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 16 19:52:39.573287 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:52:39.573264    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-jrdbq\""
Sep 16 19:52:39.573409 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:52:39.573319    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 16 19:52:39.573409 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:52:39.573335    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 16 19:52:39.580119 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:52:39.580095    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schekt25v"]
Sep 16 19:52:39.582571 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:52:39.582543    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d97fdd7e-c21b-4052-aa40-f03a259700c8-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schekt25v\" (UID: \"d97fdd7e-c21b-4052-aa40-f03a259700c8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schekt25v"
Sep 16 19:52:39.582686 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:52:39.582576    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45r6f\" (UniqueName: \"kubernetes.io/projected/d97fdd7e-c21b-4052-aa40-f03a259700c8-kube-api-access-45r6f\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schekt25v\" (UID: \"d97fdd7e-c21b-4052-aa40-f03a259700c8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schekt25v"
Sep 16 19:52:39.683584 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:52:39.683547    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d97fdd7e-c21b-4052-aa40-f03a259700c8-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schekt25v\" (UID: \"d97fdd7e-c21b-4052-aa40-f03a259700c8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schekt25v"
Sep 16 19:52:39.683769 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:52:39.683596    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45r6f\" (UniqueName: \"kubernetes.io/projected/d97fdd7e-c21b-4052-aa40-f03a259700c8-kube-api-access-45r6f\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schekt25v\" (UID: \"d97fdd7e-c21b-4052-aa40-f03a259700c8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schekt25v"
Sep 16 19:52:39.686123 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:52:39.686098    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d97fdd7e-c21b-4052-aa40-f03a259700c8-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schekt25v\" (UID: \"d97fdd7e-c21b-4052-aa40-f03a259700c8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schekt25v"
Sep 16 19:52:39.691734 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:52:39.691707    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45r6f\" (UniqueName: \"kubernetes.io/projected/d97fdd7e-c21b-4052-aa40-f03a259700c8-kube-api-access-45r6f\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schekt25v\" (UID: \"d97fdd7e-c21b-4052-aa40-f03a259700c8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schekt25v"
Sep 16 19:52:39.881307 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:52:39.881224    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schekt25v"
Sep 16 19:52:40.158197 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:52:40.158103    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schekt25v"]
Sep 16 19:52:40.161629 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:52:40.161599    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd97fdd7e_c21b_4052_aa40_f03a259700c8.slice/crio-7f8df030267c7d5c651e9333f3e49b7c77c53f4461c3b6efb5f846b5234e3c3f WatchSource:0}: Error finding container 7f8df030267c7d5c651e9333f3e49b7c77c53f4461c3b6efb5f846b5234e3c3f: Status 404 returned error can't find the container with id 7f8df030267c7d5c651e9333f3e49b7c77c53f4461c3b6efb5f846b5234e3c3f
Sep 16 19:52:40.164356 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:52:40.164331    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:52:40.570886 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:52:40.570830    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schekt25v" event={"ID":"d97fdd7e-c21b-4052-aa40-f03a259700c8","Type":"ContainerStarted","Data":"7f8df030267c7d5c651e9333f3e49b7c77c53f4461c3b6efb5f846b5234e3c3f"}
Sep 16 19:52:43.589352 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:52:43.589319    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schekt25v" event={"ID":"d97fdd7e-c21b-4052-aa40-f03a259700c8","Type":"ContainerStarted","Data":"a08c8e60b6409fee84fe1d9bb4f8abf7c1ac8cb4621f7a1cfbe1a2a28b903a28"}
Sep 16 19:52:43.589812 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:52:43.589426    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schekt25v"
Sep 16 19:52:43.608150 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:52:43.608102    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schekt25v" podStartSLOduration=2.106152774 podStartE2EDuration="4.608086203s" totalImagesPullingTime="2.501933429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:52:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:52:43.606716747 +0000 UTC m=+792.132105150" watchObservedRunningTime="2026-09-16 19:52:43.608086203 +0000 UTC m=+792.133474589"
Sep 16 19:52:52.680366 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:52:52.680330    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-lfl65"]
Sep 16 19:52:52.684026 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:52:52.684008    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-lfl65"
Sep 16 19:52:52.688109 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:52:52.688083    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-7rb5g\""
Sep 16 19:52:52.688220 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:52:52.688122    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Sep 16 19:52:52.688220 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:52:52.688158    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Sep 16 19:52:52.688220 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:52:52.688191    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Sep 16 19:52:52.695822 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:52:52.695795    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-lfl65"]
Sep 16 19:52:52.798756 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:52:52.798719    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3210ab9b-489d-402f-8aec-0a2c3971301c-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-lfl65\" (UID: \"3210ab9b-489d-402f-8aec-0a2c3971301c\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-lfl65"
Sep 16 19:52:52.798948 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:52:52.798777    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3210ab9b-489d-402f-8aec-0a2c3971301c-home\") pod \"fc-smoke-test-kserve-7f4687897-lfl65\" (UID: \"3210ab9b-489d-402f-8aec-0a2c3971301c\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-lfl65"
Sep 16 19:52:52.798948 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:52:52.798802    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3210ab9b-489d-402f-8aec-0a2c3971301c-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-lfl65\" (UID: \"3210ab9b-489d-402f-8aec-0a2c3971301c\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-lfl65"
Sep 16 19:52:52.798948 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:52:52.798827    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3210ab9b-489d-402f-8aec-0a2c3971301c-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-lfl65\" (UID: \"3210ab9b-489d-402f-8aec-0a2c3971301c\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-lfl65"
Sep 16 19:52:52.798948 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:52:52.798910    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3210ab9b-489d-402f-8aec-0a2c3971301c-dshm\") pod \"fc-smoke-test-kserve-7f4687897-lfl65\" (UID: \"3210ab9b-489d-402f-8aec-0a2c3971301c\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-lfl65"
Sep 16 19:52:52.798948 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:52:52.798945    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvtxk\" (UniqueName: \"kubernetes.io/projected/3210ab9b-489d-402f-8aec-0a2c3971301c-kube-api-access-tvtxk\") pod \"fc-smoke-test-kserve-7f4687897-lfl65\" (UID: \"3210ab9b-489d-402f-8aec-0a2c3971301c\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-lfl65"
Sep 16 19:52:52.899783 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:52:52.899746    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3210ab9b-489d-402f-8aec-0a2c3971301c-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-lfl65\" (UID: \"3210ab9b-489d-402f-8aec-0a2c3971301c\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-lfl65"
Sep 16 19:52:52.899985 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:52:52.899812    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3210ab9b-489d-402f-8aec-0a2c3971301c-home\") pod \"fc-smoke-test-kserve-7f4687897-lfl65\" (UID: \"3210ab9b-489d-402f-8aec-0a2c3971301c\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-lfl65"
Sep 16 19:52:52.899985 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:52:52.899829    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3210ab9b-489d-402f-8aec-0a2c3971301c-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-lfl65\" (UID: \"3210ab9b-489d-402f-8aec-0a2c3971301c\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-lfl65"
Sep 16 19:52:52.899985 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:52:52.899866    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3210ab9b-489d-402f-8aec-0a2c3971301c-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-lfl65\" (UID: \"3210ab9b-489d-402f-8aec-0a2c3971301c\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-lfl65"
Sep 16 19:52:52.899985 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:52:52.899888    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3210ab9b-489d-402f-8aec-0a2c3971301c-dshm\") pod \"fc-smoke-test-kserve-7f4687897-lfl65\" (UID: \"3210ab9b-489d-402f-8aec-0a2c3971301c\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-lfl65"
Sep 16 19:52:52.899985 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:52:52.899929    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvtxk\" (UniqueName: \"kubernetes.io/projected/3210ab9b-489d-402f-8aec-0a2c3971301c-kube-api-access-tvtxk\") pod \"fc-smoke-test-kserve-7f4687897-lfl65\" (UID: \"3210ab9b-489d-402f-8aec-0a2c3971301c\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-lfl65"
Sep 16 19:52:52.900257 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:52:52.900226    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3210ab9b-489d-402f-8aec-0a2c3971301c-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-lfl65\" (UID: \"3210ab9b-489d-402f-8aec-0a2c3971301c\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-lfl65"
Sep 16 19:52:52.900257 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:52:52.900261    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3210ab9b-489d-402f-8aec-0a2c3971301c-home\") pod \"fc-smoke-test-kserve-7f4687897-lfl65\" (UID: \"3210ab9b-489d-402f-8aec-0a2c3971301c\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-lfl65"
Sep 16 19:52:52.900416 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:52:52.900312    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3210ab9b-489d-402f-8aec-0a2c3971301c-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-lfl65\" (UID: \"3210ab9b-489d-402f-8aec-0a2c3971301c\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-lfl65"
Sep 16 19:52:52.902166 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:52:52.902144    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3210ab9b-489d-402f-8aec-0a2c3971301c-dshm\") pod \"fc-smoke-test-kserve-7f4687897-lfl65\" (UID: \"3210ab9b-489d-402f-8aec-0a2c3971301c\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-lfl65"
Sep 16 19:52:52.902353 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:52:52.902336    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3210ab9b-489d-402f-8aec-0a2c3971301c-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-lfl65\" (UID: \"3210ab9b-489d-402f-8aec-0a2c3971301c\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-lfl65"
Sep 16 19:52:52.908525 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:52:52.908502    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvtxk\" (UniqueName: \"kubernetes.io/projected/3210ab9b-489d-402f-8aec-0a2c3971301c-kube-api-access-tvtxk\") pod \"fc-smoke-test-kserve-7f4687897-lfl65\" (UID: \"3210ab9b-489d-402f-8aec-0a2c3971301c\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-lfl65"
Sep 16 19:52:52.994653 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:52:52.994617    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-lfl65"
Sep 16 19:52:53.273816 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:52:53.273698    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-lfl65"]
Sep 16 19:52:53.276819 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:52:53.276792    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3210ab9b_489d_402f_8aec_0a2c3971301c.slice/crio-083bcfdcf84fe62d8260c3e62ff33039fa8f71b92c631b9dbc9a628d45a1f27d WatchSource:0}: Error finding container 083bcfdcf84fe62d8260c3e62ff33039fa8f71b92c631b9dbc9a628d45a1f27d: Status 404 returned error can't find the container with id 083bcfdcf84fe62d8260c3e62ff33039fa8f71b92c631b9dbc9a628d45a1f27d
Sep 16 19:52:53.641320 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:52:53.641248    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-lfl65" event={"ID":"3210ab9b-489d-402f-8aec-0a2c3971301c","Type":"ContainerStarted","Data":"083bcfdcf84fe62d8260c3e62ff33039fa8f71b92c631b9dbc9a628d45a1f27d"}
Sep 16 19:52:55.654643 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:52:55.654607    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-lfl65" event={"ID":"3210ab9b-489d-402f-8aec-0a2c3971301c","Type":"ContainerStarted","Data":"0f859f123a127d15d5137e9fded1c5028877b8dd07f4b8e3350ff21225eb6455"}
Sep 16 19:52:55.674669 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:52:55.674616    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-lfl65" podStartSLOduration=2.127600888 podStartE2EDuration="3.67460113s" totalImagesPullingTime="1.547000242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:52:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:52:55.672410278 +0000 UTC m=+804.197798663" watchObservedRunningTime="2026-09-16 19:52:55.67460113 +0000 UTC m=+804.199989516"
Sep 16 19:52:55.995696 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:52:55.995656    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-lfl65"
Sep 16 19:53:02.994811 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:53:02.994762    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-lfl65"
Sep 16 19:53:03.007430 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:53:03.007391    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-lfl65"
Sep 16 19:53:03.710552 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:53:03.710512    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-lfl65"
Sep 16 19:53:14.596208 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:53:14.596177    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schekt25v"
Sep 16 19:53:36.618245 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:53:36.618207    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-lfl65"]
Sep 16 19:53:36.627812 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:53:36.627763    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:53:36.627812 ip-10-0-139-81 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:53:36.627812 ip-10-0-139-81 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 16 19:53:36.627812 ip-10-0-139-81 kubenswrapper[2557]:  > containerID="0f859f123a127d15d5137e9fded1c5028877b8dd07f4b8e3350ff21225eb6455" cmd=["/bin/sleep","15"]
Sep 16 19:53:36.628159 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:53:36.627827    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-lfl65" message="" err=<
Sep 16 19:53:36.628159 ip-10-0-139-81 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:53:36.628159 ip-10-0-139-81 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 16 19:53:36.628159 ip-10-0-139-81 kubenswrapper[2557]:  >
Sep 16 19:53:36.628159 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:53:36.627853    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:53:36.628159 ip-10-0-139-81 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:53:36.628159 ip-10-0-139-81 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 16 19:53:36.628159 ip-10-0-139-81 kubenswrapper[2557]:  > pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-lfl65" podUID="3210ab9b-489d-402f-8aec-0a2c3971301c" containerName="main" containerID="cri-o://0f859f123a127d15d5137e9fded1c5028877b8dd07f4b8e3350ff21225eb6455"
Sep 16 19:53:36.628159 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:53:36.627901    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-lfl65" podUID="3210ab9b-489d-402f-8aec-0a2c3971301c" containerName="main" containerID="cri-o://0f859f123a127d15d5137e9fded1c5028877b8dd07f4b8e3350ff21225eb6455" gracePeriod=60
Sep 16 19:53:36.700571 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:53:36.700246    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-lfl65" podUID="3210ab9b-489d-402f-8aec-0a2c3971301c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Sep 16 19:53:36.871876 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:53:36.871783    2557 generic.go:356] "Generic (PLEG): container finished" podID="3210ab9b-489d-402f-8aec-0a2c3971301c" containerID="0f859f123a127d15d5137e9fded1c5028877b8dd07f4b8e3350ff21225eb6455" exitCode=0
Sep 16 19:53:36.872013 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:53:36.871874    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-lfl65" event={"ID":"3210ab9b-489d-402f-8aec-0a2c3971301c","Type":"ContainerDied","Data":"0f859f123a127d15d5137e9fded1c5028877b8dd07f4b8e3350ff21225eb6455"}
Sep 16 19:53:36.986820 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:53:36.986794    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-lfl65"
Sep 16 19:53:37.092187 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:53:37.092151    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3210ab9b-489d-402f-8aec-0a2c3971301c-home\") pod \"3210ab9b-489d-402f-8aec-0a2c3971301c\" (UID: \"3210ab9b-489d-402f-8aec-0a2c3971301c\") "
Sep 16 19:53:37.092187 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:53:37.092190    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3210ab9b-489d-402f-8aec-0a2c3971301c-tls-certs\") pod \"3210ab9b-489d-402f-8aec-0a2c3971301c\" (UID: \"3210ab9b-489d-402f-8aec-0a2c3971301c\") "
Sep 16 19:53:37.092411 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:53:37.092207    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3210ab9b-489d-402f-8aec-0a2c3971301c-dshm\") pod \"3210ab9b-489d-402f-8aec-0a2c3971301c\" (UID: \"3210ab9b-489d-402f-8aec-0a2c3971301c\") "
Sep 16 19:53:37.092411 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:53:37.092245    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3210ab9b-489d-402f-8aec-0a2c3971301c-tmp-dir\") pod \"3210ab9b-489d-402f-8aec-0a2c3971301c\" (UID: \"3210ab9b-489d-402f-8aec-0a2c3971301c\") "
Sep 16 19:53:37.092411 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:53:37.092269    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvtxk\" (UniqueName: \"kubernetes.io/projected/3210ab9b-489d-402f-8aec-0a2c3971301c-kube-api-access-tvtxk\") pod \"3210ab9b-489d-402f-8aec-0a2c3971301c\" (UID: \"3210ab9b-489d-402f-8aec-0a2c3971301c\") "
Sep 16 19:53:37.092411 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:53:37.092365    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3210ab9b-489d-402f-8aec-0a2c3971301c-model-cache\") pod \"3210ab9b-489d-402f-8aec-0a2c3971301c\" (UID: \"3210ab9b-489d-402f-8aec-0a2c3971301c\") "
Sep 16 19:53:37.092686 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:53:37.092503    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3210ab9b-489d-402f-8aec-0a2c3971301c-home" (OuterVolumeSpecName: "home") pod "3210ab9b-489d-402f-8aec-0a2c3971301c" (UID: "3210ab9b-489d-402f-8aec-0a2c3971301c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:53:37.092686 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:53:37.092551    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3210ab9b-489d-402f-8aec-0a2c3971301c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3210ab9b-489d-402f-8aec-0a2c3971301c" (UID: "3210ab9b-489d-402f-8aec-0a2c3971301c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:53:37.092778 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:53:37.092680    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3210ab9b-489d-402f-8aec-0a2c3971301c-model-cache" (OuterVolumeSpecName: "model-cache") pod "3210ab9b-489d-402f-8aec-0a2c3971301c" (UID: "3210ab9b-489d-402f-8aec-0a2c3971301c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:53:37.092778 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:53:37.092714    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3210ab9b-489d-402f-8aec-0a2c3971301c-home\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 19:53:37.092778 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:53:37.092742    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3210ab9b-489d-402f-8aec-0a2c3971301c-tmp-dir\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 19:53:37.094536 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:53:37.094514    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3210ab9b-489d-402f-8aec-0a2c3971301c-dshm" (OuterVolumeSpecName: "dshm") pod "3210ab9b-489d-402f-8aec-0a2c3971301c" (UID: "3210ab9b-489d-402f-8aec-0a2c3971301c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:53:37.094536 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:53:37.094520    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3210ab9b-489d-402f-8aec-0a2c3971301c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3210ab9b-489d-402f-8aec-0a2c3971301c" (UID: "3210ab9b-489d-402f-8aec-0a2c3971301c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:53:37.094697 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:53:37.094593    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3210ab9b-489d-402f-8aec-0a2c3971301c-kube-api-access-tvtxk" (OuterVolumeSpecName: "kube-api-access-tvtxk") pod "3210ab9b-489d-402f-8aec-0a2c3971301c" (UID: "3210ab9b-489d-402f-8aec-0a2c3971301c"). InnerVolumeSpecName "kube-api-access-tvtxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:53:37.193520 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:53:37.193432    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3210ab9b-489d-402f-8aec-0a2c3971301c-model-cache\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 19:53:37.193520 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:53:37.193464    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3210ab9b-489d-402f-8aec-0a2c3971301c-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 19:53:37.193520 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:53:37.193475    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3210ab9b-489d-402f-8aec-0a2c3971301c-dshm\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 19:53:37.193520 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:53:37.193484    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvtxk\" (UniqueName: \"kubernetes.io/projected/3210ab9b-489d-402f-8aec-0a2c3971301c-kube-api-access-tvtxk\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 19:53:37.877675 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:53:37.877636    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-lfl65" event={"ID":"3210ab9b-489d-402f-8aec-0a2c3971301c","Type":"ContainerDied","Data":"083bcfdcf84fe62d8260c3e62ff33039fa8f71b92c631b9dbc9a628d45a1f27d"}
Sep 16 19:53:37.877675 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:53:37.877675    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-lfl65"
Sep 16 19:53:37.878136 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:53:37.877711    2557 scope.go:117] "RemoveContainer" containerID="0f859f123a127d15d5137e9fded1c5028877b8dd07f4b8e3350ff21225eb6455"
Sep 16 19:53:37.901991 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:53:37.901940    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-lfl65"]
Sep 16 19:53:37.905950 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:53:37.905875    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-lfl65"]
Sep 16 19:53:37.936985 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:53:37.936950    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3210ab9b-489d-402f-8aec-0a2c3971301c" path="/var/lib/kubelet/pods/3210ab9b-489d-402f-8aec-0a2c3971301c/volumes"
Sep 16 19:54:55.489003 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:54:55.488967    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xjvqn"]
Sep 16 19:54:55.491499 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:54:55.489386    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3210ab9b-489d-402f-8aec-0a2c3971301c" containerName="main"
Sep 16 19:54:55.491499 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:54:55.489397    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3210ab9b-489d-402f-8aec-0a2c3971301c" containerName="main"
Sep 16 19:54:55.491499 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:54:55.489475    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="3210ab9b-489d-402f-8aec-0a2c3971301c" containerName="main"
Sep 16 19:54:55.492528 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:54:55.492511    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xjvqn"
Sep 16 19:54:55.496853 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:54:55.496818    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Sep 16 19:54:55.496965 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:54:55.496869    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Sep 16 19:54:55.496965 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:54:55.496878    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Sep 16 19:54:55.496965 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:54:55.496820    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-6dsn6\""
Sep 16 19:54:55.502347 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:54:55.502324    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xjvqn"]
Sep 16 19:54:55.593436 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:54:55.593399    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5303ef3-c0d2-480e-bef6-9b43fbffc570-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-xjvqn\" (UID: \"f5303ef3-c0d2-480e-bef6-9b43fbffc570\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xjvqn"
Sep 16 19:54:55.593436 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:54:55.593433    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5303ef3-c0d2-480e-bef6-9b43fbffc570-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-xjvqn\" (UID: \"f5303ef3-c0d2-480e-bef6-9b43fbffc570\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xjvqn"
Sep 16 19:54:55.593644 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:54:55.593479    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5303ef3-c0d2-480e-bef6-9b43fbffc570-home\") pod \"fc-concurrency-test-kserve-b5d44467c-xjvqn\" (UID: \"f5303ef3-c0d2-480e-bef6-9b43fbffc570\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xjvqn"
Sep 16 19:54:55.593644 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:54:55.593495    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5303ef3-c0d2-480e-bef6-9b43fbffc570-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-xjvqn\" (UID: \"f5303ef3-c0d2-480e-bef6-9b43fbffc570\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xjvqn"
Sep 16 19:54:55.593644 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:54:55.593547    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5303ef3-c0d2-480e-bef6-9b43fbffc570-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-xjvqn\" (UID: \"f5303ef3-c0d2-480e-bef6-9b43fbffc570\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xjvqn"
Sep 16 19:54:55.593644 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:54:55.593599    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtptb\" (UniqueName: \"kubernetes.io/projected/f5303ef3-c0d2-480e-bef6-9b43fbffc570-kube-api-access-mtptb\") pod \"fc-concurrency-test-kserve-b5d44467c-xjvqn\" (UID: \"f5303ef3-c0d2-480e-bef6-9b43fbffc570\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xjvqn"
Sep 16 19:54:55.694162 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:54:55.694128    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5303ef3-c0d2-480e-bef6-9b43fbffc570-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-xjvqn\" (UID: \"f5303ef3-c0d2-480e-bef6-9b43fbffc570\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xjvqn"
Sep 16 19:54:55.694162 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:54:55.694165    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtptb\" (UniqueName: \"kubernetes.io/projected/f5303ef3-c0d2-480e-bef6-9b43fbffc570-kube-api-access-mtptb\") pod \"fc-concurrency-test-kserve-b5d44467c-xjvqn\" (UID: \"f5303ef3-c0d2-480e-bef6-9b43fbffc570\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xjvqn"
Sep 16 19:54:55.694392 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:54:55.694216    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5303ef3-c0d2-480e-bef6-9b43fbffc570-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-xjvqn\" (UID: \"f5303ef3-c0d2-480e-bef6-9b43fbffc570\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xjvqn"
Sep 16 19:54:55.694392 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:54:55.694231    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5303ef3-c0d2-480e-bef6-9b43fbffc570-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-xjvqn\" (UID: \"f5303ef3-c0d2-480e-bef6-9b43fbffc570\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xjvqn"
Sep 16 19:54:55.694392 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:54:55.694269    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5303ef3-c0d2-480e-bef6-9b43fbffc570-home\") pod \"fc-concurrency-test-kserve-b5d44467c-xjvqn\" (UID: \"f5303ef3-c0d2-480e-bef6-9b43fbffc570\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xjvqn"
Sep 16 19:54:55.694392 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:54:55.694291    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5303ef3-c0d2-480e-bef6-9b43fbffc570-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-xjvqn\" (UID: \"f5303ef3-c0d2-480e-bef6-9b43fbffc570\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xjvqn"
Sep 16 19:54:55.694624 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:54:55.694601    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5303ef3-c0d2-480e-bef6-9b43fbffc570-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-xjvqn\" (UID: \"f5303ef3-c0d2-480e-bef6-9b43fbffc570\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xjvqn"
Sep 16 19:54:55.694670 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:54:55.694615    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5303ef3-c0d2-480e-bef6-9b43fbffc570-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-xjvqn\" (UID: \"f5303ef3-c0d2-480e-bef6-9b43fbffc570\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xjvqn"
Sep 16 19:54:55.694709 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:54:55.694686    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5303ef3-c0d2-480e-bef6-9b43fbffc570-home\") pod \"fc-concurrency-test-kserve-b5d44467c-xjvqn\" (UID: \"f5303ef3-c0d2-480e-bef6-9b43fbffc570\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xjvqn"
Sep 16 19:54:55.696520 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:54:55.696499    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5303ef3-c0d2-480e-bef6-9b43fbffc570-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-xjvqn\" (UID: \"f5303ef3-c0d2-480e-bef6-9b43fbffc570\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xjvqn"
Sep 16 19:54:55.696784 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:54:55.696767    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5303ef3-c0d2-480e-bef6-9b43fbffc570-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-xjvqn\" (UID: \"f5303ef3-c0d2-480e-bef6-9b43fbffc570\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xjvqn"
Sep 16 19:54:55.702984 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:54:55.702959    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtptb\" (UniqueName: \"kubernetes.io/projected/f5303ef3-c0d2-480e-bef6-9b43fbffc570-kube-api-access-mtptb\") pod \"fc-concurrency-test-kserve-b5d44467c-xjvqn\" (UID: \"f5303ef3-c0d2-480e-bef6-9b43fbffc570\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xjvqn"
Sep 16 19:54:55.804079 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:54:55.803997    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xjvqn"
Sep 16 19:54:56.076050 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:54:56.076022    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xjvqn"]
Sep 16 19:54:56.078767 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:54:56.078735    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5303ef3_c0d2_480e_bef6_9b43fbffc570.slice/crio-4b552ca851e7bb8aeee580e7d22d72e7a306a21a7404e84acf8f18cedc5af80c WatchSource:0}: Error finding container 4b552ca851e7bb8aeee580e7d22d72e7a306a21a7404e84acf8f18cedc5af80c: Status 404 returned error can't find the container with id 4b552ca851e7bb8aeee580e7d22d72e7a306a21a7404e84acf8f18cedc5af80c
Sep 16 19:54:56.264114 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:54:56.264074    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xjvqn" event={"ID":"f5303ef3-c0d2-480e-bef6-9b43fbffc570","Type":"ContainerStarted","Data":"f77fb9d94a5b9cbf51377e2d3e7ac0cd9f7db52398d4977789fd66c3b505ea81"}
Sep 16 19:54:56.264309 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:54:56.264129    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xjvqn" event={"ID":"f5303ef3-c0d2-480e-bef6-9b43fbffc570","Type":"ContainerStarted","Data":"4b552ca851e7bb8aeee580e7d22d72e7a306a21a7404e84acf8f18cedc5af80c"}
Sep 16 19:54:56.282922 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:54:56.282874    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xjvqn" podStartSLOduration=1.282854946 podStartE2EDuration="1.282854946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:54:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:54:56.280518312 +0000 UTC m=+924.805906696" watchObservedRunningTime="2026-09-16 19:54:56.282854946 +0000 UTC m=+924.808243325"
Sep 16 19:54:56.804403 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:54:56.804370    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xjvqn"
Sep 16 19:55:05.804161 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:55:05.804122    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xjvqn"
Sep 16 19:55:05.816974 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:55:05.816946    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xjvqn"
Sep 16 19:55:06.328446 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:55:06.328415    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xjvqn"
Sep 16 19:55:30.450871 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:55:30.450823    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schekt25v"]
Sep 16 19:55:34.596853 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:55:34.596811    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schekt25v" podUID="d97fdd7e-c21b-4052-aa40-f03a259700c8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:55:35.034578 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:55:35.034542    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xjvqn"]
Sep 16 19:55:35.045233 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:55:35.045181    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:55:35.045233 ip-10-0-139-81 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:55:35.045233 ip-10-0-139-81 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 16 19:55:35.045233 ip-10-0-139-81 kubenswrapper[2557]:  > containerID="f77fb9d94a5b9cbf51377e2d3e7ac0cd9f7db52398d4977789fd66c3b505ea81" cmd=["/bin/sleep","15"]
Sep 16 19:55:35.045603 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:55:35.045578    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xjvqn" message="" err=<
Sep 16 19:55:35.045603 ip-10-0-139-81 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:55:35.045603 ip-10-0-139-81 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 16 19:55:35.045603 ip-10-0-139-81 kubenswrapper[2557]:  >
Sep 16 19:55:35.045833 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:55:35.045812    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:55:35.045833 ip-10-0-139-81 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:55:35.045833 ip-10-0-139-81 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 16 19:55:35.045833 ip-10-0-139-81 kubenswrapper[2557]:  > pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xjvqn" podUID="f5303ef3-c0d2-480e-bef6-9b43fbffc570" containerName="main" containerID="cri-o://f77fb9d94a5b9cbf51377e2d3e7ac0cd9f7db52398d4977789fd66c3b505ea81"
Sep 16 19:55:35.046134 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:55:35.046112    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xjvqn" podUID="f5303ef3-c0d2-480e-bef6-9b43fbffc570" containerName="main" containerID="cri-o://f77fb9d94a5b9cbf51377e2d3e7ac0cd9f7db52398d4977789fd66c3b505ea81" gracePeriod=60
Sep 16 19:55:35.318122 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:55:35.318083    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xjvqn" podUID="f5303ef3-c0d2-480e-bef6-9b43fbffc570" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Sep 16 19:55:35.408299 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:55:35.408275    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xjvqn"
Sep 16 19:55:35.476218 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:55:35.476181    2557 generic.go:356] "Generic (PLEG): container finished" podID="f5303ef3-c0d2-480e-bef6-9b43fbffc570" containerID="f77fb9d94a5b9cbf51377e2d3e7ac0cd9f7db52398d4977789fd66c3b505ea81" exitCode=0
Sep 16 19:55:35.476218 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:55:35.476219    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xjvqn"
Sep 16 19:55:35.476442 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:55:35.476241    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xjvqn" event={"ID":"f5303ef3-c0d2-480e-bef6-9b43fbffc570","Type":"ContainerDied","Data":"f77fb9d94a5b9cbf51377e2d3e7ac0cd9f7db52398d4977789fd66c3b505ea81"}
Sep 16 19:55:35.476442 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:55:35.476307    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xjvqn" event={"ID":"f5303ef3-c0d2-480e-bef6-9b43fbffc570","Type":"ContainerDied","Data":"4b552ca851e7bb8aeee580e7d22d72e7a306a21a7404e84acf8f18cedc5af80c"}
Sep 16 19:55:35.476442 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:55:35.476324    2557 scope.go:117] "RemoveContainer" containerID="f77fb9d94a5b9cbf51377e2d3e7ac0cd9f7db52398d4977789fd66c3b505ea81"
Sep 16 19:55:35.485504 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:55:35.485484    2557 scope.go:117] "RemoveContainer" containerID="f77fb9d94a5b9cbf51377e2d3e7ac0cd9f7db52398d4977789fd66c3b505ea81"
Sep 16 19:55:35.485775 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:55:35.485759    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f77fb9d94a5b9cbf51377e2d3e7ac0cd9f7db52398d4977789fd66c3b505ea81\": container with ID starting with f77fb9d94a5b9cbf51377e2d3e7ac0cd9f7db52398d4977789fd66c3b505ea81 not found: ID does not exist" containerID="f77fb9d94a5b9cbf51377e2d3e7ac0cd9f7db52398d4977789fd66c3b505ea81"
Sep 16 19:55:35.485860 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:55:35.485781    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f77fb9d94a5b9cbf51377e2d3e7ac0cd9f7db52398d4977789fd66c3b505ea81"} err="failed to get container status \"f77fb9d94a5b9cbf51377e2d3e7ac0cd9f7db52398d4977789fd66c3b505ea81\": rpc error: code = NotFound desc = could not find container \"f77fb9d94a5b9cbf51377e2d3e7ac0cd9f7db52398d4977789fd66c3b505ea81\": container with ID starting with f77fb9d94a5b9cbf51377e2d3e7ac0cd9f7db52398d4977789fd66c3b505ea81 not found: ID does not exist"
Sep 16 19:55:35.566991 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:55:35.566952    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5303ef3-c0d2-480e-bef6-9b43fbffc570-dshm\") pod \"f5303ef3-c0d2-480e-bef6-9b43fbffc570\" (UID: \"f5303ef3-c0d2-480e-bef6-9b43fbffc570\") "
Sep 16 19:55:35.567249 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:55:35.567048    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5303ef3-c0d2-480e-bef6-9b43fbffc570-model-cache\") pod \"f5303ef3-c0d2-480e-bef6-9b43fbffc570\" (UID: \"f5303ef3-c0d2-480e-bef6-9b43fbffc570\") "
Sep 16 19:55:35.567249 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:55:35.567074    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mtptb\" (UniqueName: \"kubernetes.io/projected/f5303ef3-c0d2-480e-bef6-9b43fbffc570-kube-api-access-mtptb\") pod \"f5303ef3-c0d2-480e-bef6-9b43fbffc570\" (UID: \"f5303ef3-c0d2-480e-bef6-9b43fbffc570\") "
Sep 16 19:55:35.567249 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:55:35.567156    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5303ef3-c0d2-480e-bef6-9b43fbffc570-tls-certs\") pod \"f5303ef3-c0d2-480e-bef6-9b43fbffc570\" (UID: \"f5303ef3-c0d2-480e-bef6-9b43fbffc570\") "
Sep 16 19:55:35.567249 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:55:35.567194    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5303ef3-c0d2-480e-bef6-9b43fbffc570-tmp-dir\") pod \"f5303ef3-c0d2-480e-bef6-9b43fbffc570\" (UID: \"f5303ef3-c0d2-480e-bef6-9b43fbffc570\") "
Sep 16 19:55:35.567249 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:55:35.567221    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5303ef3-c0d2-480e-bef6-9b43fbffc570-home\") pod \"f5303ef3-c0d2-480e-bef6-9b43fbffc570\" (UID: \"f5303ef3-c0d2-480e-bef6-9b43fbffc570\") "
Sep 16 19:55:35.567511 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:55:35.567339    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5303ef3-c0d2-480e-bef6-9b43fbffc570-model-cache" (OuterVolumeSpecName: "model-cache") pod "f5303ef3-c0d2-480e-bef6-9b43fbffc570" (UID: "f5303ef3-c0d2-480e-bef6-9b43fbffc570"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:55:35.567511 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:55:35.567413    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5303ef3-c0d2-480e-bef6-9b43fbffc570-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f5303ef3-c0d2-480e-bef6-9b43fbffc570" (UID: "f5303ef3-c0d2-480e-bef6-9b43fbffc570"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:55:35.567511 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:55:35.567489    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5303ef3-c0d2-480e-bef6-9b43fbffc570-home" (OuterVolumeSpecName: "home") pod "f5303ef3-c0d2-480e-bef6-9b43fbffc570" (UID: "f5303ef3-c0d2-480e-bef6-9b43fbffc570"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:55:35.567663 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:55:35.567595    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5303ef3-c0d2-480e-bef6-9b43fbffc570-home\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 19:55:35.567663 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:55:35.567614    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5303ef3-c0d2-480e-bef6-9b43fbffc570-model-cache\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 19:55:35.567663 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:55:35.567630    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5303ef3-c0d2-480e-bef6-9b43fbffc570-tmp-dir\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 19:55:35.569374 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:55:35.569349    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5303ef3-c0d2-480e-bef6-9b43fbffc570-dshm" (OuterVolumeSpecName: "dshm") pod "f5303ef3-c0d2-480e-bef6-9b43fbffc570" (UID: "f5303ef3-c0d2-480e-bef6-9b43fbffc570"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:55:35.569374 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:55:35.569350    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5303ef3-c0d2-480e-bef6-9b43fbffc570-kube-api-access-mtptb" (OuterVolumeSpecName: "kube-api-access-mtptb") pod "f5303ef3-c0d2-480e-bef6-9b43fbffc570" (UID: "f5303ef3-c0d2-480e-bef6-9b43fbffc570"). InnerVolumeSpecName "kube-api-access-mtptb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:55:35.569532 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:55:35.569401    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5303ef3-c0d2-480e-bef6-9b43fbffc570-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f5303ef3-c0d2-480e-bef6-9b43fbffc570" (UID: "f5303ef3-c0d2-480e-bef6-9b43fbffc570"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:55:35.668700 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:55:35.668661    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5303ef3-c0d2-480e-bef6-9b43fbffc570-dshm\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 19:55:35.668700 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:55:35.668692    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtptb\" (UniqueName: \"kubernetes.io/projected/f5303ef3-c0d2-480e-bef6-9b43fbffc570-kube-api-access-mtptb\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 19:55:35.668700 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:55:35.668702    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5303ef3-c0d2-480e-bef6-9b43fbffc570-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 19:55:35.798667 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:55:35.798639    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xjvqn"]
Sep 16 19:55:35.801142 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:55:35.801114    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xjvqn"]
Sep 16 19:55:35.938427 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:55:35.938353    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5303ef3-c0d2-480e-bef6-9b43fbffc570" path="/var/lib/kubelet/pods/f5303ef3-c0d2-480e-bef6-9b43fbffc570/volumes"
Sep 16 19:55:44.596608 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:55:44.596569    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schekt25v" podUID="d97fdd7e-c21b-4052-aa40-f03a259700c8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:55:45.470752 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:55:45.470711    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schekt25v" podUID="d97fdd7e-c21b-4052-aa40-f03a259700c8" containerName="main" containerID="cri-o://a08c8e60b6409fee84fe1d9bb4f8abf7c1ac8cb4621f7a1cfbe1a2a28b903a28" gracePeriod=45
Sep 16 19:55:54.596808 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:55:54.596713    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schekt25v" podUID="d97fdd7e-c21b-4052-aa40-f03a259700c8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:55:54.597193 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:55:54.596859    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schekt25v"
Sep 16 19:56:04.596095 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:56:04.596054    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schekt25v" podUID="d97fdd7e-c21b-4052-aa40-f03a259700c8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:56:14.596805 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:56:14.596764    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schekt25v" podUID="d97fdd7e-c21b-4052-aa40-f03a259700c8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:56:15.670898 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:56:15.670864    2557 generic.go:356] "Generic (PLEG): container finished" podID="d97fdd7e-c21b-4052-aa40-f03a259700c8" containerID="a08c8e60b6409fee84fe1d9bb4f8abf7c1ac8cb4621f7a1cfbe1a2a28b903a28" exitCode=0
Sep 16 19:56:15.671331 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:56:15.670930    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schekt25v" event={"ID":"d97fdd7e-c21b-4052-aa40-f03a259700c8","Type":"ContainerDied","Data":"a08c8e60b6409fee84fe1d9bb4f8abf7c1ac8cb4621f7a1cfbe1a2a28b903a28"}
Sep 16 19:56:15.725548 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:56:15.725524    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schekt25v"
Sep 16 19:56:15.813951 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:56:15.813919    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d97fdd7e-c21b-4052-aa40-f03a259700c8-tls-certs\") pod \"d97fdd7e-c21b-4052-aa40-f03a259700c8\" (UID: \"d97fdd7e-c21b-4052-aa40-f03a259700c8\") "
Sep 16 19:56:15.814118 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:56:15.813970    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45r6f\" (UniqueName: \"kubernetes.io/projected/d97fdd7e-c21b-4052-aa40-f03a259700c8-kube-api-access-45r6f\") pod \"d97fdd7e-c21b-4052-aa40-f03a259700c8\" (UID: \"d97fdd7e-c21b-4052-aa40-f03a259700c8\") "
Sep 16 19:56:15.816188 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:56:15.816156    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d97fdd7e-c21b-4052-aa40-f03a259700c8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d97fdd7e-c21b-4052-aa40-f03a259700c8" (UID: "d97fdd7e-c21b-4052-aa40-f03a259700c8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:56:15.816307 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:56:15.816231    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d97fdd7e-c21b-4052-aa40-f03a259700c8-kube-api-access-45r6f" (OuterVolumeSpecName: "kube-api-access-45r6f") pod "d97fdd7e-c21b-4052-aa40-f03a259700c8" (UID: "d97fdd7e-c21b-4052-aa40-f03a259700c8"). InnerVolumeSpecName "kube-api-access-45r6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:56:15.915249 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:56:15.915211    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d97fdd7e-c21b-4052-aa40-f03a259700c8-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 19:56:15.915249 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:56:15.915243    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45r6f\" (UniqueName: \"kubernetes.io/projected/d97fdd7e-c21b-4052-aa40-f03a259700c8-kube-api-access-45r6f\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 19:56:16.676938 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:56:16.676906    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schekt25v"
Sep 16 19:56:16.677393 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:56:16.676954    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schekt25v" event={"ID":"d97fdd7e-c21b-4052-aa40-f03a259700c8","Type":"ContainerDied","Data":"7f8df030267c7d5c651e9333f3e49b7c77c53f4461c3b6efb5f846b5234e3c3f"}
Sep 16 19:56:16.677393 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:56:16.677016    2557 scope.go:117] "RemoveContainer" containerID="a08c8e60b6409fee84fe1d9bb4f8abf7c1ac8cb4621f7a1cfbe1a2a28b903a28"
Sep 16 19:56:16.694193 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:56:16.694162    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schekt25v"]
Sep 16 19:56:16.699168 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:56:16.699146    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schekt25v"]
Sep 16 19:56:17.937497 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:56:17.937462    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d97fdd7e-c21b-4052-aa40-f03a259700c8" path="/var/lib/kubelet/pods/d97fdd7e-c21b-4052-aa40-f03a259700c8/volumes"
Sep 16 19:56:33.659444 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:56:33.659404    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-57946dmz2l"]
Sep 16 19:56:33.659811 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:56:33.659799    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d97fdd7e-c21b-4052-aa40-f03a259700c8" containerName="main"
Sep 16 19:56:33.659811 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:56:33.659813    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d97fdd7e-c21b-4052-aa40-f03a259700c8" containerName="main"
Sep 16 19:56:33.659967 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:56:33.659821    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5303ef3-c0d2-480e-bef6-9b43fbffc570" containerName="main"
Sep 16 19:56:33.659967 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:56:33.659827    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5303ef3-c0d2-480e-bef6-9b43fbffc570" containerName="main"
Sep 16 19:56:33.659967 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:56:33.659929    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d97fdd7e-c21b-4052-aa40-f03a259700c8" containerName="main"
Sep 16 19:56:33.659967 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:56:33.659937    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5303ef3-c0d2-480e-bef6-9b43fbffc570" containerName="main"
Sep 16 19:56:33.664097 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:56:33.664070    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-57946dmz2l"
Sep 16 19:56:33.668216 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:56:33.668193    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Sep 16 19:56:33.668332 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:56:33.668213    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-epp-sa-dockercfg-88sps\""
Sep 16 19:56:33.668332 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:56:33.668236    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Sep 16 19:56:33.668332 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:56:33.668251    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-cc2hp\""
Sep 16 19:56:33.668332 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:56:33.668230    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Sep 16 19:56:33.672794 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:56:33.672772    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-57946dmz2l"]
Sep 16 19:56:33.775217 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:56:33.775188    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96c9m\" (UniqueName: \"kubernetes.io/projected/87ef5eb7-4624-4dd9-b096-2281bec3aa03-kube-api-access-96c9m\") pod \"scheduler-inline-config-test-kserve-router-scheduler-57946dmz2l\" (UID: \"87ef5eb7-4624-4dd9-b096-2281bec3aa03\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-57946dmz2l"
Sep 16 19:56:33.775391 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:56:33.775239    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87ef5eb7-4624-4dd9-b096-2281bec3aa03-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-57946dmz2l\" (UID: \"87ef5eb7-4624-4dd9-b096-2281bec3aa03\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-57946dmz2l"
Sep 16 19:56:33.876100 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:56:33.876061    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96c9m\" (UniqueName: \"kubernetes.io/projected/87ef5eb7-4624-4dd9-b096-2281bec3aa03-kube-api-access-96c9m\") pod \"scheduler-inline-config-test-kserve-router-scheduler-57946dmz2l\" (UID: \"87ef5eb7-4624-4dd9-b096-2281bec3aa03\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-57946dmz2l"
Sep 16 19:56:33.876263 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:56:33.876123    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87ef5eb7-4624-4dd9-b096-2281bec3aa03-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-57946dmz2l\" (UID: \"87ef5eb7-4624-4dd9-b096-2281bec3aa03\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-57946dmz2l"
Sep 16 19:56:33.878582 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:56:33.878560    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87ef5eb7-4624-4dd9-b096-2281bec3aa03-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-57946dmz2l\" (UID: \"87ef5eb7-4624-4dd9-b096-2281bec3aa03\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-57946dmz2l"
Sep 16 19:56:33.884355 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:56:33.884336    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96c9m\" (UniqueName: \"kubernetes.io/projected/87ef5eb7-4624-4dd9-b096-2281bec3aa03-kube-api-access-96c9m\") pod \"scheduler-inline-config-test-kserve-router-scheduler-57946dmz2l\" (UID: \"87ef5eb7-4624-4dd9-b096-2281bec3aa03\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-57946dmz2l"
Sep 16 19:56:33.975654 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:56:33.975561    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-57946dmz2l"
Sep 16 19:56:34.261532 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:56:34.261504    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-57946dmz2l"]
Sep 16 19:56:34.263884 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:56:34.263856    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87ef5eb7_4624_4dd9_b096_2281bec3aa03.slice/crio-bb67282c6b852b347b5b578899010981c41af5db49f1b2c92cb1f5ff010b9eae WatchSource:0}: Error finding container bb67282c6b852b347b5b578899010981c41af5db49f1b2c92cb1f5ff010b9eae: Status 404 returned error can't find the container with id bb67282c6b852b347b5b578899010981c41af5db49f1b2c92cb1f5ff010b9eae
Sep 16 19:56:34.776628 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:56:34.776587    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-57946dmz2l" event={"ID":"87ef5eb7-4624-4dd9-b096-2281bec3aa03","Type":"ContainerStarted","Data":"1dde76837f0fad97b56cfee671db02ed151caf98f1295b93fa48eeafa3298f91"}
Sep 16 19:56:34.777077 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:56:34.776654    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-57946dmz2l" event={"ID":"87ef5eb7-4624-4dd9-b096-2281bec3aa03","Type":"ContainerStarted","Data":"bb67282c6b852b347b5b578899010981c41af5db49f1b2c92cb1f5ff010b9eae"}
Sep 16 19:56:34.777077 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:56:34.776701    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-57946dmz2l"
Sep 16 19:56:34.794200 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:56:34.794153    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-57946dmz2l" podStartSLOduration=1.7941374909999999 podStartE2EDuration="1.794137491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:56:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:56:34.792089392 +0000 UTC m=+1023.317477802" watchObservedRunningTime="2026-09-16 19:56:34.794137491 +0000 UTC m=+1023.319525876"
Sep 16 19:57:05.783793 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:57:05.783762    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-57946dmz2l"
Sep 16 19:57:24.758669 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:57:24.758580    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-57946dmz2l"]
Sep 16 19:57:25.783455 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:57:25.783403    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-57946dmz2l" podUID="87ef5eb7-4624-4dd9-b096-2281bec3aa03" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:57:35.783759 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:57:35.783722    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-57946dmz2l" podUID="87ef5eb7-4624-4dd9-b096-2281bec3aa03" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:57:39.787244 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:57:39.787196    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-57946dmz2l" podUID="87ef5eb7-4624-4dd9-b096-2281bec3aa03" containerName="main" containerID="cri-o://1dde76837f0fad97b56cfee671db02ed151caf98f1295b93fa48eeafa3298f91" gracePeriod=45
Sep 16 19:57:45.783184 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:57:45.783141    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-57946dmz2l" podUID="87ef5eb7-4624-4dd9-b096-2281bec3aa03" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:57:45.783679 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:57:45.783262    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-57946dmz2l"
Sep 16 19:57:55.783740 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:57:55.783696    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-57946dmz2l" podUID="87ef5eb7-4624-4dd9-b096-2281bec3aa03" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:58:05.783675 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:05.783631    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-57946dmz2l" podUID="87ef5eb7-4624-4dd9-b096-2281bec3aa03" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:58:10.029547 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:10.029519    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-57946dmz2l"
Sep 16 19:58:10.140373 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:10.140279    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87ef5eb7-4624-4dd9-b096-2281bec3aa03-tls-certs\") pod \"87ef5eb7-4624-4dd9-b096-2281bec3aa03\" (UID: \"87ef5eb7-4624-4dd9-b096-2281bec3aa03\") "
Sep 16 19:58:10.140533 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:10.140371    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-96c9m\" (UniqueName: \"kubernetes.io/projected/87ef5eb7-4624-4dd9-b096-2281bec3aa03-kube-api-access-96c9m\") pod \"87ef5eb7-4624-4dd9-b096-2281bec3aa03\" (UID: \"87ef5eb7-4624-4dd9-b096-2281bec3aa03\") "
Sep 16 19:58:10.142312 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:10.142282    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87ef5eb7-4624-4dd9-b096-2281bec3aa03-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "87ef5eb7-4624-4dd9-b096-2281bec3aa03" (UID: "87ef5eb7-4624-4dd9-b096-2281bec3aa03"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:58:10.142477 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:10.142461    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87ef5eb7-4624-4dd9-b096-2281bec3aa03-kube-api-access-96c9m" (OuterVolumeSpecName: "kube-api-access-96c9m") pod "87ef5eb7-4624-4dd9-b096-2281bec3aa03" (UID: "87ef5eb7-4624-4dd9-b096-2281bec3aa03"). InnerVolumeSpecName "kube-api-access-96c9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:58:10.241727 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:10.241698    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87ef5eb7-4624-4dd9-b096-2281bec3aa03-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 19:58:10.241727 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:10.241724    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96c9m\" (UniqueName: \"kubernetes.io/projected/87ef5eb7-4624-4dd9-b096-2281bec3aa03-kube-api-access-96c9m\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 19:58:10.252908 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:10.252873    2557 generic.go:356] "Generic (PLEG): container finished" podID="87ef5eb7-4624-4dd9-b096-2281bec3aa03" containerID="1dde76837f0fad97b56cfee671db02ed151caf98f1295b93fa48eeafa3298f91" exitCode=0
Sep 16 19:58:10.253051 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:10.252923    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-57946dmz2l"
Sep 16 19:58:10.253051 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:10.252950    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-57946dmz2l" event={"ID":"87ef5eb7-4624-4dd9-b096-2281bec3aa03","Type":"ContainerDied","Data":"1dde76837f0fad97b56cfee671db02ed151caf98f1295b93fa48eeafa3298f91"}
Sep 16 19:58:10.253051 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:10.253025    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-57946dmz2l" event={"ID":"87ef5eb7-4624-4dd9-b096-2281bec3aa03","Type":"ContainerDied","Data":"bb67282c6b852b347b5b578899010981c41af5db49f1b2c92cb1f5ff010b9eae"}
Sep 16 19:58:10.253184 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:10.253054    2557 scope.go:117] "RemoveContainer" containerID="1dde76837f0fad97b56cfee671db02ed151caf98f1295b93fa48eeafa3298f91"
Sep 16 19:58:10.262401 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:10.262385    2557 scope.go:117] "RemoveContainer" containerID="1dde76837f0fad97b56cfee671db02ed151caf98f1295b93fa48eeafa3298f91"
Sep 16 19:58:10.262661 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:58:10.262645    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dde76837f0fad97b56cfee671db02ed151caf98f1295b93fa48eeafa3298f91\": container with ID starting with 1dde76837f0fad97b56cfee671db02ed151caf98f1295b93fa48eeafa3298f91 not found: ID does not exist" containerID="1dde76837f0fad97b56cfee671db02ed151caf98f1295b93fa48eeafa3298f91"
Sep 16 19:58:10.262703 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:10.262667    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dde76837f0fad97b56cfee671db02ed151caf98f1295b93fa48eeafa3298f91"} err="failed to get container status \"1dde76837f0fad97b56cfee671db02ed151caf98f1295b93fa48eeafa3298f91\": rpc error: code = NotFound desc = could not find container \"1dde76837f0fad97b56cfee671db02ed151caf98f1295b93fa48eeafa3298f91\": container with ID starting with 1dde76837f0fad97b56cfee671db02ed151caf98f1295b93fa48eeafa3298f91 not found: ID does not exist"
Sep 16 19:58:10.275014 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:10.274976    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-57946dmz2l"]
Sep 16 19:58:10.279249 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:10.279228    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-57946dmz2l"]
Sep 16 19:58:11.938814 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:11.938782    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87ef5eb7-4624-4dd9-b096-2281bec3aa03" path="/var/lib/kubelet/pods/87ef5eb7-4624-4dd9-b096-2281bec3aa03/volumes"
Sep 16 19:58:33.475546 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:33.475506    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8vpn6"]
Sep 16 19:58:33.476151 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:33.476131    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87ef5eb7-4624-4dd9-b096-2281bec3aa03" containerName="main"
Sep 16 19:58:33.476250 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:33.476154    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="87ef5eb7-4624-4dd9-b096-2281bec3aa03" containerName="main"
Sep 16 19:58:33.476250 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:33.476241    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="87ef5eb7-4624-4dd9-b096-2281bec3aa03" containerName="main"
Sep 16 19:58:33.479552 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:33.479530    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8vpn6"
Sep 16 19:58:33.482406 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:33.482386    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Sep 16 19:58:33.483699 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:33.483676    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-gjfbz\""
Sep 16 19:58:33.483817 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:33.483675    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Sep 16 19:58:33.483913 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:33.483676    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Sep 16 19:58:33.485888 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:33.485815    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8vpn6"]
Sep 16 19:58:33.538826 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:33.538784    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4ad92b9-4ec5-433a-ab3d-ad72af613d94-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8vpn6\" (UID: \"d4ad92b9-4ec5-433a-ab3d-ad72af613d94\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8vpn6"
Sep 16 19:58:33.539010 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:33.538906    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4ad92b9-4ec5-433a-ab3d-ad72af613d94-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8vpn6\" (UID: \"d4ad92b9-4ec5-433a-ab3d-ad72af613d94\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8vpn6"
Sep 16 19:58:33.539010 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:33.538928    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4ad92b9-4ec5-433a-ab3d-ad72af613d94-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8vpn6\" (UID: \"d4ad92b9-4ec5-433a-ab3d-ad72af613d94\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8vpn6"
Sep 16 19:58:33.539010 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:33.538947    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4ad92b9-4ec5-433a-ab3d-ad72af613d94-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8vpn6\" (UID: \"d4ad92b9-4ec5-433a-ab3d-ad72af613d94\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8vpn6"
Sep 16 19:58:33.539010 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:33.538968    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4ad92b9-4ec5-433a-ab3d-ad72af613d94-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8vpn6\" (UID: \"d4ad92b9-4ec5-433a-ab3d-ad72af613d94\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8vpn6"
Sep 16 19:58:33.539154 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:33.539023    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4qp4\" (UniqueName: \"kubernetes.io/projected/d4ad92b9-4ec5-433a-ab3d-ad72af613d94-kube-api-access-w4qp4\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8vpn6\" (UID: \"d4ad92b9-4ec5-433a-ab3d-ad72af613d94\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8vpn6"
Sep 16 19:58:33.639613 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:33.639573    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4ad92b9-4ec5-433a-ab3d-ad72af613d94-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8vpn6\" (UID: \"d4ad92b9-4ec5-433a-ab3d-ad72af613d94\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8vpn6"
Sep 16 19:58:33.639780 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:33.639639    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4ad92b9-4ec5-433a-ab3d-ad72af613d94-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8vpn6\" (UID: \"d4ad92b9-4ec5-433a-ab3d-ad72af613d94\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8vpn6"
Sep 16 19:58:33.639780 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:33.639656    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4ad92b9-4ec5-433a-ab3d-ad72af613d94-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8vpn6\" (UID: \"d4ad92b9-4ec5-433a-ab3d-ad72af613d94\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8vpn6"
Sep 16 19:58:33.639780 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:33.639679    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4ad92b9-4ec5-433a-ab3d-ad72af613d94-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8vpn6\" (UID: \"d4ad92b9-4ec5-433a-ab3d-ad72af613d94\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8vpn6"
Sep 16 19:58:33.639780 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:33.639702    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4ad92b9-4ec5-433a-ab3d-ad72af613d94-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8vpn6\" (UID: \"d4ad92b9-4ec5-433a-ab3d-ad72af613d94\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8vpn6"
Sep 16 19:58:33.639780 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:33.639721    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4qp4\" (UniqueName: \"kubernetes.io/projected/d4ad92b9-4ec5-433a-ab3d-ad72af613d94-kube-api-access-w4qp4\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8vpn6\" (UID: \"d4ad92b9-4ec5-433a-ab3d-ad72af613d94\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8vpn6"
Sep 16 19:58:33.640213 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:33.640120    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4ad92b9-4ec5-433a-ab3d-ad72af613d94-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8vpn6\" (UID: \"d4ad92b9-4ec5-433a-ab3d-ad72af613d94\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8vpn6"
Sep 16 19:58:33.640213 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:33.640128    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4ad92b9-4ec5-433a-ab3d-ad72af613d94-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8vpn6\" (UID: \"d4ad92b9-4ec5-433a-ab3d-ad72af613d94\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8vpn6"
Sep 16 19:58:33.640310 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:33.640215    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4ad92b9-4ec5-433a-ab3d-ad72af613d94-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8vpn6\" (UID: \"d4ad92b9-4ec5-433a-ab3d-ad72af613d94\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8vpn6"
Sep 16 19:58:33.641940 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:33.641920    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4ad92b9-4ec5-433a-ab3d-ad72af613d94-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8vpn6\" (UID: \"d4ad92b9-4ec5-433a-ab3d-ad72af613d94\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8vpn6"
Sep 16 19:58:33.642120 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:33.642103    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4ad92b9-4ec5-433a-ab3d-ad72af613d94-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8vpn6\" (UID: \"d4ad92b9-4ec5-433a-ab3d-ad72af613d94\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8vpn6"
Sep 16 19:58:33.650205 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:33.650183    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4qp4\" (UniqueName: \"kubernetes.io/projected/d4ad92b9-4ec5-433a-ab3d-ad72af613d94-kube-api-access-w4qp4\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8vpn6\" (UID: \"d4ad92b9-4ec5-433a-ab3d-ad72af613d94\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8vpn6"
Sep 16 19:58:33.733389 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:33.733362    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesdqph"]
Sep 16 19:58:33.737002 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:33.736976    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesdqph"
Sep 16 19:58:33.739949 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:33.739924    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa-dockercfg-pd6rp\""
Sep 16 19:58:33.747511 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:33.747486    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesdqph"]
Sep 16 19:58:33.790857 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:33.790808    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8vpn6"
Sep 16 19:58:33.841391 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:33.841275    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4srbl\" (UniqueName: \"kubernetes.io/projected/9ee25cba-a398-437e-9e6f-d13dc2bc4350-kube-api-access-4srbl\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesdqph\" (UID: \"9ee25cba-a398-437e-9e6f-d13dc2bc4350\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesdqph"
Sep 16 19:58:33.841391 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:33.841369    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ee25cba-a398-437e-9e6f-d13dc2bc4350-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesdqph\" (UID: \"9ee25cba-a398-437e-9e6f-d13dc2bc4350\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesdqph"
Sep 16 19:58:33.942643 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:33.942610    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ee25cba-a398-437e-9e6f-d13dc2bc4350-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesdqph\" (UID: \"9ee25cba-a398-437e-9e6f-d13dc2bc4350\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesdqph"
Sep 16 19:58:33.942817 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:33.942699    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4srbl\" (UniqueName: \"kubernetes.io/projected/9ee25cba-a398-437e-9e6f-d13dc2bc4350-kube-api-access-4srbl\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesdqph\" (UID: \"9ee25cba-a398-437e-9e6f-d13dc2bc4350\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesdqph"
Sep 16 19:58:33.945127 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:33.945105    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ee25cba-a398-437e-9e6f-d13dc2bc4350-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesdqph\" (UID: \"9ee25cba-a398-437e-9e6f-d13dc2bc4350\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesdqph"
Sep 16 19:58:33.951812 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:33.951787    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4srbl\" (UniqueName: \"kubernetes.io/projected/9ee25cba-a398-437e-9e6f-d13dc2bc4350-kube-api-access-4srbl\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesdqph\" (UID: \"9ee25cba-a398-437e-9e6f-d13dc2bc4350\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesdqph"
Sep 16 19:58:34.049110 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:34.049029    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesdqph"
Sep 16 19:58:34.065869 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:34.065821    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8vpn6"]
Sep 16 19:58:34.067791 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:58:34.067763    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4ad92b9_4ec5_433a_ab3d_ad72af613d94.slice/crio-7a8a429186cdb80968e2d62aca41157fb10eab434f3aba5930722369317b2342 WatchSource:0}: Error finding container 7a8a429186cdb80968e2d62aca41157fb10eab434f3aba5930722369317b2342: Status 404 returned error can't find the container with id 7a8a429186cdb80968e2d62aca41157fb10eab434f3aba5930722369317b2342
Sep 16 19:58:34.321781 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:34.321756    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesdqph"]
Sep 16 19:58:34.323891 ip-10-0-139-81 kubenswrapper[2557]: W0916 19:58:34.323864    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ee25cba_a398_437e_9e6f_d13dc2bc4350.slice/crio-7991cc1dd0726b9a20d9d302f84a84f496e09d6a75f39f217230d11734096693 WatchSource:0}: Error finding container 7991cc1dd0726b9a20d9d302f84a84f496e09d6a75f39f217230d11734096693: Status 404 returned error can't find the container with id 7991cc1dd0726b9a20d9d302f84a84f496e09d6a75f39f217230d11734096693
Sep 16 19:58:34.372213 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:34.372178    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesdqph" event={"ID":"9ee25cba-a398-437e-9e6f-d13dc2bc4350","Type":"ContainerStarted","Data":"7991cc1dd0726b9a20d9d302f84a84f496e09d6a75f39f217230d11734096693"}
Sep 16 19:58:34.373740 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:34.373713    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8vpn6" event={"ID":"d4ad92b9-4ec5-433a-ab3d-ad72af613d94","Type":"ContainerStarted","Data":"828a72f46ca8202eea6a180f7399114a5564f1d34fcd941f97eb05f3fb6b5c47"}
Sep 16 19:58:34.373865 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:34.373762    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8vpn6" event={"ID":"d4ad92b9-4ec5-433a-ab3d-ad72af613d94","Type":"ContainerStarted","Data":"7a8a429186cdb80968e2d62aca41157fb10eab434f3aba5930722369317b2342"}
Sep 16 19:58:34.394528 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:34.394450    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8vpn6" podStartSLOduration=1.394426441 podStartE2EDuration="1.394426441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:58:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:58:34.393329511 +0000 UTC m=+1142.918717897" watchObservedRunningTime="2026-09-16 19:58:34.394426441 +0000 UTC m=+1142.919814877"
Sep 16 19:58:34.791160 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:34.791123    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8vpn6"
Sep 16 19:58:35.380721 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:35.380681    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesdqph" event={"ID":"9ee25cba-a398-437e-9e6f-d13dc2bc4350","Type":"ContainerStarted","Data":"f4f0c365747adbf588340e6faff583bea6d8caf493a802e28507eeea7fad46a2"}
Sep 16 19:58:35.380966 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:35.380947    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesdqph"
Sep 16 19:58:35.403553 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:35.403498    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesdqph" podStartSLOduration=2.403483133 podStartE2EDuration="2.403483133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:58:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:58:35.401989919 +0000 UTC m=+1143.927378303" watchObservedRunningTime="2026-09-16 19:58:35.403483133 +0000 UTC m=+1143.928871518"
Sep 16 19:58:43.791129 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:43.791090    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8vpn6"
Sep 16 19:58:43.803825 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:43.803791    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8vpn6"
Sep 16 19:58:44.437020 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:58:44.436985    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8vpn6"
Sep 16 19:59:06.388058 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:59:06.387978    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesdqph"
Sep 16 19:59:08.574727 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:59:08.574688    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8vpn6"]
Sep 16 19:59:08.584377 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:59:08.584333    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:59:08.584377 ip-10-0-139-81 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:59:08.584377 ip-10-0-139-81 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 16 19:59:08.584377 ip-10-0-139-81 kubenswrapper[2557]:  > containerID="828a72f46ca8202eea6a180f7399114a5564f1d34fcd941f97eb05f3fb6b5c47" cmd=["/bin/sleep","15"]
Sep 16 19:59:08.584596 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:59:08.584392    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8vpn6" message="" err=<
Sep 16 19:59:08.584596 ip-10-0-139-81 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:59:08.584596 ip-10-0-139-81 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 16 19:59:08.584596 ip-10-0-139-81 kubenswrapper[2557]:  >
Sep 16 19:59:08.584596 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:59:08.584404    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:59:08.584596 ip-10-0-139-81 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:59:08.584596 ip-10-0-139-81 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 16 19:59:08.584596 ip-10-0-139-81 kubenswrapper[2557]:  > pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8vpn6" podUID="d4ad92b9-4ec5-433a-ab3d-ad72af613d94" containerName="main" containerID="cri-o://828a72f46ca8202eea6a180f7399114a5564f1d34fcd941f97eb05f3fb6b5c47"
Sep 16 19:59:08.584596 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:59:08.584452    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8vpn6" podUID="d4ad92b9-4ec5-433a-ab3d-ad72af613d94" containerName="main" containerID="cri-o://828a72f46ca8202eea6a180f7399114a5564f1d34fcd941f97eb05f3fb6b5c47" gracePeriod=60
Sep 16 19:59:08.600204 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:59:08.600172    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesdqph"]
Sep 16 19:59:09.338732 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:59:09.338707    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8vpn6"
Sep 16 19:59:09.358292 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:59:09.358266    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4ad92b9-4ec5-433a-ab3d-ad72af613d94-model-cache\") pod \"d4ad92b9-4ec5-433a-ab3d-ad72af613d94\" (UID: \"d4ad92b9-4ec5-433a-ab3d-ad72af613d94\") "
Sep 16 19:59:09.358419 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:59:09.358295    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w4qp4\" (UniqueName: \"kubernetes.io/projected/d4ad92b9-4ec5-433a-ab3d-ad72af613d94-kube-api-access-w4qp4\") pod \"d4ad92b9-4ec5-433a-ab3d-ad72af613d94\" (UID: \"d4ad92b9-4ec5-433a-ab3d-ad72af613d94\") "
Sep 16 19:59:09.358419 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:59:09.358324    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4ad92b9-4ec5-433a-ab3d-ad72af613d94-dshm\") pod \"d4ad92b9-4ec5-433a-ab3d-ad72af613d94\" (UID: \"d4ad92b9-4ec5-433a-ab3d-ad72af613d94\") "
Sep 16 19:59:09.358419 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:59:09.358343    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4ad92b9-4ec5-433a-ab3d-ad72af613d94-tls-certs\") pod \"d4ad92b9-4ec5-433a-ab3d-ad72af613d94\" (UID: \"d4ad92b9-4ec5-433a-ab3d-ad72af613d94\") "
Sep 16 19:59:09.358419 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:59:09.358389    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4ad92b9-4ec5-433a-ab3d-ad72af613d94-home\") pod \"d4ad92b9-4ec5-433a-ab3d-ad72af613d94\" (UID: \"d4ad92b9-4ec5-433a-ab3d-ad72af613d94\") "
Sep 16 19:59:09.358628 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:59:09.358431    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4ad92b9-4ec5-433a-ab3d-ad72af613d94-tmp-dir\") pod \"d4ad92b9-4ec5-433a-ab3d-ad72af613d94\" (UID: \"d4ad92b9-4ec5-433a-ab3d-ad72af613d94\") "
Sep 16 19:59:09.358628 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:59:09.358531    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4ad92b9-4ec5-433a-ab3d-ad72af613d94-model-cache" (OuterVolumeSpecName: "model-cache") pod "d4ad92b9-4ec5-433a-ab3d-ad72af613d94" (UID: "d4ad92b9-4ec5-433a-ab3d-ad72af613d94"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:59:09.358735 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:59:09.358670    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4ad92b9-4ec5-433a-ab3d-ad72af613d94-home" (OuterVolumeSpecName: "home") pod "d4ad92b9-4ec5-433a-ab3d-ad72af613d94" (UID: "d4ad92b9-4ec5-433a-ab3d-ad72af613d94"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:59:09.358782 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:59:09.358718    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4ad92b9-4ec5-433a-ab3d-ad72af613d94-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d4ad92b9-4ec5-433a-ab3d-ad72af613d94" (UID: "d4ad92b9-4ec5-433a-ab3d-ad72af613d94"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:59:09.358782 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:59:09.358750    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4ad92b9-4ec5-433a-ab3d-ad72af613d94-model-cache\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 19:59:09.360479 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:59:09.360452    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4ad92b9-4ec5-433a-ab3d-ad72af613d94-dshm" (OuterVolumeSpecName: "dshm") pod "d4ad92b9-4ec5-433a-ab3d-ad72af613d94" (UID: "d4ad92b9-4ec5-433a-ab3d-ad72af613d94"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:59:09.360826 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:59:09.360804    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4ad92b9-4ec5-433a-ab3d-ad72af613d94-kube-api-access-w4qp4" (OuterVolumeSpecName: "kube-api-access-w4qp4") pod "d4ad92b9-4ec5-433a-ab3d-ad72af613d94" (UID: "d4ad92b9-4ec5-433a-ab3d-ad72af613d94"). InnerVolumeSpecName "kube-api-access-w4qp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:59:09.360826 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:59:09.360817    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4ad92b9-4ec5-433a-ab3d-ad72af613d94-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d4ad92b9-4ec5-433a-ab3d-ad72af613d94" (UID: "d4ad92b9-4ec5-433a-ab3d-ad72af613d94"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:59:09.459444 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:59:09.459368    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4qp4\" (UniqueName: \"kubernetes.io/projected/d4ad92b9-4ec5-433a-ab3d-ad72af613d94-kube-api-access-w4qp4\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 19:59:09.459444 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:59:09.459396    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4ad92b9-4ec5-433a-ab3d-ad72af613d94-dshm\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 19:59:09.459444 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:59:09.459406    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4ad92b9-4ec5-433a-ab3d-ad72af613d94-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 19:59:09.459444 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:59:09.459415    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4ad92b9-4ec5-433a-ab3d-ad72af613d94-home\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 19:59:09.459444 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:59:09.459422    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4ad92b9-4ec5-433a-ab3d-ad72af613d94-tmp-dir\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 19:59:09.549091 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:59:09.549062    2557 generic.go:356] "Generic (PLEG): container finished" podID="d4ad92b9-4ec5-433a-ab3d-ad72af613d94" containerID="828a72f46ca8202eea6a180f7399114a5564f1d34fcd941f97eb05f3fb6b5c47" exitCode=0
Sep 16 19:59:09.549216 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:59:09.549112    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8vpn6"
Sep 16 19:59:09.549216 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:59:09.549114    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8vpn6" event={"ID":"d4ad92b9-4ec5-433a-ab3d-ad72af613d94","Type":"ContainerDied","Data":"828a72f46ca8202eea6a180f7399114a5564f1d34fcd941f97eb05f3fb6b5c47"}
Sep 16 19:59:09.549284 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:59:09.549242    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8vpn6" event={"ID":"d4ad92b9-4ec5-433a-ab3d-ad72af613d94","Type":"ContainerDied","Data":"7a8a429186cdb80968e2d62aca41157fb10eab434f3aba5930722369317b2342"}
Sep 16 19:59:09.549284 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:59:09.549275    2557 scope.go:117] "RemoveContainer" containerID="828a72f46ca8202eea6a180f7399114a5564f1d34fcd941f97eb05f3fb6b5c47"
Sep 16 19:59:09.559118 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:59:09.559100    2557 scope.go:117] "RemoveContainer" containerID="828a72f46ca8202eea6a180f7399114a5564f1d34fcd941f97eb05f3fb6b5c47"
Sep 16 19:59:09.559387 ip-10-0-139-81 kubenswrapper[2557]: E0916 19:59:09.559362    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"828a72f46ca8202eea6a180f7399114a5564f1d34fcd941f97eb05f3fb6b5c47\": container with ID starting with 828a72f46ca8202eea6a180f7399114a5564f1d34fcd941f97eb05f3fb6b5c47 not found: ID does not exist" containerID="828a72f46ca8202eea6a180f7399114a5564f1d34fcd941f97eb05f3fb6b5c47"
Sep 16 19:59:09.559468 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:59:09.559396    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"828a72f46ca8202eea6a180f7399114a5564f1d34fcd941f97eb05f3fb6b5c47"} err="failed to get container status \"828a72f46ca8202eea6a180f7399114a5564f1d34fcd941f97eb05f3fb6b5c47\": rpc error: code = NotFound desc = could not find container \"828a72f46ca8202eea6a180f7399114a5564f1d34fcd941f97eb05f3fb6b5c47\": container with ID starting with 828a72f46ca8202eea6a180f7399114a5564f1d34fcd941f97eb05f3fb6b5c47 not found: ID does not exist"
Sep 16 19:59:09.578528 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:59:09.578500    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8vpn6"]
Sep 16 19:59:09.585162 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:59:09.585141    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8vpn6"]
Sep 16 19:59:09.937914 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:59:09.937878    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4ad92b9-4ec5-433a-ab3d-ad72af613d94" path="/var/lib/kubelet/pods/d4ad92b9-4ec5-433a-ab3d-ad72af613d94/volumes"
Sep 16 19:59:16.388057 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:59:16.388012    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesdqph" podUID="9ee25cba-a398-437e-9e6f-d13dc2bc4350" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:59:23.619150 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:59:23.619083    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesdqph" podUID="9ee25cba-a398-437e-9e6f-d13dc2bc4350" containerName="main" containerID="cri-o://f4f0c365747adbf588340e6faff583bea6d8caf493a802e28507eeea7fad46a2" gracePeriod=45
Sep 16 19:59:26.388091 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:59:26.388050    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesdqph" podUID="9ee25cba-a398-437e-9e6f-d13dc2bc4350" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:59:36.387677 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:59:36.387638    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesdqph" podUID="9ee25cba-a398-437e-9e6f-d13dc2bc4350" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:59:36.388105 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:59:36.387738    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesdqph"
Sep 16 19:59:46.388193 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:59:46.388154    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesdqph" podUID="9ee25cba-a398-437e-9e6f-d13dc2bc4350" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:59:53.775199 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:59:53.775166    2557 generic.go:356] "Generic (PLEG): container finished" podID="9ee25cba-a398-437e-9e6f-d13dc2bc4350" containerID="f4f0c365747adbf588340e6faff583bea6d8caf493a802e28507eeea7fad46a2" exitCode=0
Sep 16 19:59:53.775583 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:59:53.775236    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesdqph" event={"ID":"9ee25cba-a398-437e-9e6f-d13dc2bc4350","Type":"ContainerDied","Data":"f4f0c365747adbf588340e6faff583bea6d8caf493a802e28507eeea7fad46a2"}
Sep 16 19:59:53.857386 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:59:53.857353    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesdqph"
Sep 16 19:59:53.955507 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:59:53.955421    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ee25cba-a398-437e-9e6f-d13dc2bc4350-tls-certs\") pod \"9ee25cba-a398-437e-9e6f-d13dc2bc4350\" (UID: \"9ee25cba-a398-437e-9e6f-d13dc2bc4350\") "
Sep 16 19:59:53.955507 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:59:53.955494    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4srbl\" (UniqueName: \"kubernetes.io/projected/9ee25cba-a398-437e-9e6f-d13dc2bc4350-kube-api-access-4srbl\") pod \"9ee25cba-a398-437e-9e6f-d13dc2bc4350\" (UID: \"9ee25cba-a398-437e-9e6f-d13dc2bc4350\") "
Sep 16 19:59:53.957499 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:59:53.957463    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ee25cba-a398-437e-9e6f-d13dc2bc4350-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9ee25cba-a398-437e-9e6f-d13dc2bc4350" (UID: "9ee25cba-a398-437e-9e6f-d13dc2bc4350"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:59:53.957618 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:59:53.957550    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ee25cba-a398-437e-9e6f-d13dc2bc4350-kube-api-access-4srbl" (OuterVolumeSpecName: "kube-api-access-4srbl") pod "9ee25cba-a398-437e-9e6f-d13dc2bc4350" (UID: "9ee25cba-a398-437e-9e6f-d13dc2bc4350"). InnerVolumeSpecName "kube-api-access-4srbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:59:54.056231 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:59:54.056191    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ee25cba-a398-437e-9e6f-d13dc2bc4350-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 19:59:54.056231 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:59:54.056223    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4srbl\" (UniqueName: \"kubernetes.io/projected/9ee25cba-a398-437e-9e6f-d13dc2bc4350-kube-api-access-4srbl\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 19:59:54.781973 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:59:54.781940    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesdqph"
Sep 16 19:59:54.782429 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:59:54.781984    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesdqph" event={"ID":"9ee25cba-a398-437e-9e6f-d13dc2bc4350","Type":"ContainerDied","Data":"7991cc1dd0726b9a20d9d302f84a84f496e09d6a75f39f217230d11734096693"}
Sep 16 19:59:54.782429 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:59:54.782055    2557 scope.go:117] "RemoveContainer" containerID="f4f0c365747adbf588340e6faff583bea6d8caf493a802e28507eeea7fad46a2"
Sep 16 19:59:54.816057 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:59:54.816022    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesdqph"]
Sep 16 19:59:54.826851 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:59:54.826807    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesdqph"]
Sep 16 19:59:55.937225 ip-10-0-139-81 kubenswrapper[2557]: I0916 19:59:55.937186    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ee25cba-a398-437e-9e6f-d13dc2bc4350" path="/var/lib/kubelet/pods/9ee25cba-a398-437e-9e6f-d13dc2bc4350/volumes"
Sep 16 20:00:09.337674 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:00:09.337639    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-55684h9frl"]
Sep 16 20:00:09.338159 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:00:09.338142    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4ad92b9-4ec5-433a-ab3d-ad72af613d94" containerName="main"
Sep 16 20:00:09.338159 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:00:09.338160    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4ad92b9-4ec5-433a-ab3d-ad72af613d94" containerName="main"
Sep 16 20:00:09.338237 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:00:09.338174    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ee25cba-a398-437e-9e6f-d13dc2bc4350" containerName="main"
Sep 16 20:00:09.338237 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:00:09.338182    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ee25cba-a398-437e-9e6f-d13dc2bc4350" containerName="main"
Sep 16 20:00:09.338304 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:00:09.338244    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ee25cba-a398-437e-9e6f-d13dc2bc4350" containerName="main"
Sep 16 20:00:09.338304 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:00:09.338254    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4ad92b9-4ec5-433a-ab3d-ad72af613d94" containerName="main"
Sep 16 20:00:09.341217 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:00:09.341199    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-55684h9frl"
Sep 16 20:00:09.353209 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:00:09.353184    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Sep 16 20:00:09.353335 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:00:09.353224    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Sep 16 20:00:09.353400 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:00:09.353356    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Sep 16 20:00:09.353457 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:00:09.353443    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-bwx2t\""
Sep 16 20:00:09.353565 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:00:09.353547    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-cjbz8\""
Sep 16 20:00:09.368295 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:00:09.368267    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-55684h9frl"]
Sep 16 20:00:09.387939 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:00:09.387909    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h95s\" (UniqueName: \"kubernetes.io/projected/9c9fe15b-bfe9-44e6-b00f-238ecb3d3631-kube-api-access-6h95s\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-55684h9frl\" (UID: \"9c9fe15b-bfe9-44e6-b00f-238ecb3d3631\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-55684h9frl"
Sep 16 20:00:09.388079 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:00:09.387947    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c9fe15b-bfe9-44e6-b00f-238ecb3d3631-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-55684h9frl\" (UID: \"9c9fe15b-bfe9-44e6-b00f-238ecb3d3631\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-55684h9frl"
Sep 16 20:00:09.488632 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:00:09.488592    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h95s\" (UniqueName: \"kubernetes.io/projected/9c9fe15b-bfe9-44e6-b00f-238ecb3d3631-kube-api-access-6h95s\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-55684h9frl\" (UID: \"9c9fe15b-bfe9-44e6-b00f-238ecb3d3631\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-55684h9frl"
Sep 16 20:00:09.488632 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:00:09.488637    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c9fe15b-bfe9-44e6-b00f-238ecb3d3631-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-55684h9frl\" (UID: \"9c9fe15b-bfe9-44e6-b00f-238ecb3d3631\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-55684h9frl"
Sep 16 20:00:09.491138 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:00:09.491114    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c9fe15b-bfe9-44e6-b00f-238ecb3d3631-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-55684h9frl\" (UID: \"9c9fe15b-bfe9-44e6-b00f-238ecb3d3631\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-55684h9frl"
Sep 16 20:00:09.500434 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:00:09.500403    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h95s\" (UniqueName: \"kubernetes.io/projected/9c9fe15b-bfe9-44e6-b00f-238ecb3d3631-kube-api-access-6h95s\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-55684h9frl\" (UID: \"9c9fe15b-bfe9-44e6-b00f-238ecb3d3631\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-55684h9frl"
Sep 16 20:00:09.651771 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:00:09.651685    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-55684h9frl"
Sep 16 20:00:09.992980 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:00:09.992951    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-55684h9frl"]
Sep 16 20:00:09.995804 ip-10-0-139-81 kubenswrapper[2557]: W0916 20:00:09.995775    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c9fe15b_bfe9_44e6_b00f_238ecb3d3631.slice/crio-fba0d734be0c32302a5f4e568a3235294449b061ef14bed5d8cf179ee39e3262 WatchSource:0}: Error finding container fba0d734be0c32302a5f4e568a3235294449b061ef14bed5d8cf179ee39e3262: Status 404 returned error can't find the container with id fba0d734be0c32302a5f4e568a3235294449b061ef14bed5d8cf179ee39e3262
Sep 16 20:00:10.864779 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:00:10.864741    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-55684h9frl" event={"ID":"9c9fe15b-bfe9-44e6-b00f-238ecb3d3631","Type":"ContainerStarted","Data":"6ac0b48e1a3468faa2912f123caabff729b0d2678f8e49577c0d50babdb9373c"}
Sep 16 20:00:10.865202 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:00:10.864801    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-55684h9frl" event={"ID":"9c9fe15b-bfe9-44e6-b00f-238ecb3d3631","Type":"ContainerStarted","Data":"fba0d734be0c32302a5f4e568a3235294449b061ef14bed5d8cf179ee39e3262"}
Sep 16 20:00:10.865202 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:00:10.864942    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-55684h9frl"
Sep 16 20:00:10.893599 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:00:10.893541    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-55684h9frl" podStartSLOduration=1.893525074 podStartE2EDuration="1.893525074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:00:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:00:10.891932665 +0000 UTC m=+1239.417321051" watchObservedRunningTime="2026-09-16 20:00:10.893525074 +0000 UTC m=+1239.418913484"
Sep 16 20:00:41.871630 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:00:41.871600    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-55684h9frl"
Sep 16 20:00:53.870655 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:00:53.870615    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-55684h9frl"]
Sep 16 20:01:01.871737 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:01:01.871694    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-55684h9frl" podUID="9c9fe15b-bfe9-44e6-b00f-238ecb3d3631" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:01:08.890466 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:01:08.890397    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-55684h9frl" podUID="9c9fe15b-bfe9-44e6-b00f-238ecb3d3631" containerName="main" containerID="cri-o://6ac0b48e1a3468faa2912f123caabff729b0d2678f8e49577c0d50babdb9373c" gracePeriod=45
Sep 16 20:01:11.871634 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:01:11.871597    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-55684h9frl" podUID="9c9fe15b-bfe9-44e6-b00f-238ecb3d3631" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:01:21.871127 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:01:21.871083    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-55684h9frl" podUID="9c9fe15b-bfe9-44e6-b00f-238ecb3d3631" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:01:21.871510 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:01:21.871199    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-55684h9frl"
Sep 16 20:01:31.871812 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:01:31.871767    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-55684h9frl" podUID="9c9fe15b-bfe9-44e6-b00f-238ecb3d3631" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:01:39.146272 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:01:39.146245    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-55684h9frl"
Sep 16 20:01:39.285740 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:01:39.285706    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6h95s\" (UniqueName: \"kubernetes.io/projected/9c9fe15b-bfe9-44e6-b00f-238ecb3d3631-kube-api-access-6h95s\") pod \"9c9fe15b-bfe9-44e6-b00f-238ecb3d3631\" (UID: \"9c9fe15b-bfe9-44e6-b00f-238ecb3d3631\") "
Sep 16 20:01:39.285957 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:01:39.285758    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c9fe15b-bfe9-44e6-b00f-238ecb3d3631-tls-certs\") pod \"9c9fe15b-bfe9-44e6-b00f-238ecb3d3631\" (UID: \"9c9fe15b-bfe9-44e6-b00f-238ecb3d3631\") "
Sep 16 20:01:39.288005 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:01:39.287975    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c9fe15b-bfe9-44e6-b00f-238ecb3d3631-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9c9fe15b-bfe9-44e6-b00f-238ecb3d3631" (UID: "9c9fe15b-bfe9-44e6-b00f-238ecb3d3631"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:01:39.288180 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:01:39.288045    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c9fe15b-bfe9-44e6-b00f-238ecb3d3631-kube-api-access-6h95s" (OuterVolumeSpecName: "kube-api-access-6h95s") pod "9c9fe15b-bfe9-44e6-b00f-238ecb3d3631" (UID: "9c9fe15b-bfe9-44e6-b00f-238ecb3d3631"). InnerVolumeSpecName "kube-api-access-6h95s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:01:39.304134 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:01:39.304104    2557 generic.go:356] "Generic (PLEG): container finished" podID="9c9fe15b-bfe9-44e6-b00f-238ecb3d3631" containerID="6ac0b48e1a3468faa2912f123caabff729b0d2678f8e49577c0d50babdb9373c" exitCode=0
Sep 16 20:01:39.304269 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:01:39.304146    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-55684h9frl"
Sep 16 20:01:39.304269 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:01:39.304185    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-55684h9frl" event={"ID":"9c9fe15b-bfe9-44e6-b00f-238ecb3d3631","Type":"ContainerDied","Data":"6ac0b48e1a3468faa2912f123caabff729b0d2678f8e49577c0d50babdb9373c"}
Sep 16 20:01:39.304269 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:01:39.304237    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-55684h9frl" event={"ID":"9c9fe15b-bfe9-44e6-b00f-238ecb3d3631","Type":"ContainerDied","Data":"fba0d734be0c32302a5f4e568a3235294449b061ef14bed5d8cf179ee39e3262"}
Sep 16 20:01:39.304269 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:01:39.304263    2557 scope.go:117] "RemoveContainer" containerID="6ac0b48e1a3468faa2912f123caabff729b0d2678f8e49577c0d50babdb9373c"
Sep 16 20:01:39.313437 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:01:39.313419    2557 scope.go:117] "RemoveContainer" containerID="6ac0b48e1a3468faa2912f123caabff729b0d2678f8e49577c0d50babdb9373c"
Sep 16 20:01:39.313737 ip-10-0-139-81 kubenswrapper[2557]: E0916 20:01:39.313718    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ac0b48e1a3468faa2912f123caabff729b0d2678f8e49577c0d50babdb9373c\": container with ID starting with 6ac0b48e1a3468faa2912f123caabff729b0d2678f8e49577c0d50babdb9373c not found: ID does not exist" containerID="6ac0b48e1a3468faa2912f123caabff729b0d2678f8e49577c0d50babdb9373c"
Sep 16 20:01:39.313785 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:01:39.313745    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ac0b48e1a3468faa2912f123caabff729b0d2678f8e49577c0d50babdb9373c"} err="failed to get container status \"6ac0b48e1a3468faa2912f123caabff729b0d2678f8e49577c0d50babdb9373c\": rpc error: code = NotFound desc = could not find container \"6ac0b48e1a3468faa2912f123caabff729b0d2678f8e49577c0d50babdb9373c\": container with ID starting with 6ac0b48e1a3468faa2912f123caabff729b0d2678f8e49577c0d50babdb9373c not found: ID does not exist"
Sep 16 20:01:39.328996 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:01:39.328970    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-55684h9frl"]
Sep 16 20:01:39.332573 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:01:39.332547    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-55684h9frl"]
Sep 16 20:01:39.387271 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:01:39.387233    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h95s\" (UniqueName: \"kubernetes.io/projected/9c9fe15b-bfe9-44e6-b00f-238ecb3d3631-kube-api-access-6h95s\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:01:39.387271 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:01:39.387264    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c9fe15b-bfe9-44e6-b00f-238ecb3d3631-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:01:39.937751 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:01:39.937720    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c9fe15b-bfe9-44e6-b00f-238ecb3d3631" path="/var/lib/kubelet/pods/9c9fe15b-bfe9-44e6-b00f-238ecb3d3631/volumes"
Sep 16 20:01:47.366343 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:01:47.366312    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67cbd75"]
Sep 16 20:01:47.366771 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:01:47.366699    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c9fe15b-bfe9-44e6-b00f-238ecb3d3631" containerName="main"
Sep 16 20:01:47.366771 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:01:47.366709    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c9fe15b-bfe9-44e6-b00f-238ecb3d3631" containerName="main"
Sep 16 20:01:47.366862 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:01:47.366772    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c9fe15b-bfe9-44e6-b00f-238ecb3d3631" containerName="main"
Sep 16 20:01:47.371387 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:01:47.371365    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67cbd75"
Sep 16 20:01:47.377664 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:01:47.377641    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 16 20:01:47.378833 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:01:47.378811    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-c9sjl\""
Sep 16 20:01:47.378954 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:01:47.378942    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 16 20:01:47.379014 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:01:47.378969    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-k5jxh\""
Sep 16 20:01:47.379072 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:01:47.379016    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 16 20:01:47.389469 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:01:47.389433    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67cbd75"]
Sep 16 20:01:47.462513 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:01:47.462481    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phf7h\" (UniqueName: \"kubernetes.io/projected/2f30aea1-4f93-4f51-b3f4-87ec27d8310b-kube-api-access-phf7h\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67cbd75\" (UID: \"2f30aea1-4f93-4f51-b3f4-87ec27d8310b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67cbd75"
Sep 16 20:01:47.462686 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:01:47.462528    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f30aea1-4f93-4f51-b3f4-87ec27d8310b-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67cbd75\" (UID: \"2f30aea1-4f93-4f51-b3f4-87ec27d8310b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67cbd75"
Sep 16 20:01:47.563659 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:01:47.563618    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phf7h\" (UniqueName: \"kubernetes.io/projected/2f30aea1-4f93-4f51-b3f4-87ec27d8310b-kube-api-access-phf7h\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67cbd75\" (UID: \"2f30aea1-4f93-4f51-b3f4-87ec27d8310b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67cbd75"
Sep 16 20:01:47.563824 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:01:47.563676    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f30aea1-4f93-4f51-b3f4-87ec27d8310b-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67cbd75\" (UID: \"2f30aea1-4f93-4f51-b3f4-87ec27d8310b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67cbd75"
Sep 16 20:01:47.566522 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:01:47.566494    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f30aea1-4f93-4f51-b3f4-87ec27d8310b-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67cbd75\" (UID: \"2f30aea1-4f93-4f51-b3f4-87ec27d8310b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67cbd75"
Sep 16 20:01:47.572245 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:01:47.572218    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phf7h\" (UniqueName: \"kubernetes.io/projected/2f30aea1-4f93-4f51-b3f4-87ec27d8310b-kube-api-access-phf7h\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67cbd75\" (UID: \"2f30aea1-4f93-4f51-b3f4-87ec27d8310b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67cbd75"
Sep 16 20:01:47.681463 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:01:47.681366    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67cbd75"
Sep 16 20:01:47.957040 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:01:47.957015    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67cbd75"]
Sep 16 20:01:47.959311 ip-10-0-139-81 kubenswrapper[2557]: W0916 20:01:47.959276    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f30aea1_4f93_4f51_b3f4_87ec27d8310b.slice/crio-fd67cc8846196ff734d3bb080cacde4080caa479c49c62d0b97f042bd5a653e2 WatchSource:0}: Error finding container fd67cc8846196ff734d3bb080cacde4080caa479c49c62d0b97f042bd5a653e2: Status 404 returned error can't find the container with id fd67cc8846196ff734d3bb080cacde4080caa479c49c62d0b97f042bd5a653e2
Sep 16 20:01:48.352887 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:01:48.352826    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67cbd75" event={"ID":"2f30aea1-4f93-4f51-b3f4-87ec27d8310b","Type":"ContainerStarted","Data":"fb543aed17b55b012fe7102fb76f633376c19ddcbb4497f3aaeae8fdef982ce4"}
Sep 16 20:01:48.353079 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:01:48.352954    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67cbd75" event={"ID":"2f30aea1-4f93-4f51-b3f4-87ec27d8310b","Type":"ContainerStarted","Data":"fd67cc8846196ff734d3bb080cacde4080caa479c49c62d0b97f042bd5a653e2"}
Sep 16 20:01:48.353079 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:01:48.352987    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67cbd75"
Sep 16 20:01:48.371820 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:01:48.371754    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67cbd75" podStartSLOduration=1.371737641 podStartE2EDuration="1.371737641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:01:48.370872912 +0000 UTC m=+1336.896261301" watchObservedRunningTime="2026-09-16 20:01:48.371737641 +0000 UTC m=+1336.897126026"
Sep 16 20:02:19.359450 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:02:19.359420    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67cbd75"
Sep 16 20:02:21.058609 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:02:21.058570    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67cbd75"]
Sep 16 20:02:29.359888 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:02:29.359827    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67cbd75" podUID="2f30aea1-4f93-4f51-b3f4-87ec27d8310b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:02:33.058093 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:02:33.057792    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67cbd75" message="" err="command '/bin/sleep 15' exited with 137: "
Sep 16 20:02:33.058093 ip-10-0-139-81 kubenswrapper[2557]: E0916 20:02:33.057820    2557 kuberuntime_container.go:755] "PreStop hook failed" err="command '/bin/sleep 15' exited with 137: " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67cbd75" podUID="2f30aea1-4f93-4f51-b3f4-87ec27d8310b" containerName="main" containerID="cri-o://fb543aed17b55b012fe7102fb76f633376c19ddcbb4497f3aaeae8fdef982ce4"
Sep 16 20:02:33.058093 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:02:33.057893    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67cbd75" podUID="2f30aea1-4f93-4f51-b3f4-87ec27d8310b" containerName="main" containerID="cri-o://fb543aed17b55b012fe7102fb76f633376c19ddcbb4497f3aaeae8fdef982ce4" gracePeriod=49
Sep 16 20:02:33.204446 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:02:33.204423    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67cbd75_2f30aea1-4f93-4f51-b3f4-87ec27d8310b/main/0.log"
Sep 16 20:02:33.206454 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:02:33.206437    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67cbd75"
Sep 16 20:02:33.279913 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:02:33.279881    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-phf7h\" (UniqueName: \"kubernetes.io/projected/2f30aea1-4f93-4f51-b3f4-87ec27d8310b-kube-api-access-phf7h\") pod \"2f30aea1-4f93-4f51-b3f4-87ec27d8310b\" (UID: \"2f30aea1-4f93-4f51-b3f4-87ec27d8310b\") "
Sep 16 20:02:33.279913 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:02:33.279919    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f30aea1-4f93-4f51-b3f4-87ec27d8310b-tls-certs\") pod \"2f30aea1-4f93-4f51-b3f4-87ec27d8310b\" (UID: \"2f30aea1-4f93-4f51-b3f4-87ec27d8310b\") "
Sep 16 20:02:33.282227 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:02:33.282197    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f30aea1-4f93-4f51-b3f4-87ec27d8310b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2f30aea1-4f93-4f51-b3f4-87ec27d8310b" (UID: "2f30aea1-4f93-4f51-b3f4-87ec27d8310b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:02:33.282327 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:02:33.282201    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f30aea1-4f93-4f51-b3f4-87ec27d8310b-kube-api-access-phf7h" (OuterVolumeSpecName: "kube-api-access-phf7h") pod "2f30aea1-4f93-4f51-b3f4-87ec27d8310b" (UID: "2f30aea1-4f93-4f51-b3f4-87ec27d8310b"). InnerVolumeSpecName "kube-api-access-phf7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:02:33.380873 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:02:33.380761    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phf7h\" (UniqueName: \"kubernetes.io/projected/2f30aea1-4f93-4f51-b3f4-87ec27d8310b-kube-api-access-phf7h\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:02:33.380873 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:02:33.380795    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f30aea1-4f93-4f51-b3f4-87ec27d8310b-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:02:33.583167 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:02:33.583136    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67cbd75_2f30aea1-4f93-4f51-b3f4-87ec27d8310b/main/0.log"
Sep 16 20:02:33.583329 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:02:33.583177    2557 generic.go:356] "Generic (PLEG): container finished" podID="2f30aea1-4f93-4f51-b3f4-87ec27d8310b" containerID="fb543aed17b55b012fe7102fb76f633376c19ddcbb4497f3aaeae8fdef982ce4" exitCode=1
Sep 16 20:02:33.583329 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:02:33.583204    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67cbd75" event={"ID":"2f30aea1-4f93-4f51-b3f4-87ec27d8310b","Type":"ContainerDied","Data":"fb543aed17b55b012fe7102fb76f633376c19ddcbb4497f3aaeae8fdef982ce4"}
Sep 16 20:02:33.583329 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:02:33.583228    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67cbd75"
Sep 16 20:02:33.583329 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:02:33.583261    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67cbd75" event={"ID":"2f30aea1-4f93-4f51-b3f4-87ec27d8310b","Type":"ContainerDied","Data":"fd67cc8846196ff734d3bb080cacde4080caa479c49c62d0b97f042bd5a653e2"}
Sep 16 20:02:33.583329 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:02:33.583286    2557 scope.go:117] "RemoveContainer" containerID="fb543aed17b55b012fe7102fb76f633376c19ddcbb4497f3aaeae8fdef982ce4"
Sep 16 20:02:33.592174 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:02:33.592153    2557 scope.go:117] "RemoveContainer" containerID="fb543aed17b55b012fe7102fb76f633376c19ddcbb4497f3aaeae8fdef982ce4"
Sep 16 20:02:33.592435 ip-10-0-139-81 kubenswrapper[2557]: E0916 20:02:33.592413    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb543aed17b55b012fe7102fb76f633376c19ddcbb4497f3aaeae8fdef982ce4\": container with ID starting with fb543aed17b55b012fe7102fb76f633376c19ddcbb4497f3aaeae8fdef982ce4 not found: ID does not exist" containerID="fb543aed17b55b012fe7102fb76f633376c19ddcbb4497f3aaeae8fdef982ce4"
Sep 16 20:02:33.592507 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:02:33.592441    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb543aed17b55b012fe7102fb76f633376c19ddcbb4497f3aaeae8fdef982ce4"} err="failed to get container status \"fb543aed17b55b012fe7102fb76f633376c19ddcbb4497f3aaeae8fdef982ce4\": rpc error: code = NotFound desc = could not find container \"fb543aed17b55b012fe7102fb76f633376c19ddcbb4497f3aaeae8fdef982ce4\": container with ID starting with fb543aed17b55b012fe7102fb76f633376c19ddcbb4497f3aaeae8fdef982ce4 not found: ID does not exist"
Sep 16 20:02:33.604448 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:02:33.604423    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67cbd75"]
Sep 16 20:02:33.608347 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:02:33.608324    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67cbd75"]
Sep 16 20:02:33.938359 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:02:33.938321    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f30aea1-4f93-4f51-b3f4-87ec27d8310b" path="/var/lib/kubelet/pods/2f30aea1-4f93-4f51-b3f4-87ec27d8310b/volumes"
Sep 16 20:03:25.795131 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:25.795057    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t5g7l"]
Sep 16 20:03:25.795692 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:25.795676    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f30aea1-4f93-4f51-b3f4-87ec27d8310b" containerName="main"
Sep 16 20:03:25.795738 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:25.795696    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f30aea1-4f93-4f51-b3f4-87ec27d8310b" containerName="main"
Sep 16 20:03:25.795829 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:25.795818    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f30aea1-4f93-4f51-b3f4-87ec27d8310b" containerName="main"
Sep 16 20:03:25.799244 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:25.799226    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t5g7l"
Sep 16 20:03:25.802049 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:25.802029    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Sep 16 20:03:25.803429 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:25.803407    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-nckjb\""
Sep 16 20:03:25.803532 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:25.803410    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Sep 16 20:03:25.803532 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:25.803463    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Sep 16 20:03:25.808216 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:25.808192    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t5g7l"]
Sep 16 20:03:25.857718 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:25.857678    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd25c59f-aa3f-4c0b-9a77-68685c9e0e00-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-t5g7l\" (UID: \"fd25c59f-aa3f-4c0b-9a77-68685c9e0e00\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t5g7l"
Sep 16 20:03:25.857941 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:25.857748    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd25c59f-aa3f-4c0b-9a77-68685c9e0e00-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-t5g7l\" (UID: \"fd25c59f-aa3f-4c0b-9a77-68685c9e0e00\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t5g7l"
Sep 16 20:03:25.857941 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:25.857803    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd25c59f-aa3f-4c0b-9a77-68685c9e0e00-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-t5g7l\" (UID: \"fd25c59f-aa3f-4c0b-9a77-68685c9e0e00\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t5g7l"
Sep 16 20:03:25.857941 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:25.857898    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd25c59f-aa3f-4c0b-9a77-68685c9e0e00-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-t5g7l\" (UID: \"fd25c59f-aa3f-4c0b-9a77-68685c9e0e00\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t5g7l"
Sep 16 20:03:25.857941 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:25.857929    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rd5p\" (UniqueName: \"kubernetes.io/projected/fd25c59f-aa3f-4c0b-9a77-68685c9e0e00-kube-api-access-2rd5p\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-t5g7l\" (UID: \"fd25c59f-aa3f-4c0b-9a77-68685c9e0e00\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t5g7l"
Sep 16 20:03:25.858105 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:25.857988    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd25c59f-aa3f-4c0b-9a77-68685c9e0e00-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-t5g7l\" (UID: \"fd25c59f-aa3f-4c0b-9a77-68685c9e0e00\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t5g7l"
Sep 16 20:03:25.958937 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:25.958896    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd25c59f-aa3f-4c0b-9a77-68685c9e0e00-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-t5g7l\" (UID: \"fd25c59f-aa3f-4c0b-9a77-68685c9e0e00\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t5g7l"
Sep 16 20:03:25.959112 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:25.958948    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd25c59f-aa3f-4c0b-9a77-68685c9e0e00-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-t5g7l\" (UID: \"fd25c59f-aa3f-4c0b-9a77-68685c9e0e00\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t5g7l"
Sep 16 20:03:25.959112 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:25.958977    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd25c59f-aa3f-4c0b-9a77-68685c9e0e00-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-t5g7l\" (UID: \"fd25c59f-aa3f-4c0b-9a77-68685c9e0e00\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t5g7l"
Sep 16 20:03:25.959112 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:25.959004    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd25c59f-aa3f-4c0b-9a77-68685c9e0e00-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-t5g7l\" (UID: \"fd25c59f-aa3f-4c0b-9a77-68685c9e0e00\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t5g7l"
Sep 16 20:03:25.959112 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:25.959022    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rd5p\" (UniqueName: \"kubernetes.io/projected/fd25c59f-aa3f-4c0b-9a77-68685c9e0e00-kube-api-access-2rd5p\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-t5g7l\" (UID: \"fd25c59f-aa3f-4c0b-9a77-68685c9e0e00\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t5g7l"
Sep 16 20:03:25.959112 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:25.959066    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd25c59f-aa3f-4c0b-9a77-68685c9e0e00-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-t5g7l\" (UID: \"fd25c59f-aa3f-4c0b-9a77-68685c9e0e00\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t5g7l"
Sep 16 20:03:25.959456 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:25.959433    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd25c59f-aa3f-4c0b-9a77-68685c9e0e00-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-t5g7l\" (UID: \"fd25c59f-aa3f-4c0b-9a77-68685c9e0e00\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t5g7l"
Sep 16 20:03:25.959456 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:25.959449    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd25c59f-aa3f-4c0b-9a77-68685c9e0e00-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-t5g7l\" (UID: \"fd25c59f-aa3f-4c0b-9a77-68685c9e0e00\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t5g7l"
Sep 16 20:03:25.959570 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:25.959550    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd25c59f-aa3f-4c0b-9a77-68685c9e0e00-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-t5g7l\" (UID: \"fd25c59f-aa3f-4c0b-9a77-68685c9e0e00\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t5g7l"
Sep 16 20:03:25.961288 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:25.961258    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd25c59f-aa3f-4c0b-9a77-68685c9e0e00-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-t5g7l\" (UID: \"fd25c59f-aa3f-4c0b-9a77-68685c9e0e00\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t5g7l"
Sep 16 20:03:25.961618 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:25.961602    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd25c59f-aa3f-4c0b-9a77-68685c9e0e00-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-t5g7l\" (UID: \"fd25c59f-aa3f-4c0b-9a77-68685c9e0e00\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t5g7l"
Sep 16 20:03:25.967630 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:25.967604    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rd5p\" (UniqueName: \"kubernetes.io/projected/fd25c59f-aa3f-4c0b-9a77-68685c9e0e00-kube-api-access-2rd5p\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-t5g7l\" (UID: \"fd25c59f-aa3f-4c0b-9a77-68685c9e0e00\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t5g7l"
Sep 16 20:03:26.098611 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:26.098510    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-85487tw6"]
Sep 16 20:03:26.102236 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:26.102216    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-85487tw6"
Sep 16 20:03:26.105110 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:26.105083    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-epp-sa-dockercfg-hn8j8\""
Sep 16 20:03:26.111566 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:26.111545    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t5g7l"
Sep 16 20:03:26.114960 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:26.114939    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-85487tw6"]
Sep 16 20:03:26.161470 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:26.161439    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-85487tw6\" (UID: \"5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-85487tw6"
Sep 16 20:03:26.161602 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:26.161574    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9j75\" (UniqueName: \"kubernetes.io/projected/5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd-kube-api-access-w9j75\") pod \"scheduler-custom-template-test-kserve-router-scheduler-85487tw6\" (UID: \"5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-85487tw6"
Sep 16 20:03:26.161682 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:26.161611    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-85487tw6\" (UID: \"5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-85487tw6"
Sep 16 20:03:26.161682 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:26.161663    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-85487tw6\" (UID: \"5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-85487tw6"
Sep 16 20:03:26.263161 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:26.263124    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9j75\" (UniqueName: \"kubernetes.io/projected/5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd-kube-api-access-w9j75\") pod \"scheduler-custom-template-test-kserve-router-scheduler-85487tw6\" (UID: \"5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-85487tw6"
Sep 16 20:03:26.263161 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:26.263165    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-85487tw6\" (UID: \"5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-85487tw6"
Sep 16 20:03:26.263418 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:26.263202    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-85487tw6\" (UID: \"5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-85487tw6"
Sep 16 20:03:26.263418 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:26.263256    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-85487tw6\" (UID: \"5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-85487tw6"
Sep 16 20:03:26.263608 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:26.263572    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-85487tw6\" (UID: \"5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-85487tw6"
Sep 16 20:03:26.263654 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:26.263596    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-85487tw6\" (UID: \"5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-85487tw6"
Sep 16 20:03:26.265705 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:26.265684    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-85487tw6\" (UID: \"5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-85487tw6"
Sep 16 20:03:26.271912 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:26.271889    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9j75\" (UniqueName: \"kubernetes.io/projected/5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd-kube-api-access-w9j75\") pod \"scheduler-custom-template-test-kserve-router-scheduler-85487tw6\" (UID: \"5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-85487tw6"
Sep 16 20:03:26.389581 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:26.389559    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t5g7l"]
Sep 16 20:03:26.391767 ip-10-0-139-81 kubenswrapper[2557]: W0916 20:03:26.391737    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd25c59f_aa3f_4c0b_9a77_68685c9e0e00.slice/crio-a7bfaff8fa636c3920643dd54c6d989a257d1a1e2c4be39d9190e4cca0dfd42f WatchSource:0}: Error finding container a7bfaff8fa636c3920643dd54c6d989a257d1a1e2c4be39d9190e4cca0dfd42f: Status 404 returned error can't find the container with id a7bfaff8fa636c3920643dd54c6d989a257d1a1e2c4be39d9190e4cca0dfd42f
Sep 16 20:03:26.413599 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:26.413570    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-85487tw6"
Sep 16 20:03:26.694273 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:26.694176    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-85487tw6"]
Sep 16 20:03:26.698979 ip-10-0-139-81 kubenswrapper[2557]: W0916 20:03:26.698947    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fb2662e_eef6_4d19_b3fd_3ffc1d7aefbd.slice/crio-76864313cae041287033106ceafa49cfe93b2d0e2097a66753b36f77b451c06a WatchSource:0}: Error finding container 76864313cae041287033106ceafa49cfe93b2d0e2097a66753b36f77b451c06a: Status 404 returned error can't find the container with id 76864313cae041287033106ceafa49cfe93b2d0e2097a66753b36f77b451c06a
Sep 16 20:03:26.852816 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:26.852774    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-85487tw6" event={"ID":"5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd","Type":"ContainerStarted","Data":"ceffe110e18f6c627c0c8f670c0ffc9362d03afc4a17655af8590131d5c4df41"}
Sep 16 20:03:26.853255 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:26.852854    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-85487tw6" event={"ID":"5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd","Type":"ContainerStarted","Data":"76864313cae041287033106ceafa49cfe93b2d0e2097a66753b36f77b451c06a"}
Sep 16 20:03:26.853255 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:26.852884    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-85487tw6"
Sep 16 20:03:26.854387 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:26.854358    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t5g7l" event={"ID":"fd25c59f-aa3f-4c0b-9a77-68685c9e0e00","Type":"ContainerStarted","Data":"0c0b3b59446272e53032616fdb62bf5189acfc9a722637b4bc2a314b18c0c8cf"}
Sep 16 20:03:26.854520 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:26.854409    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t5g7l" event={"ID":"fd25c59f-aa3f-4c0b-9a77-68685c9e0e00","Type":"ContainerStarted","Data":"a7bfaff8fa636c3920643dd54c6d989a257d1a1e2c4be39d9190e4cca0dfd42f"}
Sep 16 20:03:26.872335 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:26.872290    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-85487tw6" podStartSLOduration=0.872276318 podStartE2EDuration="872.276318ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:03:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:03:26.870876849 +0000 UTC m=+1435.396265236" watchObservedRunningTime="2026-09-16 20:03:26.872276318 +0000 UTC m=+1435.397664702"
Sep 16 20:03:26.890533 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:26.890488    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t5g7l" podStartSLOduration=1.8904749760000001 podStartE2EDuration="1.890474976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:03:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:03:26.889240749 +0000 UTC m=+1435.414629135" watchObservedRunningTime="2026-09-16 20:03:26.890474976 +0000 UTC m=+1435.415863361"
Sep 16 20:03:27.112179 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:27.112148    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t5g7l"
Sep 16 20:03:36.112522 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:36.112480    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t5g7l"
Sep 16 20:03:36.124920 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:36.124898    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t5g7l"
Sep 16 20:03:36.916377 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:36.916347    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t5g7l"
Sep 16 20:03:57.861540 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:03:57.861509    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-85487tw6"
Sep 16 20:04:07.141158 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:07.141126    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-85487tw6"]
Sep 16 20:04:07.144014 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:07.143982    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t5g7l"]
Sep 16 20:04:07.155462 ip-10-0-139-81 kubenswrapper[2557]: E0916 20:04:07.155371    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 20:04:07.155462 ip-10-0-139-81 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:04:07.155462 ip-10-0-139-81 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 16 20:04:07.155462 ip-10-0-139-81 kubenswrapper[2557]:  > containerID="0c0b3b59446272e53032616fdb62bf5189acfc9a722637b4bc2a314b18c0c8cf" cmd=["/bin/sleep","15"]
Sep 16 20:04:07.155462 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:07.155430    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t5g7l" message="" err=<
Sep 16 20:04:07.155462 ip-10-0-139-81 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:04:07.155462 ip-10-0-139-81 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 16 20:04:07.155462 ip-10-0-139-81 kubenswrapper[2557]:  >
Sep 16 20:04:07.155462 ip-10-0-139-81 kubenswrapper[2557]: E0916 20:04:07.155447    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 20:04:07.155462 ip-10-0-139-81 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:04:07.155462 ip-10-0-139-81 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 16 20:04:07.155462 ip-10-0-139-81 kubenswrapper[2557]:  > pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t5g7l" podUID="fd25c59f-aa3f-4c0b-9a77-68685c9e0e00" containerName="main" containerID="cri-o://0c0b3b59446272e53032616fdb62bf5189acfc9a722637b4bc2a314b18c0c8cf"
Sep 16 20:04:07.156225 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:07.155501    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t5g7l" podUID="fd25c59f-aa3f-4c0b-9a77-68685c9e0e00" containerName="main" containerID="cri-o://0c0b3b59446272e53032616fdb62bf5189acfc9a722637b4bc2a314b18c0c8cf" gracePeriod=60
Sep 16 20:04:07.861531 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:07.861494    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-85487tw6" podUID="5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:04:07.909399 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:07.909372    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t5g7l"
Sep 16 20:04:07.947020 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:07.946983    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd25c59f-aa3f-4c0b-9a77-68685c9e0e00-model-cache\") pod \"fd25c59f-aa3f-4c0b-9a77-68685c9e0e00\" (UID: \"fd25c59f-aa3f-4c0b-9a77-68685c9e0e00\") "
Sep 16 20:04:07.947203 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:07.947033    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rd5p\" (UniqueName: \"kubernetes.io/projected/fd25c59f-aa3f-4c0b-9a77-68685c9e0e00-kube-api-access-2rd5p\") pod \"fd25c59f-aa3f-4c0b-9a77-68685c9e0e00\" (UID: \"fd25c59f-aa3f-4c0b-9a77-68685c9e0e00\") "
Sep 16 20:04:07.947203 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:07.947082    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd25c59f-aa3f-4c0b-9a77-68685c9e0e00-tmp-dir\") pod \"fd25c59f-aa3f-4c0b-9a77-68685c9e0e00\" (UID: \"fd25c59f-aa3f-4c0b-9a77-68685c9e0e00\") "
Sep 16 20:04:07.947203 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:07.947118    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd25c59f-aa3f-4c0b-9a77-68685c9e0e00-dshm\") pod \"fd25c59f-aa3f-4c0b-9a77-68685c9e0e00\" (UID: \"fd25c59f-aa3f-4c0b-9a77-68685c9e0e00\") "
Sep 16 20:04:07.947203 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:07.947175    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd25c59f-aa3f-4c0b-9a77-68685c9e0e00-tls-certs\") pod \"fd25c59f-aa3f-4c0b-9a77-68685c9e0e00\" (UID: \"fd25c59f-aa3f-4c0b-9a77-68685c9e0e00\") "
Sep 16 20:04:07.947203 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:07.947202    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd25c59f-aa3f-4c0b-9a77-68685c9e0e00-home\") pod \"fd25c59f-aa3f-4c0b-9a77-68685c9e0e00\" (UID: \"fd25c59f-aa3f-4c0b-9a77-68685c9e0e00\") "
Sep 16 20:04:07.947491 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:07.947256    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd25c59f-aa3f-4c0b-9a77-68685c9e0e00-model-cache" (OuterVolumeSpecName: "model-cache") pod "fd25c59f-aa3f-4c0b-9a77-68685c9e0e00" (UID: "fd25c59f-aa3f-4c0b-9a77-68685c9e0e00"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:04:07.947491 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:07.947356    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd25c59f-aa3f-4c0b-9a77-68685c9e0e00-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fd25c59f-aa3f-4c0b-9a77-68685c9e0e00" (UID: "fd25c59f-aa3f-4c0b-9a77-68685c9e0e00"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:04:07.947613 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:07.947486    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd25c59f-aa3f-4c0b-9a77-68685c9e0e00-home" (OuterVolumeSpecName: "home") pod "fd25c59f-aa3f-4c0b-9a77-68685c9e0e00" (UID: "fd25c59f-aa3f-4c0b-9a77-68685c9e0e00"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:04:07.947613 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:07.947498    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd25c59f-aa3f-4c0b-9a77-68685c9e0e00-model-cache\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:04:07.947613 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:07.947517    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd25c59f-aa3f-4c0b-9a77-68685c9e0e00-tmp-dir\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:04:07.949470 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:07.949435    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd25c59f-aa3f-4c0b-9a77-68685c9e0e00-kube-api-access-2rd5p" (OuterVolumeSpecName: "kube-api-access-2rd5p") pod "fd25c59f-aa3f-4c0b-9a77-68685c9e0e00" (UID: "fd25c59f-aa3f-4c0b-9a77-68685c9e0e00"). InnerVolumeSpecName "kube-api-access-2rd5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:04:07.949589 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:07.949475    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd25c59f-aa3f-4c0b-9a77-68685c9e0e00-dshm" (OuterVolumeSpecName: "dshm") pod "fd25c59f-aa3f-4c0b-9a77-68685c9e0e00" (UID: "fd25c59f-aa3f-4c0b-9a77-68685c9e0e00"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:04:07.949589 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:07.949484    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd25c59f-aa3f-4c0b-9a77-68685c9e0e00-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fd25c59f-aa3f-4c0b-9a77-68685c9e0e00" (UID: "fd25c59f-aa3f-4c0b-9a77-68685c9e0e00"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:04:08.048118 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:08.048014    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rd5p\" (UniqueName: \"kubernetes.io/projected/fd25c59f-aa3f-4c0b-9a77-68685c9e0e00-kube-api-access-2rd5p\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:04:08.048118 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:08.048043    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd25c59f-aa3f-4c0b-9a77-68685c9e0e00-dshm\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:04:08.048118 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:08.048056    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd25c59f-aa3f-4c0b-9a77-68685c9e0e00-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:04:08.048118 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:08.048066    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd25c59f-aa3f-4c0b-9a77-68685c9e0e00-home\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:04:08.059991 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:08.059958    2557 generic.go:356] "Generic (PLEG): container finished" podID="fd25c59f-aa3f-4c0b-9a77-68685c9e0e00" containerID="0c0b3b59446272e53032616fdb62bf5189acfc9a722637b4bc2a314b18c0c8cf" exitCode=0
Sep 16 20:04:08.060129 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:08.060003    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t5g7l"
Sep 16 20:04:08.060129 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:08.060041    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t5g7l" event={"ID":"fd25c59f-aa3f-4c0b-9a77-68685c9e0e00","Type":"ContainerDied","Data":"0c0b3b59446272e53032616fdb62bf5189acfc9a722637b4bc2a314b18c0c8cf"}
Sep 16 20:04:08.060129 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:08.060098    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t5g7l" event={"ID":"fd25c59f-aa3f-4c0b-9a77-68685c9e0e00","Type":"ContainerDied","Data":"a7bfaff8fa636c3920643dd54c6d989a257d1a1e2c4be39d9190e4cca0dfd42f"}
Sep 16 20:04:08.060129 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:08.060127    2557 scope.go:117] "RemoveContainer" containerID="0c0b3b59446272e53032616fdb62bf5189acfc9a722637b4bc2a314b18c0c8cf"
Sep 16 20:04:08.070466 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:08.070447    2557 scope.go:117] "RemoveContainer" containerID="0c0b3b59446272e53032616fdb62bf5189acfc9a722637b4bc2a314b18c0c8cf"
Sep 16 20:04:08.070757 ip-10-0-139-81 kubenswrapper[2557]: E0916 20:04:08.070739    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c0b3b59446272e53032616fdb62bf5189acfc9a722637b4bc2a314b18c0c8cf\": container with ID starting with 0c0b3b59446272e53032616fdb62bf5189acfc9a722637b4bc2a314b18c0c8cf not found: ID does not exist" containerID="0c0b3b59446272e53032616fdb62bf5189acfc9a722637b4bc2a314b18c0c8cf"
Sep 16 20:04:08.070801 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:08.070766    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c0b3b59446272e53032616fdb62bf5189acfc9a722637b4bc2a314b18c0c8cf"} err="failed to get container status \"0c0b3b59446272e53032616fdb62bf5189acfc9a722637b4bc2a314b18c0c8cf\": rpc error: code = NotFound desc = could not find container \"0c0b3b59446272e53032616fdb62bf5189acfc9a722637b4bc2a314b18c0c8cf\": container with ID starting with 0c0b3b59446272e53032616fdb62bf5189acfc9a722637b4bc2a314b18c0c8cf not found: ID does not exist"
Sep 16 20:04:08.094314 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:08.094281    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t5g7l"]
Sep 16 20:04:08.099369 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:08.099340    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t5g7l"]
Sep 16 20:04:08.905576 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:08.905522    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t5g7l" podUID="fd25c59f-aa3f-4c0b-9a77-68685c9e0e00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": context deadline exceeded"
Sep 16 20:04:09.937011 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:09.936931    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd25c59f-aa3f-4c0b-9a77-68685c9e0e00" path="/var/lib/kubelet/pods/fd25c59f-aa3f-4c0b-9a77-68685c9e0e00/volumes"
Sep 16 20:04:17.861558 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:17.861512    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-85487tw6" podUID="5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:04:22.162595 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:22.162550    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-85487tw6" podUID="5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd" containerName="main" containerID="cri-o://ceffe110e18f6c627c0c8f670c0ffc9362d03afc4a17655af8590131d5c4df41" gracePeriod=45
Sep 16 20:04:27.861627 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:27.861583    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-85487tw6" podUID="5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:04:27.862043 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:27.861719    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-85487tw6"
Sep 16 20:04:37.861220 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:37.861181    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-85487tw6" podUID="5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:04:47.861476 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:47.861435    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-85487tw6" podUID="5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:04:52.278936 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:52.278901    2557 generic.go:356] "Generic (PLEG): container finished" podID="5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd" containerID="ceffe110e18f6c627c0c8f670c0ffc9362d03afc4a17655af8590131d5c4df41" exitCode=0
Sep 16 20:04:52.279278 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:52.278995    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-85487tw6" event={"ID":"5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd","Type":"ContainerDied","Data":"ceffe110e18f6c627c0c8f670c0ffc9362d03afc4a17655af8590131d5c4df41"}
Sep 16 20:04:52.415102 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:52.415077    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-85487tw6"
Sep 16 20:04:52.443175 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:52.443105    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd-tokenizer-cache\") pod \"5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd\" (UID: \"5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd\") "
Sep 16 20:04:52.443175 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:52.443156    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd-tls-certs\") pod \"5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd\" (UID: \"5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd\") "
Sep 16 20:04:52.443175 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:52.443176    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd-cachi2-cache\") pod \"5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd\" (UID: \"5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd\") "
Sep 16 20:04:52.443441 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:52.443238    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9j75\" (UniqueName: \"kubernetes.io/projected/5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd-kube-api-access-w9j75\") pod \"5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd\" (UID: \"5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd\") "
Sep 16 20:04:52.443441 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:52.443400    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd" (UID: "5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:04:52.443553 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:52.443493    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd-cachi2-cache" (OuterVolumeSpecName: "cachi2-cache") pod "5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd" (UID: "5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd"). InnerVolumeSpecName "cachi2-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:04:52.445417 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:52.445335    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd" (UID: "5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:04:52.445417 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:52.445346    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd-kube-api-access-w9j75" (OuterVolumeSpecName: "kube-api-access-w9j75") pod "5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd" (UID: "5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd"). InnerVolumeSpecName "kube-api-access-w9j75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:04:52.544336 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:52.544297    2557 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd-tokenizer-cache\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:04:52.544336 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:52.544331    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:04:52.544336 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:52.544342    2557 reconciler_common.go:299] "Volume detached for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd-cachi2-cache\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:04:52.544566 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:52.544351    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9j75\" (UniqueName: \"kubernetes.io/projected/5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd-kube-api-access-w9j75\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:04:53.285019 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:53.284976    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-85487tw6" event={"ID":"5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd","Type":"ContainerDied","Data":"76864313cae041287033106ceafa49cfe93b2d0e2097a66753b36f77b451c06a"}
Sep 16 20:04:53.285390 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:53.285042    2557 scope.go:117] "RemoveContainer" containerID="ceffe110e18f6c627c0c8f670c0ffc9362d03afc4a17655af8590131d5c4df41"
Sep 16 20:04:53.285390 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:53.284998    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-85487tw6"
Sep 16 20:04:53.312325 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:53.312299    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-85487tw6"]
Sep 16 20:04:53.316288 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:53.316262    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-85487tw6"]
Sep 16 20:04:53.937100 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:04:53.937066    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd" path="/var/lib/kubelet/pods/5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd/volumes"
Sep 16 20:05:33.550744 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:05:33.550705    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b7rld"]
Sep 16 20:05:33.551154 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:05:33.551102    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd" containerName="main"
Sep 16 20:05:33.551154 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:05:33.551116    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd" containerName="main"
Sep 16 20:05:33.551154 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:05:33.551127    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd25c59f-aa3f-4c0b-9a77-68685c9e0e00" containerName="main"
Sep 16 20:05:33.551154 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:05:33.551132    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd25c59f-aa3f-4c0b-9a77-68685c9e0e00" containerName="main"
Sep 16 20:05:33.551288 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:05:33.551205    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd25c59f-aa3f-4c0b-9a77-68685c9e0e00" containerName="main"
Sep 16 20:05:33.551288 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:05:33.551214    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fb2662e-eef6-4d19-b3fd-3ffc1d7aefbd" containerName="main"
Sep 16 20:05:33.554690 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:05:33.554667    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b7rld"
Sep 16 20:05:33.558854 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:05:33.558813    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 16 20:05:33.558990 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:05:33.558861    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 16 20:05:33.558990 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:05:33.558905    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-q82pp\""
Sep 16 20:05:33.558990 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:05:33.558921    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 16 20:05:33.569339 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:05:33.569315    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b7rld"]
Sep 16 20:05:33.627256 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:05:33.627217    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6721f21b-fd36-4106-81f0-92461319ad77-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b7rld\" (UID: \"6721f21b-fd36-4106-81f0-92461319ad77\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b7rld"
Sep 16 20:05:33.627256 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:05:33.627263    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wx2v\" (UniqueName: \"kubernetes.io/projected/6721f21b-fd36-4106-81f0-92461319ad77-kube-api-access-5wx2v\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b7rld\" (UID: \"6721f21b-fd36-4106-81f0-92461319ad77\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b7rld"
Sep 16 20:05:33.627483 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:05:33.627377    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6721f21b-fd36-4106-81f0-92461319ad77-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b7rld\" (UID: \"6721f21b-fd36-4106-81f0-92461319ad77\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b7rld"
Sep 16 20:05:33.627483 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:05:33.627429    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6721f21b-fd36-4106-81f0-92461319ad77-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b7rld\" (UID: \"6721f21b-fd36-4106-81f0-92461319ad77\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b7rld"
Sep 16 20:05:33.627483 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:05:33.627446    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6721f21b-fd36-4106-81f0-92461319ad77-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b7rld\" (UID: \"6721f21b-fd36-4106-81f0-92461319ad77\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b7rld"
Sep 16 20:05:33.627483 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:05:33.627480    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6721f21b-fd36-4106-81f0-92461319ad77-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b7rld\" (UID: \"6721f21b-fd36-4106-81f0-92461319ad77\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b7rld"
Sep 16 20:05:33.728921 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:05:33.728868    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6721f21b-fd36-4106-81f0-92461319ad77-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b7rld\" (UID: \"6721f21b-fd36-4106-81f0-92461319ad77\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b7rld"
Sep 16 20:05:33.728921 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:05:33.728933    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wx2v\" (UniqueName: \"kubernetes.io/projected/6721f21b-fd36-4106-81f0-92461319ad77-kube-api-access-5wx2v\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b7rld\" (UID: \"6721f21b-fd36-4106-81f0-92461319ad77\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b7rld"
Sep 16 20:05:33.729127 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:05:33.728979    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6721f21b-fd36-4106-81f0-92461319ad77-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b7rld\" (UID: \"6721f21b-fd36-4106-81f0-92461319ad77\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b7rld"
Sep 16 20:05:33.729127 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:05:33.729012    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6721f21b-fd36-4106-81f0-92461319ad77-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b7rld\" (UID: \"6721f21b-fd36-4106-81f0-92461319ad77\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b7rld"
Sep 16 20:05:33.729127 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:05:33.729034    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6721f21b-fd36-4106-81f0-92461319ad77-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b7rld\" (UID: \"6721f21b-fd36-4106-81f0-92461319ad77\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b7rld"
Sep 16 20:05:33.729127 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:05:33.729068    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6721f21b-fd36-4106-81f0-92461319ad77-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b7rld\" (UID: \"6721f21b-fd36-4106-81f0-92461319ad77\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b7rld"
Sep 16 20:05:33.729439 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:05:33.729416    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6721f21b-fd36-4106-81f0-92461319ad77-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b7rld\" (UID: \"6721f21b-fd36-4106-81f0-92461319ad77\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b7rld"
Sep 16 20:05:33.729535 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:05:33.729505    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6721f21b-fd36-4106-81f0-92461319ad77-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b7rld\" (UID: \"6721f21b-fd36-4106-81f0-92461319ad77\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b7rld"
Sep 16 20:05:33.729607 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:05:33.729534    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6721f21b-fd36-4106-81f0-92461319ad77-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b7rld\" (UID: \"6721f21b-fd36-4106-81f0-92461319ad77\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b7rld"
Sep 16 20:05:33.731304 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:05:33.731283    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6721f21b-fd36-4106-81f0-92461319ad77-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b7rld\" (UID: \"6721f21b-fd36-4106-81f0-92461319ad77\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b7rld"
Sep 16 20:05:33.731617 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:05:33.731598    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6721f21b-fd36-4106-81f0-92461319ad77-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b7rld\" (UID: \"6721f21b-fd36-4106-81f0-92461319ad77\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b7rld"
Sep 16 20:05:33.738253 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:05:33.738230    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wx2v\" (UniqueName: \"kubernetes.io/projected/6721f21b-fd36-4106-81f0-92461319ad77-kube-api-access-5wx2v\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b7rld\" (UID: \"6721f21b-fd36-4106-81f0-92461319ad77\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b7rld"
Sep 16 20:05:33.822042 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:05:33.821956    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55xfqrh"]
Sep 16 20:05:33.826580 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:05:33.826562    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55xfqrh"
Sep 16 20:05:33.829565 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:05:33.829540    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-epp-sa-dockercfg-2ln6j\""
Sep 16 20:05:33.834923 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:05:33.834900    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55xfqrh"]
Sep 16 20:05:33.865534 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:05:33.865497    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b7rld"
Sep 16 20:05:33.930538 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:05:33.930496    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8mml\" (UniqueName: \"kubernetes.io/projected/b71ee6d1-40a9-48fa-b6f1-c20fc87ca5ec-kube-api-access-n8mml\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-55xfqrh\" (UID: \"b71ee6d1-40a9-48fa-b6f1-c20fc87ca5ec\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55xfqrh"
Sep 16 20:05:33.930699 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:05:33.930563    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b71ee6d1-40a9-48fa-b6f1-c20fc87ca5ec-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-55xfqrh\" (UID: \"b71ee6d1-40a9-48fa-b6f1-c20fc87ca5ec\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55xfqrh"
Sep 16 20:05:34.032023 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:05:34.031978    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8mml\" (UniqueName: \"kubernetes.io/projected/b71ee6d1-40a9-48fa-b6f1-c20fc87ca5ec-kube-api-access-n8mml\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-55xfqrh\" (UID: \"b71ee6d1-40a9-48fa-b6f1-c20fc87ca5ec\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55xfqrh"
Sep 16 20:05:34.032214 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:05:34.032032    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b71ee6d1-40a9-48fa-b6f1-c20fc87ca5ec-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-55xfqrh\" (UID: \"b71ee6d1-40a9-48fa-b6f1-c20fc87ca5ec\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55xfqrh"
Sep 16 20:05:34.034679 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:05:34.034654    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b71ee6d1-40a9-48fa-b6f1-c20fc87ca5ec-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-55xfqrh\" (UID: \"b71ee6d1-40a9-48fa-b6f1-c20fc87ca5ec\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55xfqrh"
Sep 16 20:05:34.044005 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:05:34.043979    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8mml\" (UniqueName: \"kubernetes.io/projected/b71ee6d1-40a9-48fa-b6f1-c20fc87ca5ec-kube-api-access-n8mml\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-55xfqrh\" (UID: \"b71ee6d1-40a9-48fa-b6f1-c20fc87ca5ec\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55xfqrh"
Sep 16 20:05:34.138994 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:05:34.138903    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55xfqrh"
Sep 16 20:05:34.142222 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:05:34.142200    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b7rld"]
Sep 16 20:05:34.144358 ip-10-0-139-81 kubenswrapper[2557]: W0916 20:05:34.144328    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6721f21b_fd36_4106_81f0_92461319ad77.slice/crio-22a2fbad5f79eed9d149b3d9af30773d8bfb34b72874a247bdc450c4b690c82f WatchSource:0}: Error finding container 22a2fbad5f79eed9d149b3d9af30773d8bfb34b72874a247bdc450c4b690c82f: Status 404 returned error can't find the container with id 22a2fbad5f79eed9d149b3d9af30773d8bfb34b72874a247bdc450c4b690c82f
Sep 16 20:05:34.414667 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:05:34.414642    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55xfqrh"]
Sep 16 20:05:34.416390 ip-10-0-139-81 kubenswrapper[2557]: W0916 20:05:34.416365    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb71ee6d1_40a9_48fa_b6f1_c20fc87ca5ec.slice/crio-76f8e6d70c947c5515eb676f35f82cf3a11f6b441f2ef704dad5aac5cace42c2 WatchSource:0}: Error finding container 76f8e6d70c947c5515eb676f35f82cf3a11f6b441f2ef704dad5aac5cace42c2: Status 404 returned error can't find the container with id 76f8e6d70c947c5515eb676f35f82cf3a11f6b441f2ef704dad5aac5cace42c2
Sep 16 20:05:34.498405 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:05:34.498340    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b7rld" event={"ID":"6721f21b-fd36-4106-81f0-92461319ad77","Type":"ContainerStarted","Data":"7ad46b08fca00189cafeba90f7cb76c9162dcc70b5fdaee20329430fc6afd06a"}
Sep 16 20:05:34.498606 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:05:34.498414    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b7rld" event={"ID":"6721f21b-fd36-4106-81f0-92461319ad77","Type":"ContainerStarted","Data":"22a2fbad5f79eed9d149b3d9af30773d8bfb34b72874a247bdc450c4b690c82f"}
Sep 16 20:05:34.500756 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:05:34.500721    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55xfqrh" event={"ID":"b71ee6d1-40a9-48fa-b6f1-c20fc87ca5ec","Type":"ContainerStarted","Data":"30f4feaec2e68311289cdf9167a60da0dd0b24036c78dc62a583ed6bfdfa92dd"}
Sep 16 20:05:34.500920 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:05:34.500781    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55xfqrh" event={"ID":"b71ee6d1-40a9-48fa-b6f1-c20fc87ca5ec","Type":"ContainerStarted","Data":"76f8e6d70c947c5515eb676f35f82cf3a11f6b441f2ef704dad5aac5cace42c2"}
Sep 16 20:05:34.500920 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:05:34.500809    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55xfqrh"
Sep 16 20:05:34.520587 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:05:34.520520    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b7rld" podStartSLOduration=1.520500353 podStartE2EDuration="1.520500353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:05:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:05:34.51874172 +0000 UTC m=+1563.044130106" watchObservedRunningTime="2026-09-16 20:05:34.520500353 +0000 UTC m=+1563.045888738"
Sep 16 20:05:34.540638 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:05:34.540575    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55xfqrh" podStartSLOduration=1.540554909 podStartE2EDuration="1.540554909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:05:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:05:34.537603612 +0000 UTC m=+1563.062991998" watchObservedRunningTime="2026-09-16 20:05:34.540554909 +0000 UTC m=+1563.065943295"
Sep 16 20:05:34.865783 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:05:34.865741    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b7rld"
Sep 16 20:05:43.866570 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:05:43.866527    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b7rld"
Sep 16 20:05:43.879086 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:05:43.879060    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b7rld"
Sep 16 20:05:44.564456 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:05:44.564424    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b7rld"
Sep 16 20:06:05.507792 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:06:05.507759    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55xfqrh"
Sep 16 20:06:13.042896 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:06:13.042835    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55xfqrh"]
Sep 16 20:06:13.052104 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:06:13.052075    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b7rld"]
Sep 16 20:06:13.062668 ip-10-0-139-81 kubenswrapper[2557]: E0916 20:06:13.062617    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 20:06:13.062668 ip-10-0-139-81 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:06:13.062668 ip-10-0-139-81 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 16 20:06:13.062668 ip-10-0-139-81 kubenswrapper[2557]:  > containerID="7ad46b08fca00189cafeba90f7cb76c9162dcc70b5fdaee20329430fc6afd06a" cmd=["/bin/sleep","15"]
Sep 16 20:06:13.063036 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:06:13.062697    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b7rld" message="" err=<
Sep 16 20:06:13.063036 ip-10-0-139-81 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:06:13.063036 ip-10-0-139-81 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 16 20:06:13.063036 ip-10-0-139-81 kubenswrapper[2557]:  >
Sep 16 20:06:13.063036 ip-10-0-139-81 kubenswrapper[2557]: E0916 20:06:13.062715    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 20:06:13.063036 ip-10-0-139-81 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:06:13.063036 ip-10-0-139-81 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 16 20:06:13.063036 ip-10-0-139-81 kubenswrapper[2557]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b7rld" podUID="6721f21b-fd36-4106-81f0-92461319ad77" containerName="main" containerID="cri-o://7ad46b08fca00189cafeba90f7cb76c9162dcc70b5fdaee20329430fc6afd06a"
Sep 16 20:06:13.063036 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:06:13.062779    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b7rld" podUID="6721f21b-fd36-4106-81f0-92461319ad77" containerName="main" containerID="cri-o://7ad46b08fca00189cafeba90f7cb76c9162dcc70b5fdaee20329430fc6afd06a" gracePeriod=60
Sep 16 20:06:13.417660 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:06:13.417630    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b7rld"
Sep 16 20:06:13.501276 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:06:13.501236    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6721f21b-fd36-4106-81f0-92461319ad77-tls-certs\") pod \"6721f21b-fd36-4106-81f0-92461319ad77\" (UID: \"6721f21b-fd36-4106-81f0-92461319ad77\") "
Sep 16 20:06:13.501459 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:06:13.501304    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6721f21b-fd36-4106-81f0-92461319ad77-model-cache\") pod \"6721f21b-fd36-4106-81f0-92461319ad77\" (UID: \"6721f21b-fd36-4106-81f0-92461319ad77\") "
Sep 16 20:06:13.501459 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:06:13.501325    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6721f21b-fd36-4106-81f0-92461319ad77-home\") pod \"6721f21b-fd36-4106-81f0-92461319ad77\" (UID: \"6721f21b-fd36-4106-81f0-92461319ad77\") "
Sep 16 20:06:13.501459 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:06:13.501358    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6721f21b-fd36-4106-81f0-92461319ad77-tmp-dir\") pod \"6721f21b-fd36-4106-81f0-92461319ad77\" (UID: \"6721f21b-fd36-4106-81f0-92461319ad77\") "
Sep 16 20:06:13.501459 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:06:13.501384    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6721f21b-fd36-4106-81f0-92461319ad77-dshm\") pod \"6721f21b-fd36-4106-81f0-92461319ad77\" (UID: \"6721f21b-fd36-4106-81f0-92461319ad77\") "
Sep 16 20:06:13.501459 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:06:13.501455    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wx2v\" (UniqueName: \"kubernetes.io/projected/6721f21b-fd36-4106-81f0-92461319ad77-kube-api-access-5wx2v\") pod \"6721f21b-fd36-4106-81f0-92461319ad77\" (UID: \"6721f21b-fd36-4106-81f0-92461319ad77\") "
Sep 16 20:06:13.501802 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:06:13.501540    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6721f21b-fd36-4106-81f0-92461319ad77-model-cache" (OuterVolumeSpecName: "model-cache") pod "6721f21b-fd36-4106-81f0-92461319ad77" (UID: "6721f21b-fd36-4106-81f0-92461319ad77"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:06:13.501802 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:06:13.501645    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6721f21b-fd36-4106-81f0-92461319ad77-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6721f21b-fd36-4106-81f0-92461319ad77" (UID: "6721f21b-fd36-4106-81f0-92461319ad77"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:06:13.501802 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:06:13.501657    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6721f21b-fd36-4106-81f0-92461319ad77-home" (OuterVolumeSpecName: "home") pod "6721f21b-fd36-4106-81f0-92461319ad77" (UID: "6721f21b-fd36-4106-81f0-92461319ad77"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:06:13.501802 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:06:13.501777    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6721f21b-fd36-4106-81f0-92461319ad77-model-cache\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:06:13.501802 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:06:13.501795    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6721f21b-fd36-4106-81f0-92461319ad77-home\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:06:13.502106 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:06:13.501810    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6721f21b-fd36-4106-81f0-92461319ad77-tmp-dir\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:06:13.503738 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:06:13.503667    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6721f21b-fd36-4106-81f0-92461319ad77-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6721f21b-fd36-4106-81f0-92461319ad77" (UID: "6721f21b-fd36-4106-81f0-92461319ad77"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:06:13.503910 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:06:13.503834    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6721f21b-fd36-4106-81f0-92461319ad77-dshm" (OuterVolumeSpecName: "dshm") pod "6721f21b-fd36-4106-81f0-92461319ad77" (UID: "6721f21b-fd36-4106-81f0-92461319ad77"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:06:13.504042 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:06:13.504023    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6721f21b-fd36-4106-81f0-92461319ad77-kube-api-access-5wx2v" (OuterVolumeSpecName: "kube-api-access-5wx2v") pod "6721f21b-fd36-4106-81f0-92461319ad77" (UID: "6721f21b-fd36-4106-81f0-92461319ad77"). InnerVolumeSpecName "kube-api-access-5wx2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:06:13.602739 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:06:13.602653    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6721f21b-fd36-4106-81f0-92461319ad77-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:06:13.602739 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:06:13.602684    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6721f21b-fd36-4106-81f0-92461319ad77-dshm\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:06:13.602739 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:06:13.602697    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wx2v\" (UniqueName: \"kubernetes.io/projected/6721f21b-fd36-4106-81f0-92461319ad77-kube-api-access-5wx2v\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:06:13.699986 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:06:13.699945    2557 generic.go:356] "Generic (PLEG): container finished" podID="6721f21b-fd36-4106-81f0-92461319ad77" containerID="7ad46b08fca00189cafeba90f7cb76c9162dcc70b5fdaee20329430fc6afd06a" exitCode=0
Sep 16 20:06:13.700184 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:06:13.700011    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b7rld"
Sep 16 20:06:13.700184 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:06:13.700029    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b7rld" event={"ID":"6721f21b-fd36-4106-81f0-92461319ad77","Type":"ContainerDied","Data":"7ad46b08fca00189cafeba90f7cb76c9162dcc70b5fdaee20329430fc6afd06a"}
Sep 16 20:06:13.700184 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:06:13.700125    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b7rld" event={"ID":"6721f21b-fd36-4106-81f0-92461319ad77","Type":"ContainerDied","Data":"22a2fbad5f79eed9d149b3d9af30773d8bfb34b72874a247bdc450c4b690c82f"}
Sep 16 20:06:13.700184 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:06:13.700156    2557 scope.go:117] "RemoveContainer" containerID="7ad46b08fca00189cafeba90f7cb76c9162dcc70b5fdaee20329430fc6afd06a"
Sep 16 20:06:13.710200 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:06:13.710181    2557 scope.go:117] "RemoveContainer" containerID="7ad46b08fca00189cafeba90f7cb76c9162dcc70b5fdaee20329430fc6afd06a"
Sep 16 20:06:13.710490 ip-10-0-139-81 kubenswrapper[2557]: E0916 20:06:13.710471    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ad46b08fca00189cafeba90f7cb76c9162dcc70b5fdaee20329430fc6afd06a\": container with ID starting with 7ad46b08fca00189cafeba90f7cb76c9162dcc70b5fdaee20329430fc6afd06a not found: ID does not exist" containerID="7ad46b08fca00189cafeba90f7cb76c9162dcc70b5fdaee20329430fc6afd06a"
Sep 16 20:06:13.710554 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:06:13.710500    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ad46b08fca00189cafeba90f7cb76c9162dcc70b5fdaee20329430fc6afd06a"} err="failed to get container status \"7ad46b08fca00189cafeba90f7cb76c9162dcc70b5fdaee20329430fc6afd06a\": rpc error: code = NotFound desc = could not find container \"7ad46b08fca00189cafeba90f7cb76c9162dcc70b5fdaee20329430fc6afd06a\": container with ID starting with 7ad46b08fca00189cafeba90f7cb76c9162dcc70b5fdaee20329430fc6afd06a not found: ID does not exist"
Sep 16 20:06:13.722918 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:06:13.722893    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b7rld"]
Sep 16 20:06:13.726948 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:06:13.726926    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b7rld"]
Sep 16 20:06:13.938334 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:06:13.938255    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6721f21b-fd36-4106-81f0-92461319ad77" path="/var/lib/kubelet/pods/6721f21b-fd36-4106-81f0-92461319ad77/volumes"
Sep 16 20:06:15.508520 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:06:15.508483    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55xfqrh" podUID="b71ee6d1-40a9-48fa-b6f1-c20fc87ca5ec" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:06:25.507913 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:06:25.507822    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55xfqrh" podUID="b71ee6d1-40a9-48fa-b6f1-c20fc87ca5ec" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:06:28.064336 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:06:28.064279    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55xfqrh" podUID="b71ee6d1-40a9-48fa-b6f1-c20fc87ca5ec" containerName="main" containerID="cri-o://30f4feaec2e68311289cdf9167a60da0dd0b24036c78dc62a583ed6bfdfa92dd" gracePeriod=45
Sep 16 20:06:35.508113 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:06:35.508074    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55xfqrh" podUID="b71ee6d1-40a9-48fa-b6f1-c20fc87ca5ec" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:06:35.508467 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:06:35.508192    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55xfqrh"
Sep 16 20:06:45.508388 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:06:45.508346    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55xfqrh" podUID="b71ee6d1-40a9-48fa-b6f1-c20fc87ca5ec" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:06:55.508186 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:06:55.508143    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55xfqrh" podUID="b71ee6d1-40a9-48fa-b6f1-c20fc87ca5ec" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:06:58.312349 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:06:58.312322    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55xfqrh"
Sep 16 20:06:58.510750 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:06:58.510706    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b71ee6d1-40a9-48fa-b6f1-c20fc87ca5ec-tls-certs\") pod \"b71ee6d1-40a9-48fa-b6f1-c20fc87ca5ec\" (UID: \"b71ee6d1-40a9-48fa-b6f1-c20fc87ca5ec\") "
Sep 16 20:06:58.510961 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:06:58.510901    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8mml\" (UniqueName: \"kubernetes.io/projected/b71ee6d1-40a9-48fa-b6f1-c20fc87ca5ec-kube-api-access-n8mml\") pod \"b71ee6d1-40a9-48fa-b6f1-c20fc87ca5ec\" (UID: \"b71ee6d1-40a9-48fa-b6f1-c20fc87ca5ec\") "
Sep 16 20:06:58.512962 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:06:58.512936    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b71ee6d1-40a9-48fa-b6f1-c20fc87ca5ec-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b71ee6d1-40a9-48fa-b6f1-c20fc87ca5ec" (UID: "b71ee6d1-40a9-48fa-b6f1-c20fc87ca5ec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:06:58.512962 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:06:58.512950    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b71ee6d1-40a9-48fa-b6f1-c20fc87ca5ec-kube-api-access-n8mml" (OuterVolumeSpecName: "kube-api-access-n8mml") pod "b71ee6d1-40a9-48fa-b6f1-c20fc87ca5ec" (UID: "b71ee6d1-40a9-48fa-b6f1-c20fc87ca5ec"). InnerVolumeSpecName "kube-api-access-n8mml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:06:58.611994 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:06:58.611939    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b71ee6d1-40a9-48fa-b6f1-c20fc87ca5ec-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:06:58.611994 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:06:58.611987    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8mml\" (UniqueName: \"kubernetes.io/projected/b71ee6d1-40a9-48fa-b6f1-c20fc87ca5ec-kube-api-access-n8mml\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:06:58.926621 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:06:58.926534    2557 generic.go:356] "Generic (PLEG): container finished" podID="b71ee6d1-40a9-48fa-b6f1-c20fc87ca5ec" containerID="30f4feaec2e68311289cdf9167a60da0dd0b24036c78dc62a583ed6bfdfa92dd" exitCode=0
Sep 16 20:06:58.926621 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:06:58.926578    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55xfqrh"
Sep 16 20:06:58.926819 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:06:58.926620    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55xfqrh" event={"ID":"b71ee6d1-40a9-48fa-b6f1-c20fc87ca5ec","Type":"ContainerDied","Data":"30f4feaec2e68311289cdf9167a60da0dd0b24036c78dc62a583ed6bfdfa92dd"}
Sep 16 20:06:58.926819 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:06:58.926675    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55xfqrh" event={"ID":"b71ee6d1-40a9-48fa-b6f1-c20fc87ca5ec","Type":"ContainerDied","Data":"76f8e6d70c947c5515eb676f35f82cf3a11f6b441f2ef704dad5aac5cace42c2"}
Sep 16 20:06:58.926819 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:06:58.926700    2557 scope.go:117] "RemoveContainer" containerID="30f4feaec2e68311289cdf9167a60da0dd0b24036c78dc62a583ed6bfdfa92dd"
Sep 16 20:06:58.936287 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:06:58.936271    2557 scope.go:117] "RemoveContainer" containerID="30f4feaec2e68311289cdf9167a60da0dd0b24036c78dc62a583ed6bfdfa92dd"
Sep 16 20:06:58.936561 ip-10-0-139-81 kubenswrapper[2557]: E0916 20:06:58.936540    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30f4feaec2e68311289cdf9167a60da0dd0b24036c78dc62a583ed6bfdfa92dd\": container with ID starting with 30f4feaec2e68311289cdf9167a60da0dd0b24036c78dc62a583ed6bfdfa92dd not found: ID does not exist" containerID="30f4feaec2e68311289cdf9167a60da0dd0b24036c78dc62a583ed6bfdfa92dd"
Sep 16 20:06:58.936605 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:06:58.936571    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30f4feaec2e68311289cdf9167a60da0dd0b24036c78dc62a583ed6bfdfa92dd"} err="failed to get container status \"30f4feaec2e68311289cdf9167a60da0dd0b24036c78dc62a583ed6bfdfa92dd\": rpc error: code = NotFound desc = could not find container \"30f4feaec2e68311289cdf9167a60da0dd0b24036c78dc62a583ed6bfdfa92dd\": container with ID starting with 30f4feaec2e68311289cdf9167a60da0dd0b24036c78dc62a583ed6bfdfa92dd not found: ID does not exist"
Sep 16 20:06:58.950301 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:06:58.950275    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55xfqrh"]
Sep 16 20:06:58.954686 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:06:58.954663    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55xfqrh"]
Sep 16 20:06:59.937853 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:06:59.937809    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b71ee6d1-40a9-48fa-b6f1-c20fc87ca5ec" path="/var/lib/kubelet/pods/b71ee6d1-40a9-48fa-b6f1-c20fc87ca5ec/volumes"
Sep 16 20:07:17.600392 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:17.600348    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-jcj6l"]
Sep 16 20:07:17.601010 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:17.600987    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6721f21b-fd36-4106-81f0-92461319ad77" containerName="main"
Sep 16 20:07:17.601091 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:17.601014    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6721f21b-fd36-4106-81f0-92461319ad77" containerName="main"
Sep 16 20:07:17.601091 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:17.601050    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b71ee6d1-40a9-48fa-b6f1-c20fc87ca5ec" containerName="main"
Sep 16 20:07:17.601091 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:17.601059    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b71ee6d1-40a9-48fa-b6f1-c20fc87ca5ec" containerName="main"
Sep 16 20:07:17.601238 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:17.601209    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6721f21b-fd36-4106-81f0-92461319ad77" containerName="main"
Sep 16 20:07:17.601238 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:17.601225    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b71ee6d1-40a9-48fa-b6f1-c20fc87ca5ec" containerName="main"
Sep 16 20:07:17.604117 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:17.604093    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-jcj6l"
Sep 16 20:07:17.608397 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:17.608375    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"openshift-service-ca.crt\""
Sep 16 20:07:17.608526 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:17.608378    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"kube-root-ca.crt\""
Sep 16 20:07:17.608592 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:17.608569    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"default-dockercfg-2945m\""
Sep 16 20:07:17.608655 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:17.608623    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"canary-v2-kserve-self-signed-certs\""
Sep 16 20:07:17.617055 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:17.617032    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-jcj6l"]
Sep 16 20:07:17.676493 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:17.676457    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e37031a-7901-4244-a94c-05d8b7817516-home\") pod \"canary-v2-kserve-7f65dbbccc-jcj6l\" (UID: \"0e37031a-7901-4244-a94c-05d8b7817516\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-jcj6l"
Sep 16 20:07:17.676685 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:17.676578    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e37031a-7901-4244-a94c-05d8b7817516-dshm\") pod \"canary-v2-kserve-7f65dbbccc-jcj6l\" (UID: \"0e37031a-7901-4244-a94c-05d8b7817516\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-jcj6l"
Sep 16 20:07:17.676685 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:17.676600    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e37031a-7901-4244-a94c-05d8b7817516-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-jcj6l\" (UID: \"0e37031a-7901-4244-a94c-05d8b7817516\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-jcj6l"
Sep 16 20:07:17.676685 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:17.676623    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q97zf\" (UniqueName: \"kubernetes.io/projected/0e37031a-7901-4244-a94c-05d8b7817516-kube-api-access-q97zf\") pod \"canary-v2-kserve-7f65dbbccc-jcj6l\" (UID: \"0e37031a-7901-4244-a94c-05d8b7817516\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-jcj6l"
Sep 16 20:07:17.676685 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:17.676657    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e37031a-7901-4244-a94c-05d8b7817516-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-jcj6l\" (UID: \"0e37031a-7901-4244-a94c-05d8b7817516\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-jcj6l"
Sep 16 20:07:17.676685 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:17.676675    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e37031a-7901-4244-a94c-05d8b7817516-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-jcj6l\" (UID: \"0e37031a-7901-4244-a94c-05d8b7817516\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-jcj6l"
Sep 16 20:07:17.777149 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:17.777115    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e37031a-7901-4244-a94c-05d8b7817516-dshm\") pod \"canary-v2-kserve-7f65dbbccc-jcj6l\" (UID: \"0e37031a-7901-4244-a94c-05d8b7817516\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-jcj6l"
Sep 16 20:07:17.777149 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:17.777150    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e37031a-7901-4244-a94c-05d8b7817516-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-jcj6l\" (UID: \"0e37031a-7901-4244-a94c-05d8b7817516\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-jcj6l"
Sep 16 20:07:17.777368 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:17.777169    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q97zf\" (UniqueName: \"kubernetes.io/projected/0e37031a-7901-4244-a94c-05d8b7817516-kube-api-access-q97zf\") pod \"canary-v2-kserve-7f65dbbccc-jcj6l\" (UID: \"0e37031a-7901-4244-a94c-05d8b7817516\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-jcj6l"
Sep 16 20:07:17.777368 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:17.777206    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e37031a-7901-4244-a94c-05d8b7817516-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-jcj6l\" (UID: \"0e37031a-7901-4244-a94c-05d8b7817516\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-jcj6l"
Sep 16 20:07:17.777368 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:17.777231    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e37031a-7901-4244-a94c-05d8b7817516-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-jcj6l\" (UID: \"0e37031a-7901-4244-a94c-05d8b7817516\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-jcj6l"
Sep 16 20:07:17.777368 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:17.777282    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e37031a-7901-4244-a94c-05d8b7817516-home\") pod \"canary-v2-kserve-7f65dbbccc-jcj6l\" (UID: \"0e37031a-7901-4244-a94c-05d8b7817516\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-jcj6l"
Sep 16 20:07:17.777713 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:17.777684    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e37031a-7901-4244-a94c-05d8b7817516-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-jcj6l\" (UID: \"0e37031a-7901-4244-a94c-05d8b7817516\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-jcj6l"
Sep 16 20:07:17.777809 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:17.777693    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e37031a-7901-4244-a94c-05d8b7817516-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-jcj6l\" (UID: \"0e37031a-7901-4244-a94c-05d8b7817516\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-jcj6l"
Sep 16 20:07:17.777809 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:17.777751    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e37031a-7901-4244-a94c-05d8b7817516-home\") pod \"canary-v2-kserve-7f65dbbccc-jcj6l\" (UID: \"0e37031a-7901-4244-a94c-05d8b7817516\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-jcj6l"
Sep 16 20:07:17.779502 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:17.779479    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e37031a-7901-4244-a94c-05d8b7817516-dshm\") pod \"canary-v2-kserve-7f65dbbccc-jcj6l\" (UID: \"0e37031a-7901-4244-a94c-05d8b7817516\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-jcj6l"
Sep 16 20:07:17.779679 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:17.779662    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e37031a-7901-4244-a94c-05d8b7817516-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-jcj6l\" (UID: \"0e37031a-7901-4244-a94c-05d8b7817516\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-jcj6l"
Sep 16 20:07:17.791171 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:17.791143    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q97zf\" (UniqueName: \"kubernetes.io/projected/0e37031a-7901-4244-a94c-05d8b7817516-kube-api-access-q97zf\") pod \"canary-v2-kserve-7f65dbbccc-jcj6l\" (UID: \"0e37031a-7901-4244-a94c-05d8b7817516\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-jcj6l"
Sep 16 20:07:17.914503 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:17.914400    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-jcj6l"
Sep 16 20:07:18.193231 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:18.193202    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-jcj6l"]
Sep 16 20:07:18.195386 ip-10-0-139-81 kubenswrapper[2557]: W0916 20:07:18.195357    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e37031a_7901_4244_a94c_05d8b7817516.slice/crio-a2ea859160f297a7d3536b5de4997833b5f941cf95a52833b0deb13be1038189 WatchSource:0}: Error finding container a2ea859160f297a7d3536b5de4997833b5f941cf95a52833b0deb13be1038189: Status 404 returned error can't find the container with id a2ea859160f297a7d3536b5de4997833b5f941cf95a52833b0deb13be1038189
Sep 16 20:07:18.198590 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:18.197781    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 20:07:19.030672 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:19.030632    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-jcj6l" event={"ID":"0e37031a-7901-4244-a94c-05d8b7817516","Type":"ContainerStarted","Data":"a2ea859160f297a7d3536b5de4997833b5f941cf95a52833b0deb13be1038189"}
Sep 16 20:07:20.037669 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:20.037632    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-jcj6l" event={"ID":"0e37031a-7901-4244-a94c-05d8b7817516","Type":"ContainerStarted","Data":"538a7736bbd2925ab8f0bca97997e032b602c9a435c0dbf918b377065d10b52c"}
Sep 16 20:07:20.059365 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:20.059309    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-jcj6l" podStartSLOduration=1.965889402 podStartE2EDuration="3.059292376s" totalImagesPullingTime="1.093402974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:07:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:07:20.057063921 +0000 UTC m=+1668.582452308" watchObservedRunningTime="2026-09-16 20:07:20.059292376 +0000 UTC m=+1668.584680760"
Sep 16 20:07:20.915541 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:20.915503    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-jcj6l"
Sep 16 20:07:27.915406 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:27.915362    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-jcj6l"
Sep 16 20:07:27.928119 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:27.928087    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-jcj6l"
Sep 16 20:07:28.088871 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:28.087915    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-jcj6l"
Sep 16 20:07:47.730173 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:47.730136    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched78tsf"]
Sep 16 20:07:47.732929 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:47.732911    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched78tsf"
Sep 16 20:07:47.735914 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:47.735882    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 16 20:07:47.737225 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:47.737200    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 16 20:07:47.737346 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:47.737206    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 16 20:07:47.737346 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:47.737211    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-4j5x2\""
Sep 16 20:07:47.737346 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:47.737300    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-epp-sa-dockercfg-l5cx6\""
Sep 16 20:07:47.747029 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:47.747005    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched78tsf"]
Sep 16 20:07:47.864717 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:47.864675    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d88970af-9096-4c36-b7f2-6a3df4fb1a34-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched78tsf\" (UID: \"d88970af-9096-4c36-b7f2-6a3df4fb1a34\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched78tsf"
Sep 16 20:07:47.864929 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:47.864757    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks8c7\" (UniqueName: \"kubernetes.io/projected/d88970af-9096-4c36-b7f2-6a3df4fb1a34-kube-api-access-ks8c7\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched78tsf\" (UID: \"d88970af-9096-4c36-b7f2-6a3df4fb1a34\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched78tsf"
Sep 16 20:07:47.965523 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:47.965489    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d88970af-9096-4c36-b7f2-6a3df4fb1a34-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched78tsf\" (UID: \"d88970af-9096-4c36-b7f2-6a3df4fb1a34\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched78tsf"
Sep 16 20:07:47.965710 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:47.965561    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks8c7\" (UniqueName: \"kubernetes.io/projected/d88970af-9096-4c36-b7f2-6a3df4fb1a34-kube-api-access-ks8c7\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched78tsf\" (UID: \"d88970af-9096-4c36-b7f2-6a3df4fb1a34\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched78tsf"
Sep 16 20:07:47.968158 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:47.968129    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d88970af-9096-4c36-b7f2-6a3df4fb1a34-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched78tsf\" (UID: \"d88970af-9096-4c36-b7f2-6a3df4fb1a34\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched78tsf"
Sep 16 20:07:47.975356 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:47.975327    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks8c7\" (UniqueName: \"kubernetes.io/projected/d88970af-9096-4c36-b7f2-6a3df4fb1a34-kube-api-access-ks8c7\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched78tsf\" (UID: \"d88970af-9096-4c36-b7f2-6a3df4fb1a34\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched78tsf"
Sep 16 20:07:48.043467 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:48.043420    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched78tsf"
Sep 16 20:07:48.318533 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:48.318510    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched78tsf"]
Sep 16 20:07:48.321682 ip-10-0-139-81 kubenswrapper[2557]: W0916 20:07:48.321647    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd88970af_9096_4c36_b7f2_6a3df4fb1a34.slice/crio-0ee373f1fcdd49d4d236344c39cdd7a3a852a2d12542fe8d519fe8e707b0af30 WatchSource:0}: Error finding container 0ee373f1fcdd49d4d236344c39cdd7a3a852a2d12542fe8d519fe8e707b0af30: Status 404 returned error can't find the container with id 0ee373f1fcdd49d4d236344c39cdd7a3a852a2d12542fe8d519fe8e707b0af30
Sep 16 20:07:49.178429 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:49.178391    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched78tsf" event={"ID":"d88970af-9096-4c36-b7f2-6a3df4fb1a34","Type":"ContainerStarted","Data":"66edada6cd497527c6ede3bcd5031a0805be99c9037687a1c6f355d7ce528638"}
Sep 16 20:07:49.178852 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:49.178444    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched78tsf" event={"ID":"d88970af-9096-4c36-b7f2-6a3df4fb1a34","Type":"ContainerStarted","Data":"0ee373f1fcdd49d4d236344c39cdd7a3a852a2d12542fe8d519fe8e707b0af30"}
Sep 16 20:07:49.178852 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:49.178509    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched78tsf"
Sep 16 20:07:49.197040 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:07:49.196985    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched78tsf" podStartSLOduration=2.196967628 podStartE2EDuration="2.196967628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:07:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:07:49.195775912 +0000 UTC m=+1697.721164310" watchObservedRunningTime="2026-09-16 20:07:49.196967628 +0000 UTC m=+1697.722356013"
Sep 16 20:08:20.185820 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:08:20.185788    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched78tsf"
Sep 16 20:08:28.651461 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:08:28.651410    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched78tsf"]
Sep 16 20:08:30.185808 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:08:30.185766    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched78tsf" podUID="d88970af-9096-4c36-b7f2-6a3df4fb1a34" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:08:40.185499 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:08:40.185458    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched78tsf" podUID="d88970af-9096-4c36-b7f2-6a3df4fb1a34" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:08:43.671437 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:08:43.671366    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched78tsf" podUID="d88970af-9096-4c36-b7f2-6a3df4fb1a34" containerName="main" containerID="cri-o://66edada6cd497527c6ede3bcd5031a0805be99c9037687a1c6f355d7ce528638" gracePeriod=45
Sep 16 20:08:50.185734 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:08:50.185695    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched78tsf" podUID="d88970af-9096-4c36-b7f2-6a3df4fb1a34" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:08:50.186158 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:08:50.185806    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched78tsf"
Sep 16 20:09:00.185737 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:00.185696    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched78tsf" podUID="d88970af-9096-4c36-b7f2-6a3df4fb1a34" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:09:10.185832 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:10.185792    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched78tsf" podUID="d88970af-9096-4c36-b7f2-6a3df4fb1a34" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:09:13.916819 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:13.916792    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched78tsf"
Sep 16 20:09:14.061382 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:14.061342    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d88970af-9096-4c36-b7f2-6a3df4fb1a34-tls-certs\") pod \"d88970af-9096-4c36-b7f2-6a3df4fb1a34\" (UID: \"d88970af-9096-4c36-b7f2-6a3df4fb1a34\") "
Sep 16 20:09:14.061382 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:14.061385    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ks8c7\" (UniqueName: \"kubernetes.io/projected/d88970af-9096-4c36-b7f2-6a3df4fb1a34-kube-api-access-ks8c7\") pod \"d88970af-9096-4c36-b7f2-6a3df4fb1a34\" (UID: \"d88970af-9096-4c36-b7f2-6a3df4fb1a34\") "
Sep 16 20:09:14.063697 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:14.063665    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d88970af-9096-4c36-b7f2-6a3df4fb1a34-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d88970af-9096-4c36-b7f2-6a3df4fb1a34" (UID: "d88970af-9096-4c36-b7f2-6a3df4fb1a34"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:09:14.063810 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:14.063700    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d88970af-9096-4c36-b7f2-6a3df4fb1a34-kube-api-access-ks8c7" (OuterVolumeSpecName: "kube-api-access-ks8c7") pod "d88970af-9096-4c36-b7f2-6a3df4fb1a34" (UID: "d88970af-9096-4c36-b7f2-6a3df4fb1a34"). InnerVolumeSpecName "kube-api-access-ks8c7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:09:14.162946 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:14.162910    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d88970af-9096-4c36-b7f2-6a3df4fb1a34-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:09:14.162946 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:14.162941    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks8c7\" (UniqueName: \"kubernetes.io/projected/d88970af-9096-4c36-b7f2-6a3df4fb1a34-kube-api-access-ks8c7\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:09:14.613206 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:14.613174    2557 generic.go:356] "Generic (PLEG): container finished" podID="d88970af-9096-4c36-b7f2-6a3df4fb1a34" containerID="66edada6cd497527c6ede3bcd5031a0805be99c9037687a1c6f355d7ce528638" exitCode=0
Sep 16 20:09:14.613394 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:14.613213    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched78tsf" event={"ID":"d88970af-9096-4c36-b7f2-6a3df4fb1a34","Type":"ContainerDied","Data":"66edada6cd497527c6ede3bcd5031a0805be99c9037687a1c6f355d7ce528638"}
Sep 16 20:09:14.613394 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:14.613221    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched78tsf"
Sep 16 20:09:14.613394 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:14.613256    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched78tsf" event={"ID":"d88970af-9096-4c36-b7f2-6a3df4fb1a34","Type":"ContainerDied","Data":"0ee373f1fcdd49d4d236344c39cdd7a3a852a2d12542fe8d519fe8e707b0af30"}
Sep 16 20:09:14.613394 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:14.613277    2557 scope.go:117] "RemoveContainer" containerID="66edada6cd497527c6ede3bcd5031a0805be99c9037687a1c6f355d7ce528638"
Sep 16 20:09:14.622790 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:14.622773    2557 scope.go:117] "RemoveContainer" containerID="66edada6cd497527c6ede3bcd5031a0805be99c9037687a1c6f355d7ce528638"
Sep 16 20:09:14.623062 ip-10-0-139-81 kubenswrapper[2557]: E0916 20:09:14.623038    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66edada6cd497527c6ede3bcd5031a0805be99c9037687a1c6f355d7ce528638\": container with ID starting with 66edada6cd497527c6ede3bcd5031a0805be99c9037687a1c6f355d7ce528638 not found: ID does not exist" containerID="66edada6cd497527c6ede3bcd5031a0805be99c9037687a1c6f355d7ce528638"
Sep 16 20:09:14.623146 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:14.623072    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66edada6cd497527c6ede3bcd5031a0805be99c9037687a1c6f355d7ce528638"} err="failed to get container status \"66edada6cd497527c6ede3bcd5031a0805be99c9037687a1c6f355d7ce528638\": rpc error: code = NotFound desc = could not find container \"66edada6cd497527c6ede3bcd5031a0805be99c9037687a1c6f355d7ce528638\": container with ID starting with 66edada6cd497527c6ede3bcd5031a0805be99c9037687a1c6f355d7ce528638 not found: ID does not exist"
Sep 16 20:09:14.635152 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:14.635125    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched78tsf"]
Sep 16 20:09:14.638657 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:14.638634    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched78tsf"]
Sep 16 20:09:15.937153 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:15.937119    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d88970af-9096-4c36-b7f2-6a3df4fb1a34" path="/var/lib/kubelet/pods/d88970af-9096-4c36-b7f2-6a3df4fb1a34/volumes"
Sep 16 20:09:25.809673 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:25.809585    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-jcj6l"]
Sep 16 20:09:39.000464 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:39.000417    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-6856648bm6znh"]
Sep 16 20:09:39.001059 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:39.001038    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d88970af-9096-4c36-b7f2-6a3df4fb1a34" containerName="main"
Sep 16 20:09:39.001151 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:39.001061    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d88970af-9096-4c36-b7f2-6a3df4fb1a34" containerName="main"
Sep 16 20:09:39.001206 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:39.001179    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d88970af-9096-4c36-b7f2-6a3df4fb1a34" containerName="main"
Sep 16 20:09:39.004405 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:39.004380    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-6856648bm6znh"
Sep 16 20:09:39.008681 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:39.008654    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-r7qzz\""
Sep 16 20:09:39.008681 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:39.008672    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 16 20:09:39.008897 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:39.008736    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 16 20:09:39.008897 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:39.008744    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-epp-sa-dockercfg-x58cc\""
Sep 16 20:09:39.008897 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:39.008741    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 16 20:09:39.016759 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:39.016734    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-6856648bm6znh"]
Sep 16 20:09:39.096426 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:39.096385    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq2l2\" (UniqueName: \"kubernetes.io/projected/8c99a776-241e-4502-922b-58f5cfe73bf4-kube-api-access-gq2l2\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-6856648bm6znh\" (UID: \"8c99a776-241e-4502-922b-58f5cfe73bf4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-6856648bm6znh"
Sep 16 20:09:39.096595 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:39.096501    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c99a776-241e-4502-922b-58f5cfe73bf4-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-6856648bm6znh\" (UID: \"8c99a776-241e-4502-922b-58f5cfe73bf4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-6856648bm6znh"
Sep 16 20:09:39.198040 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:39.197995    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq2l2\" (UniqueName: \"kubernetes.io/projected/8c99a776-241e-4502-922b-58f5cfe73bf4-kube-api-access-gq2l2\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-6856648bm6znh\" (UID: \"8c99a776-241e-4502-922b-58f5cfe73bf4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-6856648bm6znh"
Sep 16 20:09:39.198221 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:39.198056    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c99a776-241e-4502-922b-58f5cfe73bf4-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-6856648bm6znh\" (UID: \"8c99a776-241e-4502-922b-58f5cfe73bf4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-6856648bm6znh"
Sep 16 20:09:39.200809 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:39.200772    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c99a776-241e-4502-922b-58f5cfe73bf4-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-6856648bm6znh\" (UID: \"8c99a776-241e-4502-922b-58f5cfe73bf4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-6856648bm6znh"
Sep 16 20:09:39.207563 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:39.207533    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq2l2\" (UniqueName: \"kubernetes.io/projected/8c99a776-241e-4502-922b-58f5cfe73bf4-kube-api-access-gq2l2\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-6856648bm6znh\" (UID: \"8c99a776-241e-4502-922b-58f5cfe73bf4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-6856648bm6znh"
Sep 16 20:09:39.317947 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:39.317821    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-6856648bm6znh"
Sep 16 20:09:39.591072 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:39.591042    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-6856648bm6znh"]
Sep 16 20:09:39.593210 ip-10-0-139-81 kubenswrapper[2557]: W0916 20:09:39.593182    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c99a776_241e_4502_922b_58f5cfe73bf4.slice/crio-2a1ea80f0c75486517027409d714b7e87ed43984b3c03a77a0af75a7299fef58 WatchSource:0}: Error finding container 2a1ea80f0c75486517027409d714b7e87ed43984b3c03a77a0af75a7299fef58: Status 404 returned error can't find the container with id 2a1ea80f0c75486517027409d714b7e87ed43984b3c03a77a0af75a7299fef58
Sep 16 20:09:39.751067 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:39.751029    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-6856648bm6znh" event={"ID":"8c99a776-241e-4502-922b-58f5cfe73bf4","Type":"ContainerStarted","Data":"a8760d47db8114e9ab8e035241c03d050d7360ba0324f2697bd3aded66415406"}
Sep 16 20:09:39.751256 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:39.751091    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-6856648bm6znh" event={"ID":"8c99a776-241e-4502-922b-58f5cfe73bf4","Type":"ContainerStarted","Data":"2a1ea80f0c75486517027409d714b7e87ed43984b3c03a77a0af75a7299fef58"}
Sep 16 20:09:39.751256 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:39.751172    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-6856648bm6znh"
Sep 16 20:09:39.770139 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:39.770081    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-6856648bm6znh" podStartSLOduration=1.770064691 podStartE2EDuration="1.770064691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:09:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:09:39.767921926 +0000 UTC m=+1808.293310314" watchObservedRunningTime="2026-09-16 20:09:39.770064691 +0000 UTC m=+1808.295453077"
Sep 16 20:09:40.829829 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:40.829765    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-jcj6l" podUID="0e37031a-7901-4244-a94c-05d8b7817516" containerName="main" containerID="cri-o://538a7736bbd2925ab8f0bca97997e032b602c9a435c0dbf918b377065d10b52c" gracePeriod=45
Sep 16 20:09:41.115999 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:41.115973    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-jcj6l"
Sep 16 20:09:41.217589 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:41.217553    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e37031a-7901-4244-a94c-05d8b7817516-dshm\") pod \"0e37031a-7901-4244-a94c-05d8b7817516\" (UID: \"0e37031a-7901-4244-a94c-05d8b7817516\") "
Sep 16 20:09:41.217778 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:41.217656    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e37031a-7901-4244-a94c-05d8b7817516-home\") pod \"0e37031a-7901-4244-a94c-05d8b7817516\" (UID: \"0e37031a-7901-4244-a94c-05d8b7817516\") "
Sep 16 20:09:41.217778 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:41.217675    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e37031a-7901-4244-a94c-05d8b7817516-model-cache\") pod \"0e37031a-7901-4244-a94c-05d8b7817516\" (UID: \"0e37031a-7901-4244-a94c-05d8b7817516\") "
Sep 16 20:09:41.217778 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:41.217722    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q97zf\" (UniqueName: \"kubernetes.io/projected/0e37031a-7901-4244-a94c-05d8b7817516-kube-api-access-q97zf\") pod \"0e37031a-7901-4244-a94c-05d8b7817516\" (UID: \"0e37031a-7901-4244-a94c-05d8b7817516\") "
Sep 16 20:09:41.217982 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:41.217784    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e37031a-7901-4244-a94c-05d8b7817516-tmp-dir\") pod \"0e37031a-7901-4244-a94c-05d8b7817516\" (UID: \"0e37031a-7901-4244-a94c-05d8b7817516\") "
Sep 16 20:09:41.217982 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:41.217808    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e37031a-7901-4244-a94c-05d8b7817516-tls-certs\") pod \"0e37031a-7901-4244-a94c-05d8b7817516\" (UID: \"0e37031a-7901-4244-a94c-05d8b7817516\") "
Sep 16 20:09:41.218103 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:41.218002    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e37031a-7901-4244-a94c-05d8b7817516-home" (OuterVolumeSpecName: "home") pod "0e37031a-7901-4244-a94c-05d8b7817516" (UID: "0e37031a-7901-4244-a94c-05d8b7817516"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:09:41.218103 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:41.218037    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e37031a-7901-4244-a94c-05d8b7817516-model-cache" (OuterVolumeSpecName: "model-cache") pod "0e37031a-7901-4244-a94c-05d8b7817516" (UID: "0e37031a-7901-4244-a94c-05d8b7817516"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:09:41.218220 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:41.218132    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e37031a-7901-4244-a94c-05d8b7817516-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0e37031a-7901-4244-a94c-05d8b7817516" (UID: "0e37031a-7901-4244-a94c-05d8b7817516"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:09:41.218220 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:41.218164    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e37031a-7901-4244-a94c-05d8b7817516-home\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:09:41.218220 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:41.218179    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e37031a-7901-4244-a94c-05d8b7817516-model-cache\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:09:41.220214 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:41.220181    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e37031a-7901-4244-a94c-05d8b7817516-dshm" (OuterVolumeSpecName: "dshm") pod "0e37031a-7901-4244-a94c-05d8b7817516" (UID: "0e37031a-7901-4244-a94c-05d8b7817516"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:09:41.220355 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:41.220246    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e37031a-7901-4244-a94c-05d8b7817516-kube-api-access-q97zf" (OuterVolumeSpecName: "kube-api-access-q97zf") pod "0e37031a-7901-4244-a94c-05d8b7817516" (UID: "0e37031a-7901-4244-a94c-05d8b7817516"). InnerVolumeSpecName "kube-api-access-q97zf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:09:41.220355 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:41.220284    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e37031a-7901-4244-a94c-05d8b7817516-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0e37031a-7901-4244-a94c-05d8b7817516" (UID: "0e37031a-7901-4244-a94c-05d8b7817516"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:09:41.319286 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:41.319247    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q97zf\" (UniqueName: \"kubernetes.io/projected/0e37031a-7901-4244-a94c-05d8b7817516-kube-api-access-q97zf\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:09:41.319286 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:41.319283    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e37031a-7901-4244-a94c-05d8b7817516-tmp-dir\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:09:41.319286 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:41.319293    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e37031a-7901-4244-a94c-05d8b7817516-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:09:41.319544 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:41.319301    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e37031a-7901-4244-a94c-05d8b7817516-dshm\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:09:41.763978 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:41.763942    2557 generic.go:356] "Generic (PLEG): container finished" podID="0e37031a-7901-4244-a94c-05d8b7817516" containerID="538a7736bbd2925ab8f0bca97997e032b602c9a435c0dbf918b377065d10b52c" exitCode=0
Sep 16 20:09:41.764165 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:41.764003    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-jcj6l"
Sep 16 20:09:41.764165 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:41.764019    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-jcj6l" event={"ID":"0e37031a-7901-4244-a94c-05d8b7817516","Type":"ContainerDied","Data":"538a7736bbd2925ab8f0bca97997e032b602c9a435c0dbf918b377065d10b52c"}
Sep 16 20:09:41.764165 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:41.764105    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-jcj6l" event={"ID":"0e37031a-7901-4244-a94c-05d8b7817516","Type":"ContainerDied","Data":"a2ea859160f297a7d3536b5de4997833b5f941cf95a52833b0deb13be1038189"}
Sep 16 20:09:41.764165 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:41.764132    2557 scope.go:117] "RemoveContainer" containerID="538a7736bbd2925ab8f0bca97997e032b602c9a435c0dbf918b377065d10b52c"
Sep 16 20:09:41.775237 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:41.775212    2557 scope.go:117] "RemoveContainer" containerID="538a7736bbd2925ab8f0bca97997e032b602c9a435c0dbf918b377065d10b52c"
Sep 16 20:09:41.775722 ip-10-0-139-81 kubenswrapper[2557]: E0916 20:09:41.775688    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"538a7736bbd2925ab8f0bca97997e032b602c9a435c0dbf918b377065d10b52c\": container with ID starting with 538a7736bbd2925ab8f0bca97997e032b602c9a435c0dbf918b377065d10b52c not found: ID does not exist" containerID="538a7736bbd2925ab8f0bca97997e032b602c9a435c0dbf918b377065d10b52c"
Sep 16 20:09:41.775816 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:41.775739    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"538a7736bbd2925ab8f0bca97997e032b602c9a435c0dbf918b377065d10b52c"} err="failed to get container status \"538a7736bbd2925ab8f0bca97997e032b602c9a435c0dbf918b377065d10b52c\": rpc error: code = NotFound desc = could not find container \"538a7736bbd2925ab8f0bca97997e032b602c9a435c0dbf918b377065d10b52c\": container with ID starting with 538a7736bbd2925ab8f0bca97997e032b602c9a435c0dbf918b377065d10b52c not found: ID does not exist"
Sep 16 20:09:41.789490 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:41.789454    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-jcj6l"]
Sep 16 20:09:41.793073 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:41.793043    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-jcj6l"]
Sep 16 20:09:41.937960 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:41.937927    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e37031a-7901-4244-a94c-05d8b7817516" path="/var/lib/kubelet/pods/0e37031a-7901-4244-a94c-05d8b7817516/volumes"
Sep 16 20:09:42.076325 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:09:42.076218    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-jcj6l" podUID="0e37031a-7901-4244-a94c-05d8b7817516" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 16 20:10:10.758187 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:10:10.758151    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-6856648bm6znh"
Sep 16 20:11:05.501938 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:11:05.501849    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-6856648bm6znh"]
Sep 16 20:11:10.758419 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:11:10.758376    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-6856648bm6znh" podUID="8c99a776-241e-4502-922b-58f5cfe73bf4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:11:20.523728 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:11:20.523684    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-6856648bm6znh" podUID="8c99a776-241e-4502-922b-58f5cfe73bf4" containerName="main" containerID="cri-o://a8760d47db8114e9ab8e035241c03d050d7360ba0324f2697bd3aded66415406" gracePeriod=45
Sep 16 20:11:20.759119 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:11:20.759076    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-6856648bm6znh" podUID="8c99a776-241e-4502-922b-58f5cfe73bf4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:11:30.758441 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:11:30.758400    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-6856648bm6znh" podUID="8c99a776-241e-4502-922b-58f5cfe73bf4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:11:30.758835 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:11:30.758514    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-6856648bm6znh"
Sep 16 20:11:40.758492 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:11:40.758452    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-6856648bm6znh" podUID="8c99a776-241e-4502-922b-58f5cfe73bf4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:11:50.771691 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:11:50.771662    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-6856648bm6znh"
Sep 16 20:11:50.911563 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:11:50.911477    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gq2l2\" (UniqueName: \"kubernetes.io/projected/8c99a776-241e-4502-922b-58f5cfe73bf4-kube-api-access-gq2l2\") pod \"8c99a776-241e-4502-922b-58f5cfe73bf4\" (UID: \"8c99a776-241e-4502-922b-58f5cfe73bf4\") "
Sep 16 20:11:50.911563 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:11:50.911542    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c99a776-241e-4502-922b-58f5cfe73bf4-tls-certs\") pod \"8c99a776-241e-4502-922b-58f5cfe73bf4\" (UID: \"8c99a776-241e-4502-922b-58f5cfe73bf4\") "
Sep 16 20:11:50.913808 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:11:50.913769    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c99a776-241e-4502-922b-58f5cfe73bf4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8c99a776-241e-4502-922b-58f5cfe73bf4" (UID: "8c99a776-241e-4502-922b-58f5cfe73bf4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:11:50.913946 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:11:50.913808    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c99a776-241e-4502-922b-58f5cfe73bf4-kube-api-access-gq2l2" (OuterVolumeSpecName: "kube-api-access-gq2l2") pod "8c99a776-241e-4502-922b-58f5cfe73bf4" (UID: "8c99a776-241e-4502-922b-58f5cfe73bf4"). InnerVolumeSpecName "kube-api-access-gq2l2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:11:51.012560 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:11:51.012515    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq2l2\" (UniqueName: \"kubernetes.io/projected/8c99a776-241e-4502-922b-58f5cfe73bf4-kube-api-access-gq2l2\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:11:51.012560 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:11:51.012558    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c99a776-241e-4502-922b-58f5cfe73bf4-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:11:51.420728 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:11:51.420697    2557 generic.go:356] "Generic (PLEG): container finished" podID="8c99a776-241e-4502-922b-58f5cfe73bf4" containerID="a8760d47db8114e9ab8e035241c03d050d7360ba0324f2697bd3aded66415406" exitCode=0
Sep 16 20:11:51.420925 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:11:51.420739    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-6856648bm6znh"
Sep 16 20:11:51.420925 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:11:51.420774    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-6856648bm6znh" event={"ID":"8c99a776-241e-4502-922b-58f5cfe73bf4","Type":"ContainerDied","Data":"a8760d47db8114e9ab8e035241c03d050d7360ba0324f2697bd3aded66415406"}
Sep 16 20:11:51.420925 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:11:51.420830    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-6856648bm6znh" event={"ID":"8c99a776-241e-4502-922b-58f5cfe73bf4","Type":"ContainerDied","Data":"2a1ea80f0c75486517027409d714b7e87ed43984b3c03a77a0af75a7299fef58"}
Sep 16 20:11:51.420925 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:11:51.420864    2557 scope.go:117] "RemoveContainer" containerID="a8760d47db8114e9ab8e035241c03d050d7360ba0324f2697bd3aded66415406"
Sep 16 20:11:51.431376 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:11:51.431357    2557 scope.go:117] "RemoveContainer" containerID="a8760d47db8114e9ab8e035241c03d050d7360ba0324f2697bd3aded66415406"
Sep 16 20:11:51.433965 ip-10-0-139-81 kubenswrapper[2557]: E0916 20:11:51.433935    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8760d47db8114e9ab8e035241c03d050d7360ba0324f2697bd3aded66415406\": container with ID starting with a8760d47db8114e9ab8e035241c03d050d7360ba0324f2697bd3aded66415406 not found: ID does not exist" containerID="a8760d47db8114e9ab8e035241c03d050d7360ba0324f2697bd3aded66415406"
Sep 16 20:11:51.434096 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:11:51.433976    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8760d47db8114e9ab8e035241c03d050d7360ba0324f2697bd3aded66415406"} err="failed to get container status \"a8760d47db8114e9ab8e035241c03d050d7360ba0324f2697bd3aded66415406\": rpc error: code = NotFound desc = could not find container \"a8760d47db8114e9ab8e035241c03d050d7360ba0324f2697bd3aded66415406\": container with ID starting with a8760d47db8114e9ab8e035241c03d050d7360ba0324f2697bd3aded66415406 not found: ID does not exist"
Sep 16 20:11:51.455396 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:11:51.455366    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-6856648bm6znh"]
Sep 16 20:11:51.461854 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:11:51.461815    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-6856648bm6znh"]
Sep 16 20:11:51.757084 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:11:51.757032    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-6856648bm6znh" podUID="8c99a776-241e-4502-922b-58f5cfe73bf4" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.134.0.49:9003\" within 1s: context deadline exceeded"
Sep 16 20:11:51.757262 ip-10-0-139-81 kubenswrapper[2557]: W0916 20:11:51.757174    2557 logging.go:55] [core] [Channel #371 SubChannel #372]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.49:9003", ServerName: "10.134.0.49:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.49:9003: operation was canceled"
Sep 16 20:11:51.939232 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:11:51.939199    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c99a776-241e-4502-922b-58f5cfe73bf4" path="/var/lib/kubelet/pods/8c99a776-241e-4502-922b-58f5cfe73bf4/volumes"
Sep 16 20:11:59.894052 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:11:59.894014    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zq9j9"]
Sep 16 20:11:59.894434 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:11:59.894389    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c99a776-241e-4502-922b-58f5cfe73bf4" containerName="main"
Sep 16 20:11:59.894434 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:11:59.894400    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c99a776-241e-4502-922b-58f5cfe73bf4" containerName="main"
Sep 16 20:11:59.894434 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:11:59.894411    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e37031a-7901-4244-a94c-05d8b7817516" containerName="main"
Sep 16 20:11:59.894434 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:11:59.894416    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e37031a-7901-4244-a94c-05d8b7817516" containerName="main"
Sep 16 20:11:59.894577 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:11:59.894487    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e37031a-7901-4244-a94c-05d8b7817516" containerName="main"
Sep 16 20:11:59.894577 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:11:59.894500    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c99a776-241e-4502-922b-58f5cfe73bf4" containerName="main"
Sep 16 20:11:59.898966 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:11:59.898941    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zq9j9"
Sep 16 20:11:59.902119 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:11:59.902090    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"kube-root-ca.crt\""
Sep 16 20:11:59.903495 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:11:59.903470    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"openshift-service-ca.crt\""
Sep 16 20:11:59.903601 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:11:59.903523    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"stop-v1-kserve-self-signed-certs\""
Sep 16 20:11:59.903601 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:11:59.903589    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"default-dockercfg-rxfcp\""
Sep 16 20:11:59.908496 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:11:59.908473    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zq9j9"]
Sep 16 20:11:59.996477 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:11:59.996424    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8zdk\" (UniqueName: \"kubernetes.io/projected/0f77d2a0-45f0-4cdc-bca6-dedc79043506-kube-api-access-h8zdk\") pod \"stop-v1-kserve-5657b88c95-zq9j9\" (UID: \"0f77d2a0-45f0-4cdc-bca6-dedc79043506\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zq9j9"
Sep 16 20:11:59.996667 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:11:59.996540    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f77d2a0-45f0-4cdc-bca6-dedc79043506-dshm\") pod \"stop-v1-kserve-5657b88c95-zq9j9\" (UID: \"0f77d2a0-45f0-4cdc-bca6-dedc79043506\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zq9j9"
Sep 16 20:11:59.996667 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:11:59.996571    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f77d2a0-45f0-4cdc-bca6-dedc79043506-tmp-dir\") pod \"stop-v1-kserve-5657b88c95-zq9j9\" (UID: \"0f77d2a0-45f0-4cdc-bca6-dedc79043506\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zq9j9"
Sep 16 20:11:59.996667 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:11:59.996643    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f77d2a0-45f0-4cdc-bca6-dedc79043506-tls-certs\") pod \"stop-v1-kserve-5657b88c95-zq9j9\" (UID: \"0f77d2a0-45f0-4cdc-bca6-dedc79043506\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zq9j9"
Sep 16 20:11:59.996802 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:11:59.996704    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f77d2a0-45f0-4cdc-bca6-dedc79043506-home\") pod \"stop-v1-kserve-5657b88c95-zq9j9\" (UID: \"0f77d2a0-45f0-4cdc-bca6-dedc79043506\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zq9j9"
Sep 16 20:11:59.996802 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:11:59.996736    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f77d2a0-45f0-4cdc-bca6-dedc79043506-model-cache\") pod \"stop-v1-kserve-5657b88c95-zq9j9\" (UID: \"0f77d2a0-45f0-4cdc-bca6-dedc79043506\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zq9j9"
Sep 16 20:12:00.097646 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:12:00.097612    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f77d2a0-45f0-4cdc-bca6-dedc79043506-tls-certs\") pod \"stop-v1-kserve-5657b88c95-zq9j9\" (UID: \"0f77d2a0-45f0-4cdc-bca6-dedc79043506\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zq9j9"
Sep 16 20:12:00.097811 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:12:00.097658    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f77d2a0-45f0-4cdc-bca6-dedc79043506-home\") pod \"stop-v1-kserve-5657b88c95-zq9j9\" (UID: \"0f77d2a0-45f0-4cdc-bca6-dedc79043506\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zq9j9"
Sep 16 20:12:00.097811 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:12:00.097677    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f77d2a0-45f0-4cdc-bca6-dedc79043506-model-cache\") pod \"stop-v1-kserve-5657b88c95-zq9j9\" (UID: \"0f77d2a0-45f0-4cdc-bca6-dedc79043506\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zq9j9"
Sep 16 20:12:00.097811 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:12:00.097724    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8zdk\" (UniqueName: \"kubernetes.io/projected/0f77d2a0-45f0-4cdc-bca6-dedc79043506-kube-api-access-h8zdk\") pod \"stop-v1-kserve-5657b88c95-zq9j9\" (UID: \"0f77d2a0-45f0-4cdc-bca6-dedc79043506\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zq9j9"
Sep 16 20:12:00.097811 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:12:00.097777    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f77d2a0-45f0-4cdc-bca6-dedc79043506-dshm\") pod \"stop-v1-kserve-5657b88c95-zq9j9\" (UID: \"0f77d2a0-45f0-4cdc-bca6-dedc79043506\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zq9j9"
Sep 16 20:12:00.097811 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:12:00.097798    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f77d2a0-45f0-4cdc-bca6-dedc79043506-tmp-dir\") pod \"stop-v1-kserve-5657b88c95-zq9j9\" (UID: \"0f77d2a0-45f0-4cdc-bca6-dedc79043506\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zq9j9"
Sep 16 20:12:00.098136 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:12:00.098108    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f77d2a0-45f0-4cdc-bca6-dedc79043506-home\") pod \"stop-v1-kserve-5657b88c95-zq9j9\" (UID: \"0f77d2a0-45f0-4cdc-bca6-dedc79043506\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zq9j9"
Sep 16 20:12:00.098272 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:12:00.098181    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f77d2a0-45f0-4cdc-bca6-dedc79043506-model-cache\") pod \"stop-v1-kserve-5657b88c95-zq9j9\" (UID: \"0f77d2a0-45f0-4cdc-bca6-dedc79043506\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zq9j9"
Sep 16 20:12:00.098272 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:12:00.098201    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f77d2a0-45f0-4cdc-bca6-dedc79043506-tmp-dir\") pod \"stop-v1-kserve-5657b88c95-zq9j9\" (UID: \"0f77d2a0-45f0-4cdc-bca6-dedc79043506\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zq9j9"
Sep 16 20:12:00.100002 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:12:00.099974    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f77d2a0-45f0-4cdc-bca6-dedc79043506-dshm\") pod \"stop-v1-kserve-5657b88c95-zq9j9\" (UID: \"0f77d2a0-45f0-4cdc-bca6-dedc79043506\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zq9j9"
Sep 16 20:12:00.100168 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:12:00.100151    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f77d2a0-45f0-4cdc-bca6-dedc79043506-tls-certs\") pod \"stop-v1-kserve-5657b88c95-zq9j9\" (UID: \"0f77d2a0-45f0-4cdc-bca6-dedc79043506\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zq9j9"
Sep 16 20:12:00.106540 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:12:00.106516    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8zdk\" (UniqueName: \"kubernetes.io/projected/0f77d2a0-45f0-4cdc-bca6-dedc79043506-kube-api-access-h8zdk\") pod \"stop-v1-kserve-5657b88c95-zq9j9\" (UID: \"0f77d2a0-45f0-4cdc-bca6-dedc79043506\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zq9j9"
Sep 16 20:12:00.210358 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:12:00.210276    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zq9j9"
Sep 16 20:12:00.492240 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:12:00.492199    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zq9j9"]
Sep 16 20:12:00.494828 ip-10-0-139-81 kubenswrapper[2557]: W0916 20:12:00.494801    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f77d2a0_45f0_4cdc_bca6_dedc79043506.slice/crio-57618f069621df462b6c04fc17c01deb9f2bc0c495fc08e92a37a6cb3114544c WatchSource:0}: Error finding container 57618f069621df462b6c04fc17c01deb9f2bc0c495fc08e92a37a6cb3114544c: Status 404 returned error can't find the container with id 57618f069621df462b6c04fc17c01deb9f2bc0c495fc08e92a37a6cb3114544c
Sep 16 20:12:01.473554 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:12:01.473518    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zq9j9" event={"ID":"0f77d2a0-45f0-4cdc-bca6-dedc79043506","Type":"ContainerStarted","Data":"64a0f5727f62bcbf2022a51edae888d6f57d4de582972073825d03695dbb2471"}
Sep 16 20:12:01.473955 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:12:01.473603    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zq9j9" event={"ID":"0f77d2a0-45f0-4cdc-bca6-dedc79043506","Type":"ContainerStarted","Data":"57618f069621df462b6c04fc17c01deb9f2bc0c495fc08e92a37a6cb3114544c"}
Sep 16 20:12:01.491634 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:12:01.491580    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zq9j9" podStartSLOduration=2.49156513 podStartE2EDuration="2.49156513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:11:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:12:01.48979231 +0000 UTC m=+1950.015180698" watchObservedRunningTime="2026-09-16 20:12:01.49156513 +0000 UTC m=+1950.016953516"
Sep 16 20:12:02.211424 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:12:02.211388    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zq9j9"
Sep 16 20:12:10.211336 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:12:10.211290    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zq9j9"
Sep 16 20:12:10.224313 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:12:10.224287    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zq9j9"
Sep 16 20:12:10.530915 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:12:10.530882    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zq9j9"
Sep 16 20:12:35.015342 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:12:35.015308    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-789d76jnz"]
Sep 16 20:12:35.020646 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:12:35.020620    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-789d76jnz"
Sep 16 20:12:35.024307 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:12:35.024281    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 16 20:12:35.025599 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:12:35.025578    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 16 20:12:35.025906 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:12:35.025887    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 16 20:12:35.026264 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:12:35.026235    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-epp-sa-dockercfg-jcwg2\""
Sep 16 20:12:35.026367 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:12:35.026330    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-xzlb2\""
Sep 16 20:12:35.044958 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:12:35.044930    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-789d76jnz"]
Sep 16 20:12:35.136443 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:12:35.136401    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spdpz\" (UniqueName: \"kubernetes.io/projected/2ab23ed1-5683-4037-b4d6-0db2a56bb116-kube-api-access-spdpz\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-789d76jnz\" (UID: \"2ab23ed1-5683-4037-b4d6-0db2a56bb116\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-789d76jnz"
Sep 16 20:12:35.136627 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:12:35.136547    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ab23ed1-5683-4037-b4d6-0db2a56bb116-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-789d76jnz\" (UID: \"2ab23ed1-5683-4037-b4d6-0db2a56bb116\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-789d76jnz"
Sep 16 20:12:35.237989 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:12:35.237950    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ab23ed1-5683-4037-b4d6-0db2a56bb116-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-789d76jnz\" (UID: \"2ab23ed1-5683-4037-b4d6-0db2a56bb116\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-789d76jnz"
Sep 16 20:12:35.238176 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:12:35.238022    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spdpz\" (UniqueName: \"kubernetes.io/projected/2ab23ed1-5683-4037-b4d6-0db2a56bb116-kube-api-access-spdpz\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-789d76jnz\" (UID: \"2ab23ed1-5683-4037-b4d6-0db2a56bb116\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-789d76jnz"
Sep 16 20:12:35.240810 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:12:35.240781    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ab23ed1-5683-4037-b4d6-0db2a56bb116-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-789d76jnz\" (UID: \"2ab23ed1-5683-4037-b4d6-0db2a56bb116\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-789d76jnz"
Sep 16 20:12:35.248323 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:12:35.248294    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spdpz\" (UniqueName: \"kubernetes.io/projected/2ab23ed1-5683-4037-b4d6-0db2a56bb116-kube-api-access-spdpz\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-789d76jnz\" (UID: \"2ab23ed1-5683-4037-b4d6-0db2a56bb116\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-789d76jnz"
Sep 16 20:12:35.332231 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:12:35.332143    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-789d76jnz"
Sep 16 20:12:35.612165 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:12:35.612135    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-789d76jnz"]
Sep 16 20:12:35.614346 ip-10-0-139-81 kubenswrapper[2557]: W0916 20:12:35.614320    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ab23ed1_5683_4037_b4d6_0db2a56bb116.slice/crio-93c8dbb9370e35c410c16b3ffea8da59fb21d4c755929b7fb495f7b6f203a4df WatchSource:0}: Error finding container 93c8dbb9370e35c410c16b3ffea8da59fb21d4c755929b7fb495f7b6f203a4df: Status 404 returned error can't find the container with id 93c8dbb9370e35c410c16b3ffea8da59fb21d4c755929b7fb495f7b6f203a4df
Sep 16 20:12:35.646905 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:12:35.646875    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-789d76jnz" event={"ID":"2ab23ed1-5683-4037-b4d6-0db2a56bb116","Type":"ContainerStarted","Data":"93c8dbb9370e35c410c16b3ffea8da59fb21d4c755929b7fb495f7b6f203a4df"}
Sep 16 20:12:36.653315 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:12:36.653279    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-789d76jnz" event={"ID":"2ab23ed1-5683-4037-b4d6-0db2a56bb116","Type":"ContainerStarted","Data":"e5fa118342c1e8acbf783b75134394a1d6e5323ce072e2ef26f934d6cf94c467"}
Sep 16 20:12:36.653696 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:12:36.653363    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-789d76jnz"
Sep 16 20:12:36.675057 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:12:36.675000    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-789d76jnz" podStartSLOduration=2.674984368 podStartE2EDuration="2.674984368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:12:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:12:36.673458659 +0000 UTC m=+1985.198847047" watchObservedRunningTime="2026-09-16 20:12:36.674984368 +0000 UTC m=+1985.200372753"
Sep 16 20:13:07.665460 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:13:07.665426    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-789d76jnz"
Sep 16 20:13:40.071060 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:13:40.071028    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-789d76jnz"]
Sep 16 20:13:47.665472 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:13:47.665430    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-789d76jnz" podUID="2ab23ed1-5683-4037-b4d6-0db2a56bb116" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:13:55.091224 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:13:55.091129    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-789d76jnz" podUID="2ab23ed1-5683-4037-b4d6-0db2a56bb116" containerName="main" containerID="cri-o://e5fa118342c1e8acbf783b75134394a1d6e5323ce072e2ef26f934d6cf94c467" gracePeriod=45
Sep 16 20:13:57.665748 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:13:57.665709    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-789d76jnz" podUID="2ab23ed1-5683-4037-b4d6-0db2a56bb116" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:14:07.665295 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:07.665254    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-789d76jnz" podUID="2ab23ed1-5683-4037-b4d6-0db2a56bb116" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:14:07.665767 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:07.665389    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-789d76jnz"
Sep 16 20:14:14.473751 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:14.473715    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zq9j9"]
Sep 16 20:14:17.665025 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:17.664984    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-789d76jnz" podUID="2ab23ed1-5683-4037-b4d6-0db2a56bb116" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:14:25.340684 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:25.340654    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-789d76jnz"
Sep 16 20:14:25.354182 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:25.354127    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-spdpz\" (UniqueName: \"kubernetes.io/projected/2ab23ed1-5683-4037-b4d6-0db2a56bb116-kube-api-access-spdpz\") pod \"2ab23ed1-5683-4037-b4d6-0db2a56bb116\" (UID: \"2ab23ed1-5683-4037-b4d6-0db2a56bb116\") "
Sep 16 20:14:25.354182 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:25.354173    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ab23ed1-5683-4037-b4d6-0db2a56bb116-tls-certs\") pod \"2ab23ed1-5683-4037-b4d6-0db2a56bb116\" (UID: \"2ab23ed1-5683-4037-b4d6-0db2a56bb116\") "
Sep 16 20:14:25.356338 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:25.356312    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ab23ed1-5683-4037-b4d6-0db2a56bb116-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2ab23ed1-5683-4037-b4d6-0db2a56bb116" (UID: "2ab23ed1-5683-4037-b4d6-0db2a56bb116"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:14:25.356441 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:25.356420    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ab23ed1-5683-4037-b4d6-0db2a56bb116-kube-api-access-spdpz" (OuterVolumeSpecName: "kube-api-access-spdpz") pod "2ab23ed1-5683-4037-b4d6-0db2a56bb116" (UID: "2ab23ed1-5683-4037-b4d6-0db2a56bb116"). InnerVolumeSpecName "kube-api-access-spdpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:14:25.455089 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:25.455053    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ab23ed1-5683-4037-b4d6-0db2a56bb116-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:14:25.455089 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:25.455085    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spdpz\" (UniqueName: \"kubernetes.io/projected/2ab23ed1-5683-4037-b4d6-0db2a56bb116-kube-api-access-spdpz\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:14:26.200092 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:26.200001    2557 generic.go:356] "Generic (PLEG): container finished" podID="2ab23ed1-5683-4037-b4d6-0db2a56bb116" containerID="e5fa118342c1e8acbf783b75134394a1d6e5323ce072e2ef26f934d6cf94c467" exitCode=0
Sep 16 20:14:26.200092 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:26.200053    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-789d76jnz"
Sep 16 20:14:26.200272 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:26.200087    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-789d76jnz" event={"ID":"2ab23ed1-5683-4037-b4d6-0db2a56bb116","Type":"ContainerDied","Data":"e5fa118342c1e8acbf783b75134394a1d6e5323ce072e2ef26f934d6cf94c467"}
Sep 16 20:14:26.200272 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:26.200144    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-789d76jnz" event={"ID":"2ab23ed1-5683-4037-b4d6-0db2a56bb116","Type":"ContainerDied","Data":"93c8dbb9370e35c410c16b3ffea8da59fb21d4c755929b7fb495f7b6f203a4df"}
Sep 16 20:14:26.200272 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:26.200166    2557 scope.go:117] "RemoveContainer" containerID="e5fa118342c1e8acbf783b75134394a1d6e5323ce072e2ef26f934d6cf94c467"
Sep 16 20:14:26.208997 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:26.208820    2557 scope.go:117] "RemoveContainer" containerID="e5fa118342c1e8acbf783b75134394a1d6e5323ce072e2ef26f934d6cf94c467"
Sep 16 20:14:26.209154 ip-10-0-139-81 kubenswrapper[2557]: E0916 20:14:26.209131    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5fa118342c1e8acbf783b75134394a1d6e5323ce072e2ef26f934d6cf94c467\": container with ID starting with e5fa118342c1e8acbf783b75134394a1d6e5323ce072e2ef26f934d6cf94c467 not found: ID does not exist" containerID="e5fa118342c1e8acbf783b75134394a1d6e5323ce072e2ef26f934d6cf94c467"
Sep 16 20:14:26.209194 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:26.209161    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5fa118342c1e8acbf783b75134394a1d6e5323ce072e2ef26f934d6cf94c467"} err="failed to get container status \"e5fa118342c1e8acbf783b75134394a1d6e5323ce072e2ef26f934d6cf94c467\": rpc error: code = NotFound desc = could not find container \"e5fa118342c1e8acbf783b75134394a1d6e5323ce072e2ef26f934d6cf94c467\": container with ID starting with e5fa118342c1e8acbf783b75134394a1d6e5323ce072e2ef26f934d6cf94c467 not found: ID does not exist"
Sep 16 20:14:26.218959 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:26.218937    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-789d76jnz"]
Sep 16 20:14:26.223016 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:26.222990    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-789d76jnz"]
Sep 16 20:14:27.938448 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:27.938414    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ab23ed1-5683-4037-b4d6-0db2a56bb116" path="/var/lib/kubelet/pods/2ab23ed1-5683-4037-b4d6-0db2a56bb116/volumes"
Sep 16 20:14:29.493555 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:29.493514    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zq9j9" podUID="0f77d2a0-45f0-4cdc-bca6-dedc79043506" containerName="main" containerID="cri-o://64a0f5727f62bcbf2022a51edae888d6f57d4de582972073825d03695dbb2471" gracePeriod=45
Sep 16 20:14:29.520937 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:29.520903    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zq9j9" podUID="0f77d2a0-45f0-4cdc-bca6-dedc79043506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 16 20:14:29.751517 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:29.751444    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zq9j9"
Sep 16 20:14:29.789702 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:29.789673    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f77d2a0-45f0-4cdc-bca6-dedc79043506-tmp-dir\") pod \"0f77d2a0-45f0-4cdc-bca6-dedc79043506\" (UID: \"0f77d2a0-45f0-4cdc-bca6-dedc79043506\") "
Sep 16 20:14:29.789702 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:29.789705    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f77d2a0-45f0-4cdc-bca6-dedc79043506-tls-certs\") pod \"0f77d2a0-45f0-4cdc-bca6-dedc79043506\" (UID: \"0f77d2a0-45f0-4cdc-bca6-dedc79043506\") "
Sep 16 20:14:29.789959 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:29.789723    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f77d2a0-45f0-4cdc-bca6-dedc79043506-home\") pod \"0f77d2a0-45f0-4cdc-bca6-dedc79043506\" (UID: \"0f77d2a0-45f0-4cdc-bca6-dedc79043506\") "
Sep 16 20:14:29.789959 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:29.789767    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8zdk\" (UniqueName: \"kubernetes.io/projected/0f77d2a0-45f0-4cdc-bca6-dedc79043506-kube-api-access-h8zdk\") pod \"0f77d2a0-45f0-4cdc-bca6-dedc79043506\" (UID: \"0f77d2a0-45f0-4cdc-bca6-dedc79043506\") "
Sep 16 20:14:29.789959 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:29.789824    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f77d2a0-45f0-4cdc-bca6-dedc79043506-dshm\") pod \"0f77d2a0-45f0-4cdc-bca6-dedc79043506\" (UID: \"0f77d2a0-45f0-4cdc-bca6-dedc79043506\") "
Sep 16 20:14:29.789959 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:29.789892    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f77d2a0-45f0-4cdc-bca6-dedc79043506-model-cache\") pod \"0f77d2a0-45f0-4cdc-bca6-dedc79043506\" (UID: \"0f77d2a0-45f0-4cdc-bca6-dedc79043506\") "
Sep 16 20:14:29.790163 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:29.789993    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f77d2a0-45f0-4cdc-bca6-dedc79043506-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0f77d2a0-45f0-4cdc-bca6-dedc79043506" (UID: "0f77d2a0-45f0-4cdc-bca6-dedc79043506"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:14:29.790163 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:29.790021    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f77d2a0-45f0-4cdc-bca6-dedc79043506-home" (OuterVolumeSpecName: "home") pod "0f77d2a0-45f0-4cdc-bca6-dedc79043506" (UID: "0f77d2a0-45f0-4cdc-bca6-dedc79043506"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:14:29.790262 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:29.790225    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f77d2a0-45f0-4cdc-bca6-dedc79043506-model-cache" (OuterVolumeSpecName: "model-cache") pod "0f77d2a0-45f0-4cdc-bca6-dedc79043506" (UID: "0f77d2a0-45f0-4cdc-bca6-dedc79043506"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:14:29.790322 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:29.790256    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f77d2a0-45f0-4cdc-bca6-dedc79043506-tmp-dir\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:14:29.790322 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:29.790276    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f77d2a0-45f0-4cdc-bca6-dedc79043506-home\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:14:29.792081 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:29.792058    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f77d2a0-45f0-4cdc-bca6-dedc79043506-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0f77d2a0-45f0-4cdc-bca6-dedc79043506" (UID: "0f77d2a0-45f0-4cdc-bca6-dedc79043506"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:14:29.792181 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:29.792135    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f77d2a0-45f0-4cdc-bca6-dedc79043506-dshm" (OuterVolumeSpecName: "dshm") pod "0f77d2a0-45f0-4cdc-bca6-dedc79043506" (UID: "0f77d2a0-45f0-4cdc-bca6-dedc79043506"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:14:29.792352 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:29.792332    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f77d2a0-45f0-4cdc-bca6-dedc79043506-kube-api-access-h8zdk" (OuterVolumeSpecName: "kube-api-access-h8zdk") pod "0f77d2a0-45f0-4cdc-bca6-dedc79043506" (UID: "0f77d2a0-45f0-4cdc-bca6-dedc79043506"). InnerVolumeSpecName "kube-api-access-h8zdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:14:29.890720 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:29.890682    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f77d2a0-45f0-4cdc-bca6-dedc79043506-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:14:29.890720 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:29.890718    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8zdk\" (UniqueName: \"kubernetes.io/projected/0f77d2a0-45f0-4cdc-bca6-dedc79043506-kube-api-access-h8zdk\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:14:29.890954 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:29.890731    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f77d2a0-45f0-4cdc-bca6-dedc79043506-dshm\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:14:29.890954 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:29.890743    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f77d2a0-45f0-4cdc-bca6-dedc79043506-model-cache\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:14:30.222164 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:30.222075    2557 generic.go:356] "Generic (PLEG): container finished" podID="0f77d2a0-45f0-4cdc-bca6-dedc79043506" containerID="64a0f5727f62bcbf2022a51edae888d6f57d4de582972073825d03695dbb2471" exitCode=0
Sep 16 20:14:30.222164 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:30.222121    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zq9j9"
Sep 16 20:14:30.222354 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:30.222164    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zq9j9" event={"ID":"0f77d2a0-45f0-4cdc-bca6-dedc79043506","Type":"ContainerDied","Data":"64a0f5727f62bcbf2022a51edae888d6f57d4de582972073825d03695dbb2471"}
Sep 16 20:14:30.222354 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:30.222216    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zq9j9" event={"ID":"0f77d2a0-45f0-4cdc-bca6-dedc79043506","Type":"ContainerDied","Data":"57618f069621df462b6c04fc17c01deb9f2bc0c495fc08e92a37a6cb3114544c"}
Sep 16 20:14:30.222354 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:30.222239    2557 scope.go:117] "RemoveContainer" containerID="64a0f5727f62bcbf2022a51edae888d6f57d4de582972073825d03695dbb2471"
Sep 16 20:14:30.231165 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:30.231142    2557 scope.go:117] "RemoveContainer" containerID="64a0f5727f62bcbf2022a51edae888d6f57d4de582972073825d03695dbb2471"
Sep 16 20:14:30.231453 ip-10-0-139-81 kubenswrapper[2557]: E0916 20:14:30.231430    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64a0f5727f62bcbf2022a51edae888d6f57d4de582972073825d03695dbb2471\": container with ID starting with 64a0f5727f62bcbf2022a51edae888d6f57d4de582972073825d03695dbb2471 not found: ID does not exist" containerID="64a0f5727f62bcbf2022a51edae888d6f57d4de582972073825d03695dbb2471"
Sep 16 20:14:30.231514 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:30.231463    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64a0f5727f62bcbf2022a51edae888d6f57d4de582972073825d03695dbb2471"} err="failed to get container status \"64a0f5727f62bcbf2022a51edae888d6f57d4de582972073825d03695dbb2471\": rpc error: code = NotFound desc = could not find container \"64a0f5727f62bcbf2022a51edae888d6f57d4de582972073825d03695dbb2471\": container with ID starting with 64a0f5727f62bcbf2022a51edae888d6f57d4de582972073825d03695dbb2471 not found: ID does not exist"
Sep 16 20:14:30.243255 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:30.243228    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zq9j9"]
Sep 16 20:14:30.248041 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:30.248021    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zq9j9"]
Sep 16 20:14:31.937836 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:31.937804    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f77d2a0-45f0-4cdc-bca6-dedc79043506" path="/var/lib/kubelet/pods/0f77d2a0-45f0-4cdc-bca6-dedc79043506/volumes"
Sep 16 20:14:43.884069 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:43.884037    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v7x5x"]
Sep 16 20:14:43.884615 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:43.884594    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ab23ed1-5683-4037-b4d6-0db2a56bb116" containerName="main"
Sep 16 20:14:43.884704 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:43.884618    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ab23ed1-5683-4037-b4d6-0db2a56bb116" containerName="main"
Sep 16 20:14:43.884704 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:43.884632    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f77d2a0-45f0-4cdc-bca6-dedc79043506" containerName="main"
Sep 16 20:14:43.884704 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:43.884640    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f77d2a0-45f0-4cdc-bca6-dedc79043506" containerName="main"
Sep 16 20:14:43.884886 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:43.884753    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f77d2a0-45f0-4cdc-bca6-dedc79043506" containerName="main"
Sep 16 20:14:43.884886 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:43.884768    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ab23ed1-5683-4037-b4d6-0db2a56bb116" containerName="main"
Sep 16 20:14:43.887626 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:43.887605    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v7x5x"
Sep 16 20:14:43.890912 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:43.890892    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"default-dockercfg-jd96b\""
Sep 16 20:14:43.891028 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:43.891007    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"openshift-service-ca.crt\""
Sep 16 20:14:43.891950 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:43.891930    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"kube-root-ca.crt\""
Sep 16 20:14:43.892057 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:43.892013    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"decom-v2-kserve-self-signed-certs\""
Sep 16 20:14:43.915960 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:43.915934    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v7x5x"]
Sep 16 20:14:44.019297 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:44.019257    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8d30854-fb1f-4629-abba-2e029ffb65a1-dshm\") pod \"decom-v2-kserve-77f867d787-v7x5x\" (UID: \"e8d30854-fb1f-4629-abba-2e029ffb65a1\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v7x5x"
Sep 16 20:14:44.019297 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:44.019300    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8d30854-fb1f-4629-abba-2e029ffb65a1-tmp-dir\") pod \"decom-v2-kserve-77f867d787-v7x5x\" (UID: \"e8d30854-fb1f-4629-abba-2e029ffb65a1\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v7x5x"
Sep 16 20:14:44.019522 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:44.019465    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgr2c\" (UniqueName: \"kubernetes.io/projected/e8d30854-fb1f-4629-abba-2e029ffb65a1-kube-api-access-kgr2c\") pod \"decom-v2-kserve-77f867d787-v7x5x\" (UID: \"e8d30854-fb1f-4629-abba-2e029ffb65a1\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v7x5x"
Sep 16 20:14:44.019522 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:44.019510    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8d30854-fb1f-4629-abba-2e029ffb65a1-home\") pod \"decom-v2-kserve-77f867d787-v7x5x\" (UID: \"e8d30854-fb1f-4629-abba-2e029ffb65a1\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v7x5x"
Sep 16 20:14:44.019609 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:44.019530    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8d30854-fb1f-4629-abba-2e029ffb65a1-tls-certs\") pod \"decom-v2-kserve-77f867d787-v7x5x\" (UID: \"e8d30854-fb1f-4629-abba-2e029ffb65a1\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v7x5x"
Sep 16 20:14:44.019735 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:44.019716    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8d30854-fb1f-4629-abba-2e029ffb65a1-model-cache\") pod \"decom-v2-kserve-77f867d787-v7x5x\" (UID: \"e8d30854-fb1f-4629-abba-2e029ffb65a1\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v7x5x"
Sep 16 20:14:44.121018 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:44.120981    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgr2c\" (UniqueName: \"kubernetes.io/projected/e8d30854-fb1f-4629-abba-2e029ffb65a1-kube-api-access-kgr2c\") pod \"decom-v2-kserve-77f867d787-v7x5x\" (UID: \"e8d30854-fb1f-4629-abba-2e029ffb65a1\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v7x5x"
Sep 16 20:14:44.121018 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:44.121020    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8d30854-fb1f-4629-abba-2e029ffb65a1-home\") pod \"decom-v2-kserve-77f867d787-v7x5x\" (UID: \"e8d30854-fb1f-4629-abba-2e029ffb65a1\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v7x5x"
Sep 16 20:14:44.121246 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:44.121038    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8d30854-fb1f-4629-abba-2e029ffb65a1-tls-certs\") pod \"decom-v2-kserve-77f867d787-v7x5x\" (UID: \"e8d30854-fb1f-4629-abba-2e029ffb65a1\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v7x5x"
Sep 16 20:14:44.121246 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:44.121082    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8d30854-fb1f-4629-abba-2e029ffb65a1-model-cache\") pod \"decom-v2-kserve-77f867d787-v7x5x\" (UID: \"e8d30854-fb1f-4629-abba-2e029ffb65a1\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v7x5x"
Sep 16 20:14:44.121246 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:44.121109    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8d30854-fb1f-4629-abba-2e029ffb65a1-dshm\") pod \"decom-v2-kserve-77f867d787-v7x5x\" (UID: \"e8d30854-fb1f-4629-abba-2e029ffb65a1\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v7x5x"
Sep 16 20:14:44.121246 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:44.121131    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8d30854-fb1f-4629-abba-2e029ffb65a1-tmp-dir\") pod \"decom-v2-kserve-77f867d787-v7x5x\" (UID: \"e8d30854-fb1f-4629-abba-2e029ffb65a1\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v7x5x"
Sep 16 20:14:44.121470 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:44.121446    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8d30854-fb1f-4629-abba-2e029ffb65a1-home\") pod \"decom-v2-kserve-77f867d787-v7x5x\" (UID: \"e8d30854-fb1f-4629-abba-2e029ffb65a1\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v7x5x"
Sep 16 20:14:44.121536 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:44.121487    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8d30854-fb1f-4629-abba-2e029ffb65a1-model-cache\") pod \"decom-v2-kserve-77f867d787-v7x5x\" (UID: \"e8d30854-fb1f-4629-abba-2e029ffb65a1\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v7x5x"
Sep 16 20:14:44.121598 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:44.121549    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8d30854-fb1f-4629-abba-2e029ffb65a1-tmp-dir\") pod \"decom-v2-kserve-77f867d787-v7x5x\" (UID: \"e8d30854-fb1f-4629-abba-2e029ffb65a1\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v7x5x"
Sep 16 20:14:44.123409 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:44.123387    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8d30854-fb1f-4629-abba-2e029ffb65a1-dshm\") pod \"decom-v2-kserve-77f867d787-v7x5x\" (UID: \"e8d30854-fb1f-4629-abba-2e029ffb65a1\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v7x5x"
Sep 16 20:14:44.123601 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:44.123582    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8d30854-fb1f-4629-abba-2e029ffb65a1-tls-certs\") pod \"decom-v2-kserve-77f867d787-v7x5x\" (UID: \"e8d30854-fb1f-4629-abba-2e029ffb65a1\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v7x5x"
Sep 16 20:14:44.129880 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:44.129828    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgr2c\" (UniqueName: \"kubernetes.io/projected/e8d30854-fb1f-4629-abba-2e029ffb65a1-kube-api-access-kgr2c\") pod \"decom-v2-kserve-77f867d787-v7x5x\" (UID: \"e8d30854-fb1f-4629-abba-2e029ffb65a1\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v7x5x"
Sep 16 20:14:44.198145 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:44.198064    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v7x5x"
Sep 16 20:14:44.474555 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:44.474369    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v7x5x"]
Sep 16 20:14:44.476856 ip-10-0-139-81 kubenswrapper[2557]: W0916 20:14:44.476814    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8d30854_fb1f_4629_abba_2e029ffb65a1.slice/crio-6da810a7f1669cbf24715861178bd9c2ee32e22e45481a7d65ddab470f61ec9d WatchSource:0}: Error finding container 6da810a7f1669cbf24715861178bd9c2ee32e22e45481a7d65ddab470f61ec9d: Status 404 returned error can't find the container with id 6da810a7f1669cbf24715861178bd9c2ee32e22e45481a7d65ddab470f61ec9d
Sep 16 20:14:45.298749 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:45.298705    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v7x5x" event={"ID":"e8d30854-fb1f-4629-abba-2e029ffb65a1","Type":"ContainerStarted","Data":"e047fe85eb41c8a302a0d9573f17043b8a66218a3e9b6c4450a32a59b24e5059"}
Sep 16 20:14:45.299193 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:45.298764    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v7x5x" event={"ID":"e8d30854-fb1f-4629-abba-2e029ffb65a1","Type":"ContainerStarted","Data":"6da810a7f1669cbf24715861178bd9c2ee32e22e45481a7d65ddab470f61ec9d"}
Sep 16 20:14:45.319486 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:45.319440    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v7x5x" podStartSLOduration=2.319423545 podStartE2EDuration="2.319423545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:14:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:14:45.318725613 +0000 UTC m=+2113.844114038" watchObservedRunningTime="2026-09-16 20:14:45.319423545 +0000 UTC m=+2113.844811933"
Sep 16 20:14:46.199018 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:46.198976    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v7x5x"
Sep 16 20:14:54.199079 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:54.199043    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v7x5x"
Sep 16 20:14:54.211696 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:54.211674    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v7x5x"
Sep 16 20:14:54.353078 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:14:54.353049    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v7x5x"
Sep 16 20:15:02.170942 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:15:02.170898    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-57fb9w2vxd"]
Sep 16 20:15:02.175008 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:15:02.174977    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-57fb9w2vxd"
Sep 16 20:15:02.179763 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:15:02.179738    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-epp-sa-dockercfg-2m8pd\""
Sep 16 20:15:02.179940 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:15:02.179738    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 16 20:15:02.179940 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:15:02.179716    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 16 20:15:02.180083 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:15:02.179790    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-bn8cc\""
Sep 16 20:15:02.180083 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:15:02.179791    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 16 20:15:02.187425 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:15:02.187402    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-57fb9w2vxd"]
Sep 16 20:15:02.290118 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:15:02.290078    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/befd599e-cad2-4c9d-9576-95d4cf044119-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-57fb9w2vxd\" (UID: \"befd599e-cad2-4c9d-9576-95d4cf044119\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-57fb9w2vxd"
Sep 16 20:15:02.290293 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:15:02.290119    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2gpf\" (UniqueName: \"kubernetes.io/projected/befd599e-cad2-4c9d-9576-95d4cf044119-kube-api-access-f2gpf\") pod \"tokenizer-template-args-test-kserve-router-scheduler-57fb9w2vxd\" (UID: \"befd599e-cad2-4c9d-9576-95d4cf044119\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-57fb9w2vxd"
Sep 16 20:15:02.391300 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:15:02.391264    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/befd599e-cad2-4c9d-9576-95d4cf044119-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-57fb9w2vxd\" (UID: \"befd599e-cad2-4c9d-9576-95d4cf044119\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-57fb9w2vxd"
Sep 16 20:15:02.391300 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:15:02.391298    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2gpf\" (UniqueName: \"kubernetes.io/projected/befd599e-cad2-4c9d-9576-95d4cf044119-kube-api-access-f2gpf\") pod \"tokenizer-template-args-test-kserve-router-scheduler-57fb9w2vxd\" (UID: \"befd599e-cad2-4c9d-9576-95d4cf044119\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-57fb9w2vxd"
Sep 16 20:15:02.393820 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:15:02.393799    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/befd599e-cad2-4c9d-9576-95d4cf044119-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-57fb9w2vxd\" (UID: \"befd599e-cad2-4c9d-9576-95d4cf044119\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-57fb9w2vxd"
Sep 16 20:15:02.400191 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:15:02.400165    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2gpf\" (UniqueName: \"kubernetes.io/projected/befd599e-cad2-4c9d-9576-95d4cf044119-kube-api-access-f2gpf\") pod \"tokenizer-template-args-test-kserve-router-scheduler-57fb9w2vxd\" (UID: \"befd599e-cad2-4c9d-9576-95d4cf044119\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-57fb9w2vxd"
Sep 16 20:15:02.487915 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:15:02.487882    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-57fb9w2vxd"
Sep 16 20:15:02.759597 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:15:02.759568    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-57fb9w2vxd"]
Sep 16 20:15:02.762550 ip-10-0-139-81 kubenswrapper[2557]: W0916 20:15:02.762526    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbefd599e_cad2_4c9d_9576_95d4cf044119.slice/crio-eb18cbbfdda3a43fe0c5bad4ff8609aa6be318243735ca174ad151df4adbc928 WatchSource:0}: Error finding container eb18cbbfdda3a43fe0c5bad4ff8609aa6be318243735ca174ad151df4adbc928: Status 404 returned error can't find the container with id eb18cbbfdda3a43fe0c5bad4ff8609aa6be318243735ca174ad151df4adbc928
Sep 16 20:15:03.393323 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:15:03.393286    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-57fb9w2vxd" event={"ID":"befd599e-cad2-4c9d-9576-95d4cf044119","Type":"ContainerStarted","Data":"1c82a3e72b3d9a3624739c2ef89f4f9bd5076d4e7ec0cceefe131df0884d7d39"}
Sep 16 20:15:03.393774 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:15:03.393349    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-57fb9w2vxd" event={"ID":"befd599e-cad2-4c9d-9576-95d4cf044119","Type":"ContainerStarted","Data":"eb18cbbfdda3a43fe0c5bad4ff8609aa6be318243735ca174ad151df4adbc928"}
Sep 16 20:15:03.393774 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:15:03.393383    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-57fb9w2vxd"
Sep 16 20:15:03.413072 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:15:03.413021    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-57fb9w2vxd" podStartSLOduration=1.4130067880000001 podStartE2EDuration="1.413006788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:15:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:15:03.410682941 +0000 UTC m=+2131.936071327" watchObservedRunningTime="2026-09-16 20:15:03.413006788 +0000 UTC m=+2131.938395173"
Sep 16 20:15:34.400893 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:15:34.400795    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-57fb9w2vxd"
Sep 16 20:15:37.887314 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:15:37.887279    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v7x5x"]
Sep 16 20:15:40.941671 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:15:40.941634    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-57fb9w2vxd"]
Sep 16 20:15:44.400720 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:15:44.400673    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-57fb9w2vxd" podUID="befd599e-cad2-4c9d-9576-95d4cf044119" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:15:52.907805 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:15:52.907758    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v7x5x" podUID="e8d30854-fb1f-4629-abba-2e029ffb65a1" containerName="main" containerID="cri-o://e047fe85eb41c8a302a0d9573f17043b8a66218a3e9b6c4450a32a59b24e5059" gracePeriod=45
Sep 16 20:15:53.164050 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:15:53.163986    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v7x5x"
Sep 16 20:15:53.258852 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:15:53.258819    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8d30854-fb1f-4629-abba-2e029ffb65a1-dshm\") pod \"e8d30854-fb1f-4629-abba-2e029ffb65a1\" (UID: \"e8d30854-fb1f-4629-abba-2e029ffb65a1\") "
Sep 16 20:15:53.259100 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:15:53.258938    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8d30854-fb1f-4629-abba-2e029ffb65a1-tls-certs\") pod \"e8d30854-fb1f-4629-abba-2e029ffb65a1\" (UID: \"e8d30854-fb1f-4629-abba-2e029ffb65a1\") "
Sep 16 20:15:53.259100 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:15:53.258979    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8d30854-fb1f-4629-abba-2e029ffb65a1-tmp-dir\") pod \"e8d30854-fb1f-4629-abba-2e029ffb65a1\" (UID: \"e8d30854-fb1f-4629-abba-2e029ffb65a1\") "
Sep 16 20:15:53.259100 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:15:53.259055    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kgr2c\" (UniqueName: \"kubernetes.io/projected/e8d30854-fb1f-4629-abba-2e029ffb65a1-kube-api-access-kgr2c\") pod \"e8d30854-fb1f-4629-abba-2e029ffb65a1\" (UID: \"e8d30854-fb1f-4629-abba-2e029ffb65a1\") "
Sep 16 20:15:53.259100 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:15:53.259078    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8d30854-fb1f-4629-abba-2e029ffb65a1-home\") pod \"e8d30854-fb1f-4629-abba-2e029ffb65a1\" (UID: \"e8d30854-fb1f-4629-abba-2e029ffb65a1\") "
Sep 16 20:15:53.259100 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:15:53.259099    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8d30854-fb1f-4629-abba-2e029ffb65a1-model-cache\") pod \"e8d30854-fb1f-4629-abba-2e029ffb65a1\" (UID: \"e8d30854-fb1f-4629-abba-2e029ffb65a1\") "
Sep 16 20:15:53.259353 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:15:53.259300    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8d30854-fb1f-4629-abba-2e029ffb65a1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e8d30854-fb1f-4629-abba-2e029ffb65a1" (UID: "e8d30854-fb1f-4629-abba-2e029ffb65a1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:15:53.259408 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:15:53.259365    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8d30854-fb1f-4629-abba-2e029ffb65a1-home" (OuterVolumeSpecName: "home") pod "e8d30854-fb1f-4629-abba-2e029ffb65a1" (UID: "e8d30854-fb1f-4629-abba-2e029ffb65a1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:15:53.259455 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:15:53.259400    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8d30854-fb1f-4629-abba-2e029ffb65a1-model-cache" (OuterVolumeSpecName: "model-cache") pod "e8d30854-fb1f-4629-abba-2e029ffb65a1" (UID: "e8d30854-fb1f-4629-abba-2e029ffb65a1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:15:53.259498 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:15:53.259480    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8d30854-fb1f-4629-abba-2e029ffb65a1-tmp-dir\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:15:53.259545 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:15:53.259500    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8d30854-fb1f-4629-abba-2e029ffb65a1-home\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:15:53.261249 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:15:53.261220    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8d30854-fb1f-4629-abba-2e029ffb65a1-kube-api-access-kgr2c" (OuterVolumeSpecName: "kube-api-access-kgr2c") pod "e8d30854-fb1f-4629-abba-2e029ffb65a1" (UID: "e8d30854-fb1f-4629-abba-2e029ffb65a1"). InnerVolumeSpecName "kube-api-access-kgr2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:15:53.261470 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:15:53.261442    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8d30854-fb1f-4629-abba-2e029ffb65a1-dshm" (OuterVolumeSpecName: "dshm") pod "e8d30854-fb1f-4629-abba-2e029ffb65a1" (UID: "e8d30854-fb1f-4629-abba-2e029ffb65a1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:15:53.261533 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:15:53.261445    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8d30854-fb1f-4629-abba-2e029ffb65a1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e8d30854-fb1f-4629-abba-2e029ffb65a1" (UID: "e8d30854-fb1f-4629-abba-2e029ffb65a1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:15:53.360310 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:15:53.360278    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgr2c\" (UniqueName: \"kubernetes.io/projected/e8d30854-fb1f-4629-abba-2e029ffb65a1-kube-api-access-kgr2c\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:15:53.360310 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:15:53.360308    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8d30854-fb1f-4629-abba-2e029ffb65a1-model-cache\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:15:53.360499 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:15:53.360320    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8d30854-fb1f-4629-abba-2e029ffb65a1-dshm\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:15:53.360499 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:15:53.360330    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8d30854-fb1f-4629-abba-2e029ffb65a1-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:15:53.650362 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:15:53.650325    2557 generic.go:356] "Generic (PLEG): container finished" podID="e8d30854-fb1f-4629-abba-2e029ffb65a1" containerID="e047fe85eb41c8a302a0d9573f17043b8a66218a3e9b6c4450a32a59b24e5059" exitCode=0
Sep 16 20:15:53.650523 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:15:53.650376    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v7x5x"
Sep 16 20:15:53.650523 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:15:53.650417    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v7x5x" event={"ID":"e8d30854-fb1f-4629-abba-2e029ffb65a1","Type":"ContainerDied","Data":"e047fe85eb41c8a302a0d9573f17043b8a66218a3e9b6c4450a32a59b24e5059"}
Sep 16 20:15:53.650523 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:15:53.650475    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v7x5x" event={"ID":"e8d30854-fb1f-4629-abba-2e029ffb65a1","Type":"ContainerDied","Data":"6da810a7f1669cbf24715861178bd9c2ee32e22e45481a7d65ddab470f61ec9d"}
Sep 16 20:15:53.650523 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:15:53.650504    2557 scope.go:117] "RemoveContainer" containerID="e047fe85eb41c8a302a0d9573f17043b8a66218a3e9b6c4450a32a59b24e5059"
Sep 16 20:15:53.660008 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:15:53.659985    2557 scope.go:117] "RemoveContainer" containerID="e047fe85eb41c8a302a0d9573f17043b8a66218a3e9b6c4450a32a59b24e5059"
Sep 16 20:15:53.660279 ip-10-0-139-81 kubenswrapper[2557]: E0916 20:15:53.660262    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e047fe85eb41c8a302a0d9573f17043b8a66218a3e9b6c4450a32a59b24e5059\": container with ID starting with e047fe85eb41c8a302a0d9573f17043b8a66218a3e9b6c4450a32a59b24e5059 not found: ID does not exist" containerID="e047fe85eb41c8a302a0d9573f17043b8a66218a3e9b6c4450a32a59b24e5059"
Sep 16 20:15:53.660335 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:15:53.660286    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e047fe85eb41c8a302a0d9573f17043b8a66218a3e9b6c4450a32a59b24e5059"} err="failed to get container status \"e047fe85eb41c8a302a0d9573f17043b8a66218a3e9b6c4450a32a59b24e5059\": rpc error: code = NotFound desc = could not find container \"e047fe85eb41c8a302a0d9573f17043b8a66218a3e9b6c4450a32a59b24e5059\": container with ID starting with e047fe85eb41c8a302a0d9573f17043b8a66218a3e9b6c4450a32a59b24e5059 not found: ID does not exist"
Sep 16 20:15:53.705269 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:15:53.705231    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v7x5x"]
Sep 16 20:15:53.716152 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:15:53.716122    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v7x5x"]
Sep 16 20:15:53.937312 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:15:53.937240    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8d30854-fb1f-4629-abba-2e029ffb65a1" path="/var/lib/kubelet/pods/e8d30854-fb1f-4629-abba-2e029ffb65a1/volumes"
Sep 16 20:15:54.401092 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:15:54.401057    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-57fb9w2vxd" podUID="befd599e-cad2-4c9d-9576-95d4cf044119" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:15:55.962134 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:15:55.962080    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-57fb9w2vxd" podUID="befd599e-cad2-4c9d-9576-95d4cf044119" containerName="main" containerID="cri-o://1c82a3e72b3d9a3624739c2ef89f4f9bd5076d4e7ec0cceefe131df0884d7d39" gracePeriod=45
Sep 16 20:16:04.400787 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:04.400743    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-57fb9w2vxd" podUID="befd599e-cad2-4c9d-9576-95d4cf044119" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:16:04.401229 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:04.400895    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-57fb9w2vxd"
Sep 16 20:16:14.401293 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:14.401240    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-57fb9w2vxd" podUID="befd599e-cad2-4c9d-9576-95d4cf044119" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:16:24.401123 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:24.401081    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-57fb9w2vxd" podUID="befd599e-cad2-4c9d-9576-95d4cf044119" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:16:26.218635 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:26.218570    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-57fb9w2vxd"
Sep 16 20:16:26.364095 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:26.364056    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/befd599e-cad2-4c9d-9576-95d4cf044119-tls-certs\") pod \"befd599e-cad2-4c9d-9576-95d4cf044119\" (UID: \"befd599e-cad2-4c9d-9576-95d4cf044119\") "
Sep 16 20:16:26.364273 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:26.364150    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2gpf\" (UniqueName: \"kubernetes.io/projected/befd599e-cad2-4c9d-9576-95d4cf044119-kube-api-access-f2gpf\") pod \"befd599e-cad2-4c9d-9576-95d4cf044119\" (UID: \"befd599e-cad2-4c9d-9576-95d4cf044119\") "
Sep 16 20:16:26.366272 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:26.366233    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/befd599e-cad2-4c9d-9576-95d4cf044119-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "befd599e-cad2-4c9d-9576-95d4cf044119" (UID: "befd599e-cad2-4c9d-9576-95d4cf044119"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:16:26.366272 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:26.366266    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/befd599e-cad2-4c9d-9576-95d4cf044119-kube-api-access-f2gpf" (OuterVolumeSpecName: "kube-api-access-f2gpf") pod "befd599e-cad2-4c9d-9576-95d4cf044119" (UID: "befd599e-cad2-4c9d-9576-95d4cf044119"). InnerVolumeSpecName "kube-api-access-f2gpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:16:26.464957 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:26.464922    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/befd599e-cad2-4c9d-9576-95d4cf044119-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:16:26.464957 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:26.464951    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2gpf\" (UniqueName: \"kubernetes.io/projected/befd599e-cad2-4c9d-9576-95d4cf044119-kube-api-access-f2gpf\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:16:26.818350 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:26.818313    2557 generic.go:356] "Generic (PLEG): container finished" podID="befd599e-cad2-4c9d-9576-95d4cf044119" containerID="1c82a3e72b3d9a3624739c2ef89f4f9bd5076d4e7ec0cceefe131df0884d7d39" exitCode=0
Sep 16 20:16:26.818531 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:26.818358    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-57fb9w2vxd"
Sep 16 20:16:26.818531 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:26.818358    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-57fb9w2vxd" event={"ID":"befd599e-cad2-4c9d-9576-95d4cf044119","Type":"ContainerDied","Data":"1c82a3e72b3d9a3624739c2ef89f4f9bd5076d4e7ec0cceefe131df0884d7d39"}
Sep 16 20:16:26.818531 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:26.818487    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-57fb9w2vxd" event={"ID":"befd599e-cad2-4c9d-9576-95d4cf044119","Type":"ContainerDied","Data":"eb18cbbfdda3a43fe0c5bad4ff8609aa6be318243735ca174ad151df4adbc928"}
Sep 16 20:16:26.818531 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:26.818515    2557 scope.go:117] "RemoveContainer" containerID="1c82a3e72b3d9a3624739c2ef89f4f9bd5076d4e7ec0cceefe131df0884d7d39"
Sep 16 20:16:26.827655 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:26.827638    2557 scope.go:117] "RemoveContainer" containerID="1c82a3e72b3d9a3624739c2ef89f4f9bd5076d4e7ec0cceefe131df0884d7d39"
Sep 16 20:16:26.827959 ip-10-0-139-81 kubenswrapper[2557]: E0916 20:16:26.827940    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c82a3e72b3d9a3624739c2ef89f4f9bd5076d4e7ec0cceefe131df0884d7d39\": container with ID starting with 1c82a3e72b3d9a3624739c2ef89f4f9bd5076d4e7ec0cceefe131df0884d7d39 not found: ID does not exist" containerID="1c82a3e72b3d9a3624739c2ef89f4f9bd5076d4e7ec0cceefe131df0884d7d39"
Sep 16 20:16:26.828042 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:26.827967    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c82a3e72b3d9a3624739c2ef89f4f9bd5076d4e7ec0cceefe131df0884d7d39"} err="failed to get container status \"1c82a3e72b3d9a3624739c2ef89f4f9bd5076d4e7ec0cceefe131df0884d7d39\": rpc error: code = NotFound desc = could not find container \"1c82a3e72b3d9a3624739c2ef89f4f9bd5076d4e7ec0cceefe131df0884d7d39\": container with ID starting with 1c82a3e72b3d9a3624739c2ef89f4f9bd5076d4e7ec0cceefe131df0884d7d39 not found: ID does not exist"
Sep 16 20:16:26.840138 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:26.840114    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-57fb9w2vxd"]
Sep 16 20:16:26.844401 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:26.844377    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-57fb9w2vxd"]
Sep 16 20:16:27.936973 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:27.936940    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="befd599e-cad2-4c9d-9576-95d4cf044119" path="/var/lib/kubelet/pods/befd599e-cad2-4c9d-9576-95d4cf044119/volumes"
Sep 16 20:16:33.778309 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:33.778276    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-5vnqn"]
Sep 16 20:16:33.778697 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:33.778683    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="befd599e-cad2-4c9d-9576-95d4cf044119" containerName="main"
Sep 16 20:16:33.778697 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:33.778697    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="befd599e-cad2-4c9d-9576-95d4cf044119" containerName="main"
Sep 16 20:16:33.778783 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:33.778707    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8d30854-fb1f-4629-abba-2e029ffb65a1" containerName="main"
Sep 16 20:16:33.778783 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:33.778713    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8d30854-fb1f-4629-abba-2e029ffb65a1" containerName="main"
Sep 16 20:16:33.778783 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:33.778779    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8d30854-fb1f-4629-abba-2e029ffb65a1" containerName="main"
Sep 16 20:16:33.778908 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:33.778791    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="befd599e-cad2-4c9d-9576-95d4cf044119" containerName="main"
Sep 16 20:16:33.783636 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:33.783617    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-5vnqn"
Sep 16 20:16:33.787420 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:33.787398    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"openshift-service-ca.crt\""
Sep 16 20:16:33.787649 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:33.787431    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"default-dockercfg-djmhg\""
Sep 16 20:16:33.787743 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:33.787432    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"leave-v2-kserve-self-signed-certs\""
Sep 16 20:16:33.787743 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:33.787462    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"kube-root-ca.crt\""
Sep 16 20:16:33.793319 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:33.793291    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-5vnqn"]
Sep 16 20:16:33.933276 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:33.933246    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46125e98-1656-42dd-97a4-eab8761702a9-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-5vnqn\" (UID: \"46125e98-1656-42dd-97a4-eab8761702a9\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-5vnqn"
Sep 16 20:16:33.933443 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:33.933300    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z87r\" (UniqueName: \"kubernetes.io/projected/46125e98-1656-42dd-97a4-eab8761702a9-kube-api-access-4z87r\") pod \"leave-v2-kserve-556b4dddf9-5vnqn\" (UID: \"46125e98-1656-42dd-97a4-eab8761702a9\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-5vnqn"
Sep 16 20:16:33.933443 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:33.933321    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46125e98-1656-42dd-97a4-eab8761702a9-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-5vnqn\" (UID: \"46125e98-1656-42dd-97a4-eab8761702a9\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-5vnqn"
Sep 16 20:16:33.933443 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:33.933339    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46125e98-1656-42dd-97a4-eab8761702a9-home\") pod \"leave-v2-kserve-556b4dddf9-5vnqn\" (UID: \"46125e98-1656-42dd-97a4-eab8761702a9\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-5vnqn"
Sep 16 20:16:33.933443 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:33.933380    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46125e98-1656-42dd-97a4-eab8761702a9-dshm\") pod \"leave-v2-kserve-556b4dddf9-5vnqn\" (UID: \"46125e98-1656-42dd-97a4-eab8761702a9\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-5vnqn"
Sep 16 20:16:33.933584 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:33.933512    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46125e98-1656-42dd-97a4-eab8761702a9-model-cache\") pod \"leave-v2-kserve-556b4dddf9-5vnqn\" (UID: \"46125e98-1656-42dd-97a4-eab8761702a9\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-5vnqn"
Sep 16 20:16:34.034666 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:34.034571    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z87r\" (UniqueName: \"kubernetes.io/projected/46125e98-1656-42dd-97a4-eab8761702a9-kube-api-access-4z87r\") pod \"leave-v2-kserve-556b4dddf9-5vnqn\" (UID: \"46125e98-1656-42dd-97a4-eab8761702a9\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-5vnqn"
Sep 16 20:16:34.034666 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:34.034612    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46125e98-1656-42dd-97a4-eab8761702a9-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-5vnqn\" (UID: \"46125e98-1656-42dd-97a4-eab8761702a9\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-5vnqn"
Sep 16 20:16:34.034666 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:34.034633    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46125e98-1656-42dd-97a4-eab8761702a9-home\") pod \"leave-v2-kserve-556b4dddf9-5vnqn\" (UID: \"46125e98-1656-42dd-97a4-eab8761702a9\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-5vnqn"
Sep 16 20:16:34.034666 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:34.034653    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46125e98-1656-42dd-97a4-eab8761702a9-dshm\") pod \"leave-v2-kserve-556b4dddf9-5vnqn\" (UID: \"46125e98-1656-42dd-97a4-eab8761702a9\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-5vnqn"
Sep 16 20:16:34.035059 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:34.034714    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46125e98-1656-42dd-97a4-eab8761702a9-model-cache\") pod \"leave-v2-kserve-556b4dddf9-5vnqn\" (UID: \"46125e98-1656-42dd-97a4-eab8761702a9\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-5vnqn"
Sep 16 20:16:34.035059 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:34.034799    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46125e98-1656-42dd-97a4-eab8761702a9-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-5vnqn\" (UID: \"46125e98-1656-42dd-97a4-eab8761702a9\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-5vnqn"
Sep 16 20:16:34.035172 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:34.035057    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46125e98-1656-42dd-97a4-eab8761702a9-home\") pod \"leave-v2-kserve-556b4dddf9-5vnqn\" (UID: \"46125e98-1656-42dd-97a4-eab8761702a9\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-5vnqn"
Sep 16 20:16:34.035172 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:34.035108    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46125e98-1656-42dd-97a4-eab8761702a9-model-cache\") pod \"leave-v2-kserve-556b4dddf9-5vnqn\" (UID: \"46125e98-1656-42dd-97a4-eab8761702a9\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-5vnqn"
Sep 16 20:16:34.035172 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:34.035150    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46125e98-1656-42dd-97a4-eab8761702a9-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-5vnqn\" (UID: \"46125e98-1656-42dd-97a4-eab8761702a9\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-5vnqn"
Sep 16 20:16:34.037051 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:34.037027    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46125e98-1656-42dd-97a4-eab8761702a9-dshm\") pod \"leave-v2-kserve-556b4dddf9-5vnqn\" (UID: \"46125e98-1656-42dd-97a4-eab8761702a9\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-5vnqn"
Sep 16 20:16:34.037314 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:34.037298    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46125e98-1656-42dd-97a4-eab8761702a9-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-5vnqn\" (UID: \"46125e98-1656-42dd-97a4-eab8761702a9\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-5vnqn"
Sep 16 20:16:34.042986 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:34.042962    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z87r\" (UniqueName: \"kubernetes.io/projected/46125e98-1656-42dd-97a4-eab8761702a9-kube-api-access-4z87r\") pod \"leave-v2-kserve-556b4dddf9-5vnqn\" (UID: \"46125e98-1656-42dd-97a4-eab8761702a9\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-5vnqn"
Sep 16 20:16:34.094823 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:34.094786    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-5vnqn"
Sep 16 20:16:34.380821 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:34.380794    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-5vnqn"]
Sep 16 20:16:34.383647 ip-10-0-139-81 kubenswrapper[2557]: W0916 20:16:34.383612    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46125e98_1656_42dd_97a4_eab8761702a9.slice/crio-888f346f8722a5f273f898bdbed17a552535d70985c23711cb75a08d7d4ef5d3 WatchSource:0}: Error finding container 888f346f8722a5f273f898bdbed17a552535d70985c23711cb75a08d7d4ef5d3: Status 404 returned error can't find the container with id 888f346f8722a5f273f898bdbed17a552535d70985c23711cb75a08d7d4ef5d3
Sep 16 20:16:34.863157 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:34.863121    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-5vnqn" event={"ID":"46125e98-1656-42dd-97a4-eab8761702a9","Type":"ContainerStarted","Data":"d363969e171bf6a88cc5b76fe2a261b4e7b6694b69f98e08edbd5d788b0d85fe"}
Sep 16 20:16:34.863536 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:34.863214    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-5vnqn" event={"ID":"46125e98-1656-42dd-97a4-eab8761702a9","Type":"ContainerStarted","Data":"888f346f8722a5f273f898bdbed17a552535d70985c23711cb75a08d7d4ef5d3"}
Sep 16 20:16:34.884205 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:34.884153    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-5vnqn" podStartSLOduration=1.884138307 podStartE2EDuration="1.884138307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:16:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:16:34.882074764 +0000 UTC m=+2223.407463149" watchObservedRunningTime="2026-09-16 20:16:34.884138307 +0000 UTC m=+2223.409526692"
Sep 16 20:16:35.095858 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:35.095809    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-5vnqn"
Sep 16 20:16:44.095613 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:44.095570    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-5vnqn"
Sep 16 20:16:44.108057 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:44.108033    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-5vnqn"
Sep 16 20:16:44.924187 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:16:44.924154    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-5vnqn"
Sep 16 20:17:01.092751 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:17:01.092675    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefrth2"]
Sep 16 20:17:01.095355 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:17:01.095334    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefrth2"
Sep 16 20:17:01.099539 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:17:01.099517    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Sep 16 20:17:01.099660 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:17:01.099589    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Sep 16 20:17:01.099660 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:17:01.099615    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-5p92g\""
Sep 16 20:17:01.099660 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:17:01.099622    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisvc-router-managed-test-llm-e45d1f79-epp-sa-dockercfg-486h4\""
Sep 16 20:17:01.099822 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:17:01.099757    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Sep 16 20:17:01.105262 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:17:01.105238    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefrth2"]
Sep 16 20:17:01.157195 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:17:01.157157    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c0a4f03-7e2f-471e-a6fa-603d98fd977e-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefrth2\" (UID: \"8c0a4f03-7e2f-471e-a6fa-603d98fd977e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefrth2"
Sep 16 20:17:01.157348 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:17:01.157218    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8tgz\" (UniqueName: \"kubernetes.io/projected/8c0a4f03-7e2f-471e-a6fa-603d98fd977e-kube-api-access-h8tgz\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefrth2\" (UID: \"8c0a4f03-7e2f-471e-a6fa-603d98fd977e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefrth2"
Sep 16 20:17:01.257744 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:17:01.257710    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c0a4f03-7e2f-471e-a6fa-603d98fd977e-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefrth2\" (UID: \"8c0a4f03-7e2f-471e-a6fa-603d98fd977e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefrth2"
Sep 16 20:17:01.257942 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:17:01.257779    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8tgz\" (UniqueName: \"kubernetes.io/projected/8c0a4f03-7e2f-471e-a6fa-603d98fd977e-kube-api-access-h8tgz\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefrth2\" (UID: \"8c0a4f03-7e2f-471e-a6fa-603d98fd977e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefrth2"
Sep 16 20:17:01.260224 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:17:01.260202    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c0a4f03-7e2f-471e-a6fa-603d98fd977e-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefrth2\" (UID: \"8c0a4f03-7e2f-471e-a6fa-603d98fd977e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefrth2"
Sep 16 20:17:01.265758 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:17:01.265730    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8tgz\" (UniqueName: \"kubernetes.io/projected/8c0a4f03-7e2f-471e-a6fa-603d98fd977e-kube-api-access-h8tgz\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefrth2\" (UID: \"8c0a4f03-7e2f-471e-a6fa-603d98fd977e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefrth2"
Sep 16 20:17:01.407084 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:17:01.406992    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefrth2"
Sep 16 20:17:01.683204 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:17:01.683177    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefrth2"]
Sep 16 20:17:01.685954 ip-10-0-139-81 kubenswrapper[2557]: W0916 20:17:01.685910    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c0a4f03_7e2f_471e_a6fa_603d98fd977e.slice/crio-edf2cc87120366b96e23ef568fee9f6e9a5cdc028a774957e7ff920155eb315a WatchSource:0}: Error finding container edf2cc87120366b96e23ef568fee9f6e9a5cdc028a774957e7ff920155eb315a: Status 404 returned error can't find the container with id edf2cc87120366b96e23ef568fee9f6e9a5cdc028a774957e7ff920155eb315a
Sep 16 20:17:02.005726 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:17:02.005686    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefrth2" event={"ID":"8c0a4f03-7e2f-471e-a6fa-603d98fd977e","Type":"ContainerStarted","Data":"bac54d8d51782b97ab60804c2155fffa1e01031981a8bd566da44086a3802626"}
Sep 16 20:17:02.005943 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:17:02.005745    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefrth2" event={"ID":"8c0a4f03-7e2f-471e-a6fa-603d98fd977e","Type":"ContainerStarted","Data":"edf2cc87120366b96e23ef568fee9f6e9a5cdc028a774957e7ff920155eb315a"}
Sep 16 20:17:02.005943 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:17:02.005781    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefrth2"
Sep 16 20:17:02.026261 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:17:02.026180    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefrth2" podStartSLOduration=1.026158433 podStartE2EDuration="1.026158433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:17:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:17:02.02268527 +0000 UTC m=+2250.548073654" watchObservedRunningTime="2026-09-16 20:17:02.026158433 +0000 UTC m=+2250.551546811"
Sep 16 20:17:33.012594 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:17:33.012552    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefrth2"
Sep 16 20:17:35.773418 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:17:35.773381    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-5vnqn"]
Sep 16 20:17:50.794457 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:17:50.794410    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-5vnqn" podUID="46125e98-1656-42dd-97a4-eab8761702a9" containerName="main" containerID="cri-o://d363969e171bf6a88cc5b76fe2a261b4e7b6694b69f98e08edbd5d788b0d85fe" gracePeriod=45
Sep 16 20:17:50.914012 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:17:50.913973    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-5vnqn" podUID="46125e98-1656-42dd-97a4-eab8761702a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Sep 16 20:17:51.051208 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:17:51.051147    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-5vnqn"
Sep 16 20:17:51.114118 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:17:51.114078    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46125e98-1656-42dd-97a4-eab8761702a9-home\") pod \"46125e98-1656-42dd-97a4-eab8761702a9\" (UID: \"46125e98-1656-42dd-97a4-eab8761702a9\") "
Sep 16 20:17:51.114310 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:17:51.114153    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46125e98-1656-42dd-97a4-eab8761702a9-dshm\") pod \"46125e98-1656-42dd-97a4-eab8761702a9\" (UID: \"46125e98-1656-42dd-97a4-eab8761702a9\") "
Sep 16 20:17:51.114310 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:17:51.114187    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46125e98-1656-42dd-97a4-eab8761702a9-home" (OuterVolumeSpecName: "home") pod "46125e98-1656-42dd-97a4-eab8761702a9" (UID: "46125e98-1656-42dd-97a4-eab8761702a9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:17:51.114310 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:17:51.114194    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4z87r\" (UniqueName: \"kubernetes.io/projected/46125e98-1656-42dd-97a4-eab8761702a9-kube-api-access-4z87r\") pod \"46125e98-1656-42dd-97a4-eab8761702a9\" (UID: \"46125e98-1656-42dd-97a4-eab8761702a9\") "
Sep 16 20:17:51.114310 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:17:51.114270    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46125e98-1656-42dd-97a4-eab8761702a9-model-cache\") pod \"46125e98-1656-42dd-97a4-eab8761702a9\" (UID: \"46125e98-1656-42dd-97a4-eab8761702a9\") "
Sep 16 20:17:51.114310 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:17:51.114303    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46125e98-1656-42dd-97a4-eab8761702a9-tls-certs\") pod \"46125e98-1656-42dd-97a4-eab8761702a9\" (UID: \"46125e98-1656-42dd-97a4-eab8761702a9\") "
Sep 16 20:17:51.114582 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:17:51.114320    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46125e98-1656-42dd-97a4-eab8761702a9-tmp-dir\") pod \"46125e98-1656-42dd-97a4-eab8761702a9\" (UID: \"46125e98-1656-42dd-97a4-eab8761702a9\") "
Sep 16 20:17:51.114582 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:17:51.114572    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46125e98-1656-42dd-97a4-eab8761702a9-home\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:17:51.114668 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:17:51.114549    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46125e98-1656-42dd-97a4-eab8761702a9-model-cache" (OuterVolumeSpecName: "model-cache") pod "46125e98-1656-42dd-97a4-eab8761702a9" (UID: "46125e98-1656-42dd-97a4-eab8761702a9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:17:51.114668 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:17:51.114607    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46125e98-1656-42dd-97a4-eab8761702a9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "46125e98-1656-42dd-97a4-eab8761702a9" (UID: "46125e98-1656-42dd-97a4-eab8761702a9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:17:51.116517 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:17:51.116495    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46125e98-1656-42dd-97a4-eab8761702a9-kube-api-access-4z87r" (OuterVolumeSpecName: "kube-api-access-4z87r") pod "46125e98-1656-42dd-97a4-eab8761702a9" (UID: "46125e98-1656-42dd-97a4-eab8761702a9"). InnerVolumeSpecName "kube-api-access-4z87r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:17:51.116600 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:17:51.116510    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46125e98-1656-42dd-97a4-eab8761702a9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "46125e98-1656-42dd-97a4-eab8761702a9" (UID: "46125e98-1656-42dd-97a4-eab8761702a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:17:51.116600 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:17:51.116545    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46125e98-1656-42dd-97a4-eab8761702a9-dshm" (OuterVolumeSpecName: "dshm") pod "46125e98-1656-42dd-97a4-eab8761702a9" (UID: "46125e98-1656-42dd-97a4-eab8761702a9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:17:51.215428 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:17:51.215379    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46125e98-1656-42dd-97a4-eab8761702a9-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:17:51.215428 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:17:51.215423    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46125e98-1656-42dd-97a4-eab8761702a9-tmp-dir\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:17:51.215428 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:17:51.215432    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46125e98-1656-42dd-97a4-eab8761702a9-dshm\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:17:51.215428 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:17:51.215442    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z87r\" (UniqueName: \"kubernetes.io/projected/46125e98-1656-42dd-97a4-eab8761702a9-kube-api-access-4z87r\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:17:51.215718 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:17:51.215451    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46125e98-1656-42dd-97a4-eab8761702a9-model-cache\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:17:51.254930 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:17:51.254899    2557 generic.go:356] "Generic (PLEG): container finished" podID="46125e98-1656-42dd-97a4-eab8761702a9" containerID="d363969e171bf6a88cc5b76fe2a261b4e7b6694b69f98e08edbd5d788b0d85fe" exitCode=0
Sep 16 20:17:51.255075 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:17:51.254942    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-5vnqn"
Sep 16 20:17:51.255075 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:17:51.254989    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-5vnqn" event={"ID":"46125e98-1656-42dd-97a4-eab8761702a9","Type":"ContainerDied","Data":"d363969e171bf6a88cc5b76fe2a261b4e7b6694b69f98e08edbd5d788b0d85fe"}
Sep 16 20:17:51.255075 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:17:51.255036    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-5vnqn" event={"ID":"46125e98-1656-42dd-97a4-eab8761702a9","Type":"ContainerDied","Data":"888f346f8722a5f273f898bdbed17a552535d70985c23711cb75a08d7d4ef5d3"}
Sep 16 20:17:51.255075 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:17:51.255057    2557 scope.go:117] "RemoveContainer" containerID="d363969e171bf6a88cc5b76fe2a261b4e7b6694b69f98e08edbd5d788b0d85fe"
Sep 16 20:17:51.266681 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:17:51.266186    2557 scope.go:117] "RemoveContainer" containerID="d363969e171bf6a88cc5b76fe2a261b4e7b6694b69f98e08edbd5d788b0d85fe"
Sep 16 20:17:51.266681 ip-10-0-139-81 kubenswrapper[2557]: E0916 20:17:51.266558    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d363969e171bf6a88cc5b76fe2a261b4e7b6694b69f98e08edbd5d788b0d85fe\": container with ID starting with d363969e171bf6a88cc5b76fe2a261b4e7b6694b69f98e08edbd5d788b0d85fe not found: ID does not exist" containerID="d363969e171bf6a88cc5b76fe2a261b4e7b6694b69f98e08edbd5d788b0d85fe"
Sep 16 20:17:51.266681 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:17:51.266583    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d363969e171bf6a88cc5b76fe2a261b4e7b6694b69f98e08edbd5d788b0d85fe"} err="failed to get container status \"d363969e171bf6a88cc5b76fe2a261b4e7b6694b69f98e08edbd5d788b0d85fe\": rpc error: code = NotFound desc = could not find container \"d363969e171bf6a88cc5b76fe2a261b4e7b6694b69f98e08edbd5d788b0d85fe\": container with ID starting with d363969e171bf6a88cc5b76fe2a261b4e7b6694b69f98e08edbd5d788b0d85fe not found: ID does not exist"
Sep 16 20:17:51.277545 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:17:51.277484    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-5vnqn"]
Sep 16 20:17:51.279287 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:17:51.279266    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-5vnqn"]
Sep 16 20:17:51.938094 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:17:51.938054    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46125e98-1656-42dd-97a4-eab8761702a9" path="/var/lib/kubelet/pods/46125e98-1656-42dd-97a4-eab8761702a9/volumes"
Sep 16 20:18:06.059929 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:18:06.059894    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefrth2"]
Sep 16 20:18:13.012882 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:18:13.012825    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefrth2" podUID="8c0a4f03-7e2f-471e-a6fa-603d98fd977e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:18:21.080040 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:18:21.079991    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefrth2" podUID="8c0a4f03-7e2f-471e-a6fa-603d98fd977e" containerName="main" containerID="cri-o://bac54d8d51782b97ab60804c2155fffa1e01031981a8bd566da44086a3802626" gracePeriod=45
Sep 16 20:18:23.012794 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:18:23.012758    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefrth2" podUID="8c0a4f03-7e2f-471e-a6fa-603d98fd977e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:18:33.012878 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:18:33.012815    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefrth2" podUID="8c0a4f03-7e2f-471e-a6fa-603d98fd977e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:18:33.013274 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:18:33.012944    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefrth2"
Sep 16 20:18:43.012818 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:18:43.012768    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefrth2" podUID="8c0a4f03-7e2f-471e-a6fa-603d98fd977e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:18:51.338012 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:18:51.337981    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefrth2"
Sep 16 20:18:51.449121 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:18:51.449028    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c0a4f03-7e2f-471e-a6fa-603d98fd977e-tls-certs\") pod \"8c0a4f03-7e2f-471e-a6fa-603d98fd977e\" (UID: \"8c0a4f03-7e2f-471e-a6fa-603d98fd977e\") "
Sep 16 20:18:51.449121 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:18:51.449077    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8tgz\" (UniqueName: \"kubernetes.io/projected/8c0a4f03-7e2f-471e-a6fa-603d98fd977e-kube-api-access-h8tgz\") pod \"8c0a4f03-7e2f-471e-a6fa-603d98fd977e\" (UID: \"8c0a4f03-7e2f-471e-a6fa-603d98fd977e\") "
Sep 16 20:18:51.451261 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:18:51.451224    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c0a4f03-7e2f-471e-a6fa-603d98fd977e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8c0a4f03-7e2f-471e-a6fa-603d98fd977e" (UID: "8c0a4f03-7e2f-471e-a6fa-603d98fd977e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:18:51.451376 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:18:51.451321    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c0a4f03-7e2f-471e-a6fa-603d98fd977e-kube-api-access-h8tgz" (OuterVolumeSpecName: "kube-api-access-h8tgz") pod "8c0a4f03-7e2f-471e-a6fa-603d98fd977e" (UID: "8c0a4f03-7e2f-471e-a6fa-603d98fd977e"). InnerVolumeSpecName "kube-api-access-h8tgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:18:51.549769 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:18:51.549736    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c0a4f03-7e2f-471e-a6fa-603d98fd977e-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:18:51.549769 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:18:51.549762    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8tgz\" (UniqueName: \"kubernetes.io/projected/8c0a4f03-7e2f-471e-a6fa-603d98fd977e-kube-api-access-h8tgz\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:18:51.553107 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:18:51.553077    2557 generic.go:356] "Generic (PLEG): container finished" podID="8c0a4f03-7e2f-471e-a6fa-603d98fd977e" containerID="bac54d8d51782b97ab60804c2155fffa1e01031981a8bd566da44086a3802626" exitCode=0
Sep 16 20:18:51.553204 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:18:51.553121    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefrth2"
Sep 16 20:18:51.553204 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:18:51.553167    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefrth2" event={"ID":"8c0a4f03-7e2f-471e-a6fa-603d98fd977e","Type":"ContainerDied","Data":"bac54d8d51782b97ab60804c2155fffa1e01031981a8bd566da44086a3802626"}
Sep 16 20:18:51.553283 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:18:51.553223    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefrth2" event={"ID":"8c0a4f03-7e2f-471e-a6fa-603d98fd977e","Type":"ContainerDied","Data":"edf2cc87120366b96e23ef568fee9f6e9a5cdc028a774957e7ff920155eb315a"}
Sep 16 20:18:51.553283 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:18:51.553248    2557 scope.go:117] "RemoveContainer" containerID="bac54d8d51782b97ab60804c2155fffa1e01031981a8bd566da44086a3802626"
Sep 16 20:18:51.562941 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:18:51.562921    2557 scope.go:117] "RemoveContainer" containerID="bac54d8d51782b97ab60804c2155fffa1e01031981a8bd566da44086a3802626"
Sep 16 20:18:51.563212 ip-10-0-139-81 kubenswrapper[2557]: E0916 20:18:51.563193    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bac54d8d51782b97ab60804c2155fffa1e01031981a8bd566da44086a3802626\": container with ID starting with bac54d8d51782b97ab60804c2155fffa1e01031981a8bd566da44086a3802626 not found: ID does not exist" containerID="bac54d8d51782b97ab60804c2155fffa1e01031981a8bd566da44086a3802626"
Sep 16 20:18:51.563280 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:18:51.563225    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bac54d8d51782b97ab60804c2155fffa1e01031981a8bd566da44086a3802626"} err="failed to get container status \"bac54d8d51782b97ab60804c2155fffa1e01031981a8bd566da44086a3802626\": rpc error: code = NotFound desc = could not find container \"bac54d8d51782b97ab60804c2155fffa1e01031981a8bd566da44086a3802626\": container with ID starting with bac54d8d51782b97ab60804c2155fffa1e01031981a8bd566da44086a3802626 not found: ID does not exist"
Sep 16 20:18:51.574880 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:18:51.574833    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefrth2"]
Sep 16 20:18:51.578820 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:18:51.578800    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefrth2"]
Sep 16 20:18:51.937373 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:18:51.937341    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c0a4f03-7e2f-471e-a6fa-603d98fd977e" path="/var/lib/kubelet/pods/8c0a4f03-7e2f-471e-a6fa-603d98fd977e/volumes"
Sep 16 20:19:30.185740 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:19:30.185706    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-vdfdk"]
Sep 16 20:19:30.186207 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:19:30.186121    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46125e98-1656-42dd-97a4-eab8761702a9" containerName="main"
Sep 16 20:19:30.186207 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:19:30.186133    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="46125e98-1656-42dd-97a4-eab8761702a9" containerName="main"
Sep 16 20:19:30.186207 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:19:30.186142    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c0a4f03-7e2f-471e-a6fa-603d98fd977e" containerName="main"
Sep 16 20:19:30.186207 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:19:30.186148    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c0a4f03-7e2f-471e-a6fa-603d98fd977e" containerName="main"
Sep 16 20:19:30.186341 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:19:30.186216    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="46125e98-1656-42dd-97a4-eab8761702a9" containerName="main"
Sep 16 20:19:30.186341 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:19:30.186225    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c0a4f03-7e2f-471e-a6fa-603d98fd977e" containerName="main"
Sep 16 20:19:30.189048 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:19:30.189032    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-vdfdk"
Sep 16 20:19:30.193226 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:19:30.193190    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Sep 16 20:19:30.193226 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:19:30.193203    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-lhn98\""
Sep 16 20:19:30.193405 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:19:30.193211    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 16 20:19:30.193405 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:19:30.193194    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 16 20:19:30.199816 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:19:30.199789    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-vdfdk"]
Sep 16 20:19:30.258827 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:19:30.258785    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22b75916-cc87-4375-a17e-f93abea8d5be-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-vdfdk\" (UID: \"22b75916-cc87-4375-a17e-f93abea8d5be\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-vdfdk"
Sep 16 20:19:30.259008 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:19:30.258924    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22b75916-cc87-4375-a17e-f93abea8d5be-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-vdfdk\" (UID: \"22b75916-cc87-4375-a17e-f93abea8d5be\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-vdfdk"
Sep 16 20:19:30.259008 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:19:30.258967    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22b75916-cc87-4375-a17e-f93abea8d5be-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-vdfdk\" (UID: \"22b75916-cc87-4375-a17e-f93abea8d5be\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-vdfdk"
Sep 16 20:19:30.259102 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:19:30.259059    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22b75916-cc87-4375-a17e-f93abea8d5be-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-vdfdk\" (UID: \"22b75916-cc87-4375-a17e-f93abea8d5be\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-vdfdk"
Sep 16 20:19:30.259146 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:19:30.259116    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d47lx\" (UniqueName: \"kubernetes.io/projected/22b75916-cc87-4375-a17e-f93abea8d5be-kube-api-access-d47lx\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-vdfdk\" (UID: \"22b75916-cc87-4375-a17e-f93abea8d5be\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-vdfdk"
Sep 16 20:19:30.259193 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:19:30.259143    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22b75916-cc87-4375-a17e-f93abea8d5be-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-vdfdk\" (UID: \"22b75916-cc87-4375-a17e-f93abea8d5be\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-vdfdk"
Sep 16 20:19:30.360114 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:19:30.360069    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22b75916-cc87-4375-a17e-f93abea8d5be-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-vdfdk\" (UID: \"22b75916-cc87-4375-a17e-f93abea8d5be\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-vdfdk"
Sep 16 20:19:30.360325 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:19:30.360190    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22b75916-cc87-4375-a17e-f93abea8d5be-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-vdfdk\" (UID: \"22b75916-cc87-4375-a17e-f93abea8d5be\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-vdfdk"
Sep 16 20:19:30.360325 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:19:30.360214    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22b75916-cc87-4375-a17e-f93abea8d5be-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-vdfdk\" (UID: \"22b75916-cc87-4375-a17e-f93abea8d5be\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-vdfdk"
Sep 16 20:19:30.360448 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:19:30.360369    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22b75916-cc87-4375-a17e-f93abea8d5be-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-vdfdk\" (UID: \"22b75916-cc87-4375-a17e-f93abea8d5be\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-vdfdk"
Sep 16 20:19:30.360508 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:19:30.360460    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d47lx\" (UniqueName: \"kubernetes.io/projected/22b75916-cc87-4375-a17e-f93abea8d5be-kube-api-access-d47lx\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-vdfdk\" (UID: \"22b75916-cc87-4375-a17e-f93abea8d5be\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-vdfdk"
Sep 16 20:19:30.360508 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:19:30.360493    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22b75916-cc87-4375-a17e-f93abea8d5be-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-vdfdk\" (UID: \"22b75916-cc87-4375-a17e-f93abea8d5be\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-vdfdk"
Sep 16 20:19:30.360677 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:19:30.360545    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22b75916-cc87-4375-a17e-f93abea8d5be-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-vdfdk\" (UID: \"22b75916-cc87-4375-a17e-f93abea8d5be\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-vdfdk"
Sep 16 20:19:30.360677 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:19:30.360655    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22b75916-cc87-4375-a17e-f93abea8d5be-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-vdfdk\" (UID: \"22b75916-cc87-4375-a17e-f93abea8d5be\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-vdfdk"
Sep 16 20:19:30.360771 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:19:30.360750    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22b75916-cc87-4375-a17e-f93abea8d5be-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-vdfdk\" (UID: \"22b75916-cc87-4375-a17e-f93abea8d5be\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-vdfdk"
Sep 16 20:19:30.362352 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:19:30.362334    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22b75916-cc87-4375-a17e-f93abea8d5be-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-vdfdk\" (UID: \"22b75916-cc87-4375-a17e-f93abea8d5be\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-vdfdk"
Sep 16 20:19:30.362818 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:19:30.362799    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22b75916-cc87-4375-a17e-f93abea8d5be-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-vdfdk\" (UID: \"22b75916-cc87-4375-a17e-f93abea8d5be\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-vdfdk"
Sep 16 20:19:30.368871 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:19:30.368829    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d47lx\" (UniqueName: \"kubernetes.io/projected/22b75916-cc87-4375-a17e-f93abea8d5be-kube-api-access-d47lx\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-vdfdk\" (UID: \"22b75916-cc87-4375-a17e-f93abea8d5be\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-vdfdk"
Sep 16 20:19:30.499865 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:19:30.499802    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-vdfdk"
Sep 16 20:19:30.775033 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:19:30.775009    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-vdfdk"]
Sep 16 20:19:31.754013 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:19:31.753973    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-vdfdk" event={"ID":"22b75916-cc87-4375-a17e-f93abea8d5be","Type":"ContainerStarted","Data":"29627b7d8d842fe51e7a5267797b029fdc22e65cf1c4ba2f7d77e925b95f648f"}
Sep 16 20:19:31.754449 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:19:31.754043    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-vdfdk" event={"ID":"22b75916-cc87-4375-a17e-f93abea8d5be","Type":"ContainerStarted","Data":"509b72f9eacf71b02fd96f5541c29ca7e61f04e9480d10e4bbe0a0a144d33415"}
Sep 16 20:19:31.782157 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:19:31.782084    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-vdfdk" podStartSLOduration=1.7820622830000001 podStartE2EDuration="1.782062283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:19:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:19:31.780015484 +0000 UTC m=+2400.305403870" watchObservedRunningTime="2026-09-16 20:19:31.782062283 +0000 UTC m=+2400.307450668"
Sep 16 20:19:32.500820 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:19:32.500772    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-vdfdk"
Sep 16 20:19:40.500547 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:19:40.500505    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-vdfdk"
Sep 16 20:19:40.512789 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:19:40.512758    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-vdfdk"
Sep 16 20:19:40.807824 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:19:40.807742    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-vdfdk"
Sep 16 20:20:24.275479 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:20:24.275444    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schergq5q"]
Sep 16 20:20:24.279038 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:20:24.279019    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schergq5q"
Sep 16 20:20:24.283109 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:20:24.283089    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Sep 16 20:20:24.283230 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:20:24.283094    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa-dockercfg-5bhxr\""
Sep 16 20:20:24.287725 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:20:24.287701    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schergq5q"]
Sep 16 20:20:24.346168 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:20:24.346140    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e5c7d86-ed21-48f6-b3f0-c26f6e794f21-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schergq5q\" (UID: \"8e5c7d86-ed21-48f6-b3f0-c26f6e794f21\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schergq5q"
Sep 16 20:20:24.346333 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:20:24.346183    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64rpf\" (UniqueName: \"kubernetes.io/projected/8e5c7d86-ed21-48f6-b3f0-c26f6e794f21-kube-api-access-64rpf\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schergq5q\" (UID: \"8e5c7d86-ed21-48f6-b3f0-c26f6e794f21\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schergq5q"
Sep 16 20:20:24.447523 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:20:24.447489    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e5c7d86-ed21-48f6-b3f0-c26f6e794f21-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schergq5q\" (UID: \"8e5c7d86-ed21-48f6-b3f0-c26f6e794f21\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schergq5q"
Sep 16 20:20:24.447710 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:20:24.447547    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64rpf\" (UniqueName: \"kubernetes.io/projected/8e5c7d86-ed21-48f6-b3f0-c26f6e794f21-kube-api-access-64rpf\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schergq5q\" (UID: \"8e5c7d86-ed21-48f6-b3f0-c26f6e794f21\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schergq5q"
Sep 16 20:20:24.450275 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:20:24.450245    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e5c7d86-ed21-48f6-b3f0-c26f6e794f21-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schergq5q\" (UID: \"8e5c7d86-ed21-48f6-b3f0-c26f6e794f21\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schergq5q"
Sep 16 20:20:24.456381 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:20:24.456353    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64rpf\" (UniqueName: \"kubernetes.io/projected/8e5c7d86-ed21-48f6-b3f0-c26f6e794f21-kube-api-access-64rpf\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schergq5q\" (UID: \"8e5c7d86-ed21-48f6-b3f0-c26f6e794f21\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schergq5q"
Sep 16 20:20:24.590468 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:20:24.590385    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schergq5q"
Sep 16 20:20:24.886979 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:20:24.886825    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schergq5q"]
Sep 16 20:20:25.018608 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:20:25.018566    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schergq5q" event={"ID":"8e5c7d86-ed21-48f6-b3f0-c26f6e794f21","Type":"ContainerStarted","Data":"3c4366c3cd042f5314df0d8e6cc2d1e2cac9fabffec74500294ad6bbfcc80100"}
Sep 16 20:20:25.018766 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:20:25.018676    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schergq5q"
Sep 16 20:20:25.018766 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:20:25.018704    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schergq5q" event={"ID":"8e5c7d86-ed21-48f6-b3f0-c26f6e794f21","Type":"ContainerStarted","Data":"175989421bb525aabbf1745b27b65e97409ce6032b144fd6c454ff907e534210"}
Sep 16 20:20:25.048206 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:20:25.048142    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schergq5q" podStartSLOduration=1.048122643 podStartE2EDuration="1.048122643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:20:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:20:25.044939012 +0000 UTC m=+2453.570327398" watchObservedRunningTime="2026-09-16 20:20:25.048122643 +0000 UTC m=+2453.573511029"
Sep 16 20:20:56.026186 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:20:56.026155    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schergq5q"
Sep 16 20:21:13.053714 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:21:13.053617    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schergq5q"]
Sep 16 20:21:16.025534 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:21:16.025494    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schergq5q" podUID="8e5c7d86-ed21-48f6-b3f0-c26f6e794f21" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:21:26.025401 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:21:26.025311    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schergq5q" podUID="8e5c7d86-ed21-48f6-b3f0-c26f6e794f21" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:21:28.073219 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:21:28.073158    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schergq5q" podUID="8e5c7d86-ed21-48f6-b3f0-c26f6e794f21" containerName="main" containerID="cri-o://3c4366c3cd042f5314df0d8e6cc2d1e2cac9fabffec74500294ad6bbfcc80100" gracePeriod=45
Sep 16 20:21:36.025531 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:21:36.025491    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schergq5q" podUID="8e5c7d86-ed21-48f6-b3f0-c26f6e794f21" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:21:36.025933 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:21:36.025605    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schergq5q"
Sep 16 20:21:46.026624 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:21:46.026579    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schergq5q" podUID="8e5c7d86-ed21-48f6-b3f0-c26f6e794f21" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:21:56.026021 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:21:56.025977    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schergq5q" podUID="8e5c7d86-ed21-48f6-b3f0-c26f6e794f21" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:21:58.313775 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:21:58.313749    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schergq5q"
Sep 16 20:21:58.404555 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:21:58.404466    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64rpf\" (UniqueName: \"kubernetes.io/projected/8e5c7d86-ed21-48f6-b3f0-c26f6e794f21-kube-api-access-64rpf\") pod \"8e5c7d86-ed21-48f6-b3f0-c26f6e794f21\" (UID: \"8e5c7d86-ed21-48f6-b3f0-c26f6e794f21\") "
Sep 16 20:21:58.404725 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:21:58.404568    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e5c7d86-ed21-48f6-b3f0-c26f6e794f21-tls-certs\") pod \"8e5c7d86-ed21-48f6-b3f0-c26f6e794f21\" (UID: \"8e5c7d86-ed21-48f6-b3f0-c26f6e794f21\") "
Sep 16 20:21:58.406913 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:21:58.406883    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e5c7d86-ed21-48f6-b3f0-c26f6e794f21-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8e5c7d86-ed21-48f6-b3f0-c26f6e794f21" (UID: "8e5c7d86-ed21-48f6-b3f0-c26f6e794f21"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:21:58.407087 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:21:58.407063    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e5c7d86-ed21-48f6-b3f0-c26f6e794f21-kube-api-access-64rpf" (OuterVolumeSpecName: "kube-api-access-64rpf") pod "8e5c7d86-ed21-48f6-b3f0-c26f6e794f21" (UID: "8e5c7d86-ed21-48f6-b3f0-c26f6e794f21"). InnerVolumeSpecName "kube-api-access-64rpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:21:58.493634 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:21:58.493600    2557 generic.go:356] "Generic (PLEG): container finished" podID="8e5c7d86-ed21-48f6-b3f0-c26f6e794f21" containerID="3c4366c3cd042f5314df0d8e6cc2d1e2cac9fabffec74500294ad6bbfcc80100" exitCode=0
Sep 16 20:21:58.493804 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:21:58.493643    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schergq5q"
Sep 16 20:21:58.493804 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:21:58.493694    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schergq5q" event={"ID":"8e5c7d86-ed21-48f6-b3f0-c26f6e794f21","Type":"ContainerDied","Data":"3c4366c3cd042f5314df0d8e6cc2d1e2cac9fabffec74500294ad6bbfcc80100"}
Sep 16 20:21:58.493804 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:21:58.493747    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schergq5q" event={"ID":"8e5c7d86-ed21-48f6-b3f0-c26f6e794f21","Type":"ContainerDied","Data":"175989421bb525aabbf1745b27b65e97409ce6032b144fd6c454ff907e534210"}
Sep 16 20:21:58.493804 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:21:58.493769    2557 scope.go:117] "RemoveContainer" containerID="3c4366c3cd042f5314df0d8e6cc2d1e2cac9fabffec74500294ad6bbfcc80100"
Sep 16 20:21:58.503634 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:21:58.503616    2557 scope.go:117] "RemoveContainer" containerID="3c4366c3cd042f5314df0d8e6cc2d1e2cac9fabffec74500294ad6bbfcc80100"
Sep 16 20:21:58.503972 ip-10-0-139-81 kubenswrapper[2557]: E0916 20:21:58.503953    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c4366c3cd042f5314df0d8e6cc2d1e2cac9fabffec74500294ad6bbfcc80100\": container with ID starting with 3c4366c3cd042f5314df0d8e6cc2d1e2cac9fabffec74500294ad6bbfcc80100 not found: ID does not exist" containerID="3c4366c3cd042f5314df0d8e6cc2d1e2cac9fabffec74500294ad6bbfcc80100"
Sep 16 20:21:58.504038 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:21:58.503982    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c4366c3cd042f5314df0d8e6cc2d1e2cac9fabffec74500294ad6bbfcc80100"} err="failed to get container status \"3c4366c3cd042f5314df0d8e6cc2d1e2cac9fabffec74500294ad6bbfcc80100\": rpc error: code = NotFound desc = could not find container \"3c4366c3cd042f5314df0d8e6cc2d1e2cac9fabffec74500294ad6bbfcc80100\": container with ID starting with 3c4366c3cd042f5314df0d8e6cc2d1e2cac9fabffec74500294ad6bbfcc80100 not found: ID does not exist"
Sep 16 20:21:58.505407 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:21:58.505386    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e5c7d86-ed21-48f6-b3f0-c26f6e794f21-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:21:58.505450 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:21:58.505410    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64rpf\" (UniqueName: \"kubernetes.io/projected/8e5c7d86-ed21-48f6-b3f0-c26f6e794f21-kube-api-access-64rpf\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:21:58.516445 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:21:58.516421    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schergq5q"]
Sep 16 20:21:58.520448 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:21:58.520424    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schergq5q"]
Sep 16 20:21:59.936974 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:21:59.936942    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e5c7d86-ed21-48f6-b3f0-c26f6e794f21" path="/var/lib/kubelet/pods/8e5c7d86-ed21-48f6-b3f0-c26f6e794f21/volumes"
Sep 16 20:22:19.439119 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:22:19.439083    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-vdfdk"]
Sep 16 20:22:19.450049 ip-10-0-139-81 kubenswrapper[2557]: E0916 20:22:19.450006    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 20:22:19.450049 ip-10-0-139-81 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:22:19.450049 ip-10-0-139-81 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 16 20:22:19.450049 ip-10-0-139-81 kubenswrapper[2557]:  > containerID="29627b7d8d842fe51e7a5267797b029fdc22e65cf1c4ba2f7d77e925b95f648f" cmd=["/bin/sleep","15"]
Sep 16 20:22:19.450380 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:22:19.450069    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-vdfdk" message="" err=<
Sep 16 20:22:19.450380 ip-10-0-139-81 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:22:19.450380 ip-10-0-139-81 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 16 20:22:19.450380 ip-10-0-139-81 kubenswrapper[2557]:  >
Sep 16 20:22:19.450380 ip-10-0-139-81 kubenswrapper[2557]: E0916 20:22:19.450086    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 20:22:19.450380 ip-10-0-139-81 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:22:19.450380 ip-10-0-139-81 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 16 20:22:19.450380 ip-10-0-139-81 kubenswrapper[2557]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-vdfdk" podUID="22b75916-cc87-4375-a17e-f93abea8d5be" containerName="main" containerID="cri-o://29627b7d8d842fe51e7a5267797b029fdc22e65cf1c4ba2f7d77e925b95f648f"
Sep 16 20:22:19.450380 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:22:19.450154    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-vdfdk" podUID="22b75916-cc87-4375-a17e-f93abea8d5be" containerName="main" containerID="cri-o://29627b7d8d842fe51e7a5267797b029fdc22e65cf1c4ba2f7d77e925b95f648f" gracePeriod=60
Sep 16 20:22:19.801853 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:22:19.801814    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-vdfdk"
Sep 16 20:22:19.904301 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:22:19.904266    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22b75916-cc87-4375-a17e-f93abea8d5be-dshm\") pod \"22b75916-cc87-4375-a17e-f93abea8d5be\" (UID: \"22b75916-cc87-4375-a17e-f93abea8d5be\") "
Sep 16 20:22:19.904498 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:22:19.904333    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22b75916-cc87-4375-a17e-f93abea8d5be-home\") pod \"22b75916-cc87-4375-a17e-f93abea8d5be\" (UID: \"22b75916-cc87-4375-a17e-f93abea8d5be\") "
Sep 16 20:22:19.904498 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:22:19.904378    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22b75916-cc87-4375-a17e-f93abea8d5be-tls-certs\") pod \"22b75916-cc87-4375-a17e-f93abea8d5be\" (UID: \"22b75916-cc87-4375-a17e-f93abea8d5be\") "
Sep 16 20:22:19.904498 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:22:19.904415    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22b75916-cc87-4375-a17e-f93abea8d5be-tmp-dir\") pod \"22b75916-cc87-4375-a17e-f93abea8d5be\" (UID: \"22b75916-cc87-4375-a17e-f93abea8d5be\") "
Sep 16 20:22:19.904498 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:22:19.904462    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22b75916-cc87-4375-a17e-f93abea8d5be-model-cache\") pod \"22b75916-cc87-4375-a17e-f93abea8d5be\" (UID: \"22b75916-cc87-4375-a17e-f93abea8d5be\") "
Sep 16 20:22:19.904727 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:22:19.904536    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d47lx\" (UniqueName: \"kubernetes.io/projected/22b75916-cc87-4375-a17e-f93abea8d5be-kube-api-access-d47lx\") pod \"22b75916-cc87-4375-a17e-f93abea8d5be\" (UID: \"22b75916-cc87-4375-a17e-f93abea8d5be\") "
Sep 16 20:22:19.904727 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:22:19.904675    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22b75916-cc87-4375-a17e-f93abea8d5be-home" (OuterVolumeSpecName: "home") pod "22b75916-cc87-4375-a17e-f93abea8d5be" (UID: "22b75916-cc87-4375-a17e-f93abea8d5be"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:22:19.904858 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:22:19.904765    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22b75916-cc87-4375-a17e-f93abea8d5be-model-cache" (OuterVolumeSpecName: "model-cache") pod "22b75916-cc87-4375-a17e-f93abea8d5be" (UID: "22b75916-cc87-4375-a17e-f93abea8d5be"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:22:19.904858 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:22:19.904777    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22b75916-cc87-4375-a17e-f93abea8d5be-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "22b75916-cc87-4375-a17e-f93abea8d5be" (UID: "22b75916-cc87-4375-a17e-f93abea8d5be"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:22:19.904954 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:22:19.904832    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22b75916-cc87-4375-a17e-f93abea8d5be-home\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:22:19.906606 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:22:19.906584    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22b75916-cc87-4375-a17e-f93abea8d5be-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "22b75916-cc87-4375-a17e-f93abea8d5be" (UID: "22b75916-cc87-4375-a17e-f93abea8d5be"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:22:19.906687 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:22:19.906600    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22b75916-cc87-4375-a17e-f93abea8d5be-dshm" (OuterVolumeSpecName: "dshm") pod "22b75916-cc87-4375-a17e-f93abea8d5be" (UID: "22b75916-cc87-4375-a17e-f93abea8d5be"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:22:19.906687 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:22:19.906623    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22b75916-cc87-4375-a17e-f93abea8d5be-kube-api-access-d47lx" (OuterVolumeSpecName: "kube-api-access-d47lx") pod "22b75916-cc87-4375-a17e-f93abea8d5be" (UID: "22b75916-cc87-4375-a17e-f93abea8d5be"). InnerVolumeSpecName "kube-api-access-d47lx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:22:20.005830 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:22:20.005794    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22b75916-cc87-4375-a17e-f93abea8d5be-dshm\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:22:20.005830 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:22:20.005826    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22b75916-cc87-4375-a17e-f93abea8d5be-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:22:20.005830 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:22:20.005862    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22b75916-cc87-4375-a17e-f93abea8d5be-tmp-dir\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:22:20.005830 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:22:20.005876    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22b75916-cc87-4375-a17e-f93abea8d5be-model-cache\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:22:20.006103 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:22:20.005888    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d47lx\" (UniqueName: \"kubernetes.io/projected/22b75916-cc87-4375-a17e-f93abea8d5be-kube-api-access-d47lx\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:22:20.610254 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:22:20.610213    2557 generic.go:356] "Generic (PLEG): container finished" podID="22b75916-cc87-4375-a17e-f93abea8d5be" containerID="29627b7d8d842fe51e7a5267797b029fdc22e65cf1c4ba2f7d77e925b95f648f" exitCode=0
Sep 16 20:22:20.610729 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:22:20.610260    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-vdfdk"
Sep 16 20:22:20.610729 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:22:20.610280    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-vdfdk" event={"ID":"22b75916-cc87-4375-a17e-f93abea8d5be","Type":"ContainerDied","Data":"29627b7d8d842fe51e7a5267797b029fdc22e65cf1c4ba2f7d77e925b95f648f"}
Sep 16 20:22:20.610729 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:22:20.610330    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-vdfdk" event={"ID":"22b75916-cc87-4375-a17e-f93abea8d5be","Type":"ContainerDied","Data":"509b72f9eacf71b02fd96f5541c29ca7e61f04e9480d10e4bbe0a0a144d33415"}
Sep 16 20:22:20.610729 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:22:20.610353    2557 scope.go:117] "RemoveContainer" containerID="29627b7d8d842fe51e7a5267797b029fdc22e65cf1c4ba2f7d77e925b95f648f"
Sep 16 20:22:20.619189 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:22:20.619170    2557 scope.go:117] "RemoveContainer" containerID="29627b7d8d842fe51e7a5267797b029fdc22e65cf1c4ba2f7d77e925b95f648f"
Sep 16 20:22:20.619432 ip-10-0-139-81 kubenswrapper[2557]: E0916 20:22:20.619415    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29627b7d8d842fe51e7a5267797b029fdc22e65cf1c4ba2f7d77e925b95f648f\": container with ID starting with 29627b7d8d842fe51e7a5267797b029fdc22e65cf1c4ba2f7d77e925b95f648f not found: ID does not exist" containerID="29627b7d8d842fe51e7a5267797b029fdc22e65cf1c4ba2f7d77e925b95f648f"
Sep 16 20:22:20.619487 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:22:20.619440    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29627b7d8d842fe51e7a5267797b029fdc22e65cf1c4ba2f7d77e925b95f648f"} err="failed to get container status \"29627b7d8d842fe51e7a5267797b029fdc22e65cf1c4ba2f7d77e925b95f648f\": rpc error: code = NotFound desc = could not find container \"29627b7d8d842fe51e7a5267797b029fdc22e65cf1c4ba2f7d77e925b95f648f\": container with ID starting with 29627b7d8d842fe51e7a5267797b029fdc22e65cf1c4ba2f7d77e925b95f648f not found: ID does not exist"
Sep 16 20:22:20.627330 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:22:20.627304    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-vdfdk"]
Sep 16 20:22:20.631912 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:22:20.631888    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-vdfdk"]
Sep 16 20:22:20.796777 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:22:20.796735    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-vdfdk" podUID="22b75916-cc87-4375-a17e-f93abea8d5be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 16 20:22:21.943539 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:22:21.943502    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22b75916-cc87-4375-a17e-f93abea8d5be" path="/var/lib/kubelet/pods/22b75916-cc87-4375-a17e-f93abea8d5be/volumes"
Sep 16 20:23:23.002733 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:23:23.002700    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6w27v"]
Sep 16 20:23:23.003162 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:23:23.003091    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e5c7d86-ed21-48f6-b3f0-c26f6e794f21" containerName="main"
Sep 16 20:23:23.003162 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:23:23.003104    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e5c7d86-ed21-48f6-b3f0-c26f6e794f21" containerName="main"
Sep 16 20:23:23.003162 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:23:23.003130    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22b75916-cc87-4375-a17e-f93abea8d5be" containerName="main"
Sep 16 20:23:23.003162 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:23:23.003136    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="22b75916-cc87-4375-a17e-f93abea8d5be" containerName="main"
Sep 16 20:23:23.003566 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:23:23.003202    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e5c7d86-ed21-48f6-b3f0-c26f6e794f21" containerName="main"
Sep 16 20:23:23.003566 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:23:23.003214    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="22b75916-cc87-4375-a17e-f93abea8d5be" containerName="main"
Sep 16 20:23:23.006378 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:23:23.006360    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6w27v"
Sep 16 20:23:23.010482 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:23:23.010453    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"delw-v1-kserve-self-signed-certs\""
Sep 16 20:23:23.010599 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:23:23.010483    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"kube-root-ca.crt\""
Sep 16 20:23:23.010599 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:23:23.010454    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"default-dockercfg-lklx4\""
Sep 16 20:23:23.010599 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:23:23.010459    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"openshift-service-ca.crt\""
Sep 16 20:23:23.019069 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:23:23.019048    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6w27v"]
Sep 16 20:23:23.078598 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:23:23.078566    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j48hx\" (UniqueName: \"kubernetes.io/projected/5ea7fe6d-8037-4e73-9db8-b24900d729b9-kube-api-access-j48hx\") pod \"delw-v1-kserve-6cf8cbb847-6w27v\" (UID: \"5ea7fe6d-8037-4e73-9db8-b24900d729b9\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6w27v"
Sep 16 20:23:23.078771 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:23:23.078627    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ea7fe6d-8037-4e73-9db8-b24900d729b9-home\") pod \"delw-v1-kserve-6cf8cbb847-6w27v\" (UID: \"5ea7fe6d-8037-4e73-9db8-b24900d729b9\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6w27v"
Sep 16 20:23:23.078771 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:23:23.078666    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ea7fe6d-8037-4e73-9db8-b24900d729b9-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-6w27v\" (UID: \"5ea7fe6d-8037-4e73-9db8-b24900d729b9\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6w27v"
Sep 16 20:23:23.078771 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:23:23.078690    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ea7fe6d-8037-4e73-9db8-b24900d729b9-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-6w27v\" (UID: \"5ea7fe6d-8037-4e73-9db8-b24900d729b9\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6w27v"
Sep 16 20:23:23.078771 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:23:23.078706    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ea7fe6d-8037-4e73-9db8-b24900d729b9-dshm\") pod \"delw-v1-kserve-6cf8cbb847-6w27v\" (UID: \"5ea7fe6d-8037-4e73-9db8-b24900d729b9\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6w27v"
Sep 16 20:23:23.078946 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:23:23.078764    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ea7fe6d-8037-4e73-9db8-b24900d729b9-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-6w27v\" (UID: \"5ea7fe6d-8037-4e73-9db8-b24900d729b9\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6w27v"
Sep 16 20:23:23.179153 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:23:23.179098    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ea7fe6d-8037-4e73-9db8-b24900d729b9-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-6w27v\" (UID: \"5ea7fe6d-8037-4e73-9db8-b24900d729b9\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6w27v"
Sep 16 20:23:23.179153 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:23:23.179166    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ea7fe6d-8037-4e73-9db8-b24900d729b9-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-6w27v\" (UID: \"5ea7fe6d-8037-4e73-9db8-b24900d729b9\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6w27v"
Sep 16 20:23:23.179431 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:23:23.179190    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ea7fe6d-8037-4e73-9db8-b24900d729b9-dshm\") pod \"delw-v1-kserve-6cf8cbb847-6w27v\" (UID: \"5ea7fe6d-8037-4e73-9db8-b24900d729b9\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6w27v"
Sep 16 20:23:23.179431 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:23:23.179226    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ea7fe6d-8037-4e73-9db8-b24900d729b9-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-6w27v\" (UID: \"5ea7fe6d-8037-4e73-9db8-b24900d729b9\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6w27v"
Sep 16 20:23:23.179431 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:23:23.179282    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j48hx\" (UniqueName: \"kubernetes.io/projected/5ea7fe6d-8037-4e73-9db8-b24900d729b9-kube-api-access-j48hx\") pod \"delw-v1-kserve-6cf8cbb847-6w27v\" (UID: \"5ea7fe6d-8037-4e73-9db8-b24900d729b9\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6w27v"
Sep 16 20:23:23.179431 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:23:23.179360    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ea7fe6d-8037-4e73-9db8-b24900d729b9-home\") pod \"delw-v1-kserve-6cf8cbb847-6w27v\" (UID: \"5ea7fe6d-8037-4e73-9db8-b24900d729b9\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6w27v"
Sep 16 20:23:23.179658 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:23:23.179571    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ea7fe6d-8037-4e73-9db8-b24900d729b9-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-6w27v\" (UID: \"5ea7fe6d-8037-4e73-9db8-b24900d729b9\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6w27v"
Sep 16 20:23:23.179718 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:23:23.179669    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ea7fe6d-8037-4e73-9db8-b24900d729b9-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-6w27v\" (UID: \"5ea7fe6d-8037-4e73-9db8-b24900d729b9\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6w27v"
Sep 16 20:23:23.179770 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:23:23.179718    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ea7fe6d-8037-4e73-9db8-b24900d729b9-home\") pod \"delw-v1-kserve-6cf8cbb847-6w27v\" (UID: \"5ea7fe6d-8037-4e73-9db8-b24900d729b9\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6w27v"
Sep 16 20:23:23.181584 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:23:23.181562    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ea7fe6d-8037-4e73-9db8-b24900d729b9-dshm\") pod \"delw-v1-kserve-6cf8cbb847-6w27v\" (UID: \"5ea7fe6d-8037-4e73-9db8-b24900d729b9\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6w27v"
Sep 16 20:23:23.181860 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:23:23.181830    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ea7fe6d-8037-4e73-9db8-b24900d729b9-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-6w27v\" (UID: \"5ea7fe6d-8037-4e73-9db8-b24900d729b9\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6w27v"
Sep 16 20:23:23.188696 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:23:23.188675    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j48hx\" (UniqueName: \"kubernetes.io/projected/5ea7fe6d-8037-4e73-9db8-b24900d729b9-kube-api-access-j48hx\") pod \"delw-v1-kserve-6cf8cbb847-6w27v\" (UID: \"5ea7fe6d-8037-4e73-9db8-b24900d729b9\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6w27v"
Sep 16 20:23:23.316649 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:23:23.316566    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6w27v"
Sep 16 20:23:23.596884 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:23:23.596827    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6w27v"]
Sep 16 20:23:23.599524 ip-10-0-139-81 kubenswrapper[2557]: W0916 20:23:23.599491    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ea7fe6d_8037_4e73_9db8_b24900d729b9.slice/crio-a489977072aeda94259feea82369ebb3d80d75fa6846d86665b68a7ba4d15ca1 WatchSource:0}: Error finding container a489977072aeda94259feea82369ebb3d80d75fa6846d86665b68a7ba4d15ca1: Status 404 returned error can't find the container with id a489977072aeda94259feea82369ebb3d80d75fa6846d86665b68a7ba4d15ca1
Sep 16 20:23:23.928385 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:23:23.928295    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6w27v" event={"ID":"5ea7fe6d-8037-4e73-9db8-b24900d729b9","Type":"ContainerStarted","Data":"09dc9f159965c60b3472083ad7f37af5a8bf43eff6caf5c9ec5a4783cb3b6dd5"}
Sep 16 20:23:23.928385 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:23:23.928356    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6w27v" event={"ID":"5ea7fe6d-8037-4e73-9db8-b24900d729b9","Type":"ContainerStarted","Data":"a489977072aeda94259feea82369ebb3d80d75fa6846d86665b68a7ba4d15ca1"}
Sep 16 20:23:23.946913 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:23:23.946866    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6w27v" podStartSLOduration=1.946826985 podStartE2EDuration="1.946826985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:23:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:23:23.946120367 +0000 UTC m=+2632.471508751" watchObservedRunningTime="2026-09-16 20:23:23.946826985 +0000 UTC m=+2632.472215370"
Sep 16 20:23:24.317527 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:23:24.317490    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6w27v"
Sep 16 20:23:33.317667 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:23:33.317629    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6w27v"
Sep 16 20:23:33.330439 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:23:33.330414    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6w27v"
Sep 16 20:23:33.989431 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:23:33.989402    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6w27v"
Sep 16 20:24:01.703067 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:24:01.703037    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesrltc"]
Sep 16 20:24:01.706911 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:24:01.706892    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesrltc"
Sep 16 20:24:01.711220 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:24:01.711196    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 16 20:24:01.711377 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:24:01.711202    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Sep 16 20:24:01.711377 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:24:01.711250    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-gcz2m\""
Sep 16 20:24:01.711377 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:24:01.711204    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 16 20:24:01.711377 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:24:01.711282    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-99lhv\""
Sep 16 20:24:01.719367 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:24:01.719346    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesrltc"]
Sep 16 20:24:01.744045 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:24:01.744011    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/638a229f-de6e-4cd8-b352-19a6ec43209b-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesrltc\" (UID: \"638a229f-de6e-4cd8-b352-19a6ec43209b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesrltc"
Sep 16 20:24:01.744196 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:24:01.744087    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm4s6\" (UniqueName: \"kubernetes.io/projected/638a229f-de6e-4cd8-b352-19a6ec43209b-kube-api-access-pm4s6\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesrltc\" (UID: \"638a229f-de6e-4cd8-b352-19a6ec43209b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesrltc"
Sep 16 20:24:01.844494 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:24:01.844455    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm4s6\" (UniqueName: \"kubernetes.io/projected/638a229f-de6e-4cd8-b352-19a6ec43209b-kube-api-access-pm4s6\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesrltc\" (UID: \"638a229f-de6e-4cd8-b352-19a6ec43209b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesrltc"
Sep 16 20:24:01.844662 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:24:01.844542    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/638a229f-de6e-4cd8-b352-19a6ec43209b-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesrltc\" (UID: \"638a229f-de6e-4cd8-b352-19a6ec43209b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesrltc"
Sep 16 20:24:01.847084 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:24:01.847055    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/638a229f-de6e-4cd8-b352-19a6ec43209b-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesrltc\" (UID: \"638a229f-de6e-4cd8-b352-19a6ec43209b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesrltc"
Sep 16 20:24:01.853520 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:24:01.853495    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm4s6\" (UniqueName: \"kubernetes.io/projected/638a229f-de6e-4cd8-b352-19a6ec43209b-kube-api-access-pm4s6\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesrltc\" (UID: \"638a229f-de6e-4cd8-b352-19a6ec43209b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesrltc"
Sep 16 20:24:02.017167 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:24:02.017125    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesrltc"
Sep 16 20:24:02.293671 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:24:02.293595    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesrltc"]
Sep 16 20:24:02.297035 ip-10-0-139-81 kubenswrapper[2557]: W0916 20:24:02.296997    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod638a229f_de6e_4cd8_b352_19a6ec43209b.slice/crio-f8441cb537ef97336354daa827da7b27a27c1c14ac4552fc147dca1ef4772bbf WatchSource:0}: Error finding container f8441cb537ef97336354daa827da7b27a27c1c14ac4552fc147dca1ef4772bbf: Status 404 returned error can't find the container with id f8441cb537ef97336354daa827da7b27a27c1c14ac4552fc147dca1ef4772bbf
Sep 16 20:24:03.127500 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:24:03.127462    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesrltc" event={"ID":"638a229f-de6e-4cd8-b352-19a6ec43209b","Type":"ContainerStarted","Data":"9d0fa878332978adc7eb5854f21e2b7dd3cfcd249f1edf76aa3355af102850cd"}
Sep 16 20:24:03.128010 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:24:03.127528    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesrltc" event={"ID":"638a229f-de6e-4cd8-b352-19a6ec43209b","Type":"ContainerStarted","Data":"f8441cb537ef97336354daa827da7b27a27c1c14ac4552fc147dca1ef4772bbf"}
Sep 16 20:24:03.128010 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:24:03.127598    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesrltc"
Sep 16 20:24:03.148624 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:24:03.148563    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesrltc" podStartSLOduration=2.148545205 podStartE2EDuration="2.148545205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:24:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:24:03.145970047 +0000 UTC m=+2671.671358432" watchObservedRunningTime="2026-09-16 20:24:03.148545205 +0000 UTC m=+2671.673933590"
Sep 16 20:24:34.133898 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:24:34.133813    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesrltc"
Sep 16 20:25:09.685635 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:09.685597    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6w27v"]
Sep 16 20:25:18.745995 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:18.745961    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-4cvw8"]
Sep 16 20:25:18.749618 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:18.749594    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-4cvw8"
Sep 16 20:25:18.770133 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:18.770111    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"default-dockercfg-89679\""
Sep 16 20:25:18.770262 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:18.770217    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"openshift-service-ca.crt\""
Sep 16 20:25:18.770262 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:18.770242    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"kube-root-ca.crt\""
Sep 16 20:25:18.770734 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:18.770717    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"roll-v1-kserve-self-signed-certs\""
Sep 16 20:25:18.781244 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:18.781218    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-4cvw8"]
Sep 16 20:25:18.852756 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:18.852721    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9756521-1750-40b5-a392-652cafc93db0-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-4cvw8\" (UID: \"f9756521-1750-40b5-a392-652cafc93db0\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-4cvw8"
Sep 16 20:25:18.852756 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:18.852758    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vrtj\" (UniqueName: \"kubernetes.io/projected/f9756521-1750-40b5-a392-652cafc93db0-kube-api-access-4vrtj\") pod \"roll-v1-kserve-69cf8bcfc-4cvw8\" (UID: \"f9756521-1750-40b5-a392-652cafc93db0\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-4cvw8"
Sep 16 20:25:18.852983 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:18.852797    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9756521-1750-40b5-a392-652cafc93db0-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-4cvw8\" (UID: \"f9756521-1750-40b5-a392-652cafc93db0\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-4cvw8"
Sep 16 20:25:18.852983 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:18.852911    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9756521-1750-40b5-a392-652cafc93db0-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-4cvw8\" (UID: \"f9756521-1750-40b5-a392-652cafc93db0\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-4cvw8"
Sep 16 20:25:18.852983 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:18.852940    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9756521-1750-40b5-a392-652cafc93db0-home\") pod \"roll-v1-kserve-69cf8bcfc-4cvw8\" (UID: \"f9756521-1750-40b5-a392-652cafc93db0\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-4cvw8"
Sep 16 20:25:18.852983 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:18.852968    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9756521-1750-40b5-a392-652cafc93db0-dshm\") pod \"roll-v1-kserve-69cf8bcfc-4cvw8\" (UID: \"f9756521-1750-40b5-a392-652cafc93db0\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-4cvw8"
Sep 16 20:25:18.953464 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:18.953425    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9756521-1750-40b5-a392-652cafc93db0-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-4cvw8\" (UID: \"f9756521-1750-40b5-a392-652cafc93db0\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-4cvw8"
Sep 16 20:25:18.953653 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:18.953486    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9756521-1750-40b5-a392-652cafc93db0-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-4cvw8\" (UID: \"f9756521-1750-40b5-a392-652cafc93db0\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-4cvw8"
Sep 16 20:25:18.953653 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:18.953505    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9756521-1750-40b5-a392-652cafc93db0-home\") pod \"roll-v1-kserve-69cf8bcfc-4cvw8\" (UID: \"f9756521-1750-40b5-a392-652cafc93db0\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-4cvw8"
Sep 16 20:25:18.953653 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:18.953521    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9756521-1750-40b5-a392-652cafc93db0-dshm\") pod \"roll-v1-kserve-69cf8bcfc-4cvw8\" (UID: \"f9756521-1750-40b5-a392-652cafc93db0\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-4cvw8"
Sep 16 20:25:18.953653 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:18.953560    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9756521-1750-40b5-a392-652cafc93db0-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-4cvw8\" (UID: \"f9756521-1750-40b5-a392-652cafc93db0\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-4cvw8"
Sep 16 20:25:18.953653 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:18.953579    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vrtj\" (UniqueName: \"kubernetes.io/projected/f9756521-1750-40b5-a392-652cafc93db0-kube-api-access-4vrtj\") pod \"roll-v1-kserve-69cf8bcfc-4cvw8\" (UID: \"f9756521-1750-40b5-a392-652cafc93db0\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-4cvw8"
Sep 16 20:25:18.953955 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:18.953902    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9756521-1750-40b5-a392-652cafc93db0-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-4cvw8\" (UID: \"f9756521-1750-40b5-a392-652cafc93db0\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-4cvw8"
Sep 16 20:25:18.953955 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:18.953940    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9756521-1750-40b5-a392-652cafc93db0-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-4cvw8\" (UID: \"f9756521-1750-40b5-a392-652cafc93db0\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-4cvw8"
Sep 16 20:25:18.954037 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:18.953974    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9756521-1750-40b5-a392-652cafc93db0-home\") pod \"roll-v1-kserve-69cf8bcfc-4cvw8\" (UID: \"f9756521-1750-40b5-a392-652cafc93db0\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-4cvw8"
Sep 16 20:25:18.955871 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:18.955831    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9756521-1750-40b5-a392-652cafc93db0-dshm\") pod \"roll-v1-kserve-69cf8bcfc-4cvw8\" (UID: \"f9756521-1750-40b5-a392-652cafc93db0\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-4cvw8"
Sep 16 20:25:18.956170 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:18.956152    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9756521-1750-40b5-a392-652cafc93db0-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-4cvw8\" (UID: \"f9756521-1750-40b5-a392-652cafc93db0\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-4cvw8"
Sep 16 20:25:18.963974 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:18.963952    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vrtj\" (UniqueName: \"kubernetes.io/projected/f9756521-1750-40b5-a392-652cafc93db0-kube-api-access-4vrtj\") pod \"roll-v1-kserve-69cf8bcfc-4cvw8\" (UID: \"f9756521-1750-40b5-a392-652cafc93db0\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-4cvw8"
Sep 16 20:25:19.060579 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:19.060481    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-4cvw8"
Sep 16 20:25:19.334248 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:19.334221    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-4cvw8"]
Sep 16 20:25:19.336803 ip-10-0-139-81 kubenswrapper[2557]: W0916 20:25:19.336773    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9756521_1750_40b5_a392_652cafc93db0.slice/crio-c3431475ea18201661bff81ee0875a43430ffafb69993472b6208a68a79b3e45 WatchSource:0}: Error finding container c3431475ea18201661bff81ee0875a43430ffafb69993472b6208a68a79b3e45: Status 404 returned error can't find the container with id c3431475ea18201661bff81ee0875a43430ffafb69993472b6208a68a79b3e45
Sep 16 20:25:19.520668 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:19.520632    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-4cvw8" event={"ID":"f9756521-1750-40b5-a392-652cafc93db0","Type":"ContainerStarted","Data":"dcbb81fa49e0bb45912ffd2cd0138dac9d520419bc105a1d55ec63bcd90b1d07"}
Sep 16 20:25:19.520852 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:19.520697    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-4cvw8" event={"ID":"f9756521-1750-40b5-a392-652cafc93db0","Type":"ContainerStarted","Data":"c3431475ea18201661bff81ee0875a43430ffafb69993472b6208a68a79b3e45"}
Sep 16 20:25:19.540644 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:19.540579    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-4cvw8" podStartSLOduration=1.540559269 podStartE2EDuration="1.540559269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:25:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:25:19.538781724 +0000 UTC m=+2748.064170111" watchObservedRunningTime="2026-09-16 20:25:19.540559269 +0000 UTC m=+2748.065947655"
Sep 16 20:25:20.061154 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:20.061121    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-4cvw8"
Sep 16 20:25:24.705119 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:24.705074    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6w27v" podUID="5ea7fe6d-8037-4e73-9db8-b24900d729b9" containerName="main" containerID="cri-o://09dc9f159965c60b3472083ad7f37af5a8bf43eff6caf5c9ec5a4783cb3b6dd5" gracePeriod=45
Sep 16 20:25:24.953180 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:24.953154    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6w27v"
Sep 16 20:25:25.113653 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:25.113620    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ea7fe6d-8037-4e73-9db8-b24900d729b9-home\") pod \"5ea7fe6d-8037-4e73-9db8-b24900d729b9\" (UID: \"5ea7fe6d-8037-4e73-9db8-b24900d729b9\") "
Sep 16 20:25:25.113876 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:25.113673    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ea7fe6d-8037-4e73-9db8-b24900d729b9-tmp-dir\") pod \"5ea7fe6d-8037-4e73-9db8-b24900d729b9\" (UID: \"5ea7fe6d-8037-4e73-9db8-b24900d729b9\") "
Sep 16 20:25:25.113876 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:25.113692    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ea7fe6d-8037-4e73-9db8-b24900d729b9-model-cache\") pod \"5ea7fe6d-8037-4e73-9db8-b24900d729b9\" (UID: \"5ea7fe6d-8037-4e73-9db8-b24900d729b9\") "
Sep 16 20:25:25.113876 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:25.113750    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j48hx\" (UniqueName: \"kubernetes.io/projected/5ea7fe6d-8037-4e73-9db8-b24900d729b9-kube-api-access-j48hx\") pod \"5ea7fe6d-8037-4e73-9db8-b24900d729b9\" (UID: \"5ea7fe6d-8037-4e73-9db8-b24900d729b9\") "
Sep 16 20:25:25.113876 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:25.113787    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ea7fe6d-8037-4e73-9db8-b24900d729b9-dshm\") pod \"5ea7fe6d-8037-4e73-9db8-b24900d729b9\" (UID: \"5ea7fe6d-8037-4e73-9db8-b24900d729b9\") "
Sep 16 20:25:25.113876 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:25.113823    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ea7fe6d-8037-4e73-9db8-b24900d729b9-tls-certs\") pod \"5ea7fe6d-8037-4e73-9db8-b24900d729b9\" (UID: \"5ea7fe6d-8037-4e73-9db8-b24900d729b9\") "
Sep 16 20:25:25.114161 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:25.113972    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ea7fe6d-8037-4e73-9db8-b24900d729b9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5ea7fe6d-8037-4e73-9db8-b24900d729b9" (UID: "5ea7fe6d-8037-4e73-9db8-b24900d729b9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:25:25.114161 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:25.113991    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ea7fe6d-8037-4e73-9db8-b24900d729b9-home" (OuterVolumeSpecName: "home") pod "5ea7fe6d-8037-4e73-9db8-b24900d729b9" (UID: "5ea7fe6d-8037-4e73-9db8-b24900d729b9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:25:25.114161 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:25.114039    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ea7fe6d-8037-4e73-9db8-b24900d729b9-model-cache" (OuterVolumeSpecName: "model-cache") pod "5ea7fe6d-8037-4e73-9db8-b24900d729b9" (UID: "5ea7fe6d-8037-4e73-9db8-b24900d729b9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:25:25.114161 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:25.114151    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ea7fe6d-8037-4e73-9db8-b24900d729b9-tmp-dir\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:25:25.114335 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:25.114168    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ea7fe6d-8037-4e73-9db8-b24900d729b9-model-cache\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:25:25.114335 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:25.114179    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ea7fe6d-8037-4e73-9db8-b24900d729b9-home\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:25:25.116176 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:25.116147    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ea7fe6d-8037-4e73-9db8-b24900d729b9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5ea7fe6d-8037-4e73-9db8-b24900d729b9" (UID: "5ea7fe6d-8037-4e73-9db8-b24900d729b9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:25:25.116290 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:25.116189    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ea7fe6d-8037-4e73-9db8-b24900d729b9-dshm" (OuterVolumeSpecName: "dshm") pod "5ea7fe6d-8037-4e73-9db8-b24900d729b9" (UID: "5ea7fe6d-8037-4e73-9db8-b24900d729b9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:25:25.116290 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:25.116209    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ea7fe6d-8037-4e73-9db8-b24900d729b9-kube-api-access-j48hx" (OuterVolumeSpecName: "kube-api-access-j48hx") pod "5ea7fe6d-8037-4e73-9db8-b24900d729b9" (UID: "5ea7fe6d-8037-4e73-9db8-b24900d729b9"). InnerVolumeSpecName "kube-api-access-j48hx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:25:25.214609 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:25.214574    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ea7fe6d-8037-4e73-9db8-b24900d729b9-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:25:25.214609 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:25.214604    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j48hx\" (UniqueName: \"kubernetes.io/projected/5ea7fe6d-8037-4e73-9db8-b24900d729b9-kube-api-access-j48hx\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:25:25.214609 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:25.214613    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ea7fe6d-8037-4e73-9db8-b24900d729b9-dshm\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:25:25.553923 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:25.553887    2557 generic.go:356] "Generic (PLEG): container finished" podID="5ea7fe6d-8037-4e73-9db8-b24900d729b9" containerID="09dc9f159965c60b3472083ad7f37af5a8bf43eff6caf5c9ec5a4783cb3b6dd5" exitCode=0
Sep 16 20:25:25.554132 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:25.553936    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6w27v"
Sep 16 20:25:25.554132 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:25.553972    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6w27v" event={"ID":"5ea7fe6d-8037-4e73-9db8-b24900d729b9","Type":"ContainerDied","Data":"09dc9f159965c60b3472083ad7f37af5a8bf43eff6caf5c9ec5a4783cb3b6dd5"}
Sep 16 20:25:25.554132 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:25.554039    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6w27v" event={"ID":"5ea7fe6d-8037-4e73-9db8-b24900d729b9","Type":"ContainerDied","Data":"a489977072aeda94259feea82369ebb3d80d75fa6846d86665b68a7ba4d15ca1"}
Sep 16 20:25:25.554132 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:25.554069    2557 scope.go:117] "RemoveContainer" containerID="09dc9f159965c60b3472083ad7f37af5a8bf43eff6caf5c9ec5a4783cb3b6dd5"
Sep 16 20:25:25.563599 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:25.563583    2557 scope.go:117] "RemoveContainer" containerID="09dc9f159965c60b3472083ad7f37af5a8bf43eff6caf5c9ec5a4783cb3b6dd5"
Sep 16 20:25:25.563870 ip-10-0-139-81 kubenswrapper[2557]: E0916 20:25:25.563853    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09dc9f159965c60b3472083ad7f37af5a8bf43eff6caf5c9ec5a4783cb3b6dd5\": container with ID starting with 09dc9f159965c60b3472083ad7f37af5a8bf43eff6caf5c9ec5a4783cb3b6dd5 not found: ID does not exist" containerID="09dc9f159965c60b3472083ad7f37af5a8bf43eff6caf5c9ec5a4783cb3b6dd5"
Sep 16 20:25:25.563941 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:25.563879    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09dc9f159965c60b3472083ad7f37af5a8bf43eff6caf5c9ec5a4783cb3b6dd5"} err="failed to get container status \"09dc9f159965c60b3472083ad7f37af5a8bf43eff6caf5c9ec5a4783cb3b6dd5\": rpc error: code = NotFound desc = could not find container \"09dc9f159965c60b3472083ad7f37af5a8bf43eff6caf5c9ec5a4783cb3b6dd5\": container with ID starting with 09dc9f159965c60b3472083ad7f37af5a8bf43eff6caf5c9ec5a4783cb3b6dd5 not found: ID does not exist"
Sep 16 20:25:25.580644 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:25.580622    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6w27v"]
Sep 16 20:25:25.584243 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:25.584221    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6w27v"]
Sep 16 20:25:25.937761 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:25.937674    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ea7fe6d-8037-4e73-9db8-b24900d729b9" path="/var/lib/kubelet/pods/5ea7fe6d-8037-4e73-9db8-b24900d729b9/volumes"
Sep 16 20:25:29.060988 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:29.060945    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-4cvw8"
Sep 16 20:25:29.073554 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:29.073521    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-4cvw8"
Sep 16 20:25:29.586986 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:25:29.586957    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-4cvw8"
Sep 16 20:26:28.817242 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:26:28.817206    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesrltc"]
Sep 16 20:26:34.133416 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:26:34.133374    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesrltc" podUID="638a229f-de6e-4cd8-b352-19a6ec43209b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:26:43.839329 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:26:43.839268    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesrltc" podUID="638a229f-de6e-4cd8-b352-19a6ec43209b" containerName="main" containerID="cri-o://9d0fa878332978adc7eb5854f21e2b7dd3cfcd249f1edf76aa3355af102850cd" gracePeriod=45
Sep 16 20:26:44.134367 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:26:44.134267    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesrltc" podUID="638a229f-de6e-4cd8-b352-19a6ec43209b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:26:44.237557 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:26:44.237522    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-4cvw8"]
Sep 16 20:26:54.134027 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:26:54.133986    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesrltc" podUID="638a229f-de6e-4cd8-b352-19a6ec43209b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:26:54.134489 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:26:54.134091    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesrltc"
Sep 16 20:26:59.257182 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:26:59.257139    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-4cvw8" podUID="f9756521-1750-40b5-a392-652cafc93db0" containerName="main" containerID="cri-o://dcbb81fa49e0bb45912ffd2cd0138dac9d520419bc105a1d55ec63bcd90b1d07" gracePeriod=45
Sep 16 20:26:59.502758 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:26:59.502725    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-4cvw8"
Sep 16 20:26:59.650664 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:26:59.650582    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vrtj\" (UniqueName: \"kubernetes.io/projected/f9756521-1750-40b5-a392-652cafc93db0-kube-api-access-4vrtj\") pod \"f9756521-1750-40b5-a392-652cafc93db0\" (UID: \"f9756521-1750-40b5-a392-652cafc93db0\") "
Sep 16 20:26:59.650664 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:26:59.650633    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9756521-1750-40b5-a392-652cafc93db0-model-cache\") pod \"f9756521-1750-40b5-a392-652cafc93db0\" (UID: \"f9756521-1750-40b5-a392-652cafc93db0\") "
Sep 16 20:26:59.650664 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:26:59.650651    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9756521-1750-40b5-a392-652cafc93db0-tmp-dir\") pod \"f9756521-1750-40b5-a392-652cafc93db0\" (UID: \"f9756521-1750-40b5-a392-652cafc93db0\") "
Sep 16 20:26:59.651093 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:26:59.650668    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9756521-1750-40b5-a392-652cafc93db0-dshm\") pod \"f9756521-1750-40b5-a392-652cafc93db0\" (UID: \"f9756521-1750-40b5-a392-652cafc93db0\") "
Sep 16 20:26:59.651093 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:26:59.650754    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9756521-1750-40b5-a392-652cafc93db0-home\") pod \"f9756521-1750-40b5-a392-652cafc93db0\" (UID: \"f9756521-1750-40b5-a392-652cafc93db0\") "
Sep 16 20:26:59.651093 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:26:59.650798    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9756521-1750-40b5-a392-652cafc93db0-tls-certs\") pod \"f9756521-1750-40b5-a392-652cafc93db0\" (UID: \"f9756521-1750-40b5-a392-652cafc93db0\") "
Sep 16 20:26:59.651093 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:26:59.650941    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9756521-1750-40b5-a392-652cafc93db0-model-cache" (OuterVolumeSpecName: "model-cache") pod "f9756521-1750-40b5-a392-652cafc93db0" (UID: "f9756521-1750-40b5-a392-652cafc93db0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:26:59.651093 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:26:59.650956    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9756521-1750-40b5-a392-652cafc93db0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f9756521-1750-40b5-a392-652cafc93db0" (UID: "f9756521-1750-40b5-a392-652cafc93db0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:26:59.651285 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:26:59.651088    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9756521-1750-40b5-a392-652cafc93db0-home" (OuterVolumeSpecName: "home") pod "f9756521-1750-40b5-a392-652cafc93db0" (UID: "f9756521-1750-40b5-a392-652cafc93db0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:26:59.651285 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:26:59.651213    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9756521-1750-40b5-a392-652cafc93db0-model-cache\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:26:59.651285 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:26:59.651250    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9756521-1750-40b5-a392-652cafc93db0-tmp-dir\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:26:59.651285 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:26:59.651260    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9756521-1750-40b5-a392-652cafc93db0-home\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:26:59.653025 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:26:59.652997    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9756521-1750-40b5-a392-652cafc93db0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f9756521-1750-40b5-a392-652cafc93db0" (UID: "f9756521-1750-40b5-a392-652cafc93db0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:26:59.653270 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:26:59.653244    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9756521-1750-40b5-a392-652cafc93db0-dshm" (OuterVolumeSpecName: "dshm") pod "f9756521-1750-40b5-a392-652cafc93db0" (UID: "f9756521-1750-40b5-a392-652cafc93db0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:26:59.653365 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:26:59.653266    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9756521-1750-40b5-a392-652cafc93db0-kube-api-access-4vrtj" (OuterVolumeSpecName: "kube-api-access-4vrtj") pod "f9756521-1750-40b5-a392-652cafc93db0" (UID: "f9756521-1750-40b5-a392-652cafc93db0"). InnerVolumeSpecName "kube-api-access-4vrtj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:26:59.752640 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:26:59.752601    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vrtj\" (UniqueName: \"kubernetes.io/projected/f9756521-1750-40b5-a392-652cafc93db0-kube-api-access-4vrtj\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:26:59.752640 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:26:59.752637    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9756521-1750-40b5-a392-652cafc93db0-dshm\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:26:59.752640 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:26:59.752649    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9756521-1750-40b5-a392-652cafc93db0-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:27:00.036822 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:00.036789    2557 generic.go:356] "Generic (PLEG): container finished" podID="f9756521-1750-40b5-a392-652cafc93db0" containerID="dcbb81fa49e0bb45912ffd2cd0138dac9d520419bc105a1d55ec63bcd90b1d07" exitCode=0
Sep 16 20:27:00.036972 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:00.036850    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-4cvw8"
Sep 16 20:27:00.036972 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:00.036873    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-4cvw8" event={"ID":"f9756521-1750-40b5-a392-652cafc93db0","Type":"ContainerDied","Data":"dcbb81fa49e0bb45912ffd2cd0138dac9d520419bc105a1d55ec63bcd90b1d07"}
Sep 16 20:27:00.036972 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:00.036933    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-4cvw8" event={"ID":"f9756521-1750-40b5-a392-652cafc93db0","Type":"ContainerDied","Data":"c3431475ea18201661bff81ee0875a43430ffafb69993472b6208a68a79b3e45"}
Sep 16 20:27:00.036972 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:00.036963    2557 scope.go:117] "RemoveContainer" containerID="dcbb81fa49e0bb45912ffd2cd0138dac9d520419bc105a1d55ec63bcd90b1d07"
Sep 16 20:27:00.045781 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:00.045763    2557 scope.go:117] "RemoveContainer" containerID="dcbb81fa49e0bb45912ffd2cd0138dac9d520419bc105a1d55ec63bcd90b1d07"
Sep 16 20:27:00.046109 ip-10-0-139-81 kubenswrapper[2557]: E0916 20:27:00.046086    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcbb81fa49e0bb45912ffd2cd0138dac9d520419bc105a1d55ec63bcd90b1d07\": container with ID starting with dcbb81fa49e0bb45912ffd2cd0138dac9d520419bc105a1d55ec63bcd90b1d07 not found: ID does not exist" containerID="dcbb81fa49e0bb45912ffd2cd0138dac9d520419bc105a1d55ec63bcd90b1d07"
Sep 16 20:27:00.046218 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:00.046113    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcbb81fa49e0bb45912ffd2cd0138dac9d520419bc105a1d55ec63bcd90b1d07"} err="failed to get container status \"dcbb81fa49e0bb45912ffd2cd0138dac9d520419bc105a1d55ec63bcd90b1d07\": rpc error: code = NotFound desc = could not find container \"dcbb81fa49e0bb45912ffd2cd0138dac9d520419bc105a1d55ec63bcd90b1d07\": container with ID starting with dcbb81fa49e0bb45912ffd2cd0138dac9d520419bc105a1d55ec63bcd90b1d07 not found: ID does not exist"
Sep 16 20:27:00.056059 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:00.056032    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-4cvw8"]
Sep 16 20:27:00.059930 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:00.059908    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-4cvw8"]
Sep 16 20:27:01.937720 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:01.937686    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9756521-1750-40b5-a392-652cafc93db0" path="/var/lib/kubelet/pods/f9756521-1750-40b5-a392-652cafc93db0/volumes"
Sep 16 20:27:04.133483 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:04.133445    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesrltc" podUID="638a229f-de6e-4cd8-b352-19a6ec43209b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:27:14.097563 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:14.097489    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesrltc"
Sep 16 20:27:14.107873 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:14.107831    2557 generic.go:356] "Generic (PLEG): container finished" podID="638a229f-de6e-4cd8-b352-19a6ec43209b" containerID="9d0fa878332978adc7eb5854f21e2b7dd3cfcd249f1edf76aa3355af102850cd" exitCode=0
Sep 16 20:27:14.108004 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:14.107901    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesrltc"
Sep 16 20:27:14.108004 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:14.107927    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesrltc" event={"ID":"638a229f-de6e-4cd8-b352-19a6ec43209b","Type":"ContainerDied","Data":"9d0fa878332978adc7eb5854f21e2b7dd3cfcd249f1edf76aa3355af102850cd"}
Sep 16 20:27:14.108086 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:14.108010    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesrltc" event={"ID":"638a229f-de6e-4cd8-b352-19a6ec43209b","Type":"ContainerDied","Data":"f8441cb537ef97336354daa827da7b27a27c1c14ac4552fc147dca1ef4772bbf"}
Sep 16 20:27:14.108086 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:14.108034    2557 scope.go:117] "RemoveContainer" containerID="9d0fa878332978adc7eb5854f21e2b7dd3cfcd249f1edf76aa3355af102850cd"
Sep 16 20:27:14.116782 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:14.116764    2557 scope.go:117] "RemoveContainer" containerID="9d0fa878332978adc7eb5854f21e2b7dd3cfcd249f1edf76aa3355af102850cd"
Sep 16 20:27:14.117163 ip-10-0-139-81 kubenswrapper[2557]: E0916 20:27:14.117129    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d0fa878332978adc7eb5854f21e2b7dd3cfcd249f1edf76aa3355af102850cd\": container with ID starting with 9d0fa878332978adc7eb5854f21e2b7dd3cfcd249f1edf76aa3355af102850cd not found: ID does not exist" containerID="9d0fa878332978adc7eb5854f21e2b7dd3cfcd249f1edf76aa3355af102850cd"
Sep 16 20:27:14.117327 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:14.117174    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d0fa878332978adc7eb5854f21e2b7dd3cfcd249f1edf76aa3355af102850cd"} err="failed to get container status \"9d0fa878332978adc7eb5854f21e2b7dd3cfcd249f1edf76aa3355af102850cd\": rpc error: code = NotFound desc = could not find container \"9d0fa878332978adc7eb5854f21e2b7dd3cfcd249f1edf76aa3355af102850cd\": container with ID starting with 9d0fa878332978adc7eb5854f21e2b7dd3cfcd249f1edf76aa3355af102850cd not found: ID does not exist"
Sep 16 20:27:14.284610 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:14.284572    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pm4s6\" (UniqueName: \"kubernetes.io/projected/638a229f-de6e-4cd8-b352-19a6ec43209b-kube-api-access-pm4s6\") pod \"638a229f-de6e-4cd8-b352-19a6ec43209b\" (UID: \"638a229f-de6e-4cd8-b352-19a6ec43209b\") "
Sep 16 20:27:14.284809 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:14.284625    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/638a229f-de6e-4cd8-b352-19a6ec43209b-tls-certs\") pod \"638a229f-de6e-4cd8-b352-19a6ec43209b\" (UID: \"638a229f-de6e-4cd8-b352-19a6ec43209b\") "
Sep 16 20:27:14.286892 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:14.286860    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/638a229f-de6e-4cd8-b352-19a6ec43209b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "638a229f-de6e-4cd8-b352-19a6ec43209b" (UID: "638a229f-de6e-4cd8-b352-19a6ec43209b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:27:14.287006 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:14.286897    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/638a229f-de6e-4cd8-b352-19a6ec43209b-kube-api-access-pm4s6" (OuterVolumeSpecName: "kube-api-access-pm4s6") pod "638a229f-de6e-4cd8-b352-19a6ec43209b" (UID: "638a229f-de6e-4cd8-b352-19a6ec43209b"). InnerVolumeSpecName "kube-api-access-pm4s6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:27:14.377695 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:14.377600    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5dc9h"]
Sep 16 20:27:14.378122 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:14.378101    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ea7fe6d-8037-4e73-9db8-b24900d729b9" containerName="main"
Sep 16 20:27:14.378222 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:14.378125    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ea7fe6d-8037-4e73-9db8-b24900d729b9" containerName="main"
Sep 16 20:27:14.378222 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:14.378140    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="638a229f-de6e-4cd8-b352-19a6ec43209b" containerName="main"
Sep 16 20:27:14.378222 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:14.378150    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="638a229f-de6e-4cd8-b352-19a6ec43209b" containerName="main"
Sep 16 20:27:14.378222 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:14.378192    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9756521-1750-40b5-a392-652cafc93db0" containerName="main"
Sep 16 20:27:14.378222 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:14.378201    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9756521-1750-40b5-a392-652cafc93db0" containerName="main"
Sep 16 20:27:14.378480 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:14.378294    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ea7fe6d-8037-4e73-9db8-b24900d729b9" containerName="main"
Sep 16 20:27:14.378480 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:14.378306    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9756521-1750-40b5-a392-652cafc93db0" containerName="main"
Sep 16 20:27:14.378480 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:14.378323    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="638a229f-de6e-4cd8-b352-19a6ec43209b" containerName="main"
Sep 16 20:27:14.382743 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:14.382722    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5dc9h"
Sep 16 20:27:14.385483 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:14.385452    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"default-dockercfg-95rkl\""
Sep 16 20:27:14.385625 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:14.385578    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm4s6\" (UniqueName: \"kubernetes.io/projected/638a229f-de6e-4cd8-b352-19a6ec43209b-kube-api-access-pm4s6\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:27:14.385625 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:14.385600    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/638a229f-de6e-4cd8-b352-19a6ec43209b-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:27:14.385749 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:14.385651    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"owner-v2-kserve-self-signed-certs\""
Sep 16 20:27:14.386783 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:14.386766    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"openshift-service-ca.crt\""
Sep 16 20:27:14.386889 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:14.386768    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"kube-root-ca.crt\""
Sep 16 20:27:14.391516 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:14.391492    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5dc9h"]
Sep 16 20:27:14.431727 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:14.431674    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesrltc"]
Sep 16 20:27:14.436421 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:14.436392    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesrltc"]
Sep 16 20:27:14.486615 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:14.486587    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43828b4c-6012-4b4d-94cc-db95a3953d26-dshm\") pod \"owner-v2-kserve-5cf8959f95-5dc9h\" (UID: \"43828b4c-6012-4b4d-94cc-db95a3953d26\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5dc9h"
Sep 16 20:27:14.486773 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:14.486634    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43828b4c-6012-4b4d-94cc-db95a3953d26-tmp-dir\") pod \"owner-v2-kserve-5cf8959f95-5dc9h\" (UID: \"43828b4c-6012-4b4d-94cc-db95a3953d26\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5dc9h"
Sep 16 20:27:14.486773 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:14.486725    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43828b4c-6012-4b4d-94cc-db95a3953d26-model-cache\") pod \"owner-v2-kserve-5cf8959f95-5dc9h\" (UID: \"43828b4c-6012-4b4d-94cc-db95a3953d26\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5dc9h"
Sep 16 20:27:14.486773 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:14.486766    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvqxd\" (UniqueName: \"kubernetes.io/projected/43828b4c-6012-4b4d-94cc-db95a3953d26-kube-api-access-tvqxd\") pod \"owner-v2-kserve-5cf8959f95-5dc9h\" (UID: \"43828b4c-6012-4b4d-94cc-db95a3953d26\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5dc9h"
Sep 16 20:27:14.486978 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:14.486899    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43828b4c-6012-4b4d-94cc-db95a3953d26-tls-certs\") pod \"owner-v2-kserve-5cf8959f95-5dc9h\" (UID: \"43828b4c-6012-4b4d-94cc-db95a3953d26\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5dc9h"
Sep 16 20:27:14.486978 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:14.486955    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43828b4c-6012-4b4d-94cc-db95a3953d26-home\") pod \"owner-v2-kserve-5cf8959f95-5dc9h\" (UID: \"43828b4c-6012-4b4d-94cc-db95a3953d26\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5dc9h"
Sep 16 20:27:14.587823 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:14.587775    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43828b4c-6012-4b4d-94cc-db95a3953d26-model-cache\") pod \"owner-v2-kserve-5cf8959f95-5dc9h\" (UID: \"43828b4c-6012-4b4d-94cc-db95a3953d26\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5dc9h"
Sep 16 20:27:14.587823 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:14.587830    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvqxd\" (UniqueName: \"kubernetes.io/projected/43828b4c-6012-4b4d-94cc-db95a3953d26-kube-api-access-tvqxd\") pod \"owner-v2-kserve-5cf8959f95-5dc9h\" (UID: \"43828b4c-6012-4b4d-94cc-db95a3953d26\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5dc9h"
Sep 16 20:27:14.588065 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:14.587926    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43828b4c-6012-4b4d-94cc-db95a3953d26-tls-certs\") pod \"owner-v2-kserve-5cf8959f95-5dc9h\" (UID: \"43828b4c-6012-4b4d-94cc-db95a3953d26\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5dc9h"
Sep 16 20:27:14.588065 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:14.587971    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43828b4c-6012-4b4d-94cc-db95a3953d26-home\") pod \"owner-v2-kserve-5cf8959f95-5dc9h\" (UID: \"43828b4c-6012-4b4d-94cc-db95a3953d26\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5dc9h"
Sep 16 20:27:14.588065 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:14.588006    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43828b4c-6012-4b4d-94cc-db95a3953d26-dshm\") pod \"owner-v2-kserve-5cf8959f95-5dc9h\" (UID: \"43828b4c-6012-4b4d-94cc-db95a3953d26\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5dc9h"
Sep 16 20:27:14.588065 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:14.588040    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43828b4c-6012-4b4d-94cc-db95a3953d26-tmp-dir\") pod \"owner-v2-kserve-5cf8959f95-5dc9h\" (UID: \"43828b4c-6012-4b4d-94cc-db95a3953d26\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5dc9h"
Sep 16 20:27:14.588239 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:14.588210    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43828b4c-6012-4b4d-94cc-db95a3953d26-model-cache\") pod \"owner-v2-kserve-5cf8959f95-5dc9h\" (UID: \"43828b4c-6012-4b4d-94cc-db95a3953d26\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5dc9h"
Sep 16 20:27:14.588358 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:14.588336    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43828b4c-6012-4b4d-94cc-db95a3953d26-home\") pod \"owner-v2-kserve-5cf8959f95-5dc9h\" (UID: \"43828b4c-6012-4b4d-94cc-db95a3953d26\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5dc9h"
Sep 16 20:27:14.588456 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:14.588435    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43828b4c-6012-4b4d-94cc-db95a3953d26-tmp-dir\") pod \"owner-v2-kserve-5cf8959f95-5dc9h\" (UID: \"43828b4c-6012-4b4d-94cc-db95a3953d26\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5dc9h"
Sep 16 20:27:14.590239 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:14.590221    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43828b4c-6012-4b4d-94cc-db95a3953d26-dshm\") pod \"owner-v2-kserve-5cf8959f95-5dc9h\" (UID: \"43828b4c-6012-4b4d-94cc-db95a3953d26\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5dc9h"
Sep 16 20:27:14.590539 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:14.590522    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43828b4c-6012-4b4d-94cc-db95a3953d26-tls-certs\") pod \"owner-v2-kserve-5cf8959f95-5dc9h\" (UID: \"43828b4c-6012-4b4d-94cc-db95a3953d26\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5dc9h"
Sep 16 20:27:14.596020 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:14.596001    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvqxd\" (UniqueName: \"kubernetes.io/projected/43828b4c-6012-4b4d-94cc-db95a3953d26-kube-api-access-tvqxd\") pod \"owner-v2-kserve-5cf8959f95-5dc9h\" (UID: \"43828b4c-6012-4b4d-94cc-db95a3953d26\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5dc9h"
Sep 16 20:27:14.694524 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:14.694432    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5dc9h"
Sep 16 20:27:14.967188 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:14.967152    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5dc9h"]
Sep 16 20:27:14.970170 ip-10-0-139-81 kubenswrapper[2557]: W0916 20:27:14.970129    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43828b4c_6012_4b4d_94cc_db95a3953d26.slice/crio-882050df24771db7c3abb0ab4128242e5ac4674514c037871a099137958ee732 WatchSource:0}: Error finding container 882050df24771db7c3abb0ab4128242e5ac4674514c037871a099137958ee732: Status 404 returned error can't find the container with id 882050df24771db7c3abb0ab4128242e5ac4674514c037871a099137958ee732
Sep 16 20:27:15.117971 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:15.117931    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5dc9h" event={"ID":"43828b4c-6012-4b4d-94cc-db95a3953d26","Type":"ContainerStarted","Data":"d1874ff452355b231dd744152fbdffd27a5a836852b9bf76cf102d80074aa2f6"}
Sep 16 20:27:15.118442 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:15.118002    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5dc9h" event={"ID":"43828b4c-6012-4b4d-94cc-db95a3953d26","Type":"ContainerStarted","Data":"882050df24771db7c3abb0ab4128242e5ac4674514c037871a099137958ee732"}
Sep 16 20:27:15.139268 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:15.139210    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5dc9h" podStartSLOduration=1.139188846 podStartE2EDuration="1.139188846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:27:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:27:15.137014602 +0000 UTC m=+2863.662403012" watchObservedRunningTime="2026-09-16 20:27:15.139188846 +0000 UTC m=+2863.664577232"
Sep 16 20:27:15.694949 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:15.694913    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5dc9h"
Sep 16 20:27:15.937396 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:15.937359    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="638a229f-de6e-4cd8-b352-19a6ec43209b" path="/var/lib/kubelet/pods/638a229f-de6e-4cd8-b352-19a6ec43209b/volumes"
Sep 16 20:27:24.695128 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:24.695035    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5dc9h"
Sep 16 20:27:24.707558 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:24.707532    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5dc9h"
Sep 16 20:27:25.178971 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:25.178935    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5dc9h"
Sep 16 20:27:40.853897 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:40.853858    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8d95b"]
Sep 16 20:27:40.858895 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:40.858871    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8d95b"
Sep 16 20:27:40.861914 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:40.861885    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Sep 16 20:27:40.862067 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:40.861910    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Sep 16 20:27:40.862067 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:40.861958    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-h75ds\""
Sep 16 20:27:40.862067 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:40.861969    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Sep 16 20:27:40.863137 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:40.863114    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-r85t4\""
Sep 16 20:27:40.868167 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:40.867804    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8d95b"]
Sep 16 20:27:40.929012 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:40.928980    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56d5w\" (UniqueName: \"kubernetes.io/projected/1cd15d5f-6b6f-4e46-93ae-9982eb4014a9-kube-api-access-56d5w\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8d95b\" (UID: \"1cd15d5f-6b6f-4e46-93ae-9982eb4014a9\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8d95b"
Sep 16 20:27:40.929203 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:40.929026    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cd15d5f-6b6f-4e46-93ae-9982eb4014a9-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8d95b\" (UID: \"1cd15d5f-6b6f-4e46-93ae-9982eb4014a9\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8d95b"
Sep 16 20:27:41.029748 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:41.029708    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cd15d5f-6b6f-4e46-93ae-9982eb4014a9-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8d95b\" (UID: \"1cd15d5f-6b6f-4e46-93ae-9982eb4014a9\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8d95b"
Sep 16 20:27:41.029974 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:41.029935    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56d5w\" (UniqueName: \"kubernetes.io/projected/1cd15d5f-6b6f-4e46-93ae-9982eb4014a9-kube-api-access-56d5w\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8d95b\" (UID: \"1cd15d5f-6b6f-4e46-93ae-9982eb4014a9\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8d95b"
Sep 16 20:27:41.032287 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:41.032257    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cd15d5f-6b6f-4e46-93ae-9982eb4014a9-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8d95b\" (UID: \"1cd15d5f-6b6f-4e46-93ae-9982eb4014a9\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8d95b"
Sep 16 20:27:41.038281 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:41.038256    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56d5w\" (UniqueName: \"kubernetes.io/projected/1cd15d5f-6b6f-4e46-93ae-9982eb4014a9-kube-api-access-56d5w\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8d95b\" (UID: \"1cd15d5f-6b6f-4e46-93ae-9982eb4014a9\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8d95b"
Sep 16 20:27:41.171788 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:41.171696    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8d95b"
Sep 16 20:27:41.447948 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:41.447918    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8d95b"]
Sep 16 20:27:41.449548 ip-10-0-139-81 kubenswrapper[2557]: W0916 20:27:41.449517    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cd15d5f_6b6f_4e46_93ae_9982eb4014a9.slice/crio-2020a62d4b270ca60a40e1f6416414decb9f569d71c6aa58576b3bef47bd9e9e WatchSource:0}: Error finding container 2020a62d4b270ca60a40e1f6416414decb9f569d71c6aa58576b3bef47bd9e9e: Status 404 returned error can't find the container with id 2020a62d4b270ca60a40e1f6416414decb9f569d71c6aa58576b3bef47bd9e9e
Sep 16 20:27:42.255729 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:42.255694    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8d95b" event={"ID":"1cd15d5f-6b6f-4e46-93ae-9982eb4014a9","Type":"ContainerStarted","Data":"79de8f4405d819ffeb128da8c18a42d5e61c96642b0a66952e46f5a3f645fd2e"}
Sep 16 20:27:42.256187 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:42.255754    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8d95b" event={"ID":"1cd15d5f-6b6f-4e46-93ae-9982eb4014a9","Type":"ContainerStarted","Data":"2020a62d4b270ca60a40e1f6416414decb9f569d71c6aa58576b3bef47bd9e9e"}
Sep 16 20:27:42.256187 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:42.255853    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8d95b"
Sep 16 20:27:42.275636 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:27:42.275583    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8d95b" podStartSLOduration=2.275566243 podStartE2EDuration="2.275566243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:27:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:27:42.273931126 +0000 UTC m=+2890.799319512" watchObservedRunningTime="2026-09-16 20:27:42.275566243 +0000 UTC m=+2890.800954627"
Sep 16 20:28:13.262140 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:28:13.262105    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8d95b"
Sep 16 20:28:25.669967 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:28:25.669930    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5dc9h"]
Sep 16 20:28:40.689327 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:28:40.689283    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5dc9h" podUID="43828b4c-6012-4b4d-94cc-db95a3953d26" containerName="main" containerID="cri-o://d1874ff452355b231dd744152fbdffd27a5a836852b9bf76cf102d80074aa2f6" gracePeriod=45
Sep 16 20:28:40.947578 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:28:40.947509    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5dc9h"
Sep 16 20:28:41.080126 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:28:41.080100    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43828b4c-6012-4b4d-94cc-db95a3953d26-model-cache\") pod \"43828b4c-6012-4b4d-94cc-db95a3953d26\" (UID: \"43828b4c-6012-4b4d-94cc-db95a3953d26\") "
Sep 16 20:28:41.080291 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:28:41.080142    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43828b4c-6012-4b4d-94cc-db95a3953d26-dshm\") pod \"43828b4c-6012-4b4d-94cc-db95a3953d26\" (UID: \"43828b4c-6012-4b4d-94cc-db95a3953d26\") "
Sep 16 20:28:41.080291 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:28:41.080179    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43828b4c-6012-4b4d-94cc-db95a3953d26-tmp-dir\") pod \"43828b4c-6012-4b4d-94cc-db95a3953d26\" (UID: \"43828b4c-6012-4b4d-94cc-db95a3953d26\") "
Sep 16 20:28:41.080291 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:28:41.080239    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43828b4c-6012-4b4d-94cc-db95a3953d26-tls-certs\") pod \"43828b4c-6012-4b4d-94cc-db95a3953d26\" (UID: \"43828b4c-6012-4b4d-94cc-db95a3953d26\") "
Sep 16 20:28:41.080291 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:28:41.080274    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvqxd\" (UniqueName: \"kubernetes.io/projected/43828b4c-6012-4b4d-94cc-db95a3953d26-kube-api-access-tvqxd\") pod \"43828b4c-6012-4b4d-94cc-db95a3953d26\" (UID: \"43828b4c-6012-4b4d-94cc-db95a3953d26\") "
Sep 16 20:28:41.080520 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:28:41.080312    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43828b4c-6012-4b4d-94cc-db95a3953d26-home\") pod \"43828b4c-6012-4b4d-94cc-db95a3953d26\" (UID: \"43828b4c-6012-4b4d-94cc-db95a3953d26\") "
Sep 16 20:28:41.080520 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:28:41.080369    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43828b4c-6012-4b4d-94cc-db95a3953d26-model-cache" (OuterVolumeSpecName: "model-cache") pod "43828b4c-6012-4b4d-94cc-db95a3953d26" (UID: "43828b4c-6012-4b4d-94cc-db95a3953d26"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:28:41.080636 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:28:41.080556    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43828b4c-6012-4b4d-94cc-db95a3953d26-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "43828b4c-6012-4b4d-94cc-db95a3953d26" (UID: "43828b4c-6012-4b4d-94cc-db95a3953d26"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:28:41.080813 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:28:41.080789    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43828b4c-6012-4b4d-94cc-db95a3953d26-model-cache\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:28:41.081025 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:28:41.081008    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43828b4c-6012-4b4d-94cc-db95a3953d26-tmp-dir\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:28:41.081126 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:28:41.080853    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43828b4c-6012-4b4d-94cc-db95a3953d26-home" (OuterVolumeSpecName: "home") pod "43828b4c-6012-4b4d-94cc-db95a3953d26" (UID: "43828b4c-6012-4b4d-94cc-db95a3953d26"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:28:41.082671 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:28:41.082650    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43828b4c-6012-4b4d-94cc-db95a3953d26-dshm" (OuterVolumeSpecName: "dshm") pod "43828b4c-6012-4b4d-94cc-db95a3953d26" (UID: "43828b4c-6012-4b4d-94cc-db95a3953d26"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:28:41.082913 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:28:41.082891    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43828b4c-6012-4b4d-94cc-db95a3953d26-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "43828b4c-6012-4b4d-94cc-db95a3953d26" (UID: "43828b4c-6012-4b4d-94cc-db95a3953d26"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:28:41.083035 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:28:41.082925    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43828b4c-6012-4b4d-94cc-db95a3953d26-kube-api-access-tvqxd" (OuterVolumeSpecName: "kube-api-access-tvqxd") pod "43828b4c-6012-4b4d-94cc-db95a3953d26" (UID: "43828b4c-6012-4b4d-94cc-db95a3953d26"). InnerVolumeSpecName "kube-api-access-tvqxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:28:41.182204 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:28:41.182166    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43828b4c-6012-4b4d-94cc-db95a3953d26-dshm\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:28:41.182204 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:28:41.182198    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43828b4c-6012-4b4d-94cc-db95a3953d26-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:28:41.182204 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:28:41.182212    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvqxd\" (UniqueName: \"kubernetes.io/projected/43828b4c-6012-4b4d-94cc-db95a3953d26-kube-api-access-tvqxd\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:28:41.182433 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:28:41.182223    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43828b4c-6012-4b4d-94cc-db95a3953d26-home\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:28:41.545490 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:28:41.545451    2557 generic.go:356] "Generic (PLEG): container finished" podID="43828b4c-6012-4b4d-94cc-db95a3953d26" containerID="d1874ff452355b231dd744152fbdffd27a5a836852b9bf76cf102d80074aa2f6" exitCode=0
Sep 16 20:28:41.545664 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:28:41.545503    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5dc9h"
Sep 16 20:28:41.545664 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:28:41.545544    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5dc9h" event={"ID":"43828b4c-6012-4b4d-94cc-db95a3953d26","Type":"ContainerDied","Data":"d1874ff452355b231dd744152fbdffd27a5a836852b9bf76cf102d80074aa2f6"}
Sep 16 20:28:41.545664 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:28:41.545607    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5dc9h" event={"ID":"43828b4c-6012-4b4d-94cc-db95a3953d26","Type":"ContainerDied","Data":"882050df24771db7c3abb0ab4128242e5ac4674514c037871a099137958ee732"}
Sep 16 20:28:41.545664 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:28:41.545632    2557 scope.go:117] "RemoveContainer" containerID="d1874ff452355b231dd744152fbdffd27a5a836852b9bf76cf102d80074aa2f6"
Sep 16 20:28:41.554781 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:28:41.554763    2557 scope.go:117] "RemoveContainer" containerID="d1874ff452355b231dd744152fbdffd27a5a836852b9bf76cf102d80074aa2f6"
Sep 16 20:28:41.555081 ip-10-0-139-81 kubenswrapper[2557]: E0916 20:28:41.555061    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1874ff452355b231dd744152fbdffd27a5a836852b9bf76cf102d80074aa2f6\": container with ID starting with d1874ff452355b231dd744152fbdffd27a5a836852b9bf76cf102d80074aa2f6 not found: ID does not exist" containerID="d1874ff452355b231dd744152fbdffd27a5a836852b9bf76cf102d80074aa2f6"
Sep 16 20:28:41.555127 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:28:41.555089    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1874ff452355b231dd744152fbdffd27a5a836852b9bf76cf102d80074aa2f6"} err="failed to get container status \"d1874ff452355b231dd744152fbdffd27a5a836852b9bf76cf102d80074aa2f6\": rpc error: code = NotFound desc = could not find container \"d1874ff452355b231dd744152fbdffd27a5a836852b9bf76cf102d80074aa2f6\": container with ID starting with d1874ff452355b231dd744152fbdffd27a5a836852b9bf76cf102d80074aa2f6 not found: ID does not exist"
Sep 16 20:28:41.567306 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:28:41.567281    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5dc9h"]
Sep 16 20:28:41.573578 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:28:41.573556    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5dc9h"]
Sep 16 20:28:41.937700 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:28:41.937622    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43828b4c-6012-4b4d-94cc-db95a3953d26" path="/var/lib/kubelet/pods/43828b4c-6012-4b4d-94cc-db95a3953d26/volumes"
Sep 16 20:29:26.513487 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:29:26.513453    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8d95b"]
Sep 16 20:29:33.262508 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:29:33.262467    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8d95b" podUID="1cd15d5f-6b6f-4e46-93ae-9982eb4014a9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:29:41.532434 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:29:41.532369    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8d95b" podUID="1cd15d5f-6b6f-4e46-93ae-9982eb4014a9" containerName="main" containerID="cri-o://79de8f4405d819ffeb128da8c18a42d5e61c96642b0a66952e46f5a3f645fd2e" gracePeriod=45
Sep 16 20:29:43.261662 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:29:43.261619    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8d95b" podUID="1cd15d5f-6b6f-4e46-93ae-9982eb4014a9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:29:53.262373 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:29:53.262329    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8d95b" podUID="1cd15d5f-6b6f-4e46-93ae-9982eb4014a9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:29:53.262743 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:29:53.262442    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8d95b"
Sep 16 20:30:03.261781 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:30:03.261733    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8d95b" podUID="1cd15d5f-6b6f-4e46-93ae-9982eb4014a9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:30:11.786693 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:30:11.786665    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8d95b"
Sep 16 20:30:11.888156 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:30:11.888066    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56d5w\" (UniqueName: \"kubernetes.io/projected/1cd15d5f-6b6f-4e46-93ae-9982eb4014a9-kube-api-access-56d5w\") pod \"1cd15d5f-6b6f-4e46-93ae-9982eb4014a9\" (UID: \"1cd15d5f-6b6f-4e46-93ae-9982eb4014a9\") "
Sep 16 20:30:11.888156 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:30:11.888126    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cd15d5f-6b6f-4e46-93ae-9982eb4014a9-tls-certs\") pod \"1cd15d5f-6b6f-4e46-93ae-9982eb4014a9\" (UID: \"1cd15d5f-6b6f-4e46-93ae-9982eb4014a9\") "
Sep 16 20:30:11.890402 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:30:11.890374    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cd15d5f-6b6f-4e46-93ae-9982eb4014a9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1cd15d5f-6b6f-4e46-93ae-9982eb4014a9" (UID: "1cd15d5f-6b6f-4e46-93ae-9982eb4014a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:30:11.890402 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:30:11.890386    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cd15d5f-6b6f-4e46-93ae-9982eb4014a9-kube-api-access-56d5w" (OuterVolumeSpecName: "kube-api-access-56d5w") pod "1cd15d5f-6b6f-4e46-93ae-9982eb4014a9" (UID: "1cd15d5f-6b6f-4e46-93ae-9982eb4014a9"). InnerVolumeSpecName "kube-api-access-56d5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:30:11.989361 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:30:11.989320    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56d5w\" (UniqueName: \"kubernetes.io/projected/1cd15d5f-6b6f-4e46-93ae-9982eb4014a9-kube-api-access-56d5w\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:30:11.989361 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:30:11.989352    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cd15d5f-6b6f-4e46-93ae-9982eb4014a9-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:30:12.004696 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:30:12.004651    2557 generic.go:356] "Generic (PLEG): container finished" podID="1cd15d5f-6b6f-4e46-93ae-9982eb4014a9" containerID="79de8f4405d819ffeb128da8c18a42d5e61c96642b0a66952e46f5a3f645fd2e" exitCode=0
Sep 16 20:30:12.004882 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:30:12.004705    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8d95b"
Sep 16 20:30:12.004882 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:30:12.004711    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8d95b" event={"ID":"1cd15d5f-6b6f-4e46-93ae-9982eb4014a9","Type":"ContainerDied","Data":"79de8f4405d819ffeb128da8c18a42d5e61c96642b0a66952e46f5a3f645fd2e"}
Sep 16 20:30:12.004882 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:30:12.004794    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8d95b" event={"ID":"1cd15d5f-6b6f-4e46-93ae-9982eb4014a9","Type":"ContainerDied","Data":"2020a62d4b270ca60a40e1f6416414decb9f569d71c6aa58576b3bef47bd9e9e"}
Sep 16 20:30:12.004882 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:30:12.004818    2557 scope.go:117] "RemoveContainer" containerID="79de8f4405d819ffeb128da8c18a42d5e61c96642b0a66952e46f5a3f645fd2e"
Sep 16 20:30:12.014952 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:30:12.014925    2557 scope.go:117] "RemoveContainer" containerID="79de8f4405d819ffeb128da8c18a42d5e61c96642b0a66952e46f5a3f645fd2e"
Sep 16 20:30:12.015250 ip-10-0-139-81 kubenswrapper[2557]: E0916 20:30:12.015223    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79de8f4405d819ffeb128da8c18a42d5e61c96642b0a66952e46f5a3f645fd2e\": container with ID starting with 79de8f4405d819ffeb128da8c18a42d5e61c96642b0a66952e46f5a3f645fd2e not found: ID does not exist" containerID="79de8f4405d819ffeb128da8c18a42d5e61c96642b0a66952e46f5a3f645fd2e"
Sep 16 20:30:12.015309 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:30:12.015262    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79de8f4405d819ffeb128da8c18a42d5e61c96642b0a66952e46f5a3f645fd2e"} err="failed to get container status \"79de8f4405d819ffeb128da8c18a42d5e61c96642b0a66952e46f5a3f645fd2e\": rpc error: code = NotFound desc = could not find container \"79de8f4405d819ffeb128da8c18a42d5e61c96642b0a66952e46f5a3f645fd2e\": container with ID starting with 79de8f4405d819ffeb128da8c18a42d5e61c96642b0a66952e46f5a3f645fd2e not found: ID does not exist"
Sep 16 20:30:12.023083 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:30:12.023060    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8d95b"]
Sep 16 20:30:12.027834 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:30:12.027810    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8d95b"]
Sep 16 20:30:13.938255 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:30:13.938217    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cd15d5f-6b6f-4e46-93ae-9982eb4014a9" path="/var/lib/kubelet/pods/1cd15d5f-6b6f-4e46-93ae-9982eb4014a9/volumes"
Sep 16 20:30:34.860270 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:30:34.860238    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche92zcn"]
Sep 16 20:30:34.860654 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:30:34.860640    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cd15d5f-6b6f-4e46-93ae-9982eb4014a9" containerName="main"
Sep 16 20:30:34.860699 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:30:34.860655    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cd15d5f-6b6f-4e46-93ae-9982eb4014a9" containerName="main"
Sep 16 20:30:34.860699 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:30:34.860674    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43828b4c-6012-4b4d-94cc-db95a3953d26" containerName="main"
Sep 16 20:30:34.860699 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:30:34.860679    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="43828b4c-6012-4b4d-94cc-db95a3953d26" containerName="main"
Sep 16 20:30:34.860794 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:30:34.860744    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cd15d5f-6b6f-4e46-93ae-9982eb4014a9" containerName="main"
Sep 16 20:30:34.860794 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:30:34.860754    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="43828b4c-6012-4b4d-94cc-db95a3953d26" containerName="main"
Sep 16 20:30:34.865231 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:30:34.865215    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche92zcn"
Sep 16 20:30:34.868466 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:30:34.868446    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Sep 16 20:30:34.869790 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:30:34.869768    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-8n4ks\""
Sep 16 20:30:34.869875 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:30:34.869796    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-z4c9r\""
Sep 16 20:30:34.869875 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:30:34.869826    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Sep 16 20:30:34.869982 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:30:34.869774    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Sep 16 20:30:34.889057 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:30:34.889026    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche92zcn"]
Sep 16 20:30:34.897245 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:30:34.897215    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bz65\" (UniqueName: \"kubernetes.io/projected/2d3ba276-4ae3-4f28-bf7f-47b7c21443ea-kube-api-access-2bz65\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche92zcn\" (UID: \"2d3ba276-4ae3-4f28-bf7f-47b7c21443ea\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche92zcn"
Sep 16 20:30:34.897386 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:30:34.897256    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d3ba276-4ae3-4f28-bf7f-47b7c21443ea-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche92zcn\" (UID: \"2d3ba276-4ae3-4f28-bf7f-47b7c21443ea\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche92zcn"
Sep 16 20:30:34.997945 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:30:34.997908    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bz65\" (UniqueName: \"kubernetes.io/projected/2d3ba276-4ae3-4f28-bf7f-47b7c21443ea-kube-api-access-2bz65\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche92zcn\" (UID: \"2d3ba276-4ae3-4f28-bf7f-47b7c21443ea\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche92zcn"
Sep 16 20:30:34.997945 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:30:34.997957    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d3ba276-4ae3-4f28-bf7f-47b7c21443ea-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche92zcn\" (UID: \"2d3ba276-4ae3-4f28-bf7f-47b7c21443ea\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche92zcn"
Sep 16 20:30:35.000464 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:30:35.000438    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d3ba276-4ae3-4f28-bf7f-47b7c21443ea-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche92zcn\" (UID: \"2d3ba276-4ae3-4f28-bf7f-47b7c21443ea\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche92zcn"
Sep 16 20:30:35.007081 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:30:35.007054    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bz65\" (UniqueName: \"kubernetes.io/projected/2d3ba276-4ae3-4f28-bf7f-47b7c21443ea-kube-api-access-2bz65\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche92zcn\" (UID: \"2d3ba276-4ae3-4f28-bf7f-47b7c21443ea\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche92zcn"
Sep 16 20:30:35.180336 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:30:35.180243    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche92zcn"
Sep 16 20:30:35.456582 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:30:35.456551    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche92zcn"]
Sep 16 20:30:36.130508 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:30:36.130470    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche92zcn" event={"ID":"2d3ba276-4ae3-4f28-bf7f-47b7c21443ea","Type":"ContainerStarted","Data":"5cc24adb6174f067393ad448f6c3f42f891098fcfa69a634b16742818c2056e5"}
Sep 16 20:30:36.130921 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:30:36.130534    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche92zcn"
Sep 16 20:30:36.130921 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:30:36.130557    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche92zcn" event={"ID":"2d3ba276-4ae3-4f28-bf7f-47b7c21443ea","Type":"ContainerStarted","Data":"95d62f8fd6bc11cacc32040c47e1b21760824fedcc9655a591cd3605ed41ae77"}
Sep 16 20:30:36.148470 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:30:36.148414    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche92zcn" podStartSLOduration=2.148397539 podStartE2EDuration="2.148397539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:30:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:30:36.147269882 +0000 UTC m=+3064.672658270" watchObservedRunningTime="2026-09-16 20:30:36.148397539 +0000 UTC m=+3064.673785925"
Sep 16 20:31:07.136820 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:31:07.136786    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche92zcn"
Sep 16 20:31:57.321443 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:31:57.321364    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche92zcn"]
Sep 16 20:32:07.137072 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:32:07.137026    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche92zcn" podUID="2d3ba276-4ae3-4f28-bf7f-47b7c21443ea" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:32:12.340941 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:32:12.340871    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche92zcn" podUID="2d3ba276-4ae3-4f28-bf7f-47b7c21443ea" containerName="main" containerID="cri-o://5cc24adb6174f067393ad448f6c3f42f891098fcfa69a634b16742818c2056e5" gracePeriod=45
Sep 16 20:32:17.137043 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:32:17.136994    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche92zcn" podUID="2d3ba276-4ae3-4f28-bf7f-47b7c21443ea" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:32:27.136965 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:32:27.136925    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche92zcn" podUID="2d3ba276-4ae3-4f28-bf7f-47b7c21443ea" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:32:27.137380 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:32:27.137051    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche92zcn"
Sep 16 20:32:37.137110 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:32:37.137063    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche92zcn" podUID="2d3ba276-4ae3-4f28-bf7f-47b7c21443ea" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:32:42.586732 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:32:42.586706    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche92zcn"
Sep 16 20:32:42.629352 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:32:42.629283    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d3ba276-4ae3-4f28-bf7f-47b7c21443ea-tls-certs\") pod \"2d3ba276-4ae3-4f28-bf7f-47b7c21443ea\" (UID: \"2d3ba276-4ae3-4f28-bf7f-47b7c21443ea\") "
Sep 16 20:32:42.629352 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:32:42.629319    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2bz65\" (UniqueName: \"kubernetes.io/projected/2d3ba276-4ae3-4f28-bf7f-47b7c21443ea-kube-api-access-2bz65\") pod \"2d3ba276-4ae3-4f28-bf7f-47b7c21443ea\" (UID: \"2d3ba276-4ae3-4f28-bf7f-47b7c21443ea\") "
Sep 16 20:32:42.631452 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:32:42.631427    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d3ba276-4ae3-4f28-bf7f-47b7c21443ea-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2d3ba276-4ae3-4f28-bf7f-47b7c21443ea" (UID: "2d3ba276-4ae3-4f28-bf7f-47b7c21443ea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:32:42.631526 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:32:42.631482    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d3ba276-4ae3-4f28-bf7f-47b7c21443ea-kube-api-access-2bz65" (OuterVolumeSpecName: "kube-api-access-2bz65") pod "2d3ba276-4ae3-4f28-bf7f-47b7c21443ea" (UID: "2d3ba276-4ae3-4f28-bf7f-47b7c21443ea"). InnerVolumeSpecName "kube-api-access-2bz65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:32:42.730179 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:32:42.730143    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d3ba276-4ae3-4f28-bf7f-47b7c21443ea-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:32:42.730179 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:32:42.730175    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bz65\" (UniqueName: \"kubernetes.io/projected/2d3ba276-4ae3-4f28-bf7f-47b7c21443ea-kube-api-access-2bz65\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:32:42.761343 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:32:42.761306    2557 generic.go:356] "Generic (PLEG): container finished" podID="2d3ba276-4ae3-4f28-bf7f-47b7c21443ea" containerID="5cc24adb6174f067393ad448f6c3f42f891098fcfa69a634b16742818c2056e5" exitCode=0
Sep 16 20:32:42.761492 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:32:42.761359    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche92zcn"
Sep 16 20:32:42.761492 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:32:42.761383    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche92zcn" event={"ID":"2d3ba276-4ae3-4f28-bf7f-47b7c21443ea","Type":"ContainerDied","Data":"5cc24adb6174f067393ad448f6c3f42f891098fcfa69a634b16742818c2056e5"}
Sep 16 20:32:42.761492 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:32:42.761436    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche92zcn" event={"ID":"2d3ba276-4ae3-4f28-bf7f-47b7c21443ea","Type":"ContainerDied","Data":"95d62f8fd6bc11cacc32040c47e1b21760824fedcc9655a591cd3605ed41ae77"}
Sep 16 20:32:42.761492 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:32:42.761452    2557 scope.go:117] "RemoveContainer" containerID="5cc24adb6174f067393ad448f6c3f42f891098fcfa69a634b16742818c2056e5"
Sep 16 20:32:42.779009 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:32:42.778984    2557 scope.go:117] "RemoveContainer" containerID="5cc24adb6174f067393ad448f6c3f42f891098fcfa69a634b16742818c2056e5"
Sep 16 20:32:42.779531 ip-10-0-139-81 kubenswrapper[2557]: E0916 20:32:42.779506    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cc24adb6174f067393ad448f6c3f42f891098fcfa69a634b16742818c2056e5\": container with ID starting with 5cc24adb6174f067393ad448f6c3f42f891098fcfa69a634b16742818c2056e5 not found: ID does not exist" containerID="5cc24adb6174f067393ad448f6c3f42f891098fcfa69a634b16742818c2056e5"
Sep 16 20:32:42.779632 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:32:42.779542    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cc24adb6174f067393ad448f6c3f42f891098fcfa69a634b16742818c2056e5"} err="failed to get container status \"5cc24adb6174f067393ad448f6c3f42f891098fcfa69a634b16742818c2056e5\": rpc error: code = NotFound desc = could not find container \"5cc24adb6174f067393ad448f6c3f42f891098fcfa69a634b16742818c2056e5\": container with ID starting with 5cc24adb6174f067393ad448f6c3f42f891098fcfa69a634b16742818c2056e5 not found: ID does not exist"
Sep 16 20:32:42.788652 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:32:42.788622    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche92zcn"]
Sep 16 20:32:42.794885 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:32:42.794863    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche92zcn"]
Sep 16 20:32:43.937799 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:32:43.937759    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d3ba276-4ae3-4f28-bf7f-47b7c21443ea" path="/var/lib/kubelet/pods/2d3ba276-4ae3-4f28-bf7f-47b7c21443ea/volumes"
Sep 16 20:33:02.032815 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:02.032778    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-lqbw2"]
Sep 16 20:33:02.033204 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:02.033188    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d3ba276-4ae3-4f28-bf7f-47b7c21443ea" containerName="main"
Sep 16 20:33:02.033258 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:02.033205    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d3ba276-4ae3-4f28-bf7f-47b7c21443ea" containerName="main"
Sep 16 20:33:02.033295 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:02.033270    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d3ba276-4ae3-4f28-bf7f-47b7c21443ea" containerName="main"
Sep 16 20:33:02.035769 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:02.035745    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-lqbw2"
Sep 16 20:33:02.040456 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:02.040436    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"kube-root-ca.crt\""
Sep 16 20:33:02.040603 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:02.040433    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"default-dockercfg-czptp\""
Sep 16 20:33:02.040603 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:02.040510    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"e2e-del-blocked-svc-epp-sa-dockercfg-tpwlk\""
Sep 16 20:33:02.040603 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:02.040433    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"e2e-del-blocked-svc-kserve-self-signed-certs\""
Sep 16 20:33:02.041146 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:02.040573    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"openshift-service-ca.crt\""
Sep 16 20:33:02.049475 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:02.049442    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-lqbw2"]
Sep 16 20:33:02.086614 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:02.086575    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx7rr\" (UniqueName: \"kubernetes.io/projected/a32e6f22-e0a6-4ab2-96a9-2d95096ecbe8-kube-api-access-sx7rr\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-lqbw2\" (UID: \"a32e6f22-e0a6-4ab2-96a9-2d95096ecbe8\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-lqbw2"
Sep 16 20:33:02.086775 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:02.086632    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a32e6f22-e0a6-4ab2-96a9-2d95096ecbe8-tls-certs\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-lqbw2\" (UID: \"a32e6f22-e0a6-4ab2-96a9-2d95096ecbe8\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-lqbw2"
Sep 16 20:33:02.187194 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:02.187145    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx7rr\" (UniqueName: \"kubernetes.io/projected/a32e6f22-e0a6-4ab2-96a9-2d95096ecbe8-kube-api-access-sx7rr\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-lqbw2\" (UID: \"a32e6f22-e0a6-4ab2-96a9-2d95096ecbe8\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-lqbw2"
Sep 16 20:33:02.187365 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:02.187220    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a32e6f22-e0a6-4ab2-96a9-2d95096ecbe8-tls-certs\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-lqbw2\" (UID: \"a32e6f22-e0a6-4ab2-96a9-2d95096ecbe8\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-lqbw2"
Sep 16 20:33:02.189803 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:02.189768    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a32e6f22-e0a6-4ab2-96a9-2d95096ecbe8-tls-certs\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-lqbw2\" (UID: \"a32e6f22-e0a6-4ab2-96a9-2d95096ecbe8\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-lqbw2"
Sep 16 20:33:02.196379 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:02.196354    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx7rr\" (UniqueName: \"kubernetes.io/projected/a32e6f22-e0a6-4ab2-96a9-2d95096ecbe8-kube-api-access-sx7rr\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-lqbw2\" (UID: \"a32e6f22-e0a6-4ab2-96a9-2d95096ecbe8\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-lqbw2"
Sep 16 20:33:02.355539 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:02.355453    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-lqbw2"
Sep 16 20:33:02.629539 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:02.629510    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-lqbw2"]
Sep 16 20:33:02.632459 ip-10-0-139-81 kubenswrapper[2557]: W0916 20:33:02.632436    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda32e6f22_e0a6_4ab2_96a9_2d95096ecbe8.slice/crio-c8c79d8b58165cd91eabda54ac55cf3168a976775afab2344a6db0b2cf434d88 WatchSource:0}: Error finding container c8c79d8b58165cd91eabda54ac55cf3168a976775afab2344a6db0b2cf434d88: Status 404 returned error can't find the container with id c8c79d8b58165cd91eabda54ac55cf3168a976775afab2344a6db0b2cf434d88
Sep 16 20:33:02.873880 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:02.873818    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-lqbw2" event={"ID":"a32e6f22-e0a6-4ab2-96a9-2d95096ecbe8","Type":"ContainerStarted","Data":"1ab87c645154ca46e632004f817c4b1c94b8ceeecaec8c18e2784726eadb70f5"}
Sep 16 20:33:02.874046 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:02.873898    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-lqbw2" event={"ID":"a32e6f22-e0a6-4ab2-96a9-2d95096ecbe8","Type":"ContainerStarted","Data":"c8c79d8b58165cd91eabda54ac55cf3168a976775afab2344a6db0b2cf434d88"}
Sep 16 20:33:02.874046 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:02.873976    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-lqbw2"
Sep 16 20:33:02.894327 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:02.894224    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-lqbw2" podStartSLOduration=0.894206108 podStartE2EDuration="894.206108ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:33:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:33:02.891578636 +0000 UTC m=+3211.416967020" watchObservedRunningTime="2026-09-16 20:33:02.894206108 +0000 UTC m=+3211.419594542"
Sep 16 20:33:04.529800 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:04.529766    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-lqbw2"]
Sep 16 20:33:19.902558 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:19.902499    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-lqbw2" podUID="a32e6f22-e0a6-4ab2-96a9-2d95096ecbe8" containerName="main" containerID="cri-o://1ab87c645154ca46e632004f817c4b1c94b8ceeecaec8c18e2784726eadb70f5" gracePeriod=45
Sep 16 20:33:34.886604 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:34.886563    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-lqbw2" podUID="a32e6f22-e0a6-4ab2-96a9-2d95096ecbe8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:33:44.885961 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:44.885920    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-lqbw2" podUID="a32e6f22-e0a6-4ab2-96a9-2d95096ecbe8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:33:50.108148 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:50.108117    2557 generic.go:356] "Generic (PLEG): container finished" podID="a32e6f22-e0a6-4ab2-96a9-2d95096ecbe8" containerID="1ab87c645154ca46e632004f817c4b1c94b8ceeecaec8c18e2784726eadb70f5" exitCode=0
Sep 16 20:33:50.108505 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:50.108193    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-lqbw2" event={"ID":"a32e6f22-e0a6-4ab2-96a9-2d95096ecbe8","Type":"ContainerDied","Data":"1ab87c645154ca46e632004f817c4b1c94b8ceeecaec8c18e2784726eadb70f5"}
Sep 16 20:33:50.146222 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:50.146196    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-lqbw2"
Sep 16 20:33:50.219144 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:50.219066    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a32e6f22-e0a6-4ab2-96a9-2d95096ecbe8-tls-certs\") pod \"a32e6f22-e0a6-4ab2-96a9-2d95096ecbe8\" (UID: \"a32e6f22-e0a6-4ab2-96a9-2d95096ecbe8\") "
Sep 16 20:33:50.219398 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:50.219212    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sx7rr\" (UniqueName: \"kubernetes.io/projected/a32e6f22-e0a6-4ab2-96a9-2d95096ecbe8-kube-api-access-sx7rr\") pod \"a32e6f22-e0a6-4ab2-96a9-2d95096ecbe8\" (UID: \"a32e6f22-e0a6-4ab2-96a9-2d95096ecbe8\") "
Sep 16 20:33:50.221243 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:50.221215    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a32e6f22-e0a6-4ab2-96a9-2d95096ecbe8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a32e6f22-e0a6-4ab2-96a9-2d95096ecbe8" (UID: "a32e6f22-e0a6-4ab2-96a9-2d95096ecbe8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:33:50.221352 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:50.221297    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a32e6f22-e0a6-4ab2-96a9-2d95096ecbe8-kube-api-access-sx7rr" (OuterVolumeSpecName: "kube-api-access-sx7rr") pod "a32e6f22-e0a6-4ab2-96a9-2d95096ecbe8" (UID: "a32e6f22-e0a6-4ab2-96a9-2d95096ecbe8"). InnerVolumeSpecName "kube-api-access-sx7rr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:33:50.320420 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:50.320382    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx7rr\" (UniqueName: \"kubernetes.io/projected/a32e6f22-e0a6-4ab2-96a9-2d95096ecbe8-kube-api-access-sx7rr\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:33:50.320420 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:50.320413    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a32e6f22-e0a6-4ab2-96a9-2d95096ecbe8-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:33:51.114218 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:51.114188    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-lqbw2"
Sep 16 20:33:51.114218 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:51.114200    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-lqbw2" event={"ID":"a32e6f22-e0a6-4ab2-96a9-2d95096ecbe8","Type":"ContainerDied","Data":"c8c79d8b58165cd91eabda54ac55cf3168a976775afab2344a6db0b2cf434d88"}
Sep 16 20:33:51.114680 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:51.114254    2557 scope.go:117] "RemoveContainer" containerID="1ab87c645154ca46e632004f817c4b1c94b8ceeecaec8c18e2784726eadb70f5"
Sep 16 20:33:51.137219 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:51.137193    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-lqbw2"]
Sep 16 20:33:51.146793 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:51.144510    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-lqbw2"]
Sep 16 20:33:51.938158 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:51.938125    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a32e6f22-e0a6-4ab2-96a9-2d95096ecbe8" path="/var/lib/kubelet/pods/a32e6f22-e0a6-4ab2-96a9-2d95096ecbe8/volumes"
Sep 16 20:33:52.178620 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:52.178589    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8kh6m"]
Sep 16 20:33:52.179042 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:52.179026    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a32e6f22-e0a6-4ab2-96a9-2d95096ecbe8" containerName="main"
Sep 16 20:33:52.179090 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:52.179044    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a32e6f22-e0a6-4ab2-96a9-2d95096ecbe8" containerName="main"
Sep 16 20:33:52.179138 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:52.179126    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a32e6f22-e0a6-4ab2-96a9-2d95096ecbe8" containerName="main"
Sep 16 20:33:52.183455 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:52.183437    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8kh6m"
Sep 16 20:33:52.188081 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:52.188053    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 16 20:33:52.188552 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:52.188304    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 16 20:33:52.188552 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:52.188376    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 16 20:33:52.188552 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:52.188453    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-vq79p\""
Sep 16 20:33:52.188777 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:52.188581    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-pk8rp\""
Sep 16 20:33:52.191864 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:52.191827    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8kh6m"]
Sep 16 20:33:52.237006 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:52.236980    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxpvn\" (UniqueName: \"kubernetes.io/projected/f2fa4c4f-42c4-46ef-99bc-7c7fbc708a2f-kube-api-access-kxpvn\") pod \"stop-feature-test-kserve-router-scheduler-7468747994-8kh6m\" (UID: \"f2fa4c4f-42c4-46ef-99bc-7c7fbc708a2f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8kh6m"
Sep 16 20:33:52.237124 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:52.237018    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2fa4c4f-42c4-46ef-99bc-7c7fbc708a2f-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7468747994-8kh6m\" (UID: \"f2fa4c4f-42c4-46ef-99bc-7c7fbc708a2f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8kh6m"
Sep 16 20:33:52.337525 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:52.337470    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxpvn\" (UniqueName: \"kubernetes.io/projected/f2fa4c4f-42c4-46ef-99bc-7c7fbc708a2f-kube-api-access-kxpvn\") pod \"stop-feature-test-kserve-router-scheduler-7468747994-8kh6m\" (UID: \"f2fa4c4f-42c4-46ef-99bc-7c7fbc708a2f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8kh6m"
Sep 16 20:33:52.337706 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:52.337551    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2fa4c4f-42c4-46ef-99bc-7c7fbc708a2f-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7468747994-8kh6m\" (UID: \"f2fa4c4f-42c4-46ef-99bc-7c7fbc708a2f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8kh6m"
Sep 16 20:33:52.340265 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:52.340239    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2fa4c4f-42c4-46ef-99bc-7c7fbc708a2f-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7468747994-8kh6m\" (UID: \"f2fa4c4f-42c4-46ef-99bc-7c7fbc708a2f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8kh6m"
Sep 16 20:33:52.346350 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:52.346317    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxpvn\" (UniqueName: \"kubernetes.io/projected/f2fa4c4f-42c4-46ef-99bc-7c7fbc708a2f-kube-api-access-kxpvn\") pod \"stop-feature-test-kserve-router-scheduler-7468747994-8kh6m\" (UID: \"f2fa4c4f-42c4-46ef-99bc-7c7fbc708a2f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8kh6m"
Sep 16 20:33:52.494694 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:52.494654    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8kh6m"
Sep 16 20:33:52.771115 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:52.771022    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8kh6m"]
Sep 16 20:33:52.774256 ip-10-0-139-81 kubenswrapper[2557]: W0916 20:33:52.774227    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2fa4c4f_42c4_46ef_99bc_7c7fbc708a2f.slice/crio-378ebbf258632e499b42c097c49d1e7e45baf579d534a2b625eaadbae956e408 WatchSource:0}: Error finding container 378ebbf258632e499b42c097c49d1e7e45baf579d534a2b625eaadbae956e408: Status 404 returned error can't find the container with id 378ebbf258632e499b42c097c49d1e7e45baf579d534a2b625eaadbae956e408
Sep 16 20:33:53.126731 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:53.126624    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8kh6m" event={"ID":"f2fa4c4f-42c4-46ef-99bc-7c7fbc708a2f","Type":"ContainerStarted","Data":"1b4bb9f30625565f970a2cf4602fe73c4ab1d39137339f5e0caba700ba597740"}
Sep 16 20:33:53.126731 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:53.126705    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8kh6m" event={"ID":"f2fa4c4f-42c4-46ef-99bc-7c7fbc708a2f","Type":"ContainerStarted","Data":"378ebbf258632e499b42c097c49d1e7e45baf579d534a2b625eaadbae956e408"}
Sep 16 20:33:53.126965 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:53.126740    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8kh6m"
Sep 16 20:33:53.144111 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:33:53.144061    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8kh6m" podStartSLOduration=1.144044101 podStartE2EDuration="1.144044101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:33:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:33:53.142453973 +0000 UTC m=+3261.667842362" watchObservedRunningTime="2026-09-16 20:33:53.144044101 +0000 UTC m=+3261.669432486"
Sep 16 20:34:24.133804 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:34:24.133772    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8kh6m"
Sep 16 20:35:53.940656 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:35:53.940620    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8kh6m"]
Sep 16 20:35:54.134075 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:35:54.134036    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8kh6m" podUID="f2fa4c4f-42c4-46ef-99bc-7c7fbc708a2f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:36:04.134221 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:36:04.134181    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8kh6m" podUID="f2fa4c4f-42c4-46ef-99bc-7c7fbc708a2f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:36:06.680118 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:36:06.680077    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8dx7f"]
Sep 16 20:36:06.684347 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:36:06.684318    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8dx7f"
Sep 16 20:36:06.687123 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:36:06.687093    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-bwmgf\""
Sep 16 20:36:06.694080 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:36:06.694055    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8dx7f"]
Sep 16 20:36:06.759369 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:36:06.759332    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3652563-0ffd-4922-99ed-9e4797ea9419-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7468747994-8dx7f\" (UID: \"e3652563-0ffd-4922-99ed-9e4797ea9419\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8dx7f"
Sep 16 20:36:06.759557 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:36:06.759491    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmthb\" (UniqueName: \"kubernetes.io/projected/e3652563-0ffd-4922-99ed-9e4797ea9419-kube-api-access-wmthb\") pod \"stop-feature-test-kserve-router-scheduler-7468747994-8dx7f\" (UID: \"e3652563-0ffd-4922-99ed-9e4797ea9419\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8dx7f"
Sep 16 20:36:06.860714 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:36:06.860675    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmthb\" (UniqueName: \"kubernetes.io/projected/e3652563-0ffd-4922-99ed-9e4797ea9419-kube-api-access-wmthb\") pod \"stop-feature-test-kserve-router-scheduler-7468747994-8dx7f\" (UID: \"e3652563-0ffd-4922-99ed-9e4797ea9419\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8dx7f"
Sep 16 20:36:06.860964 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:36:06.860725    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3652563-0ffd-4922-99ed-9e4797ea9419-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7468747994-8dx7f\" (UID: \"e3652563-0ffd-4922-99ed-9e4797ea9419\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8dx7f"
Sep 16 20:36:06.863369 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:36:06.863346    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3652563-0ffd-4922-99ed-9e4797ea9419-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7468747994-8dx7f\" (UID: \"e3652563-0ffd-4922-99ed-9e4797ea9419\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8dx7f"
Sep 16 20:36:06.870094 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:36:06.870061    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmthb\" (UniqueName: \"kubernetes.io/projected/e3652563-0ffd-4922-99ed-9e4797ea9419-kube-api-access-wmthb\") pod \"stop-feature-test-kserve-router-scheduler-7468747994-8dx7f\" (UID: \"e3652563-0ffd-4922-99ed-9e4797ea9419\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8dx7f"
Sep 16 20:36:06.997553 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:36:06.997518    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8dx7f"
Sep 16 20:36:07.271815 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:36:07.271738    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8dx7f"]
Sep 16 20:36:07.275506 ip-10-0-139-81 kubenswrapper[2557]: W0916 20:36:07.275479    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3652563_0ffd_4922_99ed_9e4797ea9419.slice/crio-7d73decb89cf3f3a572cf4f64811519b33a4b2b0bd9803c4791f1192cb3eb9d7 WatchSource:0}: Error finding container 7d73decb89cf3f3a572cf4f64811519b33a4b2b0bd9803c4791f1192cb3eb9d7: Status 404 returned error can't find the container with id 7d73decb89cf3f3a572cf4f64811519b33a4b2b0bd9803c4791f1192cb3eb9d7
Sep 16 20:36:07.797559 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:36:07.797523    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8dx7f" event={"ID":"e3652563-0ffd-4922-99ed-9e4797ea9419","Type":"ContainerStarted","Data":"97375a59537e7b9f2d3a3da00e48f96b2163ae14ebd2347c5474ee8c765412da"}
Sep 16 20:36:07.797978 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:36:07.797577    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8dx7f" event={"ID":"e3652563-0ffd-4922-99ed-9e4797ea9419","Type":"ContainerStarted","Data":"7d73decb89cf3f3a572cf4f64811519b33a4b2b0bd9803c4791f1192cb3eb9d7"}
Sep 16 20:36:07.797978 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:36:07.797658    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8dx7f"
Sep 16 20:36:07.816094 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:36:07.816039    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8dx7f" podStartSLOduration=1.816023599 podStartE2EDuration="1.816023599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:36:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:36:07.814300447 +0000 UTC m=+3396.339688833" watchObservedRunningTime="2026-09-16 20:36:07.816023599 +0000 UTC m=+3396.341411984"
Sep 16 20:36:08.963711 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:36:08.963609    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8kh6m" podUID="f2fa4c4f-42c4-46ef-99bc-7c7fbc708a2f" containerName="main" containerID="cri-o://1b4bb9f30625565f970a2cf4602fe73c4ab1d39137339f5e0caba700ba597740" gracePeriod=45
Sep 16 20:36:14.134128 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:36:14.134085    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8kh6m" podUID="f2fa4c4f-42c4-46ef-99bc-7c7fbc708a2f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:36:14.134587 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:36:14.134187    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8kh6m"
Sep 16 20:36:24.133633 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:36:24.133594    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8kh6m" podUID="f2fa4c4f-42c4-46ef-99bc-7c7fbc708a2f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:36:34.134015 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:36:34.133976    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8kh6m" podUID="f2fa4c4f-42c4-46ef-99bc-7c7fbc708a2f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:36:38.804587 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:36:38.804554    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8dx7f"
Sep 16 20:36:39.209949 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:36:39.209921    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8kh6m"
Sep 16 20:36:39.261622 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:36:39.261588    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2fa4c4f-42c4-46ef-99bc-7c7fbc708a2f-tls-certs\") pod \"f2fa4c4f-42c4-46ef-99bc-7c7fbc708a2f\" (UID: \"f2fa4c4f-42c4-46ef-99bc-7c7fbc708a2f\") "
Sep 16 20:36:39.261795 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:36:39.261629    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kxpvn\" (UniqueName: \"kubernetes.io/projected/f2fa4c4f-42c4-46ef-99bc-7c7fbc708a2f-kube-api-access-kxpvn\") pod \"f2fa4c4f-42c4-46ef-99bc-7c7fbc708a2f\" (UID: \"f2fa4c4f-42c4-46ef-99bc-7c7fbc708a2f\") "
Sep 16 20:36:39.263917 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:36:39.263881    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2fa4c4f-42c4-46ef-99bc-7c7fbc708a2f-kube-api-access-kxpvn" (OuterVolumeSpecName: "kube-api-access-kxpvn") pod "f2fa4c4f-42c4-46ef-99bc-7c7fbc708a2f" (UID: "f2fa4c4f-42c4-46ef-99bc-7c7fbc708a2f"). InnerVolumeSpecName "kube-api-access-kxpvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:36:39.264034 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:36:39.263939    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2fa4c4f-42c4-46ef-99bc-7c7fbc708a2f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f2fa4c4f-42c4-46ef-99bc-7c7fbc708a2f" (UID: "f2fa4c4f-42c4-46ef-99bc-7c7fbc708a2f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:36:39.363081 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:36:39.362985    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2fa4c4f-42c4-46ef-99bc-7c7fbc708a2f-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:36:39.363081 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:36:39.363024    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxpvn\" (UniqueName: \"kubernetes.io/projected/f2fa4c4f-42c4-46ef-99bc-7c7fbc708a2f-kube-api-access-kxpvn\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:36:39.968259 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:36:39.968220    2557 generic.go:356] "Generic (PLEG): container finished" podID="f2fa4c4f-42c4-46ef-99bc-7c7fbc708a2f" containerID="1b4bb9f30625565f970a2cf4602fe73c4ab1d39137339f5e0caba700ba597740" exitCode=0
Sep 16 20:36:39.968709 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:36:39.968285    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8kh6m"
Sep 16 20:36:39.968709 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:36:39.968299    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8kh6m" event={"ID":"f2fa4c4f-42c4-46ef-99bc-7c7fbc708a2f","Type":"ContainerDied","Data":"1b4bb9f30625565f970a2cf4602fe73c4ab1d39137339f5e0caba700ba597740"}
Sep 16 20:36:39.968709 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:36:39.968358    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8kh6m" event={"ID":"f2fa4c4f-42c4-46ef-99bc-7c7fbc708a2f","Type":"ContainerDied","Data":"378ebbf258632e499b42c097c49d1e7e45baf579d534a2b625eaadbae956e408"}
Sep 16 20:36:39.968709 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:36:39.968381    2557 scope.go:117] "RemoveContainer" containerID="1b4bb9f30625565f970a2cf4602fe73c4ab1d39137339f5e0caba700ba597740"
Sep 16 20:36:39.977648 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:36:39.977629    2557 scope.go:117] "RemoveContainer" containerID="1b4bb9f30625565f970a2cf4602fe73c4ab1d39137339f5e0caba700ba597740"
Sep 16 20:36:39.977975 ip-10-0-139-81 kubenswrapper[2557]: E0916 20:36:39.977951    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b4bb9f30625565f970a2cf4602fe73c4ab1d39137339f5e0caba700ba597740\": container with ID starting with 1b4bb9f30625565f970a2cf4602fe73c4ab1d39137339f5e0caba700ba597740 not found: ID does not exist" containerID="1b4bb9f30625565f970a2cf4602fe73c4ab1d39137339f5e0caba700ba597740"
Sep 16 20:36:39.978053 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:36:39.977985    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b4bb9f30625565f970a2cf4602fe73c4ab1d39137339f5e0caba700ba597740"} err="failed to get container status \"1b4bb9f30625565f970a2cf4602fe73c4ab1d39137339f5e0caba700ba597740\": rpc error: code = NotFound desc = could not find container \"1b4bb9f30625565f970a2cf4602fe73c4ab1d39137339f5e0caba700ba597740\": container with ID starting with 1b4bb9f30625565f970a2cf4602fe73c4ab1d39137339f5e0caba700ba597740 not found: ID does not exist"
Sep 16 20:36:39.985942 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:36:39.985916    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8kh6m"]
Sep 16 20:36:39.990147 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:36:39.990122    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8kh6m"]
Sep 16 20:36:41.937352 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:36:41.937310    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2fa4c4f-42c4-46ef-99bc-7c7fbc708a2f" path="/var/lib/kubelet/pods/f2fa4c4f-42c4-46ef-99bc-7c7fbc708a2f/volumes"
Sep 16 20:38:08.215076 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:38:08.214992    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8dx7f"]
Sep 16 20:38:08.804556 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:38:08.804511    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8dx7f" podUID="e3652563-0ffd-4922-99ed-9e4797ea9419" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:38:18.804364 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:38:18.804318    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8dx7f" podUID="e3652563-0ffd-4922-99ed-9e4797ea9419" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:38:23.236751 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:38:23.236695    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8dx7f" podUID="e3652563-0ffd-4922-99ed-9e4797ea9419" containerName="main" containerID="cri-o://97375a59537e7b9f2d3a3da00e48f96b2163ae14ebd2347c5474ee8c765412da" gracePeriod=45
Sep 16 20:38:28.804210 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:38:28.804170    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8dx7f" podUID="e3652563-0ffd-4922-99ed-9e4797ea9419" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:38:28.804681 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:38:28.804284    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8dx7f"
Sep 16 20:38:38.804374 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:38:38.804329    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8dx7f" podUID="e3652563-0ffd-4922-99ed-9e4797ea9419" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:38:48.804491 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:38:48.804447    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8dx7f" podUID="e3652563-0ffd-4922-99ed-9e4797ea9419" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:38:53.501689 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:38:53.501618    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8dx7f"
Sep 16 20:38:53.643222 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:38:53.643185    2557 generic.go:356] "Generic (PLEG): container finished" podID="e3652563-0ffd-4922-99ed-9e4797ea9419" containerID="97375a59537e7b9f2d3a3da00e48f96b2163ae14ebd2347c5474ee8c765412da" exitCode=0
Sep 16 20:38:53.643222 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:38:53.643227    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8dx7f" event={"ID":"e3652563-0ffd-4922-99ed-9e4797ea9419","Type":"ContainerDied","Data":"97375a59537e7b9f2d3a3da00e48f96b2163ae14ebd2347c5474ee8c765412da"}
Sep 16 20:38:53.643445 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:38:53.643235    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8dx7f"
Sep 16 20:38:53.643445 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:38:53.643269    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8dx7f" event={"ID":"e3652563-0ffd-4922-99ed-9e4797ea9419","Type":"ContainerDied","Data":"7d73decb89cf3f3a572cf4f64811519b33a4b2b0bd9803c4791f1192cb3eb9d7"}
Sep 16 20:38:53.643445 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:38:53.643290    2557 scope.go:117] "RemoveContainer" containerID="97375a59537e7b9f2d3a3da00e48f96b2163ae14ebd2347c5474ee8c765412da"
Sep 16 20:38:53.651979 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:38:53.651948    2557 scope.go:117] "RemoveContainer" containerID="97375a59537e7b9f2d3a3da00e48f96b2163ae14ebd2347c5474ee8c765412da"
Sep 16 20:38:53.652256 ip-10-0-139-81 kubenswrapper[2557]: E0916 20:38:53.652237    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97375a59537e7b9f2d3a3da00e48f96b2163ae14ebd2347c5474ee8c765412da\": container with ID starting with 97375a59537e7b9f2d3a3da00e48f96b2163ae14ebd2347c5474ee8c765412da not found: ID does not exist" containerID="97375a59537e7b9f2d3a3da00e48f96b2163ae14ebd2347c5474ee8c765412da"
Sep 16 20:38:53.652305 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:38:53.652265    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97375a59537e7b9f2d3a3da00e48f96b2163ae14ebd2347c5474ee8c765412da"} err="failed to get container status \"97375a59537e7b9f2d3a3da00e48f96b2163ae14ebd2347c5474ee8c765412da\": rpc error: code = NotFound desc = could not find container \"97375a59537e7b9f2d3a3da00e48f96b2163ae14ebd2347c5474ee8c765412da\": container with ID starting with 97375a59537e7b9f2d3a3da00e48f96b2163ae14ebd2347c5474ee8c765412da not found: ID does not exist"
Sep 16 20:38:53.658606 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:38:53.658587    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3652563-0ffd-4922-99ed-9e4797ea9419-tls-certs\") pod \"e3652563-0ffd-4922-99ed-9e4797ea9419\" (UID: \"e3652563-0ffd-4922-99ed-9e4797ea9419\") "
Sep 16 20:38:53.658686 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:38:53.658620    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wmthb\" (UniqueName: \"kubernetes.io/projected/e3652563-0ffd-4922-99ed-9e4797ea9419-kube-api-access-wmthb\") pod \"e3652563-0ffd-4922-99ed-9e4797ea9419\" (UID: \"e3652563-0ffd-4922-99ed-9e4797ea9419\") "
Sep 16 20:38:53.660572 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:38:53.660547    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3652563-0ffd-4922-99ed-9e4797ea9419-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e3652563-0ffd-4922-99ed-9e4797ea9419" (UID: "e3652563-0ffd-4922-99ed-9e4797ea9419"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:38:53.660726 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:38:53.660706    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3652563-0ffd-4922-99ed-9e4797ea9419-kube-api-access-wmthb" (OuterVolumeSpecName: "kube-api-access-wmthb") pod "e3652563-0ffd-4922-99ed-9e4797ea9419" (UID: "e3652563-0ffd-4922-99ed-9e4797ea9419"). InnerVolumeSpecName "kube-api-access-wmthb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:38:53.760338 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:38:53.760240    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3652563-0ffd-4922-99ed-9e4797ea9419-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:38:53.760338 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:38:53.760280    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmthb\" (UniqueName: \"kubernetes.io/projected/e3652563-0ffd-4922-99ed-9e4797ea9419-kube-api-access-wmthb\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:38:53.969305 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:38:53.969255    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8dx7f"]
Sep 16 20:38:53.972884 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:38:53.972855    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-8dx7f"]
Sep 16 20:38:55.936928 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:38:55.936891    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3652563-0ffd-4922-99ed-9e4797ea9419" path="/var/lib/kubelet/pods/e3652563-0ffd-4922-99ed-9e4797ea9419/volumes"
Sep 16 20:39:56.386177 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:39:56.386137    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-56db58786c-2k7t2"]
Sep 16 20:39:56.386597 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:39:56.386529    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2fa4c4f-42c4-46ef-99bc-7c7fbc708a2f" containerName="main"
Sep 16 20:39:56.386597 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:39:56.386541    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2fa4c4f-42c4-46ef-99bc-7c7fbc708a2f" containerName="main"
Sep 16 20:39:56.386597 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:39:56.386553    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3652563-0ffd-4922-99ed-9e4797ea9419" containerName="main"
Sep 16 20:39:56.386597 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:39:56.386558    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3652563-0ffd-4922-99ed-9e4797ea9419" containerName="main"
Sep 16 20:39:56.386757 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:39:56.386623    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3652563-0ffd-4922-99ed-9e4797ea9419" containerName="main"
Sep 16 20:39:56.386757 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:39:56.386634    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2fa4c4f-42c4-46ef-99bc-7c7fbc708a2f" containerName="main"
Sep 16 20:39:56.390048 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:39:56.390026    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-56db58786c-2k7t2"
Sep 16 20:39:56.394251 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:39:56.394223    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 16 20:39:56.394383 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:39:56.394246    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 16 20:39:56.394383 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:39:56.394275    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Sep 16 20:39:56.394383 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:39:56.394224    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-cshgf\""
Sep 16 20:39:56.394383 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:39:56.394236    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-epp-sa-dockercfg-hhsx5\""
Sep 16 20:39:56.401405 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:39:56.401383    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-56db58786c-2k7t2"]
Sep 16 20:39:56.438595 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:39:56.438549    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwjtj\" (UniqueName: \"kubernetes.io/projected/aa31d54c-6f21-4067-b8f5-548fc76036e0-kube-api-access-zwjtj\") pod \"router-with-refs-test-kserve-router-scheduler-56db58786c-2k7t2\" (UID: \"aa31d54c-6f21-4067-b8f5-548fc76036e0\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-56db58786c-2k7t2"
Sep 16 20:39:56.438595 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:39:56.438596    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa31d54c-6f21-4067-b8f5-548fc76036e0-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-56db58786c-2k7t2\" (UID: \"aa31d54c-6f21-4067-b8f5-548fc76036e0\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-56db58786c-2k7t2"
Sep 16 20:39:56.539902 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:39:56.539860    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwjtj\" (UniqueName: \"kubernetes.io/projected/aa31d54c-6f21-4067-b8f5-548fc76036e0-kube-api-access-zwjtj\") pod \"router-with-refs-test-kserve-router-scheduler-56db58786c-2k7t2\" (UID: \"aa31d54c-6f21-4067-b8f5-548fc76036e0\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-56db58786c-2k7t2"
Sep 16 20:39:56.539902 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:39:56.539901    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa31d54c-6f21-4067-b8f5-548fc76036e0-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-56db58786c-2k7t2\" (UID: \"aa31d54c-6f21-4067-b8f5-548fc76036e0\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-56db58786c-2k7t2"
Sep 16 20:39:56.542408 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:39:56.542385    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa31d54c-6f21-4067-b8f5-548fc76036e0-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-56db58786c-2k7t2\" (UID: \"aa31d54c-6f21-4067-b8f5-548fc76036e0\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-56db58786c-2k7t2"
Sep 16 20:39:56.548235 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:39:56.548212    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwjtj\" (UniqueName: \"kubernetes.io/projected/aa31d54c-6f21-4067-b8f5-548fc76036e0-kube-api-access-zwjtj\") pod \"router-with-refs-test-kserve-router-scheduler-56db58786c-2k7t2\" (UID: \"aa31d54c-6f21-4067-b8f5-548fc76036e0\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-56db58786c-2k7t2"
Sep 16 20:39:56.701180 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:39:56.701086    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-56db58786c-2k7t2"
Sep 16 20:39:56.973504 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:39:56.973425    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-56db58786c-2k7t2"]
Sep 16 20:39:56.977420 ip-10-0-139-81 kubenswrapper[2557]: W0916 20:39:56.977388    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa31d54c_6f21_4067_b8f5_548fc76036e0.slice/crio-040ee9f98c8a17bacbac4571cb1812ff6f708714a39913530c89b2a64cfb3f57 WatchSource:0}: Error finding container 040ee9f98c8a17bacbac4571cb1812ff6f708714a39913530c89b2a64cfb3f57: Status 404 returned error can't find the container with id 040ee9f98c8a17bacbac4571cb1812ff6f708714a39913530c89b2a64cfb3f57
Sep 16 20:39:57.976132 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:39:57.976090    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-56db58786c-2k7t2" event={"ID":"aa31d54c-6f21-4067-b8f5-548fc76036e0","Type":"ContainerStarted","Data":"bb6d906adf37e1eebc7c2b9ed65e91bcb68b5fd718521e0f9bf5af0a65f6e682"}
Sep 16 20:39:57.976571 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:39:57.976160    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-56db58786c-2k7t2"
Sep 16 20:39:57.976571 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:39:57.976176    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-56db58786c-2k7t2" event={"ID":"aa31d54c-6f21-4067-b8f5-548fc76036e0","Type":"ContainerStarted","Data":"040ee9f98c8a17bacbac4571cb1812ff6f708714a39913530c89b2a64cfb3f57"}
Sep 16 20:39:57.992963 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:39:57.992898    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-56db58786c-2k7t2" podStartSLOduration=1.992874735 podStartE2EDuration="1.992874735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:39:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:39:57.992677479 +0000 UTC m=+3626.518065877" watchObservedRunningTime="2026-09-16 20:39:57.992874735 +0000 UTC m=+3626.518263121"
Sep 16 20:40:28.983730 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:40:28.983695    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-56db58786c-2k7t2"
Sep 16 20:42:01.430779 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:42:01.430744    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-56db58786c-2k7t2"]
Sep 16 20:42:08.983089 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:42:08.983046    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-56db58786c-2k7t2" podUID="aa31d54c-6f21-4067-b8f5-548fc76036e0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:42:16.449970 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:42:16.449920    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-56db58786c-2k7t2" podUID="aa31d54c-6f21-4067-b8f5-548fc76036e0" containerName="main" containerID="cri-o://bb6d906adf37e1eebc7c2b9ed65e91bcb68b5fd718521e0f9bf5af0a65f6e682" gracePeriod=45
Sep 16 20:42:18.983347 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:42:18.983308    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-56db58786c-2k7t2" podUID="aa31d54c-6f21-4067-b8f5-548fc76036e0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:42:28.983925 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:42:28.983821    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-56db58786c-2k7t2" podUID="aa31d54c-6f21-4067-b8f5-548fc76036e0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:42:28.984302 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:42:28.983945    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-56db58786c-2k7t2"
Sep 16 20:42:38.983274 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:42:38.983232    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-56db58786c-2k7t2" podUID="aa31d54c-6f21-4067-b8f5-548fc76036e0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:42:46.709912 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:42:46.709812    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-56db58786c-2k7t2"
Sep 16 20:42:46.803675 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:42:46.803633    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zwjtj\" (UniqueName: \"kubernetes.io/projected/aa31d54c-6f21-4067-b8f5-548fc76036e0-kube-api-access-zwjtj\") pod \"aa31d54c-6f21-4067-b8f5-548fc76036e0\" (UID: \"aa31d54c-6f21-4067-b8f5-548fc76036e0\") "
Sep 16 20:42:46.803896 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:42:46.803722    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa31d54c-6f21-4067-b8f5-548fc76036e0-tls-certs\") pod \"aa31d54c-6f21-4067-b8f5-548fc76036e0\" (UID: \"aa31d54c-6f21-4067-b8f5-548fc76036e0\") "
Sep 16 20:42:46.806000 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:42:46.805966    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa31d54c-6f21-4067-b8f5-548fc76036e0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aa31d54c-6f21-4067-b8f5-548fc76036e0" (UID: "aa31d54c-6f21-4067-b8f5-548fc76036e0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:42:46.806115 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:42:46.805977    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa31d54c-6f21-4067-b8f5-548fc76036e0-kube-api-access-zwjtj" (OuterVolumeSpecName: "kube-api-access-zwjtj") pod "aa31d54c-6f21-4067-b8f5-548fc76036e0" (UID: "aa31d54c-6f21-4067-b8f5-548fc76036e0"). InnerVolumeSpecName "kube-api-access-zwjtj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:42:46.813779 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:42:46.813753    2557 generic.go:356] "Generic (PLEG): container finished" podID="aa31d54c-6f21-4067-b8f5-548fc76036e0" containerID="bb6d906adf37e1eebc7c2b9ed65e91bcb68b5fd718521e0f9bf5af0a65f6e682" exitCode=0
Sep 16 20:42:46.813887 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:42:46.813806    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-56db58786c-2k7t2"
Sep 16 20:42:46.813887 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:42:46.813833    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-56db58786c-2k7t2" event={"ID":"aa31d54c-6f21-4067-b8f5-548fc76036e0","Type":"ContainerDied","Data":"bb6d906adf37e1eebc7c2b9ed65e91bcb68b5fd718521e0f9bf5af0a65f6e682"}
Sep 16 20:42:46.813979 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:42:46.813905    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-56db58786c-2k7t2" event={"ID":"aa31d54c-6f21-4067-b8f5-548fc76036e0","Type":"ContainerDied","Data":"040ee9f98c8a17bacbac4571cb1812ff6f708714a39913530c89b2a64cfb3f57"}
Sep 16 20:42:46.813979 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:42:46.813930    2557 scope.go:117] "RemoveContainer" containerID="bb6d906adf37e1eebc7c2b9ed65e91bcb68b5fd718521e0f9bf5af0a65f6e682"
Sep 16 20:42:46.826560 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:42:46.826542    2557 scope.go:117] "RemoveContainer" containerID="bb6d906adf37e1eebc7c2b9ed65e91bcb68b5fd718521e0f9bf5af0a65f6e682"
Sep 16 20:42:46.826895 ip-10-0-139-81 kubenswrapper[2557]: E0916 20:42:46.826874    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb6d906adf37e1eebc7c2b9ed65e91bcb68b5fd718521e0f9bf5af0a65f6e682\": container with ID starting with bb6d906adf37e1eebc7c2b9ed65e91bcb68b5fd718521e0f9bf5af0a65f6e682 not found: ID does not exist" containerID="bb6d906adf37e1eebc7c2b9ed65e91bcb68b5fd718521e0f9bf5af0a65f6e682"
Sep 16 20:42:46.826949 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:42:46.826905    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb6d906adf37e1eebc7c2b9ed65e91bcb68b5fd718521e0f9bf5af0a65f6e682"} err="failed to get container status \"bb6d906adf37e1eebc7c2b9ed65e91bcb68b5fd718521e0f9bf5af0a65f6e682\": rpc error: code = NotFound desc = could not find container \"bb6d906adf37e1eebc7c2b9ed65e91bcb68b5fd718521e0f9bf5af0a65f6e682\": container with ID starting with bb6d906adf37e1eebc7c2b9ed65e91bcb68b5fd718521e0f9bf5af0a65f6e682 not found: ID does not exist"
Sep 16 20:42:46.837291 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:42:46.837254    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-56db58786c-2k7t2"]
Sep 16 20:42:46.841119 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:42:46.841092    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-56db58786c-2k7t2"]
Sep 16 20:42:46.905190 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:42:46.905157    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwjtj\" (UniqueName: \"kubernetes.io/projected/aa31d54c-6f21-4067-b8f5-548fc76036e0-kube-api-access-zwjtj\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:42:46.905190 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:42:46.905185    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa31d54c-6f21-4067-b8f5-548fc76036e0-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:42:47.936948 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:42:47.936915    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa31d54c-6f21-4067-b8f5-548fc76036e0" path="/var/lib/kubelet/pods/aa31d54c-6f21-4067-b8f5-548fc76036e0/volumes"
Sep 16 20:42:56.735098 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:42:56.735059    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scher5k65"]
Sep 16 20:42:56.735819 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:42:56.735628    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa31d54c-6f21-4067-b8f5-548fc76036e0" containerName="main"
Sep 16 20:42:56.735819 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:42:56.735653    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa31d54c-6f21-4067-b8f5-548fc76036e0" containerName="main"
Sep 16 20:42:56.735819 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:42:56.735753    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa31d54c-6f21-4067-b8f5-548fc76036e0" containerName="main"
Sep 16 20:42:56.738953 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:42:56.738930    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scher5k65"
Sep 16 20:42:56.743425 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:42:56.743220    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 16 20:42:56.743425 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:42:56.743239    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 16 20:42:56.743425 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:42:56.743307    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 16 20:42:56.743425 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:42:56.743320    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-lq24g\""
Sep 16 20:42:56.743425 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:42:56.743394    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa-dockercfg-xxzlt\""
Sep 16 20:42:56.751219 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:42:56.751192    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scher5k65"]
Sep 16 20:42:56.789726 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:42:56.789689    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvkn8\" (UniqueName: \"kubernetes.io/projected/230b4b9d-d044-463f-864a-b3a9156b8af8-kube-api-access-gvkn8\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scher5k65\" (UID: \"230b4b9d-d044-463f-864a-b3a9156b8af8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scher5k65"
Sep 16 20:42:56.789917 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:42:56.789748    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/230b4b9d-d044-463f-864a-b3a9156b8af8-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scher5k65\" (UID: \"230b4b9d-d044-463f-864a-b3a9156b8af8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scher5k65"
Sep 16 20:42:56.891159 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:42:56.891127    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvkn8\" (UniqueName: \"kubernetes.io/projected/230b4b9d-d044-463f-864a-b3a9156b8af8-kube-api-access-gvkn8\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scher5k65\" (UID: \"230b4b9d-d044-463f-864a-b3a9156b8af8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scher5k65"
Sep 16 20:42:56.891358 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:42:56.891189    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/230b4b9d-d044-463f-864a-b3a9156b8af8-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scher5k65\" (UID: \"230b4b9d-d044-463f-864a-b3a9156b8af8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scher5k65"
Sep 16 20:42:56.893828 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:42:56.893797    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/230b4b9d-d044-463f-864a-b3a9156b8af8-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scher5k65\" (UID: \"230b4b9d-d044-463f-864a-b3a9156b8af8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scher5k65"
Sep 16 20:42:56.914096 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:42:56.914067    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvkn8\" (UniqueName: \"kubernetes.io/projected/230b4b9d-d044-463f-864a-b3a9156b8af8-kube-api-access-gvkn8\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scher5k65\" (UID: \"230b4b9d-d044-463f-864a-b3a9156b8af8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scher5k65"
Sep 16 20:42:57.052396 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:42:57.052311    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scher5k65"
Sep 16 20:42:57.331097 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:42:57.331067    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scher5k65"]
Sep 16 20:42:57.333060 ip-10-0-139-81 kubenswrapper[2557]: W0916 20:42:57.333029    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod230b4b9d_d044_463f_864a_b3a9156b8af8.slice/crio-0e5f3d1b65efdbea8d26ddc6bb52eb6016072e6805861375e4cdffeecc328258 WatchSource:0}: Error finding container 0e5f3d1b65efdbea8d26ddc6bb52eb6016072e6805861375e4cdffeecc328258: Status 404 returned error can't find the container with id 0e5f3d1b65efdbea8d26ddc6bb52eb6016072e6805861375e4cdffeecc328258
Sep 16 20:42:57.869378 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:42:57.869341    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scher5k65" event={"ID":"230b4b9d-d044-463f-864a-b3a9156b8af8","Type":"ContainerStarted","Data":"79e1a131ff3aa68980b2a9efa08efced03d153c6c7b6ed8014e2ab57ce3caeda"}
Sep 16 20:42:57.869768 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:42:57.869402    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scher5k65"
Sep 16 20:42:57.869768 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:42:57.869419    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scher5k65" event={"ID":"230b4b9d-d044-463f-864a-b3a9156b8af8","Type":"ContainerStarted","Data":"0e5f3d1b65efdbea8d26ddc6bb52eb6016072e6805861375e4cdffeecc328258"}
Sep 16 20:42:57.891778 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:42:57.891712    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scher5k65" podStartSLOduration=1.891693283 podStartE2EDuration="1.891693283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:42:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:42:57.890373107 +0000 UTC m=+3806.415761493" watchObservedRunningTime="2026-09-16 20:42:57.891693283 +0000 UTC m=+3806.417081669"
Sep 16 20:43:28.875387 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:43:28.875357    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scher5k65"
Sep 16 20:45:56.978349 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:45:56.978309    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scher5k65"]
Sep 16 20:45:58.875347 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:45:58.875306    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scher5k65" podUID="230b4b9d-d044-463f-864a-b3a9156b8af8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:46:08.876168 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:46:08.876122    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scher5k65" podUID="230b4b9d-d044-463f-864a-b3a9156b8af8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:46:11.998362 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:46:11.998314    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scher5k65" podUID="230b4b9d-d044-463f-864a-b3a9156b8af8" containerName="main" containerID="cri-o://79e1a131ff3aa68980b2a9efa08efced03d153c6c7b6ed8014e2ab57ce3caeda" gracePeriod=45
Sep 16 20:46:18.876131 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:46:18.876087    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scher5k65" podUID="230b4b9d-d044-463f-864a-b3a9156b8af8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:46:18.876591 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:46:18.876224    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scher5k65"
Sep 16 20:46:28.875407 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:46:28.875365    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scher5k65" podUID="230b4b9d-d044-463f-864a-b3a9156b8af8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:46:38.875828 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:46:38.875784    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scher5k65" podUID="230b4b9d-d044-463f-864a-b3a9156b8af8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:46:42.249198 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:46:42.249169    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scher5k65"
Sep 16 20:46:42.317803 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:46:42.317713    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/230b4b9d-d044-463f-864a-b3a9156b8af8-tls-certs\") pod \"230b4b9d-d044-463f-864a-b3a9156b8af8\" (UID: \"230b4b9d-d044-463f-864a-b3a9156b8af8\") "
Sep 16 20:46:42.317803 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:46:42.317774    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvkn8\" (UniqueName: \"kubernetes.io/projected/230b4b9d-d044-463f-864a-b3a9156b8af8-kube-api-access-gvkn8\") pod \"230b4b9d-d044-463f-864a-b3a9156b8af8\" (UID: \"230b4b9d-d044-463f-864a-b3a9156b8af8\") "
Sep 16 20:46:42.320070 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:46:42.320031    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/230b4b9d-d044-463f-864a-b3a9156b8af8-kube-api-access-gvkn8" (OuterVolumeSpecName: "kube-api-access-gvkn8") pod "230b4b9d-d044-463f-864a-b3a9156b8af8" (UID: "230b4b9d-d044-463f-864a-b3a9156b8af8"). InnerVolumeSpecName "kube-api-access-gvkn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:46:42.320208 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:46:42.320085    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/230b4b9d-d044-463f-864a-b3a9156b8af8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "230b4b9d-d044-463f-864a-b3a9156b8af8" (UID: "230b4b9d-d044-463f-864a-b3a9156b8af8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:46:42.418992 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:46:42.418954    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvkn8\" (UniqueName: \"kubernetes.io/projected/230b4b9d-d044-463f-864a-b3a9156b8af8-kube-api-access-gvkn8\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:46:42.418992 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:46:42.418985    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/230b4b9d-d044-463f-864a-b3a9156b8af8-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:46:42.972482 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:46:42.972448    2557 generic.go:356] "Generic (PLEG): container finished" podID="230b4b9d-d044-463f-864a-b3a9156b8af8" containerID="79e1a131ff3aa68980b2a9efa08efced03d153c6c7b6ed8014e2ab57ce3caeda" exitCode=0
Sep 16 20:46:42.972482 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:46:42.972491    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scher5k65"
Sep 16 20:46:42.972719 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:46:42.972503    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scher5k65" event={"ID":"230b4b9d-d044-463f-864a-b3a9156b8af8","Type":"ContainerDied","Data":"79e1a131ff3aa68980b2a9efa08efced03d153c6c7b6ed8014e2ab57ce3caeda"}
Sep 16 20:46:42.972719 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:46:42.972547    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scher5k65" event={"ID":"230b4b9d-d044-463f-864a-b3a9156b8af8","Type":"ContainerDied","Data":"0e5f3d1b65efdbea8d26ddc6bb52eb6016072e6805861375e4cdffeecc328258"}
Sep 16 20:46:42.972719 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:46:42.972574    2557 scope.go:117] "RemoveContainer" containerID="79e1a131ff3aa68980b2a9efa08efced03d153c6c7b6ed8014e2ab57ce3caeda"
Sep 16 20:46:42.981821 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:46:42.981799    2557 scope.go:117] "RemoveContainer" containerID="79e1a131ff3aa68980b2a9efa08efced03d153c6c7b6ed8014e2ab57ce3caeda"
Sep 16 20:46:42.982166 ip-10-0-139-81 kubenswrapper[2557]: E0916 20:46:42.982148    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79e1a131ff3aa68980b2a9efa08efced03d153c6c7b6ed8014e2ab57ce3caeda\": container with ID starting with 79e1a131ff3aa68980b2a9efa08efced03d153c6c7b6ed8014e2ab57ce3caeda not found: ID does not exist" containerID="79e1a131ff3aa68980b2a9efa08efced03d153c6c7b6ed8014e2ab57ce3caeda"
Sep 16 20:46:42.982210 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:46:42.982176    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79e1a131ff3aa68980b2a9efa08efced03d153c6c7b6ed8014e2ab57ce3caeda"} err="failed to get container status \"79e1a131ff3aa68980b2a9efa08efced03d153c6c7b6ed8014e2ab57ce3caeda\": rpc error: code = NotFound desc = could not find container \"79e1a131ff3aa68980b2a9efa08efced03d153c6c7b6ed8014e2ab57ce3caeda\": container with ID starting with 79e1a131ff3aa68980b2a9efa08efced03d153c6c7b6ed8014e2ab57ce3caeda not found: ID does not exist"
Sep 16 20:46:42.995074 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:46:42.995043    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scher5k65"]
Sep 16 20:46:42.998715 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:46:42.998686    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scher5k65"]
Sep 16 20:46:43.938249 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:46:43.938215    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="230b4b9d-d044-463f-864a-b3a9156b8af8" path="/var/lib/kubelet/pods/230b4b9d-d044-463f-864a-b3a9156b8af8/volumes"
Sep 16 20:46:57.325231 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:46:57.325147    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffpcfzv"]
Sep 16 20:46:57.325615 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:46:57.325600    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="230b4b9d-d044-463f-864a-b3a9156b8af8" containerName="main"
Sep 16 20:46:57.325664 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:46:57.325617    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="230b4b9d-d044-463f-864a-b3a9156b8af8" containerName="main"
Sep 16 20:46:57.325700 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:46:57.325687    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="230b4b9d-d044-463f-864a-b3a9156b8af8" containerName="main"
Sep 16 20:46:57.329038 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:46:57.329011    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffpcfzv"
Sep 16 20:46:57.332979 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:46:57.332953    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-dzph5\""
Sep 16 20:46:57.333125 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:46:57.333038    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 16 20:46:57.333125 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:46:57.333074    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 16 20:46:57.333125 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:46:57.332954    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 16 20:46:57.333281 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:46:57.333023    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-epp-sa-dockercfg-cdrt5\""
Sep 16 20:46:57.339289 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:46:57.339263    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffpcfzv"]
Sep 16 20:46:57.454897 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:46:57.454828    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6e45fbc-55cf-4686-ac25-93acae6edfdd-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-67cffpcfzv\" (UID: \"d6e45fbc-55cf-4686-ac25-93acae6edfdd\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffpcfzv"
Sep 16 20:46:57.455053 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:46:57.454942    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zzhn\" (UniqueName: \"kubernetes.io/projected/d6e45fbc-55cf-4686-ac25-93acae6edfdd-kube-api-access-7zzhn\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-67cffpcfzv\" (UID: \"d6e45fbc-55cf-4686-ac25-93acae6edfdd\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffpcfzv"
Sep 16 20:46:57.555856 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:46:57.555814    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6e45fbc-55cf-4686-ac25-93acae6edfdd-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-67cffpcfzv\" (UID: \"d6e45fbc-55cf-4686-ac25-93acae6edfdd\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffpcfzv"
Sep 16 20:46:57.556018 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:46:57.555902    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zzhn\" (UniqueName: \"kubernetes.io/projected/d6e45fbc-55cf-4686-ac25-93acae6edfdd-kube-api-access-7zzhn\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-67cffpcfzv\" (UID: \"d6e45fbc-55cf-4686-ac25-93acae6edfdd\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffpcfzv"
Sep 16 20:46:57.558774 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:46:57.558749    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6e45fbc-55cf-4686-ac25-93acae6edfdd-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-67cffpcfzv\" (UID: \"d6e45fbc-55cf-4686-ac25-93acae6edfdd\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffpcfzv"
Sep 16 20:46:57.569623 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:46:57.569589    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zzhn\" (UniqueName: \"kubernetes.io/projected/d6e45fbc-55cf-4686-ac25-93acae6edfdd-kube-api-access-7zzhn\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-67cffpcfzv\" (UID: \"d6e45fbc-55cf-4686-ac25-93acae6edfdd\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffpcfzv"
Sep 16 20:46:57.641044 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:46:57.640960    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffpcfzv"
Sep 16 20:46:57.921130 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:46:57.921101    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffpcfzv"]
Sep 16 20:46:57.923606 ip-10-0-139-81 kubenswrapper[2557]: W0916 20:46:57.923568    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6e45fbc_55cf_4686_ac25_93acae6edfdd.slice/crio-a750ab98edfda99404dce265d3096fa781197dd7a7ae8d978d67768f4f8054e4 WatchSource:0}: Error finding container a750ab98edfda99404dce265d3096fa781197dd7a7ae8d978d67768f4f8054e4: Status 404 returned error can't find the container with id a750ab98edfda99404dce265d3096fa781197dd7a7ae8d978d67768f4f8054e4
Sep 16 20:46:58.054433 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:46:58.054386    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffpcfzv" event={"ID":"d6e45fbc-55cf-4686-ac25-93acae6edfdd","Type":"ContainerStarted","Data":"a67063f31c1044127c2222100d3526b985f78ac26eaebd3a87d88ba90a0c033b"}
Sep 16 20:46:58.054627 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:46:58.054456    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffpcfzv" event={"ID":"d6e45fbc-55cf-4686-ac25-93acae6edfdd","Type":"ContainerStarted","Data":"a750ab98edfda99404dce265d3096fa781197dd7a7ae8d978d67768f4f8054e4"}
Sep 16 20:46:58.054627 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:46:58.054521    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffpcfzv"
Sep 16 20:46:58.076492 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:46:58.076390    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffpcfzv" podStartSLOduration=1.076367209 podStartE2EDuration="1.076367209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:46:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:46:58.076196254 +0000 UTC m=+4046.601584664" watchObservedRunningTime="2026-09-16 20:46:58.076367209 +0000 UTC m=+4046.601755595"
Sep 16 20:47:20.287831 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:47:20.287778    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj"]
Sep 16 20:47:20.291657 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:47:20.291624    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj"
Sep 16 20:47:20.296085 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:47:20.296049    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"kube-root-ca.crt\""
Sep 16 20:47:20.296272 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:47:20.296114    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"default-dockercfg-2plrq\""
Sep 16 20:47:20.296272 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:47:20.296052    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"lora-pvc-adapters-test-kserve-self-signed-certs\""
Sep 16 20:47:20.296272 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:47:20.296054    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"openshift-service-ca.crt\""
Sep 16 20:47:20.304883 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:47:20.304856    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj"]
Sep 16 20:47:20.469183 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:47:20.469140    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afda5388-e4be-44da-84f5-52459f2ce2f0-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj\" (UID: \"afda5388-e4be-44da-84f5-52459f2ce2f0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj"
Sep 16 20:47:20.469378 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:47:20.469205    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8rgd\" (UniqueName: \"kubernetes.io/projected/afda5388-e4be-44da-84f5-52459f2ce2f0-kube-api-access-n8rgd\") pod \"lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj\" (UID: \"afda5388-e4be-44da-84f5-52459f2ce2f0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj"
Sep 16 20:47:20.469378 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:47:20.469256    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afda5388-e4be-44da-84f5-52459f2ce2f0-home\") pod \"lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj\" (UID: \"afda5388-e4be-44da-84f5-52459f2ce2f0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj"
Sep 16 20:47:20.469378 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:47:20.469277    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afda5388-e4be-44da-84f5-52459f2ce2f0-tls-certs\") pod \"lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj\" (UID: \"afda5388-e4be-44da-84f5-52459f2ce2f0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj"
Sep 16 20:47:20.469378 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:47:20.469321    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afda5388-e4be-44da-84f5-52459f2ce2f0-dshm\") pod \"lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj\" (UID: \"afda5388-e4be-44da-84f5-52459f2ce2f0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj"
Sep 16 20:47:20.469378 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:47:20.469350    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afda5388-e4be-44da-84f5-52459f2ce2f0-model-cache\") pod \"lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj\" (UID: \"afda5388-e4be-44da-84f5-52459f2ce2f0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj"
Sep 16 20:47:20.570637 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:47:20.570534    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afda5388-e4be-44da-84f5-52459f2ce2f0-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj\" (UID: \"afda5388-e4be-44da-84f5-52459f2ce2f0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj"
Sep 16 20:47:20.570637 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:47:20.570609    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8rgd\" (UniqueName: \"kubernetes.io/projected/afda5388-e4be-44da-84f5-52459f2ce2f0-kube-api-access-n8rgd\") pod \"lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj\" (UID: \"afda5388-e4be-44da-84f5-52459f2ce2f0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj"
Sep 16 20:47:20.570878 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:47:20.570655    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afda5388-e4be-44da-84f5-52459f2ce2f0-home\") pod \"lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj\" (UID: \"afda5388-e4be-44da-84f5-52459f2ce2f0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj"
Sep 16 20:47:20.570878 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:47:20.570671    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afda5388-e4be-44da-84f5-52459f2ce2f0-tls-certs\") pod \"lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj\" (UID: \"afda5388-e4be-44da-84f5-52459f2ce2f0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj"
Sep 16 20:47:20.570878 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:47:20.570695    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afda5388-e4be-44da-84f5-52459f2ce2f0-dshm\") pod \"lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj\" (UID: \"afda5388-e4be-44da-84f5-52459f2ce2f0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj"
Sep 16 20:47:20.570878 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:47:20.570732    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afda5388-e4be-44da-84f5-52459f2ce2f0-model-cache\") pod \"lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj\" (UID: \"afda5388-e4be-44da-84f5-52459f2ce2f0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj"
Sep 16 20:47:20.571088 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:47:20.571065    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afda5388-e4be-44da-84f5-52459f2ce2f0-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj\" (UID: \"afda5388-e4be-44da-84f5-52459f2ce2f0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj"
Sep 16 20:47:20.571140 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:47:20.571114    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afda5388-e4be-44da-84f5-52459f2ce2f0-home\") pod \"lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj\" (UID: \"afda5388-e4be-44da-84f5-52459f2ce2f0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj"
Sep 16 20:47:20.571177 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:47:20.571149    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afda5388-e4be-44da-84f5-52459f2ce2f0-model-cache\") pod \"lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj\" (UID: \"afda5388-e4be-44da-84f5-52459f2ce2f0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj"
Sep 16 20:47:20.573147 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:47:20.573108    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afda5388-e4be-44da-84f5-52459f2ce2f0-dshm\") pod \"lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj\" (UID: \"afda5388-e4be-44da-84f5-52459f2ce2f0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj"
Sep 16 20:47:20.573297 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:47:20.573279    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afda5388-e4be-44da-84f5-52459f2ce2f0-tls-certs\") pod \"lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj\" (UID: \"afda5388-e4be-44da-84f5-52459f2ce2f0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj"
Sep 16 20:47:20.580630 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:47:20.580606    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8rgd\" (UniqueName: \"kubernetes.io/projected/afda5388-e4be-44da-84f5-52459f2ce2f0-kube-api-access-n8rgd\") pod \"lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj\" (UID: \"afda5388-e4be-44da-84f5-52459f2ce2f0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj"
Sep 16 20:47:23.098917 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:47:23.098870    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-5f3e01ef-3bd2-408b-a72d-686307831fd6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e686fea0c40225c9\") pod \"lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj\" (UID: \"afda5388-e4be-44da-84f5-52459f2ce2f0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj"
Sep 16 20:47:23.107283 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:47:23.107245    2557 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-5f3e01ef-3bd2-408b-a72d-686307831fd6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e686fea0c40225c9\") pod \"lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj\" (UID: \"afda5388-e4be-44da-84f5-52459f2ce2f0\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj"
Sep 16 20:47:23.199886 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:47:23.199819    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-5f3e01ef-3bd2-408b-a72d-686307831fd6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e686fea0c40225c9\") pod \"lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj\" (UID: \"afda5388-e4be-44da-84f5-52459f2ce2f0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj"
Sep 16 20:47:23.200056 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:47:23.199981    2557 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-5f3e01ef-3bd2-408b-a72d-686307831fd6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e686fea0c40225c9\") pod \"lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj\" (UID: \"afda5388-e4be-44da-84f5-52459f2ce2f0\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj"
Sep 16 20:47:23.203176 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:47:23.203155    2557 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-5f3e01ef-3bd2-408b-a72d-686307831fd6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e686fea0c40225c9\") pod \"lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj\" (UID: \"afda5388-e4be-44da-84f5-52459f2ce2f0\") DevicePath \"csi-632acd4c964c6208e6c8c9dac6484dd8a907ba3c9e41062073a67b0d8465deed\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj"
Sep 16 20:47:23.299716 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:47:23.299685    2557 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-5f3e01ef-3bd2-408b-a72d-686307831fd6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e686fea0c40225c9\") pod \"lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj\" (UID: \"afda5388-e4be-44da-84f5-52459f2ce2f0\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/74953cf68d69e61d780272c3a3e84f8d087162c0b6ab8fd3493c7738a3949f94/globalmount\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj"
Sep 16 20:47:23.321806 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:47:23.321773    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-5f3e01ef-3bd2-408b-a72d-686307831fd6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e686fea0c40225c9\") pod \"lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj\" (UID: \"afda5388-e4be-44da-84f5-52459f2ce2f0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj"
Sep 16 20:47:23.605083 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:47:23.605046    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj"
Sep 16 20:47:23.887288 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:47:23.887259    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj"]
Sep 16 20:47:23.887773 ip-10-0-139-81 kubenswrapper[2557]: W0916 20:47:23.887746    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafda5388_e4be_44da_84f5_52459f2ce2f0.slice/crio-adb31f9f79d70b3002c6aec51e5024826a1a704c8ef3df4e3cc7c25e0dc7a6f9 WatchSource:0}: Error finding container adb31f9f79d70b3002c6aec51e5024826a1a704c8ef3df4e3cc7c25e0dc7a6f9: Status 404 returned error can't find the container with id adb31f9f79d70b3002c6aec51e5024826a1a704c8ef3df4e3cc7c25e0dc7a6f9
Sep 16 20:47:23.890078 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:47:23.890059    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 20:47:24.185925 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:47:24.185816    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj" event={"ID":"afda5388-e4be-44da-84f5-52459f2ce2f0","Type":"ContainerStarted","Data":"adb31f9f79d70b3002c6aec51e5024826a1a704c8ef3df4e3cc7c25e0dc7a6f9"}
Sep 16 20:47:29.061379 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:47:29.061333    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffpcfzv"
Sep 16 20:47:54.380969 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:47:54.380875    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj" event={"ID":"afda5388-e4be-44da-84f5-52459f2ce2f0","Type":"ContainerStarted","Data":"e1fbc301667e80637673c4f65064181d6b45a55b40d175516b51958572885ad3"}
Sep 16 20:47:54.401434 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:47:54.401371    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj" podStartSLOduration=4.270498998 podStartE2EDuration="34.401354488s" totalImagesPullingTime="30.13085549s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 20:47:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:47:54.399822087 +0000 UTC m=+4102.925210510" watchObservedRunningTime="2026-09-16 20:47:54.401354488 +0000 UTC m=+4102.926742872"
Sep 16 20:47:54.605835 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:47:54.605793    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj"
Sep 16 20:48:03.605395 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:48:03.605339    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj"
Sep 16 20:48:03.606731 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:48:03.606698    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj" podUID="afda5388-e4be-44da-84f5-52459f2ce2f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Sep 16 20:48:13.605517 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:48:13.605466    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj" podUID="afda5388-e4be-44da-84f5-52459f2ce2f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Sep 16 20:48:23.605892 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:48:23.605819    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj" podUID="afda5388-e4be-44da-84f5-52459f2ce2f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Sep 16 20:48:33.605757 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:48:33.605640    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj" podUID="afda5388-e4be-44da-84f5-52459f2ce2f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Sep 16 20:48:43.605617 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:48:43.605550    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj" podUID="afda5388-e4be-44da-84f5-52459f2ce2f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Sep 16 20:48:53.606497 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:48:53.606431    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj" podUID="afda5388-e4be-44da-84f5-52459f2ce2f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Sep 16 20:49:03.606275 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:03.606218    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj" podUID="afda5388-e4be-44da-84f5-52459f2ce2f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Sep 16 20:49:06.463623 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:06.463580    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffpcfzv"]
Sep 16 20:49:09.061311 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:09.061264    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffpcfzv" podUID="d6e45fbc-55cf-4686-ac25-93acae6edfdd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:49:13.615557 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:13.615522    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj"
Sep 16 20:49:13.623151 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:13.623127    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj"
Sep 16 20:49:19.061064 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:19.061021    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffpcfzv" podUID="d6e45fbc-55cf-4686-ac25-93acae6edfdd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:49:21.486942 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:21.486877    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffpcfzv" podUID="d6e45fbc-55cf-4686-ac25-93acae6edfdd" containerName="main" containerID="cri-o://a67063f31c1044127c2222100d3526b985f78ac26eaebd3a87d88ba90a0c033b" gracePeriod=45
Sep 16 20:49:29.061940 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:29.061888    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffpcfzv" podUID="d6e45fbc-55cf-4686-ac25-93acae6edfdd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:49:29.062376 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:29.062037    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffpcfzv"
Sep 16 20:49:39.062289 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:39.062235    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffpcfzv" podUID="d6e45fbc-55cf-4686-ac25-93acae6edfdd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:49:40.034173 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:40.034142    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-a290d4b0_lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj_afda5388-e4be-44da-84f5-52459f2ce2f0/main/0.log"
Sep 16 20:49:41.014446 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:41.014412    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj"]
Sep 16 20:49:49.061550 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:49.061505    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffpcfzv" podUID="d6e45fbc-55cf-4686-ac25-93acae6edfdd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:49:51.727915 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:51.727889    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffpcfzv"
Sep 16 20:49:51.823300 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:51.823217    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7zzhn\" (UniqueName: \"kubernetes.io/projected/d6e45fbc-55cf-4686-ac25-93acae6edfdd-kube-api-access-7zzhn\") pod \"d6e45fbc-55cf-4686-ac25-93acae6edfdd\" (UID: \"d6e45fbc-55cf-4686-ac25-93acae6edfdd\") "
Sep 16 20:49:51.823300 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:51.823271    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6e45fbc-55cf-4686-ac25-93acae6edfdd-tls-certs\") pod \"d6e45fbc-55cf-4686-ac25-93acae6edfdd\" (UID: \"d6e45fbc-55cf-4686-ac25-93acae6edfdd\") "
Sep 16 20:49:51.825517 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:51.825482    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6e45fbc-55cf-4686-ac25-93acae6edfdd-kube-api-access-7zzhn" (OuterVolumeSpecName: "kube-api-access-7zzhn") pod "d6e45fbc-55cf-4686-ac25-93acae6edfdd" (UID: "d6e45fbc-55cf-4686-ac25-93acae6edfdd"). InnerVolumeSpecName "kube-api-access-7zzhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:49:51.825517 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:51.825497    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6e45fbc-55cf-4686-ac25-93acae6edfdd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d6e45fbc-55cf-4686-ac25-93acae6edfdd" (UID: "d6e45fbc-55cf-4686-ac25-93acae6edfdd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:49:51.924857 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:51.924797    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zzhn\" (UniqueName: \"kubernetes.io/projected/d6e45fbc-55cf-4686-ac25-93acae6edfdd-kube-api-access-7zzhn\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:49:51.924857 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:51.924832    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6e45fbc-55cf-4686-ac25-93acae6edfdd-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:49:51.996932 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:51.996893    2557 generic.go:356] "Generic (PLEG): container finished" podID="d6e45fbc-55cf-4686-ac25-93acae6edfdd" containerID="a67063f31c1044127c2222100d3526b985f78ac26eaebd3a87d88ba90a0c033b" exitCode=0
Sep 16 20:49:51.997108 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:51.996939    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffpcfzv"
Sep 16 20:49:51.997108 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:51.996946    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffpcfzv" event={"ID":"d6e45fbc-55cf-4686-ac25-93acae6edfdd","Type":"ContainerDied","Data":"a67063f31c1044127c2222100d3526b985f78ac26eaebd3a87d88ba90a0c033b"}
Sep 16 20:49:51.997108 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:51.997005    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffpcfzv" event={"ID":"d6e45fbc-55cf-4686-ac25-93acae6edfdd","Type":"ContainerDied","Data":"a750ab98edfda99404dce265d3096fa781197dd7a7ae8d978d67768f4f8054e4"}
Sep 16 20:49:51.997108 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:51.997030    2557 scope.go:117] "RemoveContainer" containerID="a67063f31c1044127c2222100d3526b985f78ac26eaebd3a87d88ba90a0c033b"
Sep 16 20:49:52.006329 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:52.006306    2557 scope.go:117] "RemoveContainer" containerID="a67063f31c1044127c2222100d3526b985f78ac26eaebd3a87d88ba90a0c033b"
Sep 16 20:49:52.006622 ip-10-0-139-81 kubenswrapper[2557]: E0916 20:49:52.006601    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a67063f31c1044127c2222100d3526b985f78ac26eaebd3a87d88ba90a0c033b\": container with ID starting with a67063f31c1044127c2222100d3526b985f78ac26eaebd3a87d88ba90a0c033b not found: ID does not exist" containerID="a67063f31c1044127c2222100d3526b985f78ac26eaebd3a87d88ba90a0c033b"
Sep 16 20:49:52.006696 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:52.006633    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a67063f31c1044127c2222100d3526b985f78ac26eaebd3a87d88ba90a0c033b"} err="failed to get container status \"a67063f31c1044127c2222100d3526b985f78ac26eaebd3a87d88ba90a0c033b\": rpc error: code = NotFound desc = could not find container \"a67063f31c1044127c2222100d3526b985f78ac26eaebd3a87d88ba90a0c033b\": container with ID starting with a67063f31c1044127c2222100d3526b985f78ac26eaebd3a87d88ba90a0c033b not found: ID does not exist"
Sep 16 20:49:52.013769 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:52.013740    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffpcfzv"]
Sep 16 20:49:52.019451 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:52.019429    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffpcfzv"]
Sep 16 20:49:53.937051 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:53.937017    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6e45fbc-55cf-4686-ac25-93acae6edfdd" path="/var/lib/kubelet/pods/d6e45fbc-55cf-4686-ac25-93acae6edfdd/volumes"
Sep 16 20:49:56.037095 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:56.037006    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj" podUID="afda5388-e4be-44da-84f5-52459f2ce2f0" containerName="main" containerID="cri-o://e1fbc301667e80637673c4f65064181d6b45a55b40d175516b51958572885ad3" gracePeriod=45
Sep 16 20:49:56.691887 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:56.691835    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj"
Sep 16 20:49:56.769913 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:56.769882    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afda5388-e4be-44da-84f5-52459f2ce2f0-home\") pod \"afda5388-e4be-44da-84f5-52459f2ce2f0\" (UID: \"afda5388-e4be-44da-84f5-52459f2ce2f0\") "
Sep 16 20:49:56.770076 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:56.769945    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afda5388-e4be-44da-84f5-52459f2ce2f0-tmp-dir\") pod \"afda5388-e4be-44da-84f5-52459f2ce2f0\" (UID: \"afda5388-e4be-44da-84f5-52459f2ce2f0\") "
Sep 16 20:49:56.770076 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:56.769970    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8rgd\" (UniqueName: \"kubernetes.io/projected/afda5388-e4be-44da-84f5-52459f2ce2f0-kube-api-access-n8rgd\") pod \"afda5388-e4be-44da-84f5-52459f2ce2f0\" (UID: \"afda5388-e4be-44da-84f5-52459f2ce2f0\") "
Sep 16 20:49:56.770076 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:56.770006    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afda5388-e4be-44da-84f5-52459f2ce2f0-model-cache\") pod \"afda5388-e4be-44da-84f5-52459f2ce2f0\" (UID: \"afda5388-e4be-44da-84f5-52459f2ce2f0\") "
Sep 16 20:49:56.770188 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:56.770101    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e686fea0c40225c9\") pod \"afda5388-e4be-44da-84f5-52459f2ce2f0\" (UID: \"afda5388-e4be-44da-84f5-52459f2ce2f0\") "
Sep 16 20:49:56.770188 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:56.770146    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afda5388-e4be-44da-84f5-52459f2ce2f0-tls-certs\") pod \"afda5388-e4be-44da-84f5-52459f2ce2f0\" (UID: \"afda5388-e4be-44da-84f5-52459f2ce2f0\") "
Sep 16 20:49:56.770296 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:56.770192    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afda5388-e4be-44da-84f5-52459f2ce2f0-dshm\") pod \"afda5388-e4be-44da-84f5-52459f2ce2f0\" (UID: \"afda5388-e4be-44da-84f5-52459f2ce2f0\") "
Sep 16 20:49:56.770354 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:56.770331    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afda5388-e4be-44da-84f5-52459f2ce2f0-model-cache" (OuterVolumeSpecName: "model-cache") pod "afda5388-e4be-44da-84f5-52459f2ce2f0" (UID: "afda5388-e4be-44da-84f5-52459f2ce2f0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:49:56.770413 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:56.770372    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afda5388-e4be-44da-84f5-52459f2ce2f0-home" (OuterVolumeSpecName: "home") pod "afda5388-e4be-44da-84f5-52459f2ce2f0" (UID: "afda5388-e4be-44da-84f5-52459f2ce2f0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:49:56.770543 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:56.770522    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afda5388-e4be-44da-84f5-52459f2ce2f0-home\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:49:56.770543 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:56.770545    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afda5388-e4be-44da-84f5-52459f2ce2f0-model-cache\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:49:56.772443 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:56.772414    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afda5388-e4be-44da-84f5-52459f2ce2f0-kube-api-access-n8rgd" (OuterVolumeSpecName: "kube-api-access-n8rgd") pod "afda5388-e4be-44da-84f5-52459f2ce2f0" (UID: "afda5388-e4be-44da-84f5-52459f2ce2f0"). InnerVolumeSpecName "kube-api-access-n8rgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:49:56.772557 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:56.772453    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afda5388-e4be-44da-84f5-52459f2ce2f0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "afda5388-e4be-44da-84f5-52459f2ce2f0" (UID: "afda5388-e4be-44da-84f5-52459f2ce2f0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:49:56.772557 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:56.772465    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afda5388-e4be-44da-84f5-52459f2ce2f0-dshm" (OuterVolumeSpecName: "dshm") pod "afda5388-e4be-44da-84f5-52459f2ce2f0" (UID: "afda5388-e4be-44da-84f5-52459f2ce2f0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:49:56.780618 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:56.780570    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0e686fea0c40225c9" (OuterVolumeSpecName: "kserve-pvc-source") pod "afda5388-e4be-44da-84f5-52459f2ce2f0" (UID: "afda5388-e4be-44da-84f5-52459f2ce2f0"). InnerVolumeSpecName "pvc-5f3e01ef-3bd2-408b-a72d-686307831fd6". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 16 20:49:56.784460 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:56.784433    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afda5388-e4be-44da-84f5-52459f2ce2f0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "afda5388-e4be-44da-84f5-52459f2ce2f0" (UID: "afda5388-e4be-44da-84f5-52459f2ce2f0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:49:56.871738 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:56.871700    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afda5388-e4be-44da-84f5-52459f2ce2f0-tmp-dir\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:49:56.871738 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:56.871735    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8rgd\" (UniqueName: \"kubernetes.io/projected/afda5388-e4be-44da-84f5-52459f2ce2f0-kube-api-access-n8rgd\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:49:56.871975 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:56.871768    2557 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-5f3e01ef-3bd2-408b-a72d-686307831fd6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e686fea0c40225c9\") on node \"ip-10-0-139-81.ec2.internal\" "
Sep 16 20:49:56.871975 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:56.871778    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afda5388-e4be-44da-84f5-52459f2ce2f0-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:49:56.871975 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:56.871787    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afda5388-e4be-44da-84f5-52459f2ce2f0-dshm\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:49:56.899469 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:56.899448    2557 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-5f3e01ef-3bd2-408b-a72d-686307831fd6" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0e686fea0c40225c9") on node "ip-10-0-139-81.ec2.internal"
Sep 16 20:49:56.972405 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:56.972368    2557 reconciler_common.go:299] "Volume detached for volume \"pvc-5f3e01ef-3bd2-408b-a72d-686307831fd6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e686fea0c40225c9\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"csi-632acd4c964c6208e6c8c9dac6484dd8a907ba3c9e41062073a67b0d8465deed\""
Sep 16 20:49:57.027429 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:57.027343    2557 generic.go:356] "Generic (PLEG): container finished" podID="afda5388-e4be-44da-84f5-52459f2ce2f0" containerID="e1fbc301667e80637673c4f65064181d6b45a55b40d175516b51958572885ad3" exitCode=0
Sep 16 20:49:57.027429 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:57.027388    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj"
Sep 16 20:49:57.027611 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:57.027423    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj" event={"ID":"afda5388-e4be-44da-84f5-52459f2ce2f0","Type":"ContainerDied","Data":"e1fbc301667e80637673c4f65064181d6b45a55b40d175516b51958572885ad3"}
Sep 16 20:49:57.027611 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:57.027477    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj" event={"ID":"afda5388-e4be-44da-84f5-52459f2ce2f0","Type":"ContainerDied","Data":"adb31f9f79d70b3002c6aec51e5024826a1a704c8ef3df4e3cc7c25e0dc7a6f9"}
Sep 16 20:49:57.027611 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:57.027499    2557 scope.go:117] "RemoveContainer" containerID="e1fbc301667e80637673c4f65064181d6b45a55b40d175516b51958572885ad3"
Sep 16 20:49:57.036690 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:57.036674    2557 scope.go:117] "RemoveContainer" containerID="e1fbc301667e80637673c4f65064181d6b45a55b40d175516b51958572885ad3"
Sep 16 20:49:57.036982 ip-10-0-139-81 kubenswrapper[2557]: E0916 20:49:57.036962    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1fbc301667e80637673c4f65064181d6b45a55b40d175516b51958572885ad3\": container with ID starting with e1fbc301667e80637673c4f65064181d6b45a55b40d175516b51958572885ad3 not found: ID does not exist" containerID="e1fbc301667e80637673c4f65064181d6b45a55b40d175516b51958572885ad3"
Sep 16 20:49:57.037037 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:57.036988    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1fbc301667e80637673c4f65064181d6b45a55b40d175516b51958572885ad3"} err="failed to get container status \"e1fbc301667e80637673c4f65064181d6b45a55b40d175516b51958572885ad3\": rpc error: code = NotFound desc = could not find container \"e1fbc301667e80637673c4f65064181d6b45a55b40d175516b51958572885ad3\": container with ID starting with e1fbc301667e80637673c4f65064181d6b45a55b40d175516b51958572885ad3 not found: ID does not exist"
Sep 16 20:49:57.050579 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:57.050542    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj"]
Sep 16 20:49:57.057244 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:57.057222    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj"]
Sep 16 20:49:57.616975 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:57.616912    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-ktwfj" podUID="afda5388-e4be-44da-84f5-52459f2ce2f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": context deadline exceeded"
Sep 16 20:49:57.938097 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:49:57.938059    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afda5388-e4be-44da-84f5-52459f2ce2f0" path="/var/lib/kubelet/pods/afda5388-e4be-44da-84f5-52459f2ce2f0/volumes"
Sep 16 20:50:17.462813 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:17.462781    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-85f8df497lmwf5"]
Sep 16 20:50:17.463246 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:17.463193    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6e45fbc-55cf-4686-ac25-93acae6edfdd" containerName="main"
Sep 16 20:50:17.463246 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:17.463206    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6e45fbc-55cf-4686-ac25-93acae6edfdd" containerName="main"
Sep 16 20:50:17.463246 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:17.463222    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afda5388-e4be-44da-84f5-52459f2ce2f0" containerName="main"
Sep 16 20:50:17.463246 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:17.463227    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="afda5388-e4be-44da-84f5-52459f2ce2f0" containerName="main"
Sep 16 20:50:17.463398 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:17.463286    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6e45fbc-55cf-4686-ac25-93acae6edfdd" containerName="main"
Sep 16 20:50:17.463398 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:17.463296    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="afda5388-e4be-44da-84f5-52459f2ce2f0" containerName="main"
Sep 16 20:50:17.468094 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:17.468075    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-85f8df497lmwf5"
Sep 16 20:50:17.472207 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:17.472170    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 16 20:50:17.472338 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:17.472243    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-tfs4k\""
Sep 16 20:50:17.472411 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:17.472318    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-epp-sa-dockercfg-9vvsb\""
Sep 16 20:50:17.472411 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:17.472352    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 16 20:50:17.472512 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:17.472436    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 16 20:50:17.477589 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:17.477567    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-85f8df497lmwf5"]
Sep 16 20:50:17.573262 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:17.573224    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btws8\" (UniqueName: \"kubernetes.io/projected/5f5318a4-95fc-4831-8c01-80950a7c72c0-kube-api-access-btws8\") pod \"router-with-refs-pd-test-kserve-router-scheduler-85f8df497lmwf5\" (UID: \"5f5318a4-95fc-4831-8c01-80950a7c72c0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-85f8df497lmwf5"
Sep 16 20:50:17.573262 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:17.573261    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f5318a4-95fc-4831-8c01-80950a7c72c0-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-85f8df497lmwf5\" (UID: \"5f5318a4-95fc-4831-8c01-80950a7c72c0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-85f8df497lmwf5"
Sep 16 20:50:17.674636 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:17.674594    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btws8\" (UniqueName: \"kubernetes.io/projected/5f5318a4-95fc-4831-8c01-80950a7c72c0-kube-api-access-btws8\") pod \"router-with-refs-pd-test-kserve-router-scheduler-85f8df497lmwf5\" (UID: \"5f5318a4-95fc-4831-8c01-80950a7c72c0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-85f8df497lmwf5"
Sep 16 20:50:17.674636 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:17.674640    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f5318a4-95fc-4831-8c01-80950a7c72c0-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-85f8df497lmwf5\" (UID: \"5f5318a4-95fc-4831-8c01-80950a7c72c0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-85f8df497lmwf5"
Sep 16 20:50:17.677211 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:17.677189    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f5318a4-95fc-4831-8c01-80950a7c72c0-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-85f8df497lmwf5\" (UID: \"5f5318a4-95fc-4831-8c01-80950a7c72c0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-85f8df497lmwf5"
Sep 16 20:50:17.683601 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:17.683575    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btws8\" (UniqueName: \"kubernetes.io/projected/5f5318a4-95fc-4831-8c01-80950a7c72c0-kube-api-access-btws8\") pod \"router-with-refs-pd-test-kserve-router-scheduler-85f8df497lmwf5\" (UID: \"5f5318a4-95fc-4831-8c01-80950a7c72c0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-85f8df497lmwf5"
Sep 16 20:50:17.780233 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:17.780199    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-85f8df497lmwf5"
Sep 16 20:50:18.052994 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:18.052966    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-85f8df497lmwf5"]
Sep 16 20:50:18.055124 ip-10-0-139-81 kubenswrapper[2557]: W0916 20:50:18.055094    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f5318a4_95fc_4831_8c01_80950a7c72c0.slice/crio-9b401cae5e349d6a5198e1196501ea6777ce1d6dacec535a4eb2c4de51ffc0da WatchSource:0}: Error finding container 9b401cae5e349d6a5198e1196501ea6777ce1d6dacec535a4eb2c4de51ffc0da: Status 404 returned error can't find the container with id 9b401cae5e349d6a5198e1196501ea6777ce1d6dacec535a4eb2c4de51ffc0da
Sep 16 20:50:18.133365 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:18.133327    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-85f8df497lmwf5" event={"ID":"5f5318a4-95fc-4831-8c01-80950a7c72c0","Type":"ContainerStarted","Data":"8dcb8a79a92bb59b2b68b2e9ad76d9c8a8fc49ba273d8c0cb50e0e6476233677"}
Sep 16 20:50:18.133501 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:18.133394    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-85f8df497lmwf5" event={"ID":"5f5318a4-95fc-4831-8c01-80950a7c72c0","Type":"ContainerStarted","Data":"9b401cae5e349d6a5198e1196501ea6777ce1d6dacec535a4eb2c4de51ffc0da"}
Sep 16 20:50:18.133501 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:18.133483    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-85f8df497lmwf5"
Sep 16 20:50:18.151954 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:18.151913    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-85f8df497lmwf5" podStartSLOduration=1.151898889 podStartE2EDuration="1.151898889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:50:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:50:18.149785724 +0000 UTC m=+4246.675174111" watchObservedRunningTime="2026-09-16 20:50:18.151898889 +0000 UTC m=+4246.677287274"
Sep 16 20:50:22.496622 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:22.496585    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-85f8df497lmwf5"]
Sep 16 20:50:30.636483 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:30.636446    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls"]
Sep 16 20:50:30.640349 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:30.640331    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls"
Sep 16 20:50:30.642967 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:30.642939    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"seaweedfs-s3-creds\""
Sep 16 20:50:30.643086 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:30.642939    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"openshift-service-ca.crt\""
Sep 16 20:50:30.643086 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:30.643000    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 20:50:30.643237 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:30.643221    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"tls-verification-test-kserve-dockercfg-rvw5n\""
Sep 16 20:50:30.643323 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:30.643305    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"default-dockercfg-mv9dj\""
Sep 16 20:50:30.643402 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:30.643383    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"kube-root-ca.crt\""
Sep 16 20:50:30.644534 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:30.644508    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"tls-verification-test-kserve-self-signed-certs\""
Sep 16 20:50:30.654764 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:30.654736    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls"]
Sep 16 20:50:30.695016 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:30.694984    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1a92db9-754a-445f-919b-c15b8845346f-tls-certs\") pod \"tls-verification-test-kserve-586f66667c-pzdls\" (UID: \"c1a92db9-754a-445f-919b-c15b8845346f\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls"
Sep 16 20:50:30.695174 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:30.695054    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1a92db9-754a-445f-919b-c15b8845346f-tmp-dir\") pod \"tls-verification-test-kserve-586f66667c-pzdls\" (UID: \"c1a92db9-754a-445f-919b-c15b8845346f\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls"
Sep 16 20:50:30.695174 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:30.695087    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-949w5\" (UniqueName: \"kubernetes.io/projected/c1a92db9-754a-445f-919b-c15b8845346f-kube-api-access-949w5\") pod \"tls-verification-test-kserve-586f66667c-pzdls\" (UID: \"c1a92db9-754a-445f-919b-c15b8845346f\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls"
Sep 16 20:50:30.695174 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:30.695144    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1a92db9-754a-445f-919b-c15b8845346f-model-cache\") pod \"tls-verification-test-kserve-586f66667c-pzdls\" (UID: \"c1a92db9-754a-445f-919b-c15b8845346f\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls"
Sep 16 20:50:30.695174 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:30.695163    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1a92db9-754a-445f-919b-c15b8845346f-home\") pod \"tls-verification-test-kserve-586f66667c-pzdls\" (UID: \"c1a92db9-754a-445f-919b-c15b8845346f\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls"
Sep 16 20:50:30.695315 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:30.695194    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1a92db9-754a-445f-919b-c15b8845346f-dshm\") pod \"tls-verification-test-kserve-586f66667c-pzdls\" (UID: \"c1a92db9-754a-445f-919b-c15b8845346f\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls"
Sep 16 20:50:30.695315 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:30.695267    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1a92db9-754a-445f-919b-c15b8845346f-kserve-provision-location\") pod \"tls-verification-test-kserve-586f66667c-pzdls\" (UID: \"c1a92db9-754a-445f-919b-c15b8845346f\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls"
Sep 16 20:50:30.695315 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:30.695308    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c1a92db9-754a-445f-919b-c15b8845346f-cabundle-cert\") pod \"tls-verification-test-kserve-586f66667c-pzdls\" (UID: \"c1a92db9-754a-445f-919b-c15b8845346f\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls"
Sep 16 20:50:30.795880 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:30.795818    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1a92db9-754a-445f-919b-c15b8845346f-dshm\") pod \"tls-verification-test-kserve-586f66667c-pzdls\" (UID: \"c1a92db9-754a-445f-919b-c15b8845346f\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls"
Sep 16 20:50:30.796045 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:30.795888    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1a92db9-754a-445f-919b-c15b8845346f-kserve-provision-location\") pod \"tls-verification-test-kserve-586f66667c-pzdls\" (UID: \"c1a92db9-754a-445f-919b-c15b8845346f\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls"
Sep 16 20:50:30.796045 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:30.795924    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c1a92db9-754a-445f-919b-c15b8845346f-cabundle-cert\") pod \"tls-verification-test-kserve-586f66667c-pzdls\" (UID: \"c1a92db9-754a-445f-919b-c15b8845346f\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls"
Sep 16 20:50:30.796045 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:30.795991    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1a92db9-754a-445f-919b-c15b8845346f-tls-certs\") pod \"tls-verification-test-kserve-586f66667c-pzdls\" (UID: \"c1a92db9-754a-445f-919b-c15b8845346f\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls"
Sep 16 20:50:30.796174 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:30.796053    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1a92db9-754a-445f-919b-c15b8845346f-tmp-dir\") pod \"tls-verification-test-kserve-586f66667c-pzdls\" (UID: \"c1a92db9-754a-445f-919b-c15b8845346f\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls"
Sep 16 20:50:30.796174 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:30.796080    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-949w5\" (UniqueName: \"kubernetes.io/projected/c1a92db9-754a-445f-919b-c15b8845346f-kube-api-access-949w5\") pod \"tls-verification-test-kserve-586f66667c-pzdls\" (UID: \"c1a92db9-754a-445f-919b-c15b8845346f\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls"
Sep 16 20:50:30.796174 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:30.796120    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1a92db9-754a-445f-919b-c15b8845346f-model-cache\") pod \"tls-verification-test-kserve-586f66667c-pzdls\" (UID: \"c1a92db9-754a-445f-919b-c15b8845346f\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls"
Sep 16 20:50:30.796174 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:30.796145    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1a92db9-754a-445f-919b-c15b8845346f-home\") pod \"tls-verification-test-kserve-586f66667c-pzdls\" (UID: \"c1a92db9-754a-445f-919b-c15b8845346f\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls"
Sep 16 20:50:30.796371 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:30.796331    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1a92db9-754a-445f-919b-c15b8845346f-kserve-provision-location\") pod \"tls-verification-test-kserve-586f66667c-pzdls\" (UID: \"c1a92db9-754a-445f-919b-c15b8845346f\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls"
Sep 16 20:50:30.796532 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:30.796510    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1a92db9-754a-445f-919b-c15b8845346f-model-cache\") pod \"tls-verification-test-kserve-586f66667c-pzdls\" (UID: \"c1a92db9-754a-445f-919b-c15b8845346f\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls"
Sep 16 20:50:30.796611 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:30.796557    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1a92db9-754a-445f-919b-c15b8845346f-tmp-dir\") pod \"tls-verification-test-kserve-586f66667c-pzdls\" (UID: \"c1a92db9-754a-445f-919b-c15b8845346f\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls"
Sep 16 20:50:30.796795 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:30.796776    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1a92db9-754a-445f-919b-c15b8845346f-home\") pod \"tls-verification-test-kserve-586f66667c-pzdls\" (UID: \"c1a92db9-754a-445f-919b-c15b8845346f\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls"
Sep 16 20:50:30.796888 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:30.796873    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c1a92db9-754a-445f-919b-c15b8845346f-cabundle-cert\") pod \"tls-verification-test-kserve-586f66667c-pzdls\" (UID: \"c1a92db9-754a-445f-919b-c15b8845346f\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls"
Sep 16 20:50:30.798855 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:30.798817    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1a92db9-754a-445f-919b-c15b8845346f-dshm\") pod \"tls-verification-test-kserve-586f66667c-pzdls\" (UID: \"c1a92db9-754a-445f-919b-c15b8845346f\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls"
Sep 16 20:50:30.799010 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:30.798995    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1a92db9-754a-445f-919b-c15b8845346f-tls-certs\") pod \"tls-verification-test-kserve-586f66667c-pzdls\" (UID: \"c1a92db9-754a-445f-919b-c15b8845346f\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls"
Sep 16 20:50:30.804333 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:30.804313    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-949w5\" (UniqueName: \"kubernetes.io/projected/c1a92db9-754a-445f-919b-c15b8845346f-kube-api-access-949w5\") pod \"tls-verification-test-kserve-586f66667c-pzdls\" (UID: \"c1a92db9-754a-445f-919b-c15b8845346f\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls"
Sep 16 20:50:30.862261 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:30.862231    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-774dd49f6f-7dldz"]
Sep 16 20:50:30.866177 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:30.866158    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-774dd49f6f-7dldz"
Sep 16 20:50:30.868832 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:30.868810    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"tls-verification-test-epp-sa-dockercfg-xgn9x\""
Sep 16 20:50:30.877679 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:30.877653    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-774dd49f6f-7dldz"]
Sep 16 20:50:30.896684 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:30.896605    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f00fe327-50fe-4823-af39-58504b1d2596-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-774dd49f6f-7dldz\" (UID: \"f00fe327-50fe-4823-af39-58504b1d2596\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-774dd49f6f-7dldz"
Sep 16 20:50:30.896684 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:30.896646    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzk9n\" (UniqueName: \"kubernetes.io/projected/f00fe327-50fe-4823-af39-58504b1d2596-kube-api-access-fzk9n\") pod \"tls-verification-test-kserve-router-scheduler-774dd49f6f-7dldz\" (UID: \"f00fe327-50fe-4823-af39-58504b1d2596\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-774dd49f6f-7dldz"
Sep 16 20:50:30.954127 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:30.954092    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls"
Sep 16 20:50:30.998056 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:30.997687    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f00fe327-50fe-4823-af39-58504b1d2596-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-774dd49f6f-7dldz\" (UID: \"f00fe327-50fe-4823-af39-58504b1d2596\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-774dd49f6f-7dldz"
Sep 16 20:50:30.998056 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:30.997738    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzk9n\" (UniqueName: \"kubernetes.io/projected/f00fe327-50fe-4823-af39-58504b1d2596-kube-api-access-fzk9n\") pod \"tls-verification-test-kserve-router-scheduler-774dd49f6f-7dldz\" (UID: \"f00fe327-50fe-4823-af39-58504b1d2596\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-774dd49f6f-7dldz"
Sep 16 20:50:31.000623 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:31.000590    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f00fe327-50fe-4823-af39-58504b1d2596-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-774dd49f6f-7dldz\" (UID: \"f00fe327-50fe-4823-af39-58504b1d2596\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-774dd49f6f-7dldz"
Sep 16 20:50:31.007720 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:31.007667    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzk9n\" (UniqueName: \"kubernetes.io/projected/f00fe327-50fe-4823-af39-58504b1d2596-kube-api-access-fzk9n\") pod \"tls-verification-test-kserve-router-scheduler-774dd49f6f-7dldz\" (UID: \"f00fe327-50fe-4823-af39-58504b1d2596\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-774dd49f6f-7dldz"
Sep 16 20:50:31.178234 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:31.178148    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-774dd49f6f-7dldz"
Sep 16 20:50:31.234254 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:31.234224    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls"]
Sep 16 20:50:31.236330 ip-10-0-139-81 kubenswrapper[2557]: W0916 20:50:31.236291    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1a92db9_754a_445f_919b_c15b8845346f.slice/crio-e5558f6d9acad3fb3ba7d8403f7d99c54eabfab76047c1b97934b5761af5520f WatchSource:0}: Error finding container e5558f6d9acad3fb3ba7d8403f7d99c54eabfab76047c1b97934b5761af5520f: Status 404 returned error can't find the container with id e5558f6d9acad3fb3ba7d8403f7d99c54eabfab76047c1b97934b5761af5520f
Sep 16 20:50:31.450507 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:31.450478    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-774dd49f6f-7dldz"]
Sep 16 20:50:31.453039 ip-10-0-139-81 kubenswrapper[2557]: W0916 20:50:31.453011    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf00fe327_50fe_4823_af39_58504b1d2596.slice/crio-2d40c1fa8b6bc1a94eed877f57377ec332d61db765627e873dea36497271f15f WatchSource:0}: Error finding container 2d40c1fa8b6bc1a94eed877f57377ec332d61db765627e873dea36497271f15f: Status 404 returned error can't find the container with id 2d40c1fa8b6bc1a94eed877f57377ec332d61db765627e873dea36497271f15f
Sep 16 20:50:32.205324 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:32.205288    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-774dd49f6f-7dldz" event={"ID":"f00fe327-50fe-4823-af39-58504b1d2596","Type":"ContainerStarted","Data":"feb9d357be03c278178a57eff61575e280dd5b5cc0e75b2290adf0d34d1f93dd"}
Sep 16 20:50:32.205743 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:32.205342    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-774dd49f6f-7dldz" event={"ID":"f00fe327-50fe-4823-af39-58504b1d2596","Type":"ContainerStarted","Data":"2d40c1fa8b6bc1a94eed877f57377ec332d61db765627e873dea36497271f15f"}
Sep 16 20:50:32.205743 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:32.205387    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-774dd49f6f-7dldz"
Sep 16 20:50:32.206610 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:32.206581    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls" event={"ID":"c1a92db9-754a-445f-919b-c15b8845346f","Type":"ContainerStarted","Data":"e5558f6d9acad3fb3ba7d8403f7d99c54eabfab76047c1b97934b5761af5520f"}
Sep 16 20:50:32.227712 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:32.227668    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-774dd49f6f-7dldz" podStartSLOduration=2.227653206 podStartE2EDuration="2.227653206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:50:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:50:32.22603909 +0000 UTC m=+4260.751427477" watchObservedRunningTime="2026-09-16 20:50:32.227653206 +0000 UTC m=+4260.753041591"
Sep 16 20:50:33.212942 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:33.212903    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls" event={"ID":"c1a92db9-754a-445f-919b-c15b8845346f","Type":"ContainerStarted","Data":"65f39a11ad725069be9cc5eac44509e504b2a37d1120aa301e60c9ef8e4c8cba"}
Sep 16 20:50:33.213380 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:33.213073    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls"
Sep 16 20:50:37.285372 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:37.285345    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"seaweedfs-s3-creds\""
Sep 16 20:50:37.515725 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:37.515688    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-85f8df497lmwf5" podUID="5f5318a4-95fc-4831-8c01-80950a7c72c0" containerName="main" containerID="cri-o://8dcb8a79a92bb59b2b68b2e9ad76d9c8a8fc49ba273d8c0cb50e0e6476233677" gracePeriod=45
Sep 16 20:50:38.245468 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:38.245424    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls" event={"ID":"c1a92db9-754a-445f-919b-c15b8845346f","Type":"ContainerStarted","Data":"9271cdd6f112201ea2ac31a738f82d6565b340295192dac942186bc6c5433037"}
Sep 16 20:50:40.259346 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:40.259311    2557 generic.go:356] "Generic (PLEG): container finished" podID="c1a92db9-754a-445f-919b-c15b8845346f" containerID="9271cdd6f112201ea2ac31a738f82d6565b340295192dac942186bc6c5433037" exitCode=0
Sep 16 20:50:40.259735 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:40.259381    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls" event={"ID":"c1a92db9-754a-445f-919b-c15b8845346f","Type":"ContainerDied","Data":"9271cdd6f112201ea2ac31a738f82d6565b340295192dac942186bc6c5433037"}
Sep 16 20:50:41.268358 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:41.268316    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls" event={"ID":"c1a92db9-754a-445f-919b-c15b8845346f","Type":"ContainerStarted","Data":"648a630cf01be7233350386e6cbca199d50358138a9ac6480d7b61eac618702a"}
Sep 16 20:50:41.295660 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:41.295599    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls" podStartSLOduration=3.662398257 podStartE2EDuration="11.295580424s" totalImagesPullingTime="5.470504157s" totalInitContainerRuntime="2.16267801s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:50:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:50:41.294617699 +0000 UTC m=+4269.820006098" watchObservedRunningTime="2026-09-16 20:50:41.295580424 +0000 UTC m=+4269.820968809"
Sep 16 20:50:50.954719 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:50.954680    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls"
Sep 16 20:50:50.955207 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:50.954759    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls"
Sep 16 20:50:50.955956 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:50.955924    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls" podUID="c1a92db9-754a-445f-919b-c15b8845346f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8001/health\": dial tcp 10.134.0.72:8001: connect: connection refused"
Sep 16 20:50:50.968483 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:50.968461    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls"
Sep 16 20:50:52.500427 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:50:52.500385    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-85f8df497lmwf5" podUID="5f5318a4-95fc-4831-8c01-80950a7c72c0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:51:00.955045 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:51:00.954989    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls" podUID="c1a92db9-754a-445f-919b-c15b8845346f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8001/health\": dial tcp 10.134.0.72:8001: connect: connection refused"
Sep 16 20:51:02.498723 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:51:02.498681    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-85f8df497lmwf5" podUID="5f5318a4-95fc-4831-8c01-80950a7c72c0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:51:03.215463 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:51:03.215429    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-774dd49f6f-7dldz"
Sep 16 20:51:07.762949 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:51:07.762921    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-85f8df497lmwf5"
Sep 16 20:51:07.849674 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:51:07.849586    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btws8\" (UniqueName: \"kubernetes.io/projected/5f5318a4-95fc-4831-8c01-80950a7c72c0-kube-api-access-btws8\") pod \"5f5318a4-95fc-4831-8c01-80950a7c72c0\" (UID: \"5f5318a4-95fc-4831-8c01-80950a7c72c0\") "
Sep 16 20:51:07.849674 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:51:07.849627    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f5318a4-95fc-4831-8c01-80950a7c72c0-tls-certs\") pod \"5f5318a4-95fc-4831-8c01-80950a7c72c0\" (UID: \"5f5318a4-95fc-4831-8c01-80950a7c72c0\") "
Sep 16 20:51:07.852005 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:51:07.851975    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f5318a4-95fc-4831-8c01-80950a7c72c0-kube-api-access-btws8" (OuterVolumeSpecName: "kube-api-access-btws8") pod "5f5318a4-95fc-4831-8c01-80950a7c72c0" (UID: "5f5318a4-95fc-4831-8c01-80950a7c72c0"). InnerVolumeSpecName "kube-api-access-btws8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:51:07.853339 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:51:07.853317    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f5318a4-95fc-4831-8c01-80950a7c72c0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5f5318a4-95fc-4831-8c01-80950a7c72c0" (UID: "5f5318a4-95fc-4831-8c01-80950a7c72c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:51:07.950830 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:51:07.950800    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btws8\" (UniqueName: \"kubernetes.io/projected/5f5318a4-95fc-4831-8c01-80950a7c72c0-kube-api-access-btws8\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:51:07.950830 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:51:07.950828    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f5318a4-95fc-4831-8c01-80950a7c72c0-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:51:08.413135 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:51:08.413100    2557 generic.go:356] "Generic (PLEG): container finished" podID="5f5318a4-95fc-4831-8c01-80950a7c72c0" containerID="8dcb8a79a92bb59b2b68b2e9ad76d9c8a8fc49ba273d8c0cb50e0e6476233677" exitCode=0
Sep 16 20:51:08.413317 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:51:08.413150    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-85f8df497lmwf5"
Sep 16 20:51:08.413317 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:51:08.413176    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-85f8df497lmwf5" event={"ID":"5f5318a4-95fc-4831-8c01-80950a7c72c0","Type":"ContainerDied","Data":"8dcb8a79a92bb59b2b68b2e9ad76d9c8a8fc49ba273d8c0cb50e0e6476233677"}
Sep 16 20:51:08.413317 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:51:08.413249    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-85f8df497lmwf5" event={"ID":"5f5318a4-95fc-4831-8c01-80950a7c72c0","Type":"ContainerDied","Data":"9b401cae5e349d6a5198e1196501ea6777ce1d6dacec535a4eb2c4de51ffc0da"}
Sep 16 20:51:08.413317 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:51:08.413277    2557 scope.go:117] "RemoveContainer" containerID="8dcb8a79a92bb59b2b68b2e9ad76d9c8a8fc49ba273d8c0cb50e0e6476233677"
Sep 16 20:51:08.423034 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:51:08.422939    2557 scope.go:117] "RemoveContainer" containerID="8dcb8a79a92bb59b2b68b2e9ad76d9c8a8fc49ba273d8c0cb50e0e6476233677"
Sep 16 20:51:08.423324 ip-10-0-139-81 kubenswrapper[2557]: E0916 20:51:08.423294    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dcb8a79a92bb59b2b68b2e9ad76d9c8a8fc49ba273d8c0cb50e0e6476233677\": container with ID starting with 8dcb8a79a92bb59b2b68b2e9ad76d9c8a8fc49ba273d8c0cb50e0e6476233677 not found: ID does not exist" containerID="8dcb8a79a92bb59b2b68b2e9ad76d9c8a8fc49ba273d8c0cb50e0e6476233677"
Sep 16 20:51:08.423451 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:51:08.423326    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dcb8a79a92bb59b2b68b2e9ad76d9c8a8fc49ba273d8c0cb50e0e6476233677"} err="failed to get container status \"8dcb8a79a92bb59b2b68b2e9ad76d9c8a8fc49ba273d8c0cb50e0e6476233677\": rpc error: code = NotFound desc = could not find container \"8dcb8a79a92bb59b2b68b2e9ad76d9c8a8fc49ba273d8c0cb50e0e6476233677\": container with ID starting with 8dcb8a79a92bb59b2b68b2e9ad76d9c8a8fc49ba273d8c0cb50e0e6476233677 not found: ID does not exist"
Sep 16 20:51:08.433661 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:51:08.433624    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-85f8df497lmwf5"]
Sep 16 20:51:08.436334 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:51:08.436309    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-85f8df497lmwf5"]
Sep 16 20:51:09.938207 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:51:09.938122    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f5318a4-95fc-4831-8c01-80950a7c72c0" path="/var/lib/kubelet/pods/5f5318a4-95fc-4831-8c01-80950a7c72c0/volumes"
Sep 16 20:51:10.955057 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:51:10.954998    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls" podUID="c1a92db9-754a-445f-919b-c15b8845346f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8001/health\": dial tcp 10.134.0.72:8001: connect: connection refused"
Sep 16 20:51:20.955149 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:51:20.955094    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls" podUID="c1a92db9-754a-445f-919b-c15b8845346f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8001/health\": dial tcp 10.134.0.72:8001: connect: connection refused"
Sep 16 20:51:30.955338 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:51:30.955220    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls" podUID="c1a92db9-754a-445f-919b-c15b8845346f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8001/health\": dial tcp 10.134.0.72:8001: connect: connection refused"
Sep 16 20:51:40.955532 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:51:40.955484    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls" podUID="c1a92db9-754a-445f-919b-c15b8845346f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8001/health\": dial tcp 10.134.0.72:8001: connect: connection refused"
Sep 16 20:51:50.955360 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:51:50.955310    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls" podUID="c1a92db9-754a-445f-919b-c15b8845346f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8001/health\": dial tcp 10.134.0.72:8001: connect: connection refused"
Sep 16 20:52:00.954815 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:52:00.954760    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls" podUID="c1a92db9-754a-445f-919b-c15b8845346f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8001/health\": dial tcp 10.134.0.72:8001: connect: connection refused"
Sep 16 20:52:10.954808 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:52:10.954757    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls" podUID="c1a92db9-754a-445f-919b-c15b8845346f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8001/health\": dial tcp 10.134.0.72:8001: connect: connection refused"
Sep 16 20:52:20.955488 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:52:20.955432    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls" podUID="c1a92db9-754a-445f-919b-c15b8845346f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8001/health\": dial tcp 10.134.0.72:8001: connect: connection refused"
Sep 16 20:52:30.954689 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:52:30.954647    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls" podUID="c1a92db9-754a-445f-919b-c15b8845346f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8001/health\": dial tcp 10.134.0.72:8001: connect: connection refused"
Sep 16 20:52:40.964266 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:52:40.964233    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls"
Sep 16 20:52:40.975744 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:52:40.975716    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls"
Sep 16 20:53:50.884759 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:53:50.884714    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-774dd49f6f-7dldz"]
Sep 16 20:53:50.895044 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:53:50.894997    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls"]
Sep 16 20:53:53.215019 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:53:53.214981    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-774dd49f6f-7dldz" podUID="f00fe327-50fe-4823-af39-58504b1d2596" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:54:03.214663 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:03.214624    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-774dd49f6f-7dldz" podUID="f00fe327-50fe-4823-af39-58504b1d2596" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:54:05.912970 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:05.912894    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-774dd49f6f-7dldz" podUID="f00fe327-50fe-4823-af39-58504b1d2596" containerName="main" containerID="cri-o://feb9d357be03c278178a57eff61575e280dd5b5cc0e75b2290adf0d34d1f93dd" gracePeriod=45
Sep 16 20:54:05.922894 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:05.922854    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls" podUID="c1a92db9-754a-445f-919b-c15b8845346f" containerName="main" containerID="cri-o://648a630cf01be7233350386e6cbca199d50358138a9ac6480d7b61eac618702a" gracePeriod=45
Sep 16 20:54:06.467151 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:06.467119    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls" podUID="c1a92db9-754a-445f-919b-c15b8845346f" containerName="llm-d-routing-sidecar" containerID="cri-o://65f39a11ad725069be9cc5eac44509e504b2a37d1120aa301e60c9ef8e4c8cba" gracePeriod=45
Sep 16 20:54:06.726754 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:06.726729    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls"
Sep 16 20:54:06.813989 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:06.813951    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1a92db9-754a-445f-919b-c15b8845346f-kserve-provision-location\") pod \"c1a92db9-754a-445f-919b-c15b8845346f\" (UID: \"c1a92db9-754a-445f-919b-c15b8845346f\") "
Sep 16 20:54:06.814178 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:06.814028    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-949w5\" (UniqueName: \"kubernetes.io/projected/c1a92db9-754a-445f-919b-c15b8845346f-kube-api-access-949w5\") pod \"c1a92db9-754a-445f-919b-c15b8845346f\" (UID: \"c1a92db9-754a-445f-919b-c15b8845346f\") "
Sep 16 20:54:06.814178 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:06.814062    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1a92db9-754a-445f-919b-c15b8845346f-tmp-dir\") pod \"c1a92db9-754a-445f-919b-c15b8845346f\" (UID: \"c1a92db9-754a-445f-919b-c15b8845346f\") "
Sep 16 20:54:06.814178 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:06.814098    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1a92db9-754a-445f-919b-c15b8845346f-model-cache\") pod \"c1a92db9-754a-445f-919b-c15b8845346f\" (UID: \"c1a92db9-754a-445f-919b-c15b8845346f\") "
Sep 16 20:54:06.814178 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:06.814116    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1a92db9-754a-445f-919b-c15b8845346f-dshm\") pod \"c1a92db9-754a-445f-919b-c15b8845346f\" (UID: \"c1a92db9-754a-445f-919b-c15b8845346f\") "
Sep 16 20:54:06.814178 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:06.814144    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1a92db9-754a-445f-919b-c15b8845346f-tls-certs\") pod \"c1a92db9-754a-445f-919b-c15b8845346f\" (UID: \"c1a92db9-754a-445f-919b-c15b8845346f\") "
Sep 16 20:54:06.814178 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:06.814162    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1a92db9-754a-445f-919b-c15b8845346f-home\") pod \"c1a92db9-754a-445f-919b-c15b8845346f\" (UID: \"c1a92db9-754a-445f-919b-c15b8845346f\") "
Sep 16 20:54:06.814178 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:06.814177    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c1a92db9-754a-445f-919b-c15b8845346f-cabundle-cert\") pod \"c1a92db9-754a-445f-919b-c15b8845346f\" (UID: \"c1a92db9-754a-445f-919b-c15b8845346f\") "
Sep 16 20:54:06.814563 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:06.814306    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1a92db9-754a-445f-919b-c15b8845346f-model-cache" (OuterVolumeSpecName: "model-cache") pod "c1a92db9-754a-445f-919b-c15b8845346f" (UID: "c1a92db9-754a-445f-919b-c15b8845346f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:54:06.814563 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:06.814420    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1a92db9-754a-445f-919b-c15b8845346f-model-cache\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:54:06.814718 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:06.814691    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1a92db9-754a-445f-919b-c15b8845346f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c1a92db9-754a-445f-919b-c15b8845346f" (UID: "c1a92db9-754a-445f-919b-c15b8845346f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:54:06.814894 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:06.814829    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1a92db9-754a-445f-919b-c15b8845346f-home" (OuterVolumeSpecName: "home") pod "c1a92db9-754a-445f-919b-c15b8845346f" (UID: "c1a92db9-754a-445f-919b-c15b8845346f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:54:06.816506 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:06.816479    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1a92db9-754a-445f-919b-c15b8845346f-kube-api-access-949w5" (OuterVolumeSpecName: "kube-api-access-949w5") pod "c1a92db9-754a-445f-919b-c15b8845346f" (UID: "c1a92db9-754a-445f-919b-c15b8845346f"). InnerVolumeSpecName "kube-api-access-949w5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:54:06.816717 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:06.816689    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1a92db9-754a-445f-919b-c15b8845346f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c1a92db9-754a-445f-919b-c15b8845346f" (UID: "c1a92db9-754a-445f-919b-c15b8845346f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:54:06.816865 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:06.816824    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1a92db9-754a-445f-919b-c15b8845346f-dshm" (OuterVolumeSpecName: "dshm") pod "c1a92db9-754a-445f-919b-c15b8845346f" (UID: "c1a92db9-754a-445f-919b-c15b8845346f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:54:06.826267 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:06.826244    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1a92db9-754a-445f-919b-c15b8845346f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c1a92db9-754a-445f-919b-c15b8845346f" (UID: "c1a92db9-754a-445f-919b-c15b8845346f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:54:06.835567 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:06.835539    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1a92db9-754a-445f-919b-c15b8845346f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c1a92db9-754a-445f-919b-c15b8845346f" (UID: "c1a92db9-754a-445f-919b-c15b8845346f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:54:06.915426 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:06.915384    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1a92db9-754a-445f-919b-c15b8845346f-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:54:06.915426 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:06.915419    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1a92db9-754a-445f-919b-c15b8845346f-home\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:54:06.915426 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:06.915429    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c1a92db9-754a-445f-919b-c15b8845346f-cabundle-cert\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:54:06.915949 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:06.915438    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1a92db9-754a-445f-919b-c15b8845346f-kserve-provision-location\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:54:06.915949 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:06.915450    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-949w5\" (UniqueName: \"kubernetes.io/projected/c1a92db9-754a-445f-919b-c15b8845346f-kube-api-access-949w5\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:54:06.915949 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:06.915458    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1a92db9-754a-445f-919b-c15b8845346f-tmp-dir\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:54:06.915949 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:06.915468    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1a92db9-754a-445f-919b-c15b8845346f-dshm\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:54:07.373480 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:07.373445    2557 generic.go:356] "Generic (PLEG): container finished" podID="c1a92db9-754a-445f-919b-c15b8845346f" containerID="648a630cf01be7233350386e6cbca199d50358138a9ac6480d7b61eac618702a" exitCode=0
Sep 16 20:54:07.373480 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:07.373475    2557 generic.go:356] "Generic (PLEG): container finished" podID="c1a92db9-754a-445f-919b-c15b8845346f" containerID="65f39a11ad725069be9cc5eac44509e504b2a37d1120aa301e60c9ef8e4c8cba" exitCode=0
Sep 16 20:54:07.373727 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:07.373510    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls"
Sep 16 20:54:07.373727 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:07.373530    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls" event={"ID":"c1a92db9-754a-445f-919b-c15b8845346f","Type":"ContainerDied","Data":"648a630cf01be7233350386e6cbca199d50358138a9ac6480d7b61eac618702a"}
Sep 16 20:54:07.373727 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:07.373599    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls" event={"ID":"c1a92db9-754a-445f-919b-c15b8845346f","Type":"ContainerDied","Data":"65f39a11ad725069be9cc5eac44509e504b2a37d1120aa301e60c9ef8e4c8cba"}
Sep 16 20:54:07.373727 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:07.373626    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls" event={"ID":"c1a92db9-754a-445f-919b-c15b8845346f","Type":"ContainerDied","Data":"e5558f6d9acad3fb3ba7d8403f7d99c54eabfab76047c1b97934b5761af5520f"}
Sep 16 20:54:07.373727 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:07.373656    2557 scope.go:117] "RemoveContainer" containerID="648a630cf01be7233350386e6cbca199d50358138a9ac6480d7b61eac618702a"
Sep 16 20:54:07.384076 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:07.384050    2557 scope.go:117] "RemoveContainer" containerID="9271cdd6f112201ea2ac31a738f82d6565b340295192dac942186bc6c5433037"
Sep 16 20:54:07.391879 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:07.391859    2557 scope.go:117] "RemoveContainer" containerID="65f39a11ad725069be9cc5eac44509e504b2a37d1120aa301e60c9ef8e4c8cba"
Sep 16 20:54:07.399044 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:07.399020    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls"]
Sep 16 20:54:07.400213 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:07.400197    2557 scope.go:117] "RemoveContainer" containerID="648a630cf01be7233350386e6cbca199d50358138a9ac6480d7b61eac618702a"
Sep 16 20:54:07.400495 ip-10-0-139-81 kubenswrapper[2557]: E0916 20:54:07.400474    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"648a630cf01be7233350386e6cbca199d50358138a9ac6480d7b61eac618702a\": container with ID starting with 648a630cf01be7233350386e6cbca199d50358138a9ac6480d7b61eac618702a not found: ID does not exist" containerID="648a630cf01be7233350386e6cbca199d50358138a9ac6480d7b61eac618702a"
Sep 16 20:54:07.400543 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:07.400507    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"648a630cf01be7233350386e6cbca199d50358138a9ac6480d7b61eac618702a"} err="failed to get container status \"648a630cf01be7233350386e6cbca199d50358138a9ac6480d7b61eac618702a\": rpc error: code = NotFound desc = could not find container \"648a630cf01be7233350386e6cbca199d50358138a9ac6480d7b61eac618702a\": container with ID starting with 648a630cf01be7233350386e6cbca199d50358138a9ac6480d7b61eac618702a not found: ID does not exist"
Sep 16 20:54:07.400543 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:07.400525    2557 scope.go:117] "RemoveContainer" containerID="9271cdd6f112201ea2ac31a738f82d6565b340295192dac942186bc6c5433037"
Sep 16 20:54:07.400766 ip-10-0-139-81 kubenswrapper[2557]: E0916 20:54:07.400752    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9271cdd6f112201ea2ac31a738f82d6565b340295192dac942186bc6c5433037\": container with ID starting with 9271cdd6f112201ea2ac31a738f82d6565b340295192dac942186bc6c5433037 not found: ID does not exist" containerID="9271cdd6f112201ea2ac31a738f82d6565b340295192dac942186bc6c5433037"
Sep 16 20:54:07.400815 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:07.400769    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9271cdd6f112201ea2ac31a738f82d6565b340295192dac942186bc6c5433037"} err="failed to get container status \"9271cdd6f112201ea2ac31a738f82d6565b340295192dac942186bc6c5433037\": rpc error: code = NotFound desc = could not find container \"9271cdd6f112201ea2ac31a738f82d6565b340295192dac942186bc6c5433037\": container with ID starting with 9271cdd6f112201ea2ac31a738f82d6565b340295192dac942186bc6c5433037 not found: ID does not exist"
Sep 16 20:54:07.400815 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:07.400784    2557 scope.go:117] "RemoveContainer" containerID="65f39a11ad725069be9cc5eac44509e504b2a37d1120aa301e60c9ef8e4c8cba"
Sep 16 20:54:07.401046 ip-10-0-139-81 kubenswrapper[2557]: E0916 20:54:07.401031    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65f39a11ad725069be9cc5eac44509e504b2a37d1120aa301e60c9ef8e4c8cba\": container with ID starting with 65f39a11ad725069be9cc5eac44509e504b2a37d1120aa301e60c9ef8e4c8cba not found: ID does not exist" containerID="65f39a11ad725069be9cc5eac44509e504b2a37d1120aa301e60c9ef8e4c8cba"
Sep 16 20:54:07.401096 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:07.401051    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65f39a11ad725069be9cc5eac44509e504b2a37d1120aa301e60c9ef8e4c8cba"} err="failed to get container status \"65f39a11ad725069be9cc5eac44509e504b2a37d1120aa301e60c9ef8e4c8cba\": rpc error: code = NotFound desc = could not find container \"65f39a11ad725069be9cc5eac44509e504b2a37d1120aa301e60c9ef8e4c8cba\": container with ID starting with 65f39a11ad725069be9cc5eac44509e504b2a37d1120aa301e60c9ef8e4c8cba not found: ID does not exist"
Sep 16 20:54:07.401096 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:07.401064    2557 scope.go:117] "RemoveContainer" containerID="648a630cf01be7233350386e6cbca199d50358138a9ac6480d7b61eac618702a"
Sep 16 20:54:07.401311 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:07.401289    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"648a630cf01be7233350386e6cbca199d50358138a9ac6480d7b61eac618702a"} err="failed to get container status \"648a630cf01be7233350386e6cbca199d50358138a9ac6480d7b61eac618702a\": rpc error: code = NotFound desc = could not find container \"648a630cf01be7233350386e6cbca199d50358138a9ac6480d7b61eac618702a\": container with ID starting with 648a630cf01be7233350386e6cbca199d50358138a9ac6480d7b61eac618702a not found: ID does not exist"
Sep 16 20:54:07.401386 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:07.401313    2557 scope.go:117] "RemoveContainer" containerID="9271cdd6f112201ea2ac31a738f82d6565b340295192dac942186bc6c5433037"
Sep 16 20:54:07.401520 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:07.401503    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9271cdd6f112201ea2ac31a738f82d6565b340295192dac942186bc6c5433037"} err="failed to get container status \"9271cdd6f112201ea2ac31a738f82d6565b340295192dac942186bc6c5433037\": rpc error: code = NotFound desc = could not find container \"9271cdd6f112201ea2ac31a738f82d6565b340295192dac942186bc6c5433037\": container with ID starting with 9271cdd6f112201ea2ac31a738f82d6565b340295192dac942186bc6c5433037 not found: ID does not exist"
Sep 16 20:54:07.401562 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:07.401520    2557 scope.go:117] "RemoveContainer" containerID="65f39a11ad725069be9cc5eac44509e504b2a37d1120aa301e60c9ef8e4c8cba"
Sep 16 20:54:07.401757 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:07.401738    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65f39a11ad725069be9cc5eac44509e504b2a37d1120aa301e60c9ef8e4c8cba"} err="failed to get container status \"65f39a11ad725069be9cc5eac44509e504b2a37d1120aa301e60c9ef8e4c8cba\": rpc error: code = NotFound desc = could not find container \"65f39a11ad725069be9cc5eac44509e504b2a37d1120aa301e60c9ef8e4c8cba\": container with ID starting with 65f39a11ad725069be9cc5eac44509e504b2a37d1120aa301e60c9ef8e4c8cba not found: ID does not exist"
Sep 16 20:54:07.405423 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:07.405401    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-586f66667c-pzdls"]
Sep 16 20:54:07.936897 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:07.936860    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1a92db9-754a-445f-919b-c15b8845346f" path="/var/lib/kubelet/pods/c1a92db9-754a-445f-919b-c15b8845346f/volumes"
Sep 16 20:54:13.214805 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:13.214760    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-774dd49f6f-7dldz" podUID="f00fe327-50fe-4823-af39-58504b1d2596" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:54:13.215324 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:13.214937    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-774dd49f6f-7dldz"
Sep 16 20:54:23.214855 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:23.214802    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-774dd49f6f-7dldz" podUID="f00fe327-50fe-4823-af39-58504b1d2596" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:54:33.214619 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:33.214529    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-774dd49f6f-7dldz" podUID="f00fe327-50fe-4823-af39-58504b1d2596" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:54:36.177508 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:36.177447    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-774dd49f6f-7dldz"
Sep 16 20:54:36.281575 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:36.281536    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fzk9n\" (UniqueName: \"kubernetes.io/projected/f00fe327-50fe-4823-af39-58504b1d2596-kube-api-access-fzk9n\") pod \"f00fe327-50fe-4823-af39-58504b1d2596\" (UID: \"f00fe327-50fe-4823-af39-58504b1d2596\") "
Sep 16 20:54:36.281771 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:36.281629    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f00fe327-50fe-4823-af39-58504b1d2596-tls-certs\") pod \"f00fe327-50fe-4823-af39-58504b1d2596\" (UID: \"f00fe327-50fe-4823-af39-58504b1d2596\") "
Sep 16 20:54:36.283791 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:36.283760    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f00fe327-50fe-4823-af39-58504b1d2596-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f00fe327-50fe-4823-af39-58504b1d2596" (UID: "f00fe327-50fe-4823-af39-58504b1d2596"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:54:36.283914 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:36.283855    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f00fe327-50fe-4823-af39-58504b1d2596-kube-api-access-fzk9n" (OuterVolumeSpecName: "kube-api-access-fzk9n") pod "f00fe327-50fe-4823-af39-58504b1d2596" (UID: "f00fe327-50fe-4823-af39-58504b1d2596"). InnerVolumeSpecName "kube-api-access-fzk9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:54:36.382537 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:36.382493    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f00fe327-50fe-4823-af39-58504b1d2596-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:54:36.382537 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:36.382532    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzk9n\" (UniqueName: \"kubernetes.io/projected/f00fe327-50fe-4823-af39-58504b1d2596-kube-api-access-fzk9n\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:54:36.516831 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:36.516799    2557 generic.go:356] "Generic (PLEG): container finished" podID="f00fe327-50fe-4823-af39-58504b1d2596" containerID="feb9d357be03c278178a57eff61575e280dd5b5cc0e75b2290adf0d34d1f93dd" exitCode=0
Sep 16 20:54:36.517008 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:36.516859    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-774dd49f6f-7dldz"
Sep 16 20:54:36.517008 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:36.516890    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-774dd49f6f-7dldz" event={"ID":"f00fe327-50fe-4823-af39-58504b1d2596","Type":"ContainerDied","Data":"feb9d357be03c278178a57eff61575e280dd5b5cc0e75b2290adf0d34d1f93dd"}
Sep 16 20:54:36.517008 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:36.516966    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-774dd49f6f-7dldz" event={"ID":"f00fe327-50fe-4823-af39-58504b1d2596","Type":"ContainerDied","Data":"2d40c1fa8b6bc1a94eed877f57377ec332d61db765627e873dea36497271f15f"}
Sep 16 20:54:36.517008 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:36.516997    2557 scope.go:117] "RemoveContainer" containerID="feb9d357be03c278178a57eff61575e280dd5b5cc0e75b2290adf0d34d1f93dd"
Sep 16 20:54:36.526769 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:36.526744    2557 scope.go:117] "RemoveContainer" containerID="feb9d357be03c278178a57eff61575e280dd5b5cc0e75b2290adf0d34d1f93dd"
Sep 16 20:54:36.527066 ip-10-0-139-81 kubenswrapper[2557]: E0916 20:54:36.527046    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"feb9d357be03c278178a57eff61575e280dd5b5cc0e75b2290adf0d34d1f93dd\": container with ID starting with feb9d357be03c278178a57eff61575e280dd5b5cc0e75b2290adf0d34d1f93dd not found: ID does not exist" containerID="feb9d357be03c278178a57eff61575e280dd5b5cc0e75b2290adf0d34d1f93dd"
Sep 16 20:54:36.527135 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:36.527074    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"feb9d357be03c278178a57eff61575e280dd5b5cc0e75b2290adf0d34d1f93dd"} err="failed to get container status \"feb9d357be03c278178a57eff61575e280dd5b5cc0e75b2290adf0d34d1f93dd\": rpc error: code = NotFound desc = could not find container \"feb9d357be03c278178a57eff61575e280dd5b5cc0e75b2290adf0d34d1f93dd\": container with ID starting with feb9d357be03c278178a57eff61575e280dd5b5cc0e75b2290adf0d34d1f93dd not found: ID does not exist"
Sep 16 20:54:36.539407 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:36.539384    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-774dd49f6f-7dldz"]
Sep 16 20:54:36.543271 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:36.543248    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-774dd49f6f-7dldz"]
Sep 16 20:54:37.937895 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:54:37.937827    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f00fe327-50fe-4823-af39-58504b1d2596" path="/var/lib/kubelet/pods/f00fe327-50fe-4823-af39-58504b1d2596/volumes"
Sep 16 20:55:26.596673 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:26.596632    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h"]
Sep 16 20:55:26.597256 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:26.597234    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1a92db9-754a-445f-919b-c15b8845346f" containerName="main"
Sep 16 20:55:26.597340 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:26.597260    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1a92db9-754a-445f-919b-c15b8845346f" containerName="main"
Sep 16 20:55:26.597340 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:26.597286    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f5318a4-95fc-4831-8c01-80950a7c72c0" containerName="main"
Sep 16 20:55:26.597340 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:26.597296    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f5318a4-95fc-4831-8c01-80950a7c72c0" containerName="main"
Sep 16 20:55:26.597340 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:26.597314    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1a92db9-754a-445f-919b-c15b8845346f" containerName="storage-initializer"
Sep 16 20:55:26.597340 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:26.597324    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1a92db9-754a-445f-919b-c15b8845346f" containerName="storage-initializer"
Sep 16 20:55:26.597592 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:26.597349    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f00fe327-50fe-4823-af39-58504b1d2596" containerName="main"
Sep 16 20:55:26.597592 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:26.597359    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f00fe327-50fe-4823-af39-58504b1d2596" containerName="main"
Sep 16 20:55:26.597592 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:26.597373    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1a92db9-754a-445f-919b-c15b8845346f" containerName="llm-d-routing-sidecar"
Sep 16 20:55:26.597592 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:26.597382    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1a92db9-754a-445f-919b-c15b8845346f" containerName="llm-d-routing-sidecar"
Sep 16 20:55:26.597592 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:26.597484    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1a92db9-754a-445f-919b-c15b8845346f" containerName="main"
Sep 16 20:55:26.597592 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:26.597495    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="f00fe327-50fe-4823-af39-58504b1d2596" containerName="main"
Sep 16 20:55:26.597592 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:26.597510    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f5318a4-95fc-4831-8c01-80950a7c72c0" containerName="main"
Sep 16 20:55:26.597592 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:26.597521    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1a92db9-754a-445f-919b-c15b8845346f" containerName="llm-d-routing-sidecar"
Sep 16 20:55:26.600812 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:26.600789    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h"
Sep 16 20:55:26.603901 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:26.603871    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"seaweedfs-s3-creds\""
Sep 16 20:55:26.604030 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:26.603871    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 20:55:26.604030 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:26.603915    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"kube-root-ca.crt\""
Sep 16 20:55:26.604030 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:26.603957    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"openshift-service-ca.crt\""
Sep 16 20:55:26.604171 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:26.604098    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"default-dockercfg-qz9dr\""
Sep 16 20:55:26.604219 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:26.604177    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Sep 16 20:55:26.616452 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:26.616425    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h"]
Sep 16 20:55:26.711024 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:26.710980    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf63809e-33cd-4cca-9a98-b8272241e2a7-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h\" (UID: \"cf63809e-33cd-4cca-9a98-b8272241e2a7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h"
Sep 16 20:55:26.711024 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:26.711021    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf63809e-33cd-4cca-9a98-b8272241e2a7-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h\" (UID: \"cf63809e-33cd-4cca-9a98-b8272241e2a7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h"
Sep 16 20:55:26.711277 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:26.711055    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf63809e-33cd-4cca-9a98-b8272241e2a7-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h\" (UID: \"cf63809e-33cd-4cca-9a98-b8272241e2a7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h"
Sep 16 20:55:26.711277 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:26.711098    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf63809e-33cd-4cca-9a98-b8272241e2a7-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h\" (UID: \"cf63809e-33cd-4cca-9a98-b8272241e2a7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h"
Sep 16 20:55:26.711277 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:26.711146    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf63809e-33cd-4cca-9a98-b8272241e2a7-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h\" (UID: \"cf63809e-33cd-4cca-9a98-b8272241e2a7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h"
Sep 16 20:55:26.711277 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:26.711177    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf63809e-33cd-4cca-9a98-b8272241e2a7-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h\" (UID: \"cf63809e-33cd-4cca-9a98-b8272241e2a7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h"
Sep 16 20:55:26.711277 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:26.711194    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk2c6\" (UniqueName: \"kubernetes.io/projected/cf63809e-33cd-4cca-9a98-b8272241e2a7-kube-api-access-tk2c6\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h\" (UID: \"cf63809e-33cd-4cca-9a98-b8272241e2a7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h"
Sep 16 20:55:26.711277 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:26.711266    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf63809e-33cd-4cca-9a98-b8272241e2a7-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h\" (UID: \"cf63809e-33cd-4cca-9a98-b8272241e2a7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h"
Sep 16 20:55:26.812601 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:26.812562    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf63809e-33cd-4cca-9a98-b8272241e2a7-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h\" (UID: \"cf63809e-33cd-4cca-9a98-b8272241e2a7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h"
Sep 16 20:55:26.812601 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:26.812598    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf63809e-33cd-4cca-9a98-b8272241e2a7-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h\" (UID: \"cf63809e-33cd-4cca-9a98-b8272241e2a7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h"
Sep 16 20:55:26.812883 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:26.812618    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf63809e-33cd-4cca-9a98-b8272241e2a7-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h\" (UID: \"cf63809e-33cd-4cca-9a98-b8272241e2a7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h"
Sep 16 20:55:26.812883 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:26.812643    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf63809e-33cd-4cca-9a98-b8272241e2a7-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h\" (UID: \"cf63809e-33cd-4cca-9a98-b8272241e2a7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h"
Sep 16 20:55:26.812883 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:26.812690    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf63809e-33cd-4cca-9a98-b8272241e2a7-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h\" (UID: \"cf63809e-33cd-4cca-9a98-b8272241e2a7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h"
Sep 16 20:55:26.812883 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:26.812731    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf63809e-33cd-4cca-9a98-b8272241e2a7-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h\" (UID: \"cf63809e-33cd-4cca-9a98-b8272241e2a7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h"
Sep 16 20:55:26.812883 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:26.812754    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk2c6\" (UniqueName: \"kubernetes.io/projected/cf63809e-33cd-4cca-9a98-b8272241e2a7-kube-api-access-tk2c6\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h\" (UID: \"cf63809e-33cd-4cca-9a98-b8272241e2a7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h"
Sep 16 20:55:26.812883 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:26.812820    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf63809e-33cd-4cca-9a98-b8272241e2a7-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h\" (UID: \"cf63809e-33cd-4cca-9a98-b8272241e2a7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h"
Sep 16 20:55:26.813240 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:26.813060    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf63809e-33cd-4cca-9a98-b8272241e2a7-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h\" (UID: \"cf63809e-33cd-4cca-9a98-b8272241e2a7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h"
Sep 16 20:55:26.813296 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:26.813265    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf63809e-33cd-4cca-9a98-b8272241e2a7-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h\" (UID: \"cf63809e-33cd-4cca-9a98-b8272241e2a7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h"
Sep 16 20:55:26.813352 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:26.813305    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf63809e-33cd-4cca-9a98-b8272241e2a7-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h\" (UID: \"cf63809e-33cd-4cca-9a98-b8272241e2a7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h"
Sep 16 20:55:26.813352 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:26.813325    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf63809e-33cd-4cca-9a98-b8272241e2a7-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h\" (UID: \"cf63809e-33cd-4cca-9a98-b8272241e2a7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h"
Sep 16 20:55:26.813523 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:26.813503    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf63809e-33cd-4cca-9a98-b8272241e2a7-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h\" (UID: \"cf63809e-33cd-4cca-9a98-b8272241e2a7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h"
Sep 16 20:55:26.815043 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:26.815019    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf63809e-33cd-4cca-9a98-b8272241e2a7-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h\" (UID: \"cf63809e-33cd-4cca-9a98-b8272241e2a7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h"
Sep 16 20:55:26.815234 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:26.815215    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf63809e-33cd-4cca-9a98-b8272241e2a7-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h\" (UID: \"cf63809e-33cd-4cca-9a98-b8272241e2a7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h"
Sep 16 20:55:26.821032 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:26.821012    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk2c6\" (UniqueName: \"kubernetes.io/projected/cf63809e-33cd-4cca-9a98-b8272241e2a7-kube-api-access-tk2c6\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h\" (UID: \"cf63809e-33cd-4cca-9a98-b8272241e2a7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h"
Sep 16 20:55:26.913521 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:26.913431    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h"
Sep 16 20:55:27.194209 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:27.194167    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h"]
Sep 16 20:55:27.774806 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:27.774764    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h" event={"ID":"cf63809e-33cd-4cca-9a98-b8272241e2a7","Type":"ContainerStarted","Data":"95b4011328c96d079091415e3643cbe69508e34b899bbb0c6632c1356c6a4da3"}
Sep 16 20:55:27.775167 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:27.774831    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h" event={"ID":"cf63809e-33cd-4cca-9a98-b8272241e2a7","Type":"ContainerStarted","Data":"58b9b4bee339aac331782e821672df205bb6b04b430886f6746218a7c6de01e4"}
Sep 16 20:55:29.787082 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:29.787045    2557 generic.go:356] "Generic (PLEG): container finished" podID="cf63809e-33cd-4cca-9a98-b8272241e2a7" containerID="95b4011328c96d079091415e3643cbe69508e34b899bbb0c6632c1356c6a4da3" exitCode=0
Sep 16 20:55:29.787461 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:29.787102    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h" event={"ID":"cf63809e-33cd-4cca-9a98-b8272241e2a7","Type":"ContainerDied","Data":"95b4011328c96d079091415e3643cbe69508e34b899bbb0c6632c1356c6a4da3"}
Sep 16 20:55:30.795192 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:30.795148    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h" event={"ID":"cf63809e-33cd-4cca-9a98-b8272241e2a7","Type":"ContainerStarted","Data":"f529fc72286c9352b531d71dc0fb72390534de9cbff539e4cbdd3eff1986db61"}
Sep 16 20:55:30.821241 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:30.821184    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h" podStartSLOduration=2.654937445 podStartE2EDuration="4.821168957s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.166231512s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:55:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:55:30.820223311 +0000 UTC m=+4559.345611696" watchObservedRunningTime="2026-09-16 20:55:30.821168957 +0000 UTC m=+4559.346557342"
Sep 16 20:55:30.913582 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:30.913540    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h"
Sep 16 20:55:36.914638 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:36.914593    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h"
Sep 16 20:55:36.916081 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:36.916051    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h" podUID="cf63809e-33cd-4cca-9a98-b8272241e2a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 16 20:55:46.914649 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:46.914608    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h" podUID="cf63809e-33cd-4cca-9a98-b8272241e2a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 16 20:55:56.914145 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:55:56.914054    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h" podUID="cf63809e-33cd-4cca-9a98-b8272241e2a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 16 20:56:06.914365 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:56:06.914316    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h" podUID="cf63809e-33cd-4cca-9a98-b8272241e2a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 16 20:56:16.913894 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:56:16.913825    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h" podUID="cf63809e-33cd-4cca-9a98-b8272241e2a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 16 20:56:26.914119 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:56:26.914060    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h" podUID="cf63809e-33cd-4cca-9a98-b8272241e2a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 16 20:56:36.914924 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:56:36.914876    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h" podUID="cf63809e-33cd-4cca-9a98-b8272241e2a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 16 20:56:46.914910 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:56:46.914859    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h" podUID="cf63809e-33cd-4cca-9a98-b8272241e2a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 16 20:56:56.914297 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:56:56.914238    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h" podUID="cf63809e-33cd-4cca-9a98-b8272241e2a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 16 20:57:06.914558 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:06.914495    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h" podUID="cf63809e-33cd-4cca-9a98-b8272241e2a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 16 20:57:07.204043 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:07.203943    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h"]
Sep 16 20:57:22.226597 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:22.226552    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h" podUID="cf63809e-33cd-4cca-9a98-b8272241e2a7" containerName="main" containerID="cri-o://f529fc72286c9352b531d71dc0fb72390534de9cbff539e4cbdd3eff1986db61" gracePeriod=45
Sep 16 20:57:22.909500 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:22.909473    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h"
Sep 16 20:57:22.992787 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:22.992755    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf63809e-33cd-4cca-9a98-b8272241e2a7-cabundle-cert\") pod \"cf63809e-33cd-4cca-9a98-b8272241e2a7\" (UID: \"cf63809e-33cd-4cca-9a98-b8272241e2a7\") "
Sep 16 20:57:22.992942 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:22.992806    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tk2c6\" (UniqueName: \"kubernetes.io/projected/cf63809e-33cd-4cca-9a98-b8272241e2a7-kube-api-access-tk2c6\") pod \"cf63809e-33cd-4cca-9a98-b8272241e2a7\" (UID: \"cf63809e-33cd-4cca-9a98-b8272241e2a7\") "
Sep 16 20:57:22.992942 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:22.992877    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf63809e-33cd-4cca-9a98-b8272241e2a7-tls-certs\") pod \"cf63809e-33cd-4cca-9a98-b8272241e2a7\" (UID: \"cf63809e-33cd-4cca-9a98-b8272241e2a7\") "
Sep 16 20:57:22.992942 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:22.992925    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf63809e-33cd-4cca-9a98-b8272241e2a7-tmp-dir\") pod \"cf63809e-33cd-4cca-9a98-b8272241e2a7\" (UID: \"cf63809e-33cd-4cca-9a98-b8272241e2a7\") "
Sep 16 20:57:22.993090 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:22.992964    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf63809e-33cd-4cca-9a98-b8272241e2a7-dshm\") pod \"cf63809e-33cd-4cca-9a98-b8272241e2a7\" (UID: \"cf63809e-33cd-4cca-9a98-b8272241e2a7\") "
Sep 16 20:57:22.993090 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:22.993006    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf63809e-33cd-4cca-9a98-b8272241e2a7-home\") pod \"cf63809e-33cd-4cca-9a98-b8272241e2a7\" (UID: \"cf63809e-33cd-4cca-9a98-b8272241e2a7\") "
Sep 16 20:57:22.993090 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:22.993030    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf63809e-33cd-4cca-9a98-b8272241e2a7-kserve-provision-location\") pod \"cf63809e-33cd-4cca-9a98-b8272241e2a7\" (UID: \"cf63809e-33cd-4cca-9a98-b8272241e2a7\") "
Sep 16 20:57:22.993090 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:22.993052    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf63809e-33cd-4cca-9a98-b8272241e2a7-model-cache\") pod \"cf63809e-33cd-4cca-9a98-b8272241e2a7\" (UID: \"cf63809e-33cd-4cca-9a98-b8272241e2a7\") "
Sep 16 20:57:22.993329 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:22.993297    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf63809e-33cd-4cca-9a98-b8272241e2a7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cf63809e-33cd-4cca-9a98-b8272241e2a7" (UID: "cf63809e-33cd-4cca-9a98-b8272241e2a7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:57:22.993596 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:22.993557    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf63809e-33cd-4cca-9a98-b8272241e2a7-model-cache" (OuterVolumeSpecName: "model-cache") pod "cf63809e-33cd-4cca-9a98-b8272241e2a7" (UID: "cf63809e-33cd-4cca-9a98-b8272241e2a7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:57:22.993767 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:22.993739    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf63809e-33cd-4cca-9a98-b8272241e2a7-home" (OuterVolumeSpecName: "home") pod "cf63809e-33cd-4cca-9a98-b8272241e2a7" (UID: "cf63809e-33cd-4cca-9a98-b8272241e2a7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:57:22.995354 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:22.995323    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf63809e-33cd-4cca-9a98-b8272241e2a7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cf63809e-33cd-4cca-9a98-b8272241e2a7" (UID: "cf63809e-33cd-4cca-9a98-b8272241e2a7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:57:22.995472 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:22.995431    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf63809e-33cd-4cca-9a98-b8272241e2a7-kube-api-access-tk2c6" (OuterVolumeSpecName: "kube-api-access-tk2c6") pod "cf63809e-33cd-4cca-9a98-b8272241e2a7" (UID: "cf63809e-33cd-4cca-9a98-b8272241e2a7"). InnerVolumeSpecName "kube-api-access-tk2c6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:57:22.995584 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:22.995567    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf63809e-33cd-4cca-9a98-b8272241e2a7-dshm" (OuterVolumeSpecName: "dshm") pod "cf63809e-33cd-4cca-9a98-b8272241e2a7" (UID: "cf63809e-33cd-4cca-9a98-b8272241e2a7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:57:23.005424 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:23.005400    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf63809e-33cd-4cca-9a98-b8272241e2a7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cf63809e-33cd-4cca-9a98-b8272241e2a7" (UID: "cf63809e-33cd-4cca-9a98-b8272241e2a7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:57:23.016689 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:23.016661    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf63809e-33cd-4cca-9a98-b8272241e2a7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cf63809e-33cd-4cca-9a98-b8272241e2a7" (UID: "cf63809e-33cd-4cca-9a98-b8272241e2a7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:57:23.094279 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:23.094240    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf63809e-33cd-4cca-9a98-b8272241e2a7-home\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:57:23.094279 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:23.094273    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf63809e-33cd-4cca-9a98-b8272241e2a7-kserve-provision-location\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:57:23.094279 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:23.094284    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf63809e-33cd-4cca-9a98-b8272241e2a7-model-cache\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:57:23.094494 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:23.094294    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf63809e-33cd-4cca-9a98-b8272241e2a7-cabundle-cert\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:57:23.094494 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:23.094303    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk2c6\" (UniqueName: \"kubernetes.io/projected/cf63809e-33cd-4cca-9a98-b8272241e2a7-kube-api-access-tk2c6\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:57:23.094494 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:23.094313    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf63809e-33cd-4cca-9a98-b8272241e2a7-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:57:23.094494 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:23.094339    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf63809e-33cd-4cca-9a98-b8272241e2a7-tmp-dir\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:57:23.094494 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:23.094348    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf63809e-33cd-4cca-9a98-b8272241e2a7-dshm\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:57:23.382647 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:23.382557    2557 generic.go:356] "Generic (PLEG): container finished" podID="cf63809e-33cd-4cca-9a98-b8272241e2a7" containerID="f529fc72286c9352b531d71dc0fb72390534de9cbff539e4cbdd3eff1986db61" exitCode=0
Sep 16 20:57:23.382647 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:23.382623    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h"
Sep 16 20:57:23.382647 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:23.382628    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h" event={"ID":"cf63809e-33cd-4cca-9a98-b8272241e2a7","Type":"ContainerDied","Data":"f529fc72286c9352b531d71dc0fb72390534de9cbff539e4cbdd3eff1986db61"}
Sep 16 20:57:23.383159 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:23.382679    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h" event={"ID":"cf63809e-33cd-4cca-9a98-b8272241e2a7","Type":"ContainerDied","Data":"58b9b4bee339aac331782e821672df205bb6b04b430886f6746218a7c6de01e4"}
Sep 16 20:57:23.383159 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:23.382703    2557 scope.go:117] "RemoveContainer" containerID="f529fc72286c9352b531d71dc0fb72390534de9cbff539e4cbdd3eff1986db61"
Sep 16 20:57:23.394511 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:23.394490    2557 scope.go:117] "RemoveContainer" containerID="95b4011328c96d079091415e3643cbe69508e34b899bbb0c6632c1356c6a4da3"
Sep 16 20:57:23.402336 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:23.402313    2557 scope.go:117] "RemoveContainer" containerID="f529fc72286c9352b531d71dc0fb72390534de9cbff539e4cbdd3eff1986db61"
Sep 16 20:57:23.402600 ip-10-0-139-81 kubenswrapper[2557]: E0916 20:57:23.402580    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f529fc72286c9352b531d71dc0fb72390534de9cbff539e4cbdd3eff1986db61\": container with ID starting with f529fc72286c9352b531d71dc0fb72390534de9cbff539e4cbdd3eff1986db61 not found: ID does not exist" containerID="f529fc72286c9352b531d71dc0fb72390534de9cbff539e4cbdd3eff1986db61"
Sep 16 20:57:23.402653 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:23.402611    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f529fc72286c9352b531d71dc0fb72390534de9cbff539e4cbdd3eff1986db61"} err="failed to get container status \"f529fc72286c9352b531d71dc0fb72390534de9cbff539e4cbdd3eff1986db61\": rpc error: code = NotFound desc = could not find container \"f529fc72286c9352b531d71dc0fb72390534de9cbff539e4cbdd3eff1986db61\": container with ID starting with f529fc72286c9352b531d71dc0fb72390534de9cbff539e4cbdd3eff1986db61 not found: ID does not exist"
Sep 16 20:57:23.402653 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:23.402628    2557 scope.go:117] "RemoveContainer" containerID="95b4011328c96d079091415e3643cbe69508e34b899bbb0c6632c1356c6a4da3"
Sep 16 20:57:23.402908 ip-10-0-139-81 kubenswrapper[2557]: E0916 20:57:23.402890    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95b4011328c96d079091415e3643cbe69508e34b899bbb0c6632c1356c6a4da3\": container with ID starting with 95b4011328c96d079091415e3643cbe69508e34b899bbb0c6632c1356c6a4da3 not found: ID does not exist" containerID="95b4011328c96d079091415e3643cbe69508e34b899bbb0c6632c1356c6a4da3"
Sep 16 20:57:23.402964 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:23.402914    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95b4011328c96d079091415e3643cbe69508e34b899bbb0c6632c1356c6a4da3"} err="failed to get container status \"95b4011328c96d079091415e3643cbe69508e34b899bbb0c6632c1356c6a4da3\": rpc error: code = NotFound desc = could not find container \"95b4011328c96d079091415e3643cbe69508e34b899bbb0c6632c1356c6a4da3\": container with ID starting with 95b4011328c96d079091415e3643cbe69508e34b899bbb0c6632c1356c6a4da3 not found: ID does not exist"
Sep 16 20:57:23.409104 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:23.409081    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h"]
Sep 16 20:57:23.412454 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:23.412433    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5d676f6c-nhp6h"]
Sep 16 20:57:23.937762 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:23.937729    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf63809e-33cd-4cca-9a98-b8272241e2a7" path="/var/lib/kubelet/pods/cf63809e-33cd-4cca-9a98-b8272241e2a7/volumes"
Sep 16 20:57:32.189253 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:32.189175    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-lm5zl"]
Sep 16 20:57:32.189596 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:32.189550    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf63809e-33cd-4cca-9a98-b8272241e2a7" containerName="main"
Sep 16 20:57:32.189596 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:32.189561    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf63809e-33cd-4cca-9a98-b8272241e2a7" containerName="main"
Sep 16 20:57:32.189596 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:32.189570    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf63809e-33cd-4cca-9a98-b8272241e2a7" containerName="storage-initializer"
Sep 16 20:57:32.189596 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:32.189576    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf63809e-33cd-4cca-9a98-b8272241e2a7" containerName="storage-initializer"
Sep 16 20:57:32.189727 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:32.189659    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf63809e-33cd-4cca-9a98-b8272241e2a7" containerName="main"
Sep 16 20:57:32.192777 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:32.192758    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-lm5zl"
Sep 16 20:57:32.195753 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:32.195721    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 16 20:57:32.195905 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:32.195873    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 20:57:32.195991 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:32.195879    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-2t9hg\""
Sep 16 20:57:32.196051 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:32.195987    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Sep 16 20:57:32.196199 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:32.196180    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 16 20:57:32.196932 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:32.196910    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 16 20:57:32.204657 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:32.204632    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-lm5zl"]
Sep 16 20:57:32.272413 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:32.272379    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b79cb39-9868-457b-8d38-4de111ac90b8-tmp-dir\") pod \"prestop-hook-test-kserve-5c5b674df-lm5zl\" (UID: \"6b79cb39-9868-457b-8d38-4de111ac90b8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-lm5zl"
Sep 16 20:57:32.272606 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:32.272419    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b79cb39-9868-457b-8d38-4de111ac90b8-model-cache\") pod \"prestop-hook-test-kserve-5c5b674df-lm5zl\" (UID: \"6b79cb39-9868-457b-8d38-4de111ac90b8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-lm5zl"
Sep 16 20:57:32.272606 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:32.272499    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6b79cb39-9868-457b-8d38-4de111ac90b8-cabundle-cert\") pod \"prestop-hook-test-kserve-5c5b674df-lm5zl\" (UID: \"6b79cb39-9868-457b-8d38-4de111ac90b8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-lm5zl"
Sep 16 20:57:32.272606 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:32.272538    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b79cb39-9868-457b-8d38-4de111ac90b8-dshm\") pod \"prestop-hook-test-kserve-5c5b674df-lm5zl\" (UID: \"6b79cb39-9868-457b-8d38-4de111ac90b8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-lm5zl"
Sep 16 20:57:32.272606 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:32.272572    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b79cb39-9868-457b-8d38-4de111ac90b8-tls-certs\") pod \"prestop-hook-test-kserve-5c5b674df-lm5zl\" (UID: \"6b79cb39-9868-457b-8d38-4de111ac90b8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-lm5zl"
Sep 16 20:57:32.272606 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:32.272593    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b79cb39-9868-457b-8d38-4de111ac90b8-kserve-provision-location\") pod \"prestop-hook-test-kserve-5c5b674df-lm5zl\" (UID: \"6b79cb39-9868-457b-8d38-4de111ac90b8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-lm5zl"
Sep 16 20:57:32.272831 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:32.272626    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7qrx\" (UniqueName: \"kubernetes.io/projected/6b79cb39-9868-457b-8d38-4de111ac90b8-kube-api-access-l7qrx\") pod \"prestop-hook-test-kserve-5c5b674df-lm5zl\" (UID: \"6b79cb39-9868-457b-8d38-4de111ac90b8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-lm5zl"
Sep 16 20:57:32.272831 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:32.272724    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b79cb39-9868-457b-8d38-4de111ac90b8-home\") pod \"prestop-hook-test-kserve-5c5b674df-lm5zl\" (UID: \"6b79cb39-9868-457b-8d38-4de111ac90b8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-lm5zl"
Sep 16 20:57:32.373462 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:32.373429    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b79cb39-9868-457b-8d38-4de111ac90b8-home\") pod \"prestop-hook-test-kserve-5c5b674df-lm5zl\" (UID: \"6b79cb39-9868-457b-8d38-4de111ac90b8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-lm5zl"
Sep 16 20:57:32.373611 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:32.373470    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b79cb39-9868-457b-8d38-4de111ac90b8-tmp-dir\") pod \"prestop-hook-test-kserve-5c5b674df-lm5zl\" (UID: \"6b79cb39-9868-457b-8d38-4de111ac90b8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-lm5zl"
Sep 16 20:57:32.373611 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:32.373493    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b79cb39-9868-457b-8d38-4de111ac90b8-model-cache\") pod \"prestop-hook-test-kserve-5c5b674df-lm5zl\" (UID: \"6b79cb39-9868-457b-8d38-4de111ac90b8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-lm5zl"
Sep 16 20:57:32.373611 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:32.373547    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6b79cb39-9868-457b-8d38-4de111ac90b8-cabundle-cert\") pod \"prestop-hook-test-kserve-5c5b674df-lm5zl\" (UID: \"6b79cb39-9868-457b-8d38-4de111ac90b8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-lm5zl"
Sep 16 20:57:32.373611 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:32.373574    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b79cb39-9868-457b-8d38-4de111ac90b8-dshm\") pod \"prestop-hook-test-kserve-5c5b674df-lm5zl\" (UID: \"6b79cb39-9868-457b-8d38-4de111ac90b8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-lm5zl"
Sep 16 20:57:32.373611 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:32.373605    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b79cb39-9868-457b-8d38-4de111ac90b8-tls-certs\") pod \"prestop-hook-test-kserve-5c5b674df-lm5zl\" (UID: \"6b79cb39-9868-457b-8d38-4de111ac90b8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-lm5zl"
Sep 16 20:57:32.373931 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:32.373623    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b79cb39-9868-457b-8d38-4de111ac90b8-kserve-provision-location\") pod \"prestop-hook-test-kserve-5c5b674df-lm5zl\" (UID: \"6b79cb39-9868-457b-8d38-4de111ac90b8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-lm5zl"
Sep 16 20:57:32.373931 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:32.373648    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7qrx\" (UniqueName: \"kubernetes.io/projected/6b79cb39-9868-457b-8d38-4de111ac90b8-kube-api-access-l7qrx\") pod \"prestop-hook-test-kserve-5c5b674df-lm5zl\" (UID: \"6b79cb39-9868-457b-8d38-4de111ac90b8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-lm5zl"
Sep 16 20:57:32.373931 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:32.373918    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b79cb39-9868-457b-8d38-4de111ac90b8-home\") pod \"prestop-hook-test-kserve-5c5b674df-lm5zl\" (UID: \"6b79cb39-9868-457b-8d38-4de111ac90b8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-lm5zl"
Sep 16 20:57:32.374055 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:32.373942    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b79cb39-9868-457b-8d38-4de111ac90b8-tmp-dir\") pod \"prestop-hook-test-kserve-5c5b674df-lm5zl\" (UID: \"6b79cb39-9868-457b-8d38-4de111ac90b8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-lm5zl"
Sep 16 20:57:32.374055 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:32.373973    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b79cb39-9868-457b-8d38-4de111ac90b8-model-cache\") pod \"prestop-hook-test-kserve-5c5b674df-lm5zl\" (UID: \"6b79cb39-9868-457b-8d38-4de111ac90b8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-lm5zl"
Sep 16 20:57:32.374182 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:32.374157    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b79cb39-9868-457b-8d38-4de111ac90b8-kserve-provision-location\") pod \"prestop-hook-test-kserve-5c5b674df-lm5zl\" (UID: \"6b79cb39-9868-457b-8d38-4de111ac90b8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-lm5zl"
Sep 16 20:57:32.374359 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:32.374343    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6b79cb39-9868-457b-8d38-4de111ac90b8-cabundle-cert\") pod \"prestop-hook-test-kserve-5c5b674df-lm5zl\" (UID: \"6b79cb39-9868-457b-8d38-4de111ac90b8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-lm5zl"
Sep 16 20:57:32.376032 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:32.376008    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b79cb39-9868-457b-8d38-4de111ac90b8-dshm\") pod \"prestop-hook-test-kserve-5c5b674df-lm5zl\" (UID: \"6b79cb39-9868-457b-8d38-4de111ac90b8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-lm5zl"
Sep 16 20:57:32.376295 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:32.376276    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b79cb39-9868-457b-8d38-4de111ac90b8-tls-certs\") pod \"prestop-hook-test-kserve-5c5b674df-lm5zl\" (UID: \"6b79cb39-9868-457b-8d38-4de111ac90b8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-lm5zl"
Sep 16 20:57:32.382123 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:32.382099    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7qrx\" (UniqueName: \"kubernetes.io/projected/6b79cb39-9868-457b-8d38-4de111ac90b8-kube-api-access-l7qrx\") pod \"prestop-hook-test-kserve-5c5b674df-lm5zl\" (UID: \"6b79cb39-9868-457b-8d38-4de111ac90b8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-lm5zl"
Sep 16 20:57:32.505057 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:32.505013    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-lm5zl"
Sep 16 20:57:32.787730 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:32.787703    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-lm5zl"]
Sep 16 20:57:32.790199 ip-10-0-139-81 kubenswrapper[2557]: W0916 20:57:32.790167    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b79cb39_9868_457b_8d38_4de111ac90b8.slice/crio-3f718eef61a5fd0f2a8a3f12b9e21dbd5cb04b3ad3818dcd43316f098cec51bd WatchSource:0}: Error finding container 3f718eef61a5fd0f2a8a3f12b9e21dbd5cb04b3ad3818dcd43316f098cec51bd: Status 404 returned error can't find the container with id 3f718eef61a5fd0f2a8a3f12b9e21dbd5cb04b3ad3818dcd43316f098cec51bd
Sep 16 20:57:33.436317 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:33.436275    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-lm5zl" event={"ID":"6b79cb39-9868-457b-8d38-4de111ac90b8","Type":"ContainerStarted","Data":"5035eb033a663fd1604ad15cbcce9e21a8b473ecb4e6e2a0c69491e740ec917b"}
Sep 16 20:57:33.436860 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:33.436337    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-lm5zl" event={"ID":"6b79cb39-9868-457b-8d38-4de111ac90b8","Type":"ContainerStarted","Data":"3f718eef61a5fd0f2a8a3f12b9e21dbd5cb04b3ad3818dcd43316f098cec51bd"}
Sep 16 20:57:35.457558 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:35.457460    2557 generic.go:356] "Generic (PLEG): container finished" podID="6b79cb39-9868-457b-8d38-4de111ac90b8" containerID="5035eb033a663fd1604ad15cbcce9e21a8b473ecb4e6e2a0c69491e740ec917b" exitCode=0
Sep 16 20:57:35.457558 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:35.457536    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-lm5zl" event={"ID":"6b79cb39-9868-457b-8d38-4de111ac90b8","Type":"ContainerDied","Data":"5035eb033a663fd1604ad15cbcce9e21a8b473ecb4e6e2a0c69491e740ec917b"}
Sep 16 20:57:36.466243 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:36.466211    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-lm5zl" event={"ID":"6b79cb39-9868-457b-8d38-4de111ac90b8","Type":"ContainerStarted","Data":"9793552aa1b0cc5b854e7b1d0d4e79e7cd23f26bb1a9c7829879d78a89abab2c"}
Sep 16 20:57:36.489350 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:36.489295    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-lm5zl" podStartSLOduration=2.305380163 podStartE2EDuration="4.489278742s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.183898579s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:57:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:57:36.48794653 +0000 UTC m=+4685.013334915" watchObservedRunningTime="2026-09-16 20:57:36.489278742 +0000 UTC m=+4685.014667127"
Sep 16 20:57:36.505288 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:36.505257    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-lm5zl"
Sep 16 20:57:37.904323 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:37.904278    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-sj2z5"]
Sep 16 20:57:37.909369 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:37.909338    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-sj2z5"
Sep 16 20:57:37.912123 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:37.912094    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 16 20:57:37.913297 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:37.913273    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 16 20:57:37.913467 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:37.913334    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-epp-sa-dockercfg-dpqb4\""
Sep 16 20:57:37.913633 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:37.913360    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-d8rcr\""
Sep 16 20:57:37.913741 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:37.913459    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 16 20:57:37.920459 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:37.920435    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-sj2z5"]
Sep 16 20:57:38.029300 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:38.029259    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkdb2\" (UniqueName: \"kubernetes.io/projected/d5fd032e-4840-4056-a567-fad8ff662e9c-kube-api-access-dkdb2\") pod \"rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-sj2z5\" (UID: \"d5fd032e-4840-4056-a567-fad8ff662e9c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-sj2z5"
Sep 16 20:57:38.029493 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:38.029358    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5fd032e-4840-4056-a567-fad8ff662e9c-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-sj2z5\" (UID: \"d5fd032e-4840-4056-a567-fad8ff662e9c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-sj2z5"
Sep 16 20:57:38.130145 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:38.130106    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5fd032e-4840-4056-a567-fad8ff662e9c-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-sj2z5\" (UID: \"d5fd032e-4840-4056-a567-fad8ff662e9c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-sj2z5"
Sep 16 20:57:38.130347 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:38.130210    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkdb2\" (UniqueName: \"kubernetes.io/projected/d5fd032e-4840-4056-a567-fad8ff662e9c-kube-api-access-dkdb2\") pod \"rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-sj2z5\" (UID: \"d5fd032e-4840-4056-a567-fad8ff662e9c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-sj2z5"
Sep 16 20:57:38.132696 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:38.132668    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5fd032e-4840-4056-a567-fad8ff662e9c-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-sj2z5\" (UID: \"d5fd032e-4840-4056-a567-fad8ff662e9c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-sj2z5"
Sep 16 20:57:38.138296 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:38.138276    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkdb2\" (UniqueName: \"kubernetes.io/projected/d5fd032e-4840-4056-a567-fad8ff662e9c-kube-api-access-dkdb2\") pod \"rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-sj2z5\" (UID: \"d5fd032e-4840-4056-a567-fad8ff662e9c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-sj2z5"
Sep 16 20:57:38.222737 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:38.222647    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-sj2z5"
Sep 16 20:57:38.508228 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:38.508104    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-sj2z5"]
Sep 16 20:57:38.510869 ip-10-0-139-81 kubenswrapper[2557]: W0916 20:57:38.510819    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5fd032e_4840_4056_a567_fad8ff662e9c.slice/crio-b290df9a9969da8cdf7bd47a8d99534a0bbd0aa4ef95c2e423f7939a839f5ab3 WatchSource:0}: Error finding container b290df9a9969da8cdf7bd47a8d99534a0bbd0aa4ef95c2e423f7939a839f5ab3: Status 404 returned error can't find the container with id b290df9a9969da8cdf7bd47a8d99534a0bbd0aa4ef95c2e423f7939a839f5ab3
Sep 16 20:57:39.484590 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:39.484550    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-sj2z5" event={"ID":"d5fd032e-4840-4056-a567-fad8ff662e9c","Type":"ContainerStarted","Data":"ca9ed168bbeea9957e801de0a5482fb5011438d11d6e049c81641c9eaff95785"}
Sep 16 20:57:39.484983 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:39.484608    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-sj2z5" event={"ID":"d5fd032e-4840-4056-a567-fad8ff662e9c","Type":"ContainerStarted","Data":"b290df9a9969da8cdf7bd47a8d99534a0bbd0aa4ef95c2e423f7939a839f5ab3"}
Sep 16 20:57:39.484983 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:39.484662    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-sj2z5"
Sep 16 20:57:39.503466 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:39.503405    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-sj2z5" podStartSLOduration=2.503380604 podStartE2EDuration="2.503380604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:57:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:57:39.50030075 +0000 UTC m=+4688.025689137" watchObservedRunningTime="2026-09-16 20:57:39.503380604 +0000 UTC m=+4688.028768989"
Sep 16 20:57:42.505384 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:42.505338    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-lm5zl"
Sep 16 20:57:42.506954 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:42.506924    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-lm5zl" podUID="6b79cb39-9868-457b-8d38-4de111ac90b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 16 20:57:52.505790 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:57:52.505743    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-lm5zl" podUID="6b79cb39-9868-457b-8d38-4de111ac90b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 16 20:58:02.506112 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:58:02.506058    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-lm5zl" podUID="6b79cb39-9868-457b-8d38-4de111ac90b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 16 20:58:10.492303 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:58:10.492267    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-sj2z5"
Sep 16 20:58:12.505699 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:58:12.505648    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-lm5zl" podUID="6b79cb39-9868-457b-8d38-4de111ac90b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 16 20:58:14.011189 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:58:14.011152    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-lm5zl"]
Sep 16 20:58:25.163607 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:58:25.163559    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-sj2z5"]
Sep 16 20:58:29.034145 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:58:29.034101    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-lm5zl" podUID="6b79cb39-9868-457b-8d38-4de111ac90b8" containerName="main" containerID="cri-o://9793552aa1b0cc5b854e7b1d0d4e79e7cd23f26bb1a9c7829879d78a89abab2c" gracePeriod=45
Sep 16 20:58:30.492362 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:58:30.492312    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-sj2z5" podUID="d5fd032e-4840-4056-a567-fad8ff662e9c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:58:30.581584 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:58:30.579718    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-5c5b674df-lm5zl_6b79cb39-9868-457b-8d38-4de111ac90b8/main/0.log"
Sep 16 20:58:30.583869 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:58:30.583822    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-lm5zl"
Sep 16 20:58:30.633594 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:58:30.633555    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7qrx\" (UniqueName: \"kubernetes.io/projected/6b79cb39-9868-457b-8d38-4de111ac90b8-kube-api-access-l7qrx\") pod \"6b79cb39-9868-457b-8d38-4de111ac90b8\" (UID: \"6b79cb39-9868-457b-8d38-4de111ac90b8\") "
Sep 16 20:58:30.633754 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:58:30.633605    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b79cb39-9868-457b-8d38-4de111ac90b8-kserve-provision-location\") pod \"6b79cb39-9868-457b-8d38-4de111ac90b8\" (UID: \"6b79cb39-9868-457b-8d38-4de111ac90b8\") "
Sep 16 20:58:30.633754 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:58:30.633657    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b79cb39-9868-457b-8d38-4de111ac90b8-tmp-dir\") pod \"6b79cb39-9868-457b-8d38-4de111ac90b8\" (UID: \"6b79cb39-9868-457b-8d38-4de111ac90b8\") "
Sep 16 20:58:30.633754 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:58:30.633715    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b79cb39-9868-457b-8d38-4de111ac90b8-model-cache\") pod \"6b79cb39-9868-457b-8d38-4de111ac90b8\" (UID: \"6b79cb39-9868-457b-8d38-4de111ac90b8\") "
Sep 16 20:58:30.633754 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:58:30.633741    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b79cb39-9868-457b-8d38-4de111ac90b8-dshm\") pod \"6b79cb39-9868-457b-8d38-4de111ac90b8\" (UID: \"6b79cb39-9868-457b-8d38-4de111ac90b8\") "
Sep 16 20:58:30.634014 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:58:30.633788    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6b79cb39-9868-457b-8d38-4de111ac90b8-cabundle-cert\") pod \"6b79cb39-9868-457b-8d38-4de111ac90b8\" (UID: \"6b79cb39-9868-457b-8d38-4de111ac90b8\") "
Sep 16 20:58:30.634014 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:58:30.633870    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b79cb39-9868-457b-8d38-4de111ac90b8-tls-certs\") pod \"6b79cb39-9868-457b-8d38-4de111ac90b8\" (UID: \"6b79cb39-9868-457b-8d38-4de111ac90b8\") "
Sep 16 20:58:30.634014 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:58:30.633918    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b79cb39-9868-457b-8d38-4de111ac90b8-home\") pod \"6b79cb39-9868-457b-8d38-4de111ac90b8\" (UID: \"6b79cb39-9868-457b-8d38-4de111ac90b8\") "
Sep 16 20:58:30.634171 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:58:30.634104    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b79cb39-9868-457b-8d38-4de111ac90b8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6b79cb39-9868-457b-8d38-4de111ac90b8" (UID: "6b79cb39-9868-457b-8d38-4de111ac90b8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:58:30.634171 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:58:30.634111    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b79cb39-9868-457b-8d38-4de111ac90b8-model-cache" (OuterVolumeSpecName: "model-cache") pod "6b79cb39-9868-457b-8d38-4de111ac90b8" (UID: "6b79cb39-9868-457b-8d38-4de111ac90b8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:58:30.634283 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:58:30.634187    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b79cb39-9868-457b-8d38-4de111ac90b8-tmp-dir\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:58:30.634283 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:58:30.634209    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b79cb39-9868-457b-8d38-4de111ac90b8-model-cache\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:58:30.634388 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:58:30.634317    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b79cb39-9868-457b-8d38-4de111ac90b8-home" (OuterVolumeSpecName: "home") pod "6b79cb39-9868-457b-8d38-4de111ac90b8" (UID: "6b79cb39-9868-457b-8d38-4de111ac90b8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:58:30.634480 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:58:30.634449    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b79cb39-9868-457b-8d38-4de111ac90b8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6b79cb39-9868-457b-8d38-4de111ac90b8" (UID: "6b79cb39-9868-457b-8d38-4de111ac90b8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:58:30.635936 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:58:30.635908    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b79cb39-9868-457b-8d38-4de111ac90b8-kube-api-access-l7qrx" (OuterVolumeSpecName: "kube-api-access-l7qrx") pod "6b79cb39-9868-457b-8d38-4de111ac90b8" (UID: "6b79cb39-9868-457b-8d38-4de111ac90b8"). InnerVolumeSpecName "kube-api-access-l7qrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:58:30.636056 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:58:30.636033    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b79cb39-9868-457b-8d38-4de111ac90b8-dshm" (OuterVolumeSpecName: "dshm") pod "6b79cb39-9868-457b-8d38-4de111ac90b8" (UID: "6b79cb39-9868-457b-8d38-4de111ac90b8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:58:30.636144 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:58:30.636120    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b79cb39-9868-457b-8d38-4de111ac90b8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6b79cb39-9868-457b-8d38-4de111ac90b8" (UID: "6b79cb39-9868-457b-8d38-4de111ac90b8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:58:30.657507 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:58:30.657440    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b79cb39-9868-457b-8d38-4de111ac90b8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6b79cb39-9868-457b-8d38-4de111ac90b8" (UID: "6b79cb39-9868-457b-8d38-4de111ac90b8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:58:30.735630 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:58:30.735583    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b79cb39-9868-457b-8d38-4de111ac90b8-dshm\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:58:30.735630 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:58:30.735625    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6b79cb39-9868-457b-8d38-4de111ac90b8-cabundle-cert\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:58:30.735630 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:58:30.735638    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b79cb39-9868-457b-8d38-4de111ac90b8-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:58:30.735870 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:58:30.735646    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b79cb39-9868-457b-8d38-4de111ac90b8-home\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:58:30.735870 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:58:30.735655    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7qrx\" (UniqueName: \"kubernetes.io/projected/6b79cb39-9868-457b-8d38-4de111ac90b8-kube-api-access-l7qrx\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:58:30.735870 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:58:30.735664    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b79cb39-9868-457b-8d38-4de111ac90b8-kserve-provision-location\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:58:30.760307 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:58:30.760282    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-5c5b674df-lm5zl_6b79cb39-9868-457b-8d38-4de111ac90b8/main/0.log"
Sep 16 20:58:30.760652 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:58:30.760624    2557 generic.go:356] "Generic (PLEG): container finished" podID="6b79cb39-9868-457b-8d38-4de111ac90b8" containerID="9793552aa1b0cc5b854e7b1d0d4e79e7cd23f26bb1a9c7829879d78a89abab2c" exitCode=130
Sep 16 20:58:30.760784 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:58:30.760678    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-lm5zl"
Sep 16 20:58:30.760784 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:58:30.760716    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-lm5zl" event={"ID":"6b79cb39-9868-457b-8d38-4de111ac90b8","Type":"ContainerDied","Data":"9793552aa1b0cc5b854e7b1d0d4e79e7cd23f26bb1a9c7829879d78a89abab2c"}
Sep 16 20:58:30.760784 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:58:30.760779    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-lm5zl" event={"ID":"6b79cb39-9868-457b-8d38-4de111ac90b8","Type":"ContainerDied","Data":"3f718eef61a5fd0f2a8a3f12b9e21dbd5cb04b3ad3818dcd43316f098cec51bd"}
Sep 16 20:58:30.760935 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:58:30.760804    2557 scope.go:117] "RemoveContainer" containerID="9793552aa1b0cc5b854e7b1d0d4e79e7cd23f26bb1a9c7829879d78a89abab2c"
Sep 16 20:58:30.770371 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:58:30.770349    2557 scope.go:117] "RemoveContainer" containerID="5035eb033a663fd1604ad15cbcce9e21a8b473ecb4e6e2a0c69491e740ec917b"
Sep 16 20:58:30.778678 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:58:30.778661    2557 scope.go:117] "RemoveContainer" containerID="9793552aa1b0cc5b854e7b1d0d4e79e7cd23f26bb1a9c7829879d78a89abab2c"
Sep 16 20:58:30.778965 ip-10-0-139-81 kubenswrapper[2557]: E0916 20:58:30.778946    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9793552aa1b0cc5b854e7b1d0d4e79e7cd23f26bb1a9c7829879d78a89abab2c\": container with ID starting with 9793552aa1b0cc5b854e7b1d0d4e79e7cd23f26bb1a9c7829879d78a89abab2c not found: ID does not exist" containerID="9793552aa1b0cc5b854e7b1d0d4e79e7cd23f26bb1a9c7829879d78a89abab2c"
Sep 16 20:58:30.779017 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:58:30.778974    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9793552aa1b0cc5b854e7b1d0d4e79e7cd23f26bb1a9c7829879d78a89abab2c"} err="failed to get container status \"9793552aa1b0cc5b854e7b1d0d4e79e7cd23f26bb1a9c7829879d78a89abab2c\": rpc error: code = NotFound desc = could not find container \"9793552aa1b0cc5b854e7b1d0d4e79e7cd23f26bb1a9c7829879d78a89abab2c\": container with ID starting with 9793552aa1b0cc5b854e7b1d0d4e79e7cd23f26bb1a9c7829879d78a89abab2c not found: ID does not exist"
Sep 16 20:58:30.779017 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:58:30.778990    2557 scope.go:117] "RemoveContainer" containerID="5035eb033a663fd1604ad15cbcce9e21a8b473ecb4e6e2a0c69491e740ec917b"
Sep 16 20:58:30.779263 ip-10-0-139-81 kubenswrapper[2557]: E0916 20:58:30.779244    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5035eb033a663fd1604ad15cbcce9e21a8b473ecb4e6e2a0c69491e740ec917b\": container with ID starting with 5035eb033a663fd1604ad15cbcce9e21a8b473ecb4e6e2a0c69491e740ec917b not found: ID does not exist" containerID="5035eb033a663fd1604ad15cbcce9e21a8b473ecb4e6e2a0c69491e740ec917b"
Sep 16 20:58:30.779311 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:58:30.779270    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5035eb033a663fd1604ad15cbcce9e21a8b473ecb4e6e2a0c69491e740ec917b"} err="failed to get container status \"5035eb033a663fd1604ad15cbcce9e21a8b473ecb4e6e2a0c69491e740ec917b\": rpc error: code = NotFound desc = could not find container \"5035eb033a663fd1604ad15cbcce9e21a8b473ecb4e6e2a0c69491e740ec917b\": container with ID starting with 5035eb033a663fd1604ad15cbcce9e21a8b473ecb4e6e2a0c69491e740ec917b not found: ID does not exist"
Sep 16 20:58:30.783746 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:58:30.783720    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-lm5zl"]
Sep 16 20:58:30.786969 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:58:30.786947    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-lm5zl"]
Sep 16 20:58:31.938525 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:58:31.938480    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b79cb39-9868-457b-8d38-4de111ac90b8" path="/var/lib/kubelet/pods/6b79cb39-9868-457b-8d38-4de111ac90b8/volumes"
Sep 16 20:58:40.183578 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:58:40.183536    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-sj2z5" podUID="d5fd032e-4840-4056-a567-fad8ff662e9c" containerName="main" containerID="cri-o://ca9ed168bbeea9957e801de0a5482fb5011438d11d6e049c81641c9eaff95785" gracePeriod=45
Sep 16 20:58:40.492284 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:58:40.492249    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-sj2z5" podUID="d5fd032e-4840-4056-a567-fad8ff662e9c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:58:50.492074 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:58:50.492035    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-sj2z5" podUID="d5fd032e-4840-4056-a567-fad8ff662e9c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:58:50.492513 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:58:50.492171    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-sj2z5"
Sep 16 20:59:00.492613 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:59:00.492532    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-sj2z5" podUID="d5fd032e-4840-4056-a567-fad8ff662e9c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:59:10.437833 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:59:10.437801    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-sj2z5"
Sep 16 20:59:10.490183 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:59:10.490145    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5fd032e-4840-4056-a567-fad8ff662e9c-tls-certs\") pod \"d5fd032e-4840-4056-a567-fad8ff662e9c\" (UID: \"d5fd032e-4840-4056-a567-fad8ff662e9c\") "
Sep 16 20:59:10.490344 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:59:10.490242    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dkdb2\" (UniqueName: \"kubernetes.io/projected/d5fd032e-4840-4056-a567-fad8ff662e9c-kube-api-access-dkdb2\") pod \"d5fd032e-4840-4056-a567-fad8ff662e9c\" (UID: \"d5fd032e-4840-4056-a567-fad8ff662e9c\") "
Sep 16 20:59:10.492500 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:59:10.492459    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5fd032e-4840-4056-a567-fad8ff662e9c-kube-api-access-dkdb2" (OuterVolumeSpecName: "kube-api-access-dkdb2") pod "d5fd032e-4840-4056-a567-fad8ff662e9c" (UID: "d5fd032e-4840-4056-a567-fad8ff662e9c"). InnerVolumeSpecName "kube-api-access-dkdb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:59:10.492500 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:59:10.492483    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5fd032e-4840-4056-a567-fad8ff662e9c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d5fd032e-4840-4056-a567-fad8ff662e9c" (UID: "d5fd032e-4840-4056-a567-fad8ff662e9c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:59:10.590908 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:59:10.590827    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkdb2\" (UniqueName: \"kubernetes.io/projected/d5fd032e-4840-4056-a567-fad8ff662e9c-kube-api-access-dkdb2\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:59:10.590908 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:59:10.590904    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5fd032e-4840-4056-a567-fad8ff662e9c-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Sep 16 20:59:10.958817 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:59:10.958724    2557 generic.go:356] "Generic (PLEG): container finished" podID="d5fd032e-4840-4056-a567-fad8ff662e9c" containerID="ca9ed168bbeea9957e801de0a5482fb5011438d11d6e049c81641c9eaff95785" exitCode=0
Sep 16 20:59:10.958817 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:59:10.958781    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-sj2z5"
Sep 16 20:59:10.959033 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:59:10.958819    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-sj2z5" event={"ID":"d5fd032e-4840-4056-a567-fad8ff662e9c","Type":"ContainerDied","Data":"ca9ed168bbeea9957e801de0a5482fb5011438d11d6e049c81641c9eaff95785"}
Sep 16 20:59:10.959033 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:59:10.958899    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-sj2z5" event={"ID":"d5fd032e-4840-4056-a567-fad8ff662e9c","Type":"ContainerDied","Data":"b290df9a9969da8cdf7bd47a8d99534a0bbd0aa4ef95c2e423f7939a839f5ab3"}
Sep 16 20:59:10.959033 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:59:10.958924    2557 scope.go:117] "RemoveContainer" containerID="ca9ed168bbeea9957e801de0a5482fb5011438d11d6e049c81641c9eaff95785"
Sep 16 20:59:10.968135 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:59:10.968117    2557 scope.go:117] "RemoveContainer" containerID="ca9ed168bbeea9957e801de0a5482fb5011438d11d6e049c81641c9eaff95785"
Sep 16 20:59:10.968438 ip-10-0-139-81 kubenswrapper[2557]: E0916 20:59:10.968416    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca9ed168bbeea9957e801de0a5482fb5011438d11d6e049c81641c9eaff95785\": container with ID starting with ca9ed168bbeea9957e801de0a5482fb5011438d11d6e049c81641c9eaff95785 not found: ID does not exist" containerID="ca9ed168bbeea9957e801de0a5482fb5011438d11d6e049c81641c9eaff95785"
Sep 16 20:59:10.968545 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:59:10.968450    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca9ed168bbeea9957e801de0a5482fb5011438d11d6e049c81641c9eaff95785"} err="failed to get container status \"ca9ed168bbeea9957e801de0a5482fb5011438d11d6e049c81641c9eaff95785\": rpc error: code = NotFound desc = could not find container \"ca9ed168bbeea9957e801de0a5482fb5011438d11d6e049c81641c9eaff95785\": container with ID starting with ca9ed168bbeea9957e801de0a5482fb5011438d11d6e049c81641c9eaff95785 not found: ID does not exist"
Sep 16 20:59:10.981143 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:59:10.981118    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-sj2z5"]
Sep 16 20:59:10.985280 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:59:10.985259    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-sj2z5"]
Sep 16 20:59:11.941037 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:59:11.940998    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5fd032e-4840-4056-a567-fad8ff662e9c" path="/var/lib/kubelet/pods/d5fd032e-4840-4056-a567-fad8ff662e9c/volumes"
Sep 16 20:59:39.682484 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:59:39.682451    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-b4b87b796-l78vs_25b2d6d3-b32d-4e8f-96c1-7701d5609d6f/router/0.log"
Sep 16 20:59:40.467979 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:59:40.467946    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-b4b87b796-l78vs_25b2d6d3-b32d-4e8f-96c1-7701d5609d6f/router/0.log"
Sep 16 20:59:41.803367 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:59:41.803333    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-gwnhm_de06a41d-3376-4092-b1f6-d5e588104656/authorino/0.log"
Sep 16 20:59:42.047170 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:59:42.047130    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-49qfs_c65b34e0-35f4-4d8a-8ddb-558df53723de/manager/0.log"
Sep 16 20:59:51.146353 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:59:51.146321    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qpp69_3428bc6c-ab5c-47f1-8f1d-f5923fdacfce/global-pull-secret-syncer/0.log"
Sep 16 20:59:51.278165 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:59:51.278136    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-v57p4_85c2f2c0-d2a8-454e-9e28-498c0c8d3c21/konnectivity-agent/0.log"
Sep 16 20:59:51.372746 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:59:51.372709    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-81.ec2.internal_a47431da3c311f72143c9d580d811b73/haproxy/0.log"
Sep 16 20:59:56.712166 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:59:56.712129    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-gwnhm_de06a41d-3376-4092-b1f6-d5e588104656/authorino/0.log"
Sep 16 20:59:57.181313 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:59:57.181279    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-49qfs_c65b34e0-35f4-4d8a-8ddb-558df53723de/manager/0.log"
Sep 16 20:59:58.234197 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:59:58.234169    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9213f9dd-3f19-4700-9586-3b9cb8616647/alertmanager/0.log"
Sep 16 20:59:58.260240 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:59:58.260212    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9213f9dd-3f19-4700-9586-3b9cb8616647/config-reloader/0.log"
Sep 16 20:59:58.281301 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:59:58.281269    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9213f9dd-3f19-4700-9586-3b9cb8616647/kube-rbac-proxy-web/0.log"
Sep 16 20:59:58.306602 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:59:58.306575    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9213f9dd-3f19-4700-9586-3b9cb8616647/kube-rbac-proxy/0.log"
Sep 16 20:59:58.329024 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:59:58.328999    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9213f9dd-3f19-4700-9586-3b9cb8616647/kube-rbac-proxy-metric/0.log"
Sep 16 20:59:58.353190 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:59:58.353163    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9213f9dd-3f19-4700-9586-3b9cb8616647/prom-label-proxy/0.log"
Sep 16 20:59:58.378995 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:59:58.378964    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9213f9dd-3f19-4700-9586-3b9cb8616647/init-config-reloader/0.log"
Sep 16 20:59:58.442069 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:59:58.441990    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-fbswk_d6e55a03-bcc6-42f0-9b0c-b3c62b89d3ad/cluster-monitoring-operator/0.log"
Sep 16 20:59:58.604819 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:59:58.604794    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-zpbj5_4b06b88d-9541-4e30-9b3d-24d6121f17ab/monitoring-plugin/0.log"
Sep 16 20:59:58.808501 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:59:58.808467    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rsdqh_215db735-5f5f-4194-ba50-d75b6ca11eaa/node-exporter/0.log"
Sep 16 20:59:58.829714 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:59:58.829681    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rsdqh_215db735-5f5f-4194-ba50-d75b6ca11eaa/kube-rbac-proxy/0.log"
Sep 16 20:59:58.873239 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:59:58.873209    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rsdqh_215db735-5f5f-4194-ba50-d75b6ca11eaa/init-textfile/0.log"
Sep 16 20:59:58.900151 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:59:58.900120    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-267mk_ad1e0f37-fcd1-4038-8d64-8b2aeeac356c/kube-rbac-proxy-main/0.log"
Sep 16 20:59:58.921429 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:59:58.921401    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-267mk_ad1e0f37-fcd1-4038-8d64-8b2aeeac356c/kube-rbac-proxy-self/0.log"
Sep 16 20:59:58.942014 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:59:58.941990    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-267mk_ad1e0f37-fcd1-4038-8d64-8b2aeeac356c/openshift-state-metrics/0.log"
Sep 16 20:59:58.988896 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:59:58.988867    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f99c26bc-67a2-411b-8fff-e09d444fdd1c/prometheus/0.log"
Sep 16 20:59:59.008551 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:59:59.008525    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f99c26bc-67a2-411b-8fff-e09d444fdd1c/config-reloader/0.log"
Sep 16 20:59:59.028215 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:59:59.028187    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f99c26bc-67a2-411b-8fff-e09d444fdd1c/thanos-sidecar/0.log"
Sep 16 20:59:59.054541 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:59:59.054510    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f99c26bc-67a2-411b-8fff-e09d444fdd1c/kube-rbac-proxy-web/0.log"
Sep 16 20:59:59.075025 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:59:59.074949    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f99c26bc-67a2-411b-8fff-e09d444fdd1c/kube-rbac-proxy/0.log"
Sep 16 20:59:59.095329 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:59:59.095303    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f99c26bc-67a2-411b-8fff-e09d444fdd1c/kube-rbac-proxy-thanos/0.log"
Sep 16 20:59:59.117019 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:59:59.116985    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f99c26bc-67a2-411b-8fff-e09d444fdd1c/init-config-reloader/0.log"
Sep 16 20:59:59.199291 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:59:59.199260    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-9wz5v_09c71130-048a-4881-b61a-5fb73b2d007d/prometheus-operator-admission-webhook/0.log"
Sep 16 20:59:59.238897 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:59:59.238863    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-68bf5dcbc6-k6gkh_7d4ad5c1-bf25-43ca-af03-bafda1baddfa/telemeter-client/0.log"
Sep 16 20:59:59.259015 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:59:59.258988    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-68bf5dcbc6-k6gkh_7d4ad5c1-bf25-43ca-af03-bafda1baddfa/reload/0.log"
Sep 16 20:59:59.280713 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:59:59.280687    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-68bf5dcbc6-k6gkh_7d4ad5c1-bf25-43ca-af03-bafda1baddfa/kube-rbac-proxy/0.log"
Sep 16 20:59:59.315703 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:59:59.315676    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84dbb595c9-fgcq7_3a10832d-3f24-4464-9291-a8b8c78adfbb/thanos-query/0.log"
Sep 16 20:59:59.338431 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:59:59.338348    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84dbb595c9-fgcq7_3a10832d-3f24-4464-9291-a8b8c78adfbb/kube-rbac-proxy-web/0.log"
Sep 16 20:59:59.357567 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:59:59.357540    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84dbb595c9-fgcq7_3a10832d-3f24-4464-9291-a8b8c78adfbb/kube-rbac-proxy/0.log"
Sep 16 20:59:59.376408 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:59:59.376382    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84dbb595c9-fgcq7_3a10832d-3f24-4464-9291-a8b8c78adfbb/prom-label-proxy/0.log"
Sep 16 20:59:59.397178 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:59:59.397148    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84dbb595c9-fgcq7_3a10832d-3f24-4464-9291-a8b8c78adfbb/kube-rbac-proxy-rules/0.log"
Sep 16 20:59:59.416100 ip-10-0-139-81 kubenswrapper[2557]: I0916 20:59:59.416071    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84dbb595c9-fgcq7_3a10832d-3f24-4464-9291-a8b8c78adfbb/kube-rbac-proxy-metrics/0.log"
Sep 16 21:00:00.546200 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:00.546170    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-2kj2q_e60fe99d-8dc5-46bb-bd98-0003a5aca1e8/networking-console-plugin/0.log"
Sep 16 21:00:01.190308 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:01.190276    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9wkzj/perf-node-gather-daemonset-h4fwh"]
Sep 16 21:00:01.190675 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:01.190662    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b79cb39-9868-457b-8d38-4de111ac90b8" containerName="main"
Sep 16 21:00:01.190727 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:01.190676    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b79cb39-9868-457b-8d38-4de111ac90b8" containerName="main"
Sep 16 21:00:01.190727 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:01.190688    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5fd032e-4840-4056-a567-fad8ff662e9c" containerName="main"
Sep 16 21:00:01.190727 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:01.190693    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5fd032e-4840-4056-a567-fad8ff662e9c" containerName="main"
Sep 16 21:00:01.190727 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:01.190712    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b79cb39-9868-457b-8d38-4de111ac90b8" containerName="storage-initializer"
Sep 16 21:00:01.190727 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:01.190718    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b79cb39-9868-457b-8d38-4de111ac90b8" containerName="storage-initializer"
Sep 16 21:00:01.190912 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:01.190770    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b79cb39-9868-457b-8d38-4de111ac90b8" containerName="main"
Sep 16 21:00:01.190912 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:01.190779    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5fd032e-4840-4056-a567-fad8ff662e9c" containerName="main"
Sep 16 21:00:01.194033 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:01.194010    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-h4fwh"
Sep 16 21:00:01.196581 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:01.196557    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9wkzj\"/\"openshift-service-ca.crt\""
Sep 16 21:00:01.197926 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:01.197909    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9wkzj\"/\"kube-root-ca.crt\""
Sep 16 21:00:01.197992 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:01.197927    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9wkzj\"/\"default-dockercfg-qhzj8\""
Sep 16 21:00:01.202438 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:01.202417    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9wkzj/perf-node-gather-daemonset-h4fwh"]
Sep 16 21:00:01.240529 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:01.240496    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1b7e1933-9331-4b49-bdab-068850c651a2-podres\") pod \"perf-node-gather-daemonset-h4fwh\" (UID: \"1b7e1933-9331-4b49-bdab-068850c651a2\") " pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-h4fwh"
Sep 16 21:00:01.240701 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:01.240599    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bvrl\" (UniqueName: \"kubernetes.io/projected/1b7e1933-9331-4b49-bdab-068850c651a2-kube-api-access-7bvrl\") pod \"perf-node-gather-daemonset-h4fwh\" (UID: \"1b7e1933-9331-4b49-bdab-068850c651a2\") " pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-h4fwh"
Sep 16 21:00:01.240701 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:01.240689    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b7e1933-9331-4b49-bdab-068850c651a2-sys\") pod \"perf-node-gather-daemonset-h4fwh\" (UID: \"1b7e1933-9331-4b49-bdab-068850c651a2\") " pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-h4fwh"
Sep 16 21:00:01.240781 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:01.240718    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1b7e1933-9331-4b49-bdab-068850c651a2-lib-modules\") pod \"perf-node-gather-daemonset-h4fwh\" (UID: \"1b7e1933-9331-4b49-bdab-068850c651a2\") " pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-h4fwh"
Sep 16 21:00:01.240781 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:01.240744    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1b7e1933-9331-4b49-bdab-068850c651a2-proc\") pod \"perf-node-gather-daemonset-h4fwh\" (UID: \"1b7e1933-9331-4b49-bdab-068850c651a2\") " pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-h4fwh"
Sep 16 21:00:01.341686 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:01.341649    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1b7e1933-9331-4b49-bdab-068850c651a2-podres\") pod \"perf-node-gather-daemonset-h4fwh\" (UID: \"1b7e1933-9331-4b49-bdab-068850c651a2\") " pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-h4fwh"
Sep 16 21:00:01.341921 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:01.341713    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bvrl\" (UniqueName: \"kubernetes.io/projected/1b7e1933-9331-4b49-bdab-068850c651a2-kube-api-access-7bvrl\") pod \"perf-node-gather-daemonset-h4fwh\" (UID: \"1b7e1933-9331-4b49-bdab-068850c651a2\") " pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-h4fwh"
Sep 16 21:00:01.341921 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:01.341765    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b7e1933-9331-4b49-bdab-068850c651a2-sys\") pod \"perf-node-gather-daemonset-h4fwh\" (UID: \"1b7e1933-9331-4b49-bdab-068850c651a2\") " pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-h4fwh"
Sep 16 21:00:01.341921 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:01.341781    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1b7e1933-9331-4b49-bdab-068850c651a2-lib-modules\") pod \"perf-node-gather-daemonset-h4fwh\" (UID: \"1b7e1933-9331-4b49-bdab-068850c651a2\") " pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-h4fwh"
Sep 16 21:00:01.341921 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:01.341807    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1b7e1933-9331-4b49-bdab-068850c651a2-proc\") pod \"perf-node-gather-daemonset-h4fwh\" (UID: \"1b7e1933-9331-4b49-bdab-068850c651a2\") " pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-h4fwh"
Sep 16 21:00:01.341921 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:01.341855    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1b7e1933-9331-4b49-bdab-068850c651a2-podres\") pod \"perf-node-gather-daemonset-h4fwh\" (UID: \"1b7e1933-9331-4b49-bdab-068850c651a2\") " pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-h4fwh"
Sep 16 21:00:01.341921 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:01.341905    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b7e1933-9331-4b49-bdab-068850c651a2-sys\") pod \"perf-node-gather-daemonset-h4fwh\" (UID: \"1b7e1933-9331-4b49-bdab-068850c651a2\") " pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-h4fwh"
Sep 16 21:00:01.342158 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:01.341930    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1b7e1933-9331-4b49-bdab-068850c651a2-proc\") pod \"perf-node-gather-daemonset-h4fwh\" (UID: \"1b7e1933-9331-4b49-bdab-068850c651a2\") " pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-h4fwh"
Sep 16 21:00:01.342158 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:01.341965    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1b7e1933-9331-4b49-bdab-068850c651a2-lib-modules\") pod \"perf-node-gather-daemonset-h4fwh\" (UID: \"1b7e1933-9331-4b49-bdab-068850c651a2\") " pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-h4fwh"
Sep 16 21:00:01.350409 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:01.350384    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bvrl\" (UniqueName: \"kubernetes.io/projected/1b7e1933-9331-4b49-bdab-068850c651a2-kube-api-access-7bvrl\") pod \"perf-node-gather-daemonset-h4fwh\" (UID: \"1b7e1933-9331-4b49-bdab-068850c651a2\") " pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-h4fwh"
Sep 16 21:00:01.505327 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:01.505290    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-h4fwh"
Sep 16 21:00:01.554855 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:01.554805    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-7kt88_f4c1d81c-50de-47c5-a944-cf1571f49abb/download-server/0.log"
Sep 16 21:00:01.787764 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:01.787717    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9wkzj/perf-node-gather-daemonset-h4fwh"]
Sep 16 21:00:01.790193 ip-10-0-139-81 kubenswrapper[2557]: W0916 21:00:01.790163    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1b7e1933_9331_4b49_bdab_068850c651a2.slice/crio-cf52f5be8291b25c041ea968f4a4b79adaf3bd9543e709df729b422720beafe0 WatchSource:0}: Error finding container cf52f5be8291b25c041ea968f4a4b79adaf3bd9543e709df729b422720beafe0: Status 404 returned error can't find the container with id cf52f5be8291b25c041ea968f4a4b79adaf3bd9543e709df729b422720beafe0
Sep 16 21:00:02.215198 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:02.215105    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-h4fwh" event={"ID":"1b7e1933-9331-4b49-bdab-068850c651a2","Type":"ContainerStarted","Data":"6e81925364e758b69f7bbff0ceb61381e8af43f2b59b5fe3e34804c2892eee66"}
Sep 16 21:00:02.215198 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:02.215163    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-h4fwh" event={"ID":"1b7e1933-9331-4b49-bdab-068850c651a2","Type":"ContainerStarted","Data":"cf52f5be8291b25c041ea968f4a4b79adaf3bd9543e709df729b422720beafe0"}
Sep 16 21:00:02.215198 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:02.215195    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-h4fwh"
Sep 16 21:00:02.233232 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:02.233184    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-h4fwh" podStartSLOduration=1.233169372 podStartE2EDuration="1.233169372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 21:00:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:00:02.23210731 +0000 UTC m=+4830.757495708" watchObservedRunningTime="2026-09-16 21:00:02.233169372 +0000 UTC m=+4830.758557757"
Sep 16 21:00:03.152204 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:03.152170    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-d8hqp_955ae2fa-b138-46de-b15d-a15850b54bb0/dns/0.log"
Sep 16 21:00:03.171909 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:03.171883    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-d8hqp_955ae2fa-b138-46de-b15d-a15850b54bb0/kube-rbac-proxy/0.log"
Sep 16 21:00:03.242192 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:03.242162    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-clfb8_142b9c47-4826-4fa8-b228-5500d98fee67/dns-node-resolver/0.log"
Sep 16 21:00:03.874761 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:03.874722    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hhvcc_bb2436af-e07f-464b-8973-1b0370888a65/node-ca/0.log"
Sep 16 21:00:05.087304 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:05.087276    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-b4b87b796-l78vs_25b2d6d3-b32d-4e8f-96c1-7701d5609d6f/router/0.log"
Sep 16 21:00:05.603509 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:05.603474    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-r2rtx_af904ddd-40b6-48f8-9917-8ed2eb4b256a/serve-healthcheck-canary/0.log"
Sep 16 21:00:06.127394 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:06.127357    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-9tm7s_8ebbee05-192c-4db3-ba7a-f0de3c33983c/insights-operator/0.log"
Sep 16 21:00:06.129347 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:06.129322    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-9tm7s_8ebbee05-192c-4db3-ba7a-f0de3c33983c/insights-operator/1.log"
Sep 16 21:00:06.306065 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:06.306027    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-v5rhc_9489a4e3-e69c-4f07-a65a-9debfc38d24c/kube-rbac-proxy/0.log"
Sep 16 21:00:06.327033 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:06.326997    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-v5rhc_9489a4e3-e69c-4f07-a65a-9debfc38d24c/exporter/0.log"
Sep 16 21:00:06.348871 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:06.348819    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-v5rhc_9489a4e3-e69c-4f07-a65a-9debfc38d24c/extractor/0.log"
Sep 16 21:00:08.229935 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:08.229909    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-h4fwh"
Sep 16 21:00:09.351136 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:09.351076    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-b4645d7c8-qpgls_d88a58d2-fcbc-4de1-8402-aa3c03d9dcbf/manager/0.log"
Sep 16 21:00:09.377569 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:09.377536    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-cgl9l_9af5e15a-b673-47ab-aa06-a9902c7bf091/openshift-lws-operator/0.log"
Sep 16 21:00:10.451488 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:10.451454    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-2hb7c_f606e249-5ba6-4f99-9c67-13efa9033d09/server/0.log"
Sep 16 21:00:10.891757 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:10.891712    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-b4b87b796-l78vs_25b2d6d3-b32d-4e8f-96c1-7701d5609d6f/router/0.log"
Sep 16 21:00:12.128420 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:12.128390    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-b4b87b796-l78vs_25b2d6d3-b32d-4e8f-96c1-7701d5609d6f/router/0.log"
Sep 16 21:00:12.227063 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:12.227020    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-ps5ff_d729dd68-1fe0-420f-9cf4-a049f927044b/manager/0.log"
Sep 16 21:00:12.307665 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:12.307632    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-8q7vx_4d120a48-88a6-49f3-82b2-af6cdd4a0642/seaweedfs/0.log"
Sep 16 21:00:16.930321 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:16.930288    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-wlg99_d8ce792d-c238-411a-9b23-80b06844837d/migrator/0.log"
Sep 16 21:00:16.948498 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:16.948446    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-wlg99_d8ce792d-c238-411a-9b23-80b06844837d/graceful-termination/0.log"
Sep 16 21:00:18.309570 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:18.309542    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-58kk4_43cea22a-1cba-450a-8a7d-86beb5de6d75/kube-multus/0.log"
Sep 16 21:00:18.705969 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:18.705899    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xv4vd_fb2ab49a-ef77-4425-91b4-2b6a6ca89630/kube-multus-additional-cni-plugins/0.log"
Sep 16 21:00:18.725514 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:18.725488    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xv4vd_fb2ab49a-ef77-4425-91b4-2b6a6ca89630/egress-router-binary-copy/0.log"
Sep 16 21:00:18.748164 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:18.748141    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xv4vd_fb2ab49a-ef77-4425-91b4-2b6a6ca89630/cni-plugins/0.log"
Sep 16 21:00:18.768928 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:18.768901    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xv4vd_fb2ab49a-ef77-4425-91b4-2b6a6ca89630/bond-cni-plugin/0.log"
Sep 16 21:00:18.787498 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:18.787471    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xv4vd_fb2ab49a-ef77-4425-91b4-2b6a6ca89630/routeoverride-cni/0.log"
Sep 16 21:00:18.806240 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:18.806199    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xv4vd_fb2ab49a-ef77-4425-91b4-2b6a6ca89630/whereabouts-cni-bincopy/0.log"
Sep 16 21:00:18.824973 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:18.824950    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xv4vd_fb2ab49a-ef77-4425-91b4-2b6a6ca89630/whereabouts-cni/0.log"
Sep 16 21:00:18.993971 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:18.993937    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-drgl8_b4ea9fa1-48c7-45be-bfcf-4594ee751086/network-metrics-daemon/0.log"
Sep 16 21:00:19.011101 ip-10-0-139-81 kubenswrapper[2557]: I0916 21:00:19.011074    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-drgl8_b4ea9fa1-48c7-45be-bfcf-4594ee751086/kube-rbac-proxy/0.log"