Jul 28 10:07:43.925248 ip-10-0-131-88 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 10:07:44.318899 ip-10-0-131-88 kubenswrapper[2557]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 10:07:44.318899 ip-10-0-131-88 kubenswrapper[2557]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 10:07:44.318899 ip-10-0-131-88 kubenswrapper[2557]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 10:07:44.318899 ip-10-0-131-88 kubenswrapper[2557]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 10:07:44.318899 ip-10-0-131-88 kubenswrapper[2557]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 10:07:44.320828 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.320741    2557 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 10:07:44.325799 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325781    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 10:07:44.325799 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325800    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 10:07:44.325867 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325804    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 10:07:44.325867 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325808    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 10:07:44.325867 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325811    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 10:07:44.325867 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325814    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 10:07:44.325867 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325817    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 10:07:44.325867 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325820    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 10:07:44.325867 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325823    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 10:07:44.325867 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325826    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 10:07:44.325867 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325829    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 10:07:44.325867 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325831    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 10:07:44.325867 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325834    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 10:07:44.325867 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325837    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 10:07:44.325867 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325840    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 10:07:44.325867 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325842    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 10:07:44.325867 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325845    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 10:07:44.325867 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325852    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 10:07:44.325867 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325855    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 10:07:44.325867 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325857    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 10:07:44.325867 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325860    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 10:07:44.325867 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325863    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 10:07:44.326348 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325866    2557 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 10:07:44.326348 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325868    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 10:07:44.326348 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325872    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 10:07:44.326348 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325875    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 10:07:44.326348 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325878    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 10:07:44.326348 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325880    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 10:07:44.326348 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325884    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 10:07:44.326348 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325888    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 10:07:44.326348 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325892    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 10:07:44.326348 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325895    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 10:07:44.326348 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325898    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 10:07:44.326348 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325901    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 10:07:44.326348 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325904    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 10:07:44.326348 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325907    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 10:07:44.326348 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325909    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 10:07:44.326348 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325913    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 10:07:44.326348 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325916    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 10:07:44.326348 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325919    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 10:07:44.326348 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325922    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 10:07:44.326825 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325926    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 10:07:44.326825 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325929    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 10:07:44.326825 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325932    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 10:07:44.326825 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325935    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 10:07:44.326825 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325939    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 10:07:44.326825 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325942    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 10:07:44.326825 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325945    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 10:07:44.326825 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325947    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 10:07:44.326825 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325949    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 10:07:44.326825 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325952    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 10:07:44.326825 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325954    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 10:07:44.326825 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325957    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 10:07:44.326825 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325960    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 10:07:44.326825 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325964    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 10:07:44.326825 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325968    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 10:07:44.326825 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325972    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 10:07:44.326825 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325974    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 10:07:44.326825 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325977    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 10:07:44.326825 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325980    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 10:07:44.326825 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325982    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 10:07:44.327330 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325985    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 10:07:44.327330 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325987    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 10:07:44.327330 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325990    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 10:07:44.327330 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325992    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 10:07:44.327330 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325996    2557 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 10:07:44.327330 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.325999    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 10:07:44.327330 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326001    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 10:07:44.327330 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326004    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 10:07:44.327330 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326006    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 10:07:44.327330 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326010    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 10:07:44.327330 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326013    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 10:07:44.327330 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326015    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 10:07:44.327330 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326018    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 10:07:44.327330 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326020    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 10:07:44.327330 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326023    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 10:07:44.327330 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326025    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 10:07:44.327330 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326028    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 10:07:44.327330 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326030    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 10:07:44.327330 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326033    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 10:07:44.327330 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326035    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 10:07:44.327835 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326038    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 10:07:44.327835 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326040    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 10:07:44.327835 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326043    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 10:07:44.327835 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326045    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 10:07:44.327835 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326048    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 10:07:44.327835 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326051    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 10:07:44.327835 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326054    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 10:07:44.327835 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326057    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 10:07:44.327835 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326060    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 10:07:44.327835 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326062    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 10:07:44.327835 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326065    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 10:07:44.327835 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326067    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 10:07:44.327835 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326070    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 10:07:44.327835 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326072    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 10:07:44.327835 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326075    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 10:07:44.327835 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326078    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 10:07:44.327835 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326080    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 10:07:44.327835 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326083    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 10:07:44.327835 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326842    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 10:07:44.327835 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326849    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 10:07:44.328333 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326852    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 10:07:44.328333 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326855    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 10:07:44.328333 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326858    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 10:07:44.328333 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326861    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 10:07:44.328333 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326863    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 10:07:44.328333 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326866    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 10:07:44.328333 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326868    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 10:07:44.328333 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326871    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 10:07:44.328333 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326874    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 10:07:44.328333 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326876    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 10:07:44.328333 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326879    2557 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 10:07:44.328333 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326881    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 10:07:44.328333 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326884    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 10:07:44.328333 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326887    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 10:07:44.328333 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326889    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 10:07:44.328333 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326892    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 10:07:44.328333 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326895    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 10:07:44.328333 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326898    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 10:07:44.328333 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326901    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 10:07:44.328333 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326904    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 10:07:44.328878 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326907    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 10:07:44.328878 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326909    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 10:07:44.328878 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326911    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 10:07:44.328878 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326915    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 10:07:44.328878 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326917    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 10:07:44.328878 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326920    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 10:07:44.328878 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326922    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 10:07:44.328878 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326925    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 10:07:44.328878 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326927    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 10:07:44.328878 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326930    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 10:07:44.328878 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326932    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 10:07:44.328878 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326935    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 10:07:44.328878 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326937    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 10:07:44.328878 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326940    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 10:07:44.328878 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326942    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 10:07:44.328878 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326945    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 10:07:44.328878 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326948    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 10:07:44.328878 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326950    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 10:07:44.328878 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326953    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 10:07:44.328878 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326955    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 10:07:44.329408 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326958    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 10:07:44.329408 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326961    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 10:07:44.329408 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326963    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 10:07:44.329408 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326965    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 10:07:44.329408 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326968    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 10:07:44.329408 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326971    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 10:07:44.329408 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326973    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 10:07:44.329408 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326976    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 10:07:44.329408 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326979    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 10:07:44.329408 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326981    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 10:07:44.329408 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.326998    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 10:07:44.329408 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327001    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 10:07:44.329408 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327004    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 10:07:44.329408 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327006    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 10:07:44.329408 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327009    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 10:07:44.329408 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327012    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 10:07:44.329408 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327014    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 10:07:44.329408 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327017    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 10:07:44.329408 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327019    2557 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 10:07:44.329408 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327022    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 10:07:44.329908 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327024    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 10:07:44.329908 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327027    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 10:07:44.329908 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327029    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 10:07:44.329908 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327032    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 10:07:44.329908 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327034    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 10:07:44.329908 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327037    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 10:07:44.329908 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327039    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 10:07:44.329908 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327042    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 10:07:44.329908 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327045    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 10:07:44.329908 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327048    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 10:07:44.329908 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327050    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 10:07:44.329908 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327053    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 10:07:44.329908 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327055    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 10:07:44.329908 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327058    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 10:07:44.329908 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327060    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 10:07:44.329908 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327063    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 10:07:44.329908 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327068    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 10:07:44.329908 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327071    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 10:07:44.329908 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327074    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 10:07:44.330367 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327077    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 10:07:44.330367 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327080    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 10:07:44.330367 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327084    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 10:07:44.330367 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327088    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 10:07:44.330367 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327090    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 10:07:44.330367 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327093    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 10:07:44.330367 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327097    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 10:07:44.330367 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327100    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 10:07:44.330367 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327103    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 10:07:44.330367 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327106    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 10:07:44.330367 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327109    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 10:07:44.330367 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327112    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 10:07:44.330367 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327115    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 10:07:44.330367 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327117    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 10:07:44.330367 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327120    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 10:07:44.330367 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327123    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 10:07:44.330367 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327125    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 10:07:44.330367 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327128    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 10:07:44.330367 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327577    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 10:07:44.330844 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327583    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 10:07:44.330844 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327586    2557 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 10:07:44.330844 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327589    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 10:07:44.330844 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327592    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 10:07:44.330844 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327595    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 10:07:44.330844 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327614    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 10:07:44.330844 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327618    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 10:07:44.330844 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327620    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 10:07:44.330844 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327623    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 10:07:44.330844 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327626    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 10:07:44.330844 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327629    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 10:07:44.330844 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327632    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 10:07:44.330844 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327635    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 10:07:44.330844 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327637    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 10:07:44.330844 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327641    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 10:07:44.330844 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327645    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 10:07:44.330844 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327648    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 10:07:44.330844 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327651    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 10:07:44.330844 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327654    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 10:07:44.331304 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327657    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 10:07:44.331304 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327659    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 10:07:44.331304 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327662    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 10:07:44.331304 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327664    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 10:07:44.331304 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327667    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 10:07:44.331304 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327669    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 10:07:44.331304 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327672    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 10:07:44.331304 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327675    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 10:07:44.331304 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327677    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 10:07:44.331304 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327680    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 10:07:44.331304 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327683    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 10:07:44.331304 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327686    2557 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 10:07:44.331304 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327688    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 10:07:44.331304 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327691    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 10:07:44.331304 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327693    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 10:07:44.331304 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327696    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 10:07:44.331304 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327699    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 10:07:44.331304 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327701    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 10:07:44.331304 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327704    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 10:07:44.331304 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327707    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 10:07:44.331800 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327709    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 10:07:44.331800 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327712    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 10:07:44.331800 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327714    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 10:07:44.331800 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327717    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 10:07:44.331800 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327720    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 10:07:44.331800 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327722    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 10:07:44.331800 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327725    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 10:07:44.331800 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327728    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 10:07:44.331800 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327730    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 10:07:44.331800 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327733    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 10:07:44.331800 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327735    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 10:07:44.331800 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327738    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 10:07:44.331800 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327740    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 10:07:44.331800 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327743    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 10:07:44.331800 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327745    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 10:07:44.331800 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327748    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 10:07:44.331800 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327752    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 10:07:44.331800 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327755    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 10:07:44.331800 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327758    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 10:07:44.332265 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327761    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 10:07:44.332265 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327764    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 10:07:44.332265 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327766    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 10:07:44.332265 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327769    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 10:07:44.332265 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327772    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 10:07:44.332265 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327775    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 10:07:44.332265 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327777    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 10:07:44.332265 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327780    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 10:07:44.332265 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327783    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 10:07:44.332265 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327785    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 10:07:44.332265 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327788    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 10:07:44.332265 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327791    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 10:07:44.332265 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327794    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 10:07:44.332265 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327796    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 10:07:44.332265 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327798    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 10:07:44.332265 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327801    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 10:07:44.332265 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327803    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 10:07:44.332265 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327806    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 10:07:44.332265 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327811    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 10:07:44.332265 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327813    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 10:07:44.332757 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327817    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 10:07:44.332757 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327819    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 10:07:44.332757 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327824    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 10:07:44.332757 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327827    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 10:07:44.332757 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327830    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 10:07:44.332757 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327833    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 10:07:44.332757 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327835    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 10:07:44.332757 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327839    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 10:07:44.332757 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327841    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 10:07:44.332757 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327844    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 10:07:44.332757 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327847    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 10:07:44.332757 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327849    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 10:07:44.332757 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327852    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 10:07:44.332757 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327854    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 10:07:44.332757 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327857    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 10:07:44.332757 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327860    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 10:07:44.332757 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327862    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 10:07:44.332757 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327865    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 10:07:44.332757 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327867    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 10:07:44.332757 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.327870    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 10:07:44.333239 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.327937    2557 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 10:07:44.333239 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.327945    2557 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 10:07:44.333239 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.327951    2557 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 10:07:44.333239 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.327957    2557 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 10:07:44.333239 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.327961    2557 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 10:07:44.333239 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.327964    2557 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 10:07:44.333239 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.327969    2557 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 10:07:44.333239 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.327973    2557 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 10:07:44.333239 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.327976    2557 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 10:07:44.333239 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.327979    2557 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 10:07:44.333239 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.327984    2557 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 10:07:44.333239 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.327988    2557 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 10:07:44.333239 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.327991    2557 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 10:07:44.333239 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.327994    2557 flags.go:64] FLAG: --cgroup-root=""
Jul 28 10:07:44.333239 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.327998    2557 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 10:07:44.333239 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328001    2557 flags.go:64] FLAG: --client-ca-file=""
Jul 28 10:07:44.333239 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328004    2557 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 10:07:44.333239 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328007    2557 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 10:07:44.333239 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328012    2557 flags.go:64] FLAG: --cluster-domain=""
Jul 28 10:07:44.333239 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328015    2557 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 10:07:44.333239 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328018    2557 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 10:07:44.333239 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328021    2557 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 10:07:44.333239 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328024    2557 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 10:07:44.333239 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328037    2557 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 10:07:44.333830 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328040    2557 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 10:07:44.333830 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328044    2557 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 10:07:44.333830 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328047    2557 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 10:07:44.333830 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328050    2557 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 10:07:44.333830 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328053    2557 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 10:07:44.333830 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328056    2557 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 10:07:44.333830 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328059    2557 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 10:07:44.333830 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328062    2557 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 10:07:44.333830 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328066    2557 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 10:07:44.333830 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328076    2557 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 10:07:44.333830 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328080    2557 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 10:07:44.333830 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328083    2557 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 10:07:44.333830 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328086    2557 flags.go:64] FLAG: --enable-server="true"
Jul 28 10:07:44.333830 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328090    2557 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 10:07:44.333830 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328095    2557 flags.go:64] FLAG: --event-burst="100"
Jul 28 10:07:44.333830 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328098    2557 flags.go:64] FLAG: --event-qps="50"
Jul 28 10:07:44.333830 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328101    2557 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 10:07:44.333830 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328104    2557 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 10:07:44.333830 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328108    2557 flags.go:64] FLAG: --eviction-hard=""
Jul 28 10:07:44.333830 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328120    2557 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 10:07:44.333830 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328123    2557 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 10:07:44.333830 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328126    2557 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 10:07:44.333830 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328130    2557 flags.go:64] FLAG: --eviction-soft=""
Jul 28 10:07:44.333830 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328134    2557 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 10:07:44.333830 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328137    2557 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 10:07:44.334508 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328140    2557 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 10:07:44.334508 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328143    2557 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 10:07:44.334508 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328146    2557 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 10:07:44.334508 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328148    2557 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 10:07:44.334508 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328151    2557 flags.go:64] FLAG: --feature-gates=""
Jul 28 10:07:44.334508 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328160    2557 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 10:07:44.334508 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328163    2557 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 10:07:44.334508 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328166    2557 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 10:07:44.334508 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328170    2557 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 10:07:44.334508 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328173    2557 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 10:07:44.334508 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328176    2557 flags.go:64] FLAG: --help="false"
Jul 28 10:07:44.334508 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328179    2557 flags.go:64] FLAG: --hostname-override="ip-10-0-131-88.ec2.internal"
Jul 28 10:07:44.334508 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328182    2557 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 10:07:44.334508 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328185    2557 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 10:07:44.334508 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328187    2557 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 10:07:44.334508 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328191    2557 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 10:07:44.334508 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328194    2557 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 10:07:44.334508 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328197    2557 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 10:07:44.334508 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328200    2557 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 10:07:44.334508 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328202    2557 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 10:07:44.334508 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328206    2557 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 10:07:44.334508 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328209    2557 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 10:07:44.334508 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328212    2557 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 10:07:44.334508 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328214    2557 flags.go:64] FLAG: --kube-reserved=""
Jul 28 10:07:44.335307 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328217    2557 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 10:07:44.335307 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328221    2557 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 10:07:44.335307 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328224    2557 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 10:07:44.335307 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328227    2557 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 10:07:44.335307 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328230    2557 flags.go:64] FLAG: --lock-file=""
Jul 28 10:07:44.335307 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328234    2557 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 10:07:44.335307 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328237    2557 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 10:07:44.335307 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328240    2557 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 10:07:44.335307 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328245    2557 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 10:07:44.335307 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328248    2557 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 10:07:44.335307 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328251    2557 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 10:07:44.335307 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328254    2557 flags.go:64] FLAG: --logging-format="text"
Jul 28 10:07:44.335307 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328256    2557 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 10:07:44.335307 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328260    2557 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 10:07:44.335307 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328263    2557 flags.go:64] FLAG: --manifest-url=""
Jul 28 10:07:44.335307 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328266    2557 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 10:07:44.335307 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328271    2557 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 10:07:44.335307 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328274    2557 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 10:07:44.335307 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328282    2557 flags.go:64] FLAG: --max-pods="110"
Jul 28 10:07:44.335307 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328285    2557 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 10:07:44.335307 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328288    2557 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 10:07:44.335307 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328291    2557 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 10:07:44.335307 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328293    2557 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 10:07:44.335307 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328296    2557 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 10:07:44.335307 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328300    2557 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 10:07:44.336306 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328303    2557 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 10:07:44.336306 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328310    2557 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 10:07:44.336306 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328313    2557 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 10:07:44.336306 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328315    2557 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 10:07:44.336306 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328319    2557 flags.go:64] FLAG: --pod-cidr=""
Jul 28 10:07:44.336306 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328322    2557 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 10:07:44.336306 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328327    2557 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 10:07:44.336306 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328330    2557 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 10:07:44.336306 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328334    2557 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 10:07:44.336306 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328337    2557 flags.go:64] FLAG: --port="10250"
Jul 28 10:07:44.336306 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328341    2557 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 10:07:44.336306 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328344    2557 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ebd8824d2f8a4750"
Jul 28 10:07:44.336306 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328347    2557 flags.go:64] FLAG: --qos-reserved=""
Jul 28 10:07:44.336306 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328351    2557 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 10:07:44.336306 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328354    2557 flags.go:64] FLAG: --register-node="true"
Jul 28 10:07:44.336306 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328357    2557 flags.go:64] FLAG: --register-with-taints=""
Jul 28 10:07:44.336306 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328361    2557 flags.go:64] FLAG: --registry-burst="10"
Jul 28 10:07:44.336306 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328364    2557 flags.go:64] FLAG: --registry-qps="5"
Jul 28 10:07:44.336306 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328366    2557 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 10:07:44.336306 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328369    2557 flags.go:64] FLAG: --reserved-memory=""
Jul 28 10:07:44.336306 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328375    2557 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 10:07:44.336306 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328378    2557 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 10:07:44.336306 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328381    2557 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 10:07:44.336306 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328384    2557 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 10:07:44.336306 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328387    2557 flags.go:64] FLAG: --runonce="false"
Jul 28 10:07:44.337025 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328390    2557 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 10:07:44.337025 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328393    2557 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 10:07:44.337025 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328395    2557 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 10:07:44.337025 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328398    2557 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 10:07:44.337025 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328401    2557 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 10:07:44.337025 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328404    2557 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 10:07:44.337025 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328407    2557 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 10:07:44.337025 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328410    2557 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 10:07:44.337025 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328413    2557 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 10:07:44.337025 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328415    2557 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 10:07:44.337025 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328418    2557 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 10:07:44.337025 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328421    2557 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 10:07:44.337025 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328424    2557 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 10:07:44.337025 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328427    2557 flags.go:64] FLAG: --system-cgroups=""
Jul 28 10:07:44.337025 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328430    2557 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 10:07:44.337025 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328437    2557 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 10:07:44.337025 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328440    2557 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 10:07:44.337025 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328442    2557 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 10:07:44.337025 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328446    2557 flags.go:64] FLAG: --tls-min-version=""
Jul 28 10:07:44.337025 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328449    2557 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 10:07:44.337025 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328453    2557 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 10:07:44.337025 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328456    2557 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 10:07:44.337025 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328459    2557 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 10:07:44.337025 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328462    2557 flags.go:64] FLAG: --v="2"
Jul 28 10:07:44.337025 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328466    2557 flags.go:64] FLAG: --version="false"
Jul 28 10:07:44.337650 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328470    2557 flags.go:64] FLAG: --vmodule=""
Jul 28 10:07:44.337650 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328474    2557 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 10:07:44.337650 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328477    2557 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 10:07:44.337650 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328594    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 10:07:44.337650 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328610    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 10:07:44.337650 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328613    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 10:07:44.337650 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328616    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 10:07:44.337650 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328620    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 10:07:44.337650 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328623    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 10:07:44.337650 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328626    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 10:07:44.337650 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328629    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 10:07:44.337650 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328632    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 10:07:44.337650 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328635    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 10:07:44.337650 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328637    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 10:07:44.337650 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328640    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 10:07:44.337650 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328643    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 10:07:44.337650 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328646    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 10:07:44.337650 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328648    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 10:07:44.337650 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328651    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 10:07:44.337650 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328653    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 10:07:44.338141 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328656    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 10:07:44.338141 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328659    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 10:07:44.338141 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328663    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 10:07:44.338141 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328666    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 10:07:44.338141 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328668    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 10:07:44.338141 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328671    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 10:07:44.338141 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328673    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 10:07:44.338141 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328677    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 10:07:44.338141 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328679    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 10:07:44.338141 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328682    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 10:07:44.338141 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328684    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 10:07:44.338141 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328687    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 10:07:44.338141 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328690    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 10:07:44.338141 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328693    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 10:07:44.338141 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328695    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 10:07:44.338141 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328698    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 10:07:44.338141 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328700    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 10:07:44.338141 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328703    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 10:07:44.338141 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328705    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 10:07:44.338593 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328708    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 10:07:44.338593 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328711    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 10:07:44.338593 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328713    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 10:07:44.338593 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328715    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 10:07:44.338593 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328718    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 10:07:44.338593 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328720    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 10:07:44.338593 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328723    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 10:07:44.338593 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328727    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 10:07:44.338593 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328730    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 10:07:44.338593 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328733    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 10:07:44.338593 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328735    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 10:07:44.338593 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328738    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 10:07:44.338593 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328740    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 10:07:44.338593 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328743    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 10:07:44.338593 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328746    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 10:07:44.338593 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328750    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 10:07:44.338593 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328752    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 10:07:44.338593 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328754    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 10:07:44.338593 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328757    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 10:07:44.339110 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328760    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 10:07:44.339110 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328764    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 10:07:44.339110 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328767    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 10:07:44.339110 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328770    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 10:07:44.339110 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328772    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 10:07:44.339110 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328775    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 10:07:44.339110 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328777    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 10:07:44.339110 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328780    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 10:07:44.339110 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328782    2557 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 10:07:44.339110 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328785    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 10:07:44.339110 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328787    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 10:07:44.339110 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328790    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 10:07:44.339110 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328792    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 10:07:44.339110 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328795    2557 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 10:07:44.339110 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328798    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 10:07:44.339110 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328800    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 10:07:44.339110 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328803    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 10:07:44.339110 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328805    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 10:07:44.339110 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328808    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 10:07:44.339110 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328810    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 10:07:44.339110 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328813    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 10:07:44.339652 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328815    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 10:07:44.339652 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328818    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 10:07:44.339652 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328820    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 10:07:44.339652 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328822    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 10:07:44.339652 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328825    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 10:07:44.339652 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328827    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 10:07:44.339652 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328830    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 10:07:44.339652 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328834    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 10:07:44.339652 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328837    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 10:07:44.339652 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328839    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 10:07:44.339652 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328842    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 10:07:44.339652 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328844    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 10:07:44.339652 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328848    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 10:07:44.339652 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328851    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 10:07:44.339652 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328853    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 10:07:44.339652 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328856    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 10:07:44.339652 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328858    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 10:07:44.339652 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328861    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 10:07:44.339652 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328864    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 10:07:44.339652 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328868    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 10:07:44.340210 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328871    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 10:07:44.340210 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328873    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 10:07:44.340210 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.328877    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 10:07:44.340210 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.328883    2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 10:07:44.340210 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.336618    2557 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 10:07:44.340210 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.336634    2557 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 10:07:44.340210 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.336654    2557 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 10:07:44.340210 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.336660    2557 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 10:07:44.340210 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336717    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 10:07:44.340210 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336723    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 10:07:44.340210 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336726    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 10:07:44.340210 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336731    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 10:07:44.340210 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336735    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 10:07:44.340210 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336738    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 10:07:44.340562 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336741    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 10:07:44.340562 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336744    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 10:07:44.340562 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336747    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 10:07:44.340562 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336750    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 10:07:44.340562 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336752    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 10:07:44.340562 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336755    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 10:07:44.340562 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336758    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 10:07:44.340562 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336761    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 10:07:44.340562 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336764    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 10:07:44.340562 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336766    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 10:07:44.340562 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336769    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 10:07:44.340562 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336773    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 10:07:44.340562 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336776    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 10:07:44.340562 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336780    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 10:07:44.340562 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336784    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 10:07:44.340562 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336787    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 10:07:44.340562 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336790    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 10:07:44.340562 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336793    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 10:07:44.340562 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336796    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 10:07:44.341060 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336799    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 10:07:44.341060 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336801    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 10:07:44.341060 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336804    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 10:07:44.341060 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336807    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 10:07:44.341060 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336811    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 10:07:44.341060 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336814    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 10:07:44.341060 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336816    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 10:07:44.341060 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336819    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 10:07:44.341060 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336822    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 10:07:44.341060 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336824    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 10:07:44.341060 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336827    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 10:07:44.341060 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336829    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 10:07:44.341060 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336832    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 10:07:44.341060 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336834    2557 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 10:07:44.341060 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336837    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 10:07:44.341060 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336839    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 10:07:44.341060 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336842    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 10:07:44.341060 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336845    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 10:07:44.341060 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336847    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 10:07:44.341060 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336850    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 10:07:44.341544 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336852    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 10:07:44.341544 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336855    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 10:07:44.341544 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336857    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 10:07:44.341544 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336860    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 10:07:44.341544 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336862    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 10:07:44.341544 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336865    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 10:07:44.341544 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336868    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 10:07:44.341544 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336871    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 10:07:44.341544 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336873    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 10:07:44.341544 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336876    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 10:07:44.341544 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336879    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 10:07:44.341544 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336882    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 10:07:44.341544 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336884    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 10:07:44.341544 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336887    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 10:07:44.341544 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336890    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 10:07:44.341544 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336892    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 10:07:44.341544 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336895    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 10:07:44.341544 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336898    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 10:07:44.341544 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336901    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 10:07:44.342154 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336903    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 10:07:44.342154 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336906    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 10:07:44.342154 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336909    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 10:07:44.342154 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336911    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 10:07:44.342154 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336914    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 10:07:44.342154 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336917    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 10:07:44.342154 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336919    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 10:07:44.342154 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336922    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 10:07:44.342154 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336925    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 10:07:44.342154 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336927    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 10:07:44.342154 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336929    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 10:07:44.342154 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336932    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 10:07:44.342154 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336935    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 10:07:44.342154 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336937    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 10:07:44.342154 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336940    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 10:07:44.342154 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336943    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 10:07:44.342154 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336946    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 10:07:44.342154 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336948    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 10:07:44.342154 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336951    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 10:07:44.342154 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336953    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 10:07:44.342693 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336956    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 10:07:44.342693 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336958    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 10:07:44.342693 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336961    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 10:07:44.342693 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336963    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 10:07:44.342693 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336966    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 10:07:44.342693 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336968    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 10:07:44.342693 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336971    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 10:07:44.342693 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336973    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 10:07:44.342693 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336976    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 10:07:44.342693 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336979    2557 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 10:07:44.342693 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336982    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 10:07:44.342693 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336985    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 10:07:44.342693 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336987    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 10:07:44.342693 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336990    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 10:07:44.342693 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.336992    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 10:07:44.343080 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.336998    2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 10:07:44.343080 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337140    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 10:07:44.343080 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337146    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 10:07:44.343080 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337149    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 10:07:44.343080 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337152    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 10:07:44.343080 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337155    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 10:07:44.343080 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337158    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 10:07:44.343080 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337161    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 10:07:44.343080 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337164    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 10:07:44.343080 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337166    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 10:07:44.343080 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337169    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 10:07:44.343080 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337172    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 10:07:44.343080 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337175    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 10:07:44.343080 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337178    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 10:07:44.343080 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337181    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 10:07:44.343456 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337183    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 10:07:44.343456 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337186    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 10:07:44.343456 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337189    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 10:07:44.343456 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337191    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 10:07:44.343456 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337194    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 10:07:44.343456 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337196    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 10:07:44.343456 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337199    2557 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 10:07:44.343456 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337201    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 10:07:44.343456 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337204    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 10:07:44.343456 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337207    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 10:07:44.343456 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337209    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 10:07:44.343456 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337212    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 10:07:44.343456 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337215    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 10:07:44.343456 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337218    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 10:07:44.343456 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337221    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 10:07:44.343456 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337224    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 10:07:44.343456 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337226    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 10:07:44.343456 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337229    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 10:07:44.343456 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337231    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 10:07:44.343456 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337234    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 10:07:44.343953 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337237    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 10:07:44.343953 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337239    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 10:07:44.343953 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337242    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 10:07:44.343953 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337244    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 10:07:44.343953 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337248    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 10:07:44.343953 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337252    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 10:07:44.343953 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337255    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 10:07:44.343953 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337258    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 10:07:44.343953 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337260    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 10:07:44.343953 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337264    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 10:07:44.343953 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337267    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 10:07:44.343953 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337271    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 10:07:44.343953 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337273    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 10:07:44.343953 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337276    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 10:07:44.343953 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337278    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 10:07:44.343953 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337281    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 10:07:44.343953 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337283    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 10:07:44.343953 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337286    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 10:07:44.343953 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337288    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 10:07:44.344431 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337291    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 10:07:44.344431 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337293    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 10:07:44.344431 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337296    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 10:07:44.344431 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337298    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 10:07:44.344431 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337301    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 10:07:44.344431 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337304    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 10:07:44.344431 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337307    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 10:07:44.344431 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337309    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 10:07:44.344431 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337312    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 10:07:44.344431 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337315    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 10:07:44.344431 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337317    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 10:07:44.344431 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337320    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 10:07:44.344431 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337322    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 10:07:44.344431 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337325    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 10:07:44.344431 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337327    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 10:07:44.344431 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337330    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 10:07:44.344431 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337332    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 10:07:44.344431 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337334    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 10:07:44.344431 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337337    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 10:07:44.344431 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337340    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 10:07:44.345052 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337342    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 10:07:44.345052 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337345    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 10:07:44.345052 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337347    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 10:07:44.345052 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337350    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 10:07:44.345052 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337352    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 10:07:44.345052 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337355    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 10:07:44.345052 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337358    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 10:07:44.345052 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337360    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 10:07:44.345052 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337363    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 10:07:44.345052 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337365    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 10:07:44.345052 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337368    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 10:07:44.345052 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337370    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 10:07:44.345052 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337373    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 10:07:44.345052 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337377    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 10:07:44.345052 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337380    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 10:07:44.345052 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337383    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 10:07:44.345052 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337386    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 10:07:44.345052 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337389    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 10:07:44.345052 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337392    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 10:07:44.345542 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337394    2557 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 10:07:44.345542 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337397    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 10:07:44.345542 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337399    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 10:07:44.345542 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337402    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 10:07:44.345542 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337404    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 10:07:44.345542 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337407    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 10:07:44.345542 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:44.337410    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 10:07:44.345542 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.337414    2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 10:07:44.345542 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.337525    2557 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 10:07:44.345542 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.339548    2557 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 10:07:44.345542 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.340726    2557 server.go:1019] "Starting client certificate rotation"
Jul 28 10:07:44.345542 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.340825    2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 10:07:44.345542 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.340860    2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 10:07:44.345932 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.343003    2557 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 10:07:44.345932 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.345841    2557 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 10:07:44.354538 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.354520    2557 log.go:25] "Validated CRI v1 runtime API"
Jul 28 10:07:44.358358 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.358332    2557 log.go:25] "Validated CRI v1 image API"
Jul 28 10:07:44.359182 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.359167    2557 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 10:07:44.361646 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.361624    2557 fs.go:135] Filesystem UUIDs: map[05164aa0-5976-4d15-815d-2d8ba88987eb:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 ef47b067-8992-40a7-8f29-df18060b6d6f:/dev/nvme0n1p4]
Jul 28 10:07:44.361707 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.361646    2557 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 10:07:44.368937 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.368787    2557 manager.go:216] Machine: {Timestamp:2026-07-28 10:07:44.367430933 +0000 UTC m=+0.326199306 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100736 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27e94cf58659894d8bea6060661abe SystemUUID:ec27e94c-f586-5989-4d8b-ea6060661abe BootID:a36fb1dd-3189-452a-af18-fdd49782428f Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:1e:0a:1f:eb:65 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:1e:0a:1f:eb:65 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9e:e9:d7:b5:cf:3f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 10:07:44.368937 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.368932    2557 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 10:07:44.369074 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.369025    2557 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 10:07:44.370609 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.370574    2557 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 10:07:44.370792 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.370620    2557 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-88.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 10:07:44.371433 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.371415    2557 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 10:07:44.371433 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.371433    2557 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 10:07:44.371558 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.371444    2557 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 10:07:44.371558 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.371533    2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 10:07:44.373348 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.373262    2557 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 10:07:44.374321 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.374295    2557 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 10:07:44.376435 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.376418    2557 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 10:07:44.376591 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.376579    2557 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 10:07:44.376654 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.376622    2557 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 10:07:44.376654 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.376638    2557 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 10:07:44.376779 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.376676    2557 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 10:07:44.376779 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.376690    2557 kubelet.go:398] "Adding apiserver pod source"
Jul 28 10:07:44.376779 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.376706    2557 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 10:07:44.377136 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.377112    2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 10:07:44.377136 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.377137    2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 10:07:44.379491 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.379472    2557 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 10:07:44.381560 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.381545    2557 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 10:07:44.381644 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.381579    2557 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 10:07:44.383091 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.383079    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 10:07:44.383145 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.383096    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 10:07:44.383145 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.383104    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 10:07:44.383145 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.383110    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 10:07:44.383145 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.383121    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 10:07:44.383145 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.383127    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 10:07:44.383145 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.383135    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 10:07:44.383145 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.383141    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 10:07:44.383145 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.383148    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 10:07:44.383376 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.383154    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 10:07:44.383376 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.383163    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 10:07:44.383376 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.383168    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 10:07:44.384029 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.384019    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 10:07:44.384069 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.384031    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 10:07:44.387498 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.387483    2557 server.go:1268] "Started kubelet"
Jul 28 10:07:44.387644 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.387592    2557 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 10:07:44.387794 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.387664    2557 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 10:07:44.388212 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.388187    2557 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 10:07:44.388435 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:44.388404    2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 10:07:44.388521 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.388487    2557 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-88.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 10:07:44.388590 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:44.388559    2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-88.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 10:07:44.388643 ip-10-0-131-88 systemd[1]: Started Kubernetes Kubelet.
Jul 28 10:07:44.389904 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.389885    2557 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 10:07:44.390693 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.390662    2557 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 10:07:44.393799 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:44.393772    2557 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 10:07:44.396847 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.396823    2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lmsrd"
Jul 28 10:07:44.397236 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.397211    2557 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 10:07:44.397236 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.397228    2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 10:07:44.397379 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:44.397347    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-88.ec2.internal\" not found"
Jul 28 10:07:44.397816 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:44.396605    2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-88.ec2.internal.18c66c3205067258  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-88.ec2.internal,UID:ip-10-0-131-88.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-88.ec2.internal,},FirstTimestamp:2026-07-28 10:07:44.387453528 +0000 UTC m=+0.346221900,LastTimestamp:2026-07-28 10:07:44.387453528 +0000 UTC m=+0.346221900,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-88.ec2.internal,}"
Jul 28 10:07:44.397967 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.397937    2557 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 10:07:44.397967 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.397956    2557 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 10:07:44.398154 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.398138    2557 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 10:07:44.398154 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.398152    2557 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 10:07:44.398274 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.398207    2557 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 10:07:44.398385 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.398365    2557 factory.go:55] Registering systemd factory
Jul 28 10:07:44.398473 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.398409    2557 factory.go:223] Registration of the systemd container factory successfully
Jul 28 10:07:44.398708 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.398689    2557 factory.go:153] Registering CRI-O factory
Jul 28 10:07:44.398708 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.398710    2557 factory.go:223] Registration of the crio container factory successfully
Jul 28 10:07:44.398870 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.398767    2557 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 10:07:44.398870 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.398796    2557 factory.go:103] Registering Raw factory
Jul 28 10:07:44.398870 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.398812    2557 manager.go:1196] Started watching for new ooms in manager
Jul 28 10:07:44.400040 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.400020    2557 manager.go:349] Starting recovery of all containers
Jul 28 10:07:44.400968 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:44.400938    2557 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-88.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 10:07:44.401055 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:44.401003    2557 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 10:07:44.405125 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.405095    2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lmsrd"
Jul 28 10:07:44.413010 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.412848    2557 manager.go:354] Recovery completed
Jul 28 10:07:44.417290 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.417272    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 10:07:44.419287 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.419270    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 10:07:44.419367 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.419305    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 10:07:44.419367 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.419322    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-88.ec2.internal" event="NodeHasSufficientPID"
Jul 28 10:07:44.419971 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.419955    2557 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 10:07:44.419971 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.419968    2557 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 10:07:44.419971 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.419973    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 10:07:44.420113 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.419984    2557 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 10:07:44.420731 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.420713    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 10:07:44.420786 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.420750    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 10:07:44.420786 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.420770    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-88.ec2.internal" event="NodeHasSufficientPID"
Jul 28 10:07:44.421914 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:44.421795    2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-88.ec2.internal.18c66c3206ec2f6f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-88.ec2.internal,UID:ip-10-0-131-88.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-88.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-88.ec2.internal,},FirstTimestamp:2026-07-28 10:07:44.419286895 +0000 UTC m=+0.378055280,LastTimestamp:2026-07-28 10:07:44.419286895 +0000 UTC m=+0.378055280,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-88.ec2.internal,}"
Jul 28 10:07:44.422003 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.421932    2557 policy_none.go:49] "None policy: Start"
Jul 28 10:07:44.422003 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.421948    2557 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 10:07:44.422003 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.421963    2557 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 10:07:44.423307 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.423294    2557 policy_none.go:47] "Start"
Jul 28 10:07:44.479396 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.479369    2557 manager.go:337] "Starting Device Plugin manager"
Jul 28 10:07:44.479559 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:44.479418    2557 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 10:07:44.479559 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.479432    2557 server.go:87] "Starting device plugin registration server"
Jul 28 10:07:44.479798 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.479782    2557 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 10:07:44.479864 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.479799    2557 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 10:07:44.479914 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.479876    2557 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 10:07:44.479990 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.479970    2557 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 10:07:44.479990 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.479984    2557 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 10:07:44.480680 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:44.480653    2557 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 10:07:44.480794 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:44.480711    2557 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-88.ec2.internal\" not found"
Jul 28 10:07:44.572331 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.572244    2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 10:07:44.573629 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.573598    2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 10:07:44.573698 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.573652    2557 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 10:07:44.573698 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.573678    2557 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 10:07:44.573787 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:44.573721    2557 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 10:07:44.576591 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.576570    2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 10:07:44.580695 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.580669    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 10:07:44.584541 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.584522    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 10:07:44.584649 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.584556    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 10:07:44.584649 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.584568    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-88.ec2.internal" event="NodeHasSufficientPID"
Jul 28 10:07:44.584649 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.584597    2557 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-88.ec2.internal"
Jul 28 10:07:44.591855 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.591839    2557 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-88.ec2.internal"
Jul 28 10:07:44.591913 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:44.591864    2557 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-88.ec2.internal\": node \"ip-10-0-131-88.ec2.internal\" not found"
Jul 28 10:07:44.611712 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:44.611688    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-88.ec2.internal\" not found"
Jul 28 10:07:44.674894 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.674110    2557 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-131-88.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-88.ec2.internal"]
Jul 28 10:07:44.675013 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.674946    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 10:07:44.676819 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.676800    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 10:07:44.676902 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.676840    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 10:07:44.676902 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.676856    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-88.ec2.internal" event="NodeHasSufficientPID"
Jul 28 10:07:44.678266 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.678249    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 10:07:44.678368 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.678353    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-88.ec2.internal"
Jul 28 10:07:44.678429 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.678383    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 10:07:44.679073 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.679054    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 10:07:44.679150 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.679087    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 10:07:44.679150 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.679098    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-88.ec2.internal" event="NodeHasSufficientPID"
Jul 28 10:07:44.679150 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.679054    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 10:07:44.679272 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.679170    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 10:07:44.679272 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.679184    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-88.ec2.internal" event="NodeHasSufficientPID"
Jul 28 10:07:44.680367 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.680353    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-88.ec2.internal"
Jul 28 10:07:44.680426 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.680379    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 10:07:44.681848 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.681828    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 10:07:44.681922 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.681864    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 10:07:44.681922 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.681881    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-88.ec2.internal" event="NodeHasSufficientPID"
Jul 28 10:07:44.695344 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:44.695319    2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-88.ec2.internal\" not found" node="ip-10-0-131-88.ec2.internal"
Jul 28 10:07:44.699569 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.699546    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae61811891b0aefce4345602b66dee13-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-88.ec2.internal\" (UID: \"ae61811891b0aefce4345602b66dee13\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-88.ec2.internal"
Jul 28 10:07:44.699691 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.699574    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6ff9d18249fb2db5835b1f8658874702-config\") pod \"kube-apiserver-proxy-ip-10-0-131-88.ec2.internal\" (UID: \"6ff9d18249fb2db5835b1f8658874702\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-88.ec2.internal"
Jul 28 10:07:44.699691 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:44.699577    2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-88.ec2.internal\" not found" node="ip-10-0-131-88.ec2.internal"
Jul 28 10:07:44.699691 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.699591    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ae61811891b0aefce4345602b66dee13-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-88.ec2.internal\" (UID: \"ae61811891b0aefce4345602b66dee13\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-88.ec2.internal"
Jul 28 10:07:44.712853 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:44.712810    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-88.ec2.internal\" not found"
Jul 28 10:07:44.800684 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.800651    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6ff9d18249fb2db5835b1f8658874702-config\") pod \"kube-apiserver-proxy-ip-10-0-131-88.ec2.internal\" (UID: \"6ff9d18249fb2db5835b1f8658874702\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-88.ec2.internal"
Jul 28 10:07:44.800684 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.800682    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ae61811891b0aefce4345602b66dee13-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-88.ec2.internal\" (UID: \"ae61811891b0aefce4345602b66dee13\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-88.ec2.internal"
Jul 28 10:07:44.800889 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.800701    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae61811891b0aefce4345602b66dee13-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-88.ec2.internal\" (UID: \"ae61811891b0aefce4345602b66dee13\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-88.ec2.internal"
Jul 28 10:07:44.800889 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.800733    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae61811891b0aefce4345602b66dee13-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-88.ec2.internal\" (UID: \"ae61811891b0aefce4345602b66dee13\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-88.ec2.internal"
Jul 28 10:07:44.800889 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.800759    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6ff9d18249fb2db5835b1f8658874702-config\") pod \"kube-apiserver-proxy-ip-10-0-131-88.ec2.internal\" (UID: \"6ff9d18249fb2db5835b1f8658874702\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-88.ec2.internal"
Jul 28 10:07:44.800889 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:44.800759    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ae61811891b0aefce4345602b66dee13-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-88.ec2.internal\" (UID: \"ae61811891b0aefce4345602b66dee13\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-88.ec2.internal"
Jul 28 10:07:44.813681 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:44.813650    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-88.ec2.internal\" not found"
Jul 28 10:07:44.913947 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:44.913863    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-88.ec2.internal\" not found"
Jul 28 10:07:45.003206 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:45.003159    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-88.ec2.internal"
Jul 28 10:07:45.003319 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:45.003166    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-88.ec2.internal"
Jul 28 10:07:45.014233 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:45.014205    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-88.ec2.internal\" not found"
Jul 28 10:07:45.114910 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:45.114861    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-88.ec2.internal\" not found"
Jul 28 10:07:45.210659 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:45.210562    2557 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 10:07:45.215151 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:45.215129    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-88.ec2.internal\" not found"
Jul 28 10:07:45.315432 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:45.315391    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-88.ec2.internal\" not found"
Jul 28 10:07:45.340702 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:45.340677    2557 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 10:07:45.341363 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:45.340861    2557 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 10:07:45.341363 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:45.340900    2557 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 10:07:45.412137 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:45.412089    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 10:02:45 +0000 UTC" deadline="2028-04-10 23:04:20.81833712 +0000 UTC"
Jul 28 10:07:45.412137 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:45.412132    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14940h56m35.406209495s"
Jul 28 10:07:45.416342 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:45.416315    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-88.ec2.internal\" not found"
Jul 28 10:07:45.517098 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:45.517072    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-88.ec2.internal\" not found"
Jul 28 10:07:45.594515 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:45.594484    2557 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 10:07:45.617471 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:45.617435    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-88.ec2.internal\" not found"
Jul 28 10:07:45.646457 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:45.646412    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ff9d18249fb2db5835b1f8658874702.slice/crio-14960905c879d472cc750540742c5aa567033a8be31131794e8cb33148896a1b WatchSource:0}: Error finding container 14960905c879d472cc750540742c5aa567033a8be31131794e8cb33148896a1b: Status 404 returned error can't find the container with id 14960905c879d472cc750540742c5aa567033a8be31131794e8cb33148896a1b
Jul 28 10:07:45.646644 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:45.646628    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae61811891b0aefce4345602b66dee13.slice/crio-61f3947314d17a187c67ba33f68b313efbd29431803ab44b29b4f97d77a483b4 WatchSource:0}: Error finding container 61f3947314d17a187c67ba33f68b313efbd29431803ab44b29b4f97d77a483b4: Status 404 returned error can't find the container with id 61f3947314d17a187c67ba33f68b313efbd29431803ab44b29b4f97d77a483b4
Jul 28 10:07:45.651222 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:45.651171    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 10:07:45.670377 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:45.670346    2557 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 10:07:45.699065 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:45.699020    2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-88.ec2.internal"
Jul 28 10:07:45.711024 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:45.710995    2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 10:07:45.712191 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:45.712176    2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-88.ec2.internal"
Jul 28 10:07:45.728666 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:45.728634    2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 10:07:46.379536 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.379264    2557 apiserver.go:52] "Watching apiserver"
Jul 28 10:07:46.387917 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.386847    2557 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 10:07:46.387917 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.387427    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-jhkxw","kube-system/kube-apiserver-proxy-ip-10-0-131-88.ec2.internal","openshift-cluster-node-tuning-operator/tuned-msf7n","openshift-image-registry/node-ca-h8krb","openshift-multus/multus-additional-cni-plugins-gzgcl","openshift-network-diagnostics/network-check-target-ddrhj","openshift-network-operator/iptables-alerter-vgktv","openshift-ovn-kubernetes/ovnkube-node-hdqls","kube-system/konnectivity-agent-v7gsk","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hh49h","openshift-dns/node-resolver-5z8xh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-88.ec2.internal","openshift-multus/multus-p2rv8","openshift-multus/network-metrics-daemon-8fltb"]
Jul 28 10:07:46.389292 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.389236    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vgktv"
Jul 28 10:07:46.391801 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.391056    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-p2rv8"
Jul 28 10:07:46.395031 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.391913    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 10:07:46.395031 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.392169    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 10:07:46.395031 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.392344    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-p92mv\""
Jul 28 10:07:46.395031 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.393146    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 10:07:46.395031 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.393428    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 10:07:46.397720 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.395446    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 10:07:46.397720 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.395988    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-q7gqj\""
Jul 28 10:07:46.397720 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.396017    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 10:07:46.397720 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.397617    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 10:07:46.399934 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.398173    2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 10:07:46.399934 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.399319    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-msf7n"
Jul 28 10:07:46.405249 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.401879    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-h8krb"
Jul 28 10:07:46.405249 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.403715    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gzgcl"
Jul 28 10:07:46.405249 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.404568    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 10:07:46.405249 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.404679    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xxcmz\""
Jul 28 10:07:46.405249 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.404927    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 10:07:46.405249 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.404984    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 10:07:46.405249 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.404993    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 10:07:46.405249 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.404931    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dfdvg\""
Jul 28 10:07:46.405249 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.405227    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 10:07:46.405899 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.405826    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddrhj"
Jul 28 10:07:46.405961 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:46.405918    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddrhj" podUID="0bc74d3d-3abf-47f5-8149-7d07a9a34e7c"
Jul 28 10:07:46.406012 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.405959    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 10:07:46.406012 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.405961    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rcwzq\""
Jul 28 10:07:46.410220 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.406540    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 10:07:46.410220 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.407477    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v7gsk"
Jul 28 10:07:46.410220 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.409507    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jhkxw"
Jul 28 10:07:46.410220 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:46.409576    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jhkxw" podUID="ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb"
Jul 28 10:07:46.410654 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.410637    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8fltb"
Jul 28 10:07:46.410812 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:46.410703    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8fltb" podUID="d23b40b0-9299-4a0b-a87c-d2f2cb921fd9"
Jul 28 10:07:46.413245 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.412309    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hh49h"
Jul 28 10:07:46.413245 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.412347    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2c5a2bd4-8e86-493f-9479-1a183cc866e6-etc-sysctl-d\") pod \"tuned-msf7n\" (UID: \"2c5a2bd4-8e86-493f-9479-1a183cc866e6\") " pod="openshift-cluster-node-tuning-operator/tuned-msf7n"
Jul 28 10:07:46.413245 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.412384    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a1a543f9-5413-4260-b48d-f767a66f23c7-host\") pod \"node-ca-h8krb\" (UID: \"a1a543f9-5413-4260-b48d-f767a66f23c7\") " pod="openshift-image-registry/node-ca-h8krb"
Jul 28 10:07:46.413245 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.412412    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/58c000f3-06db-4572-8abb-89ca11078c3c-iptables-alerter-script\") pod \"iptables-alerter-vgktv\" (UID: \"58c000f3-06db-4572-8abb-89ca11078c3c\") " pod="openshift-network-operator/iptables-alerter-vgktv"
Jul 28 10:07:46.413245 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.412430    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4e80f436-0cf4-45a3-8fdf-53bdae6a7fad-host-run-netns\") pod \"multus-p2rv8\" (UID: \"4e80f436-0cf4-45a3-8fdf-53bdae6a7fad\") " pod="openshift-multus/multus-p2rv8"
Jul 28 10:07:46.413245 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.412446    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4e80f436-0cf4-45a3-8fdf-53bdae6a7fad-host-var-lib-cni-multus\") pod \"multus-p2rv8\" (UID: \"4e80f436-0cf4-45a3-8fdf-53bdae6a7fad\") " pod="openshift-multus/multus-p2rv8"
Jul 28 10:07:46.413245 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.412466    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2c5a2bd4-8e86-493f-9479-1a183cc866e6-etc-sysctl-conf\") pod \"tuned-msf7n\" (UID: \"2c5a2bd4-8e86-493f-9479-1a183cc866e6\") " pod="openshift-cluster-node-tuning-operator/tuned-msf7n"
Jul 28 10:07:46.413245 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.412485    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2c5a2bd4-8e86-493f-9479-1a183cc866e6-tmp\") pod \"tuned-msf7n\" (UID: \"2c5a2bd4-8e86-493f-9479-1a183cc866e6\") " pod="openshift-cluster-node-tuning-operator/tuned-msf7n"
Jul 28 10:07:46.413245 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.412501    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e80f436-0cf4-45a3-8fdf-53bdae6a7fad-system-cni-dir\") pod \"multus-p2rv8\" (UID: \"4e80f436-0cf4-45a3-8fdf-53bdae6a7fad\") " pod="openshift-multus/multus-p2rv8"
Jul 28 10:07:46.413245 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.412519    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4e80f436-0cf4-45a3-8fdf-53bdae6a7fad-multus-daemon-config\") pod \"multus-p2rv8\" (UID: \"4e80f436-0cf4-45a3-8fdf-53bdae6a7fad\") " pod="openshift-multus/multus-p2rv8"
Jul 28 10:07:46.413245 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.412548    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2c5a2bd4-8e86-493f-9479-1a183cc866e6-host\") pod \"tuned-msf7n\" (UID: \"2c5a2bd4-8e86-493f-9479-1a183cc866e6\") " pod="openshift-cluster-node-tuning-operator/tuned-msf7n"
Jul 28 10:07:46.413245 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.412567    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2c5a2bd4-8e86-493f-9479-1a183cc866e6-etc-tuned\") pod \"tuned-msf7n\" (UID: \"2c5a2bd4-8e86-493f-9479-1a183cc866e6\") " pod="openshift-cluster-node-tuning-operator/tuned-msf7n"
Jul 28 10:07:46.413245 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.412582    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/58c000f3-06db-4572-8abb-89ca11078c3c-host-slash\") pod \"iptables-alerter-vgktv\" (UID: \"58c000f3-06db-4572-8abb-89ca11078c3c\") " pod="openshift-network-operator/iptables-alerter-vgktv"
Jul 28 10:07:46.413245 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.412597    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4e80f436-0cf4-45a3-8fdf-53bdae6a7fad-host-var-lib-cni-bin\") pod \"multus-p2rv8\" (UID: \"4e80f436-0cf4-45a3-8fdf-53bdae6a7fad\") " pod="openshift-multus/multus-p2rv8"
Jul 28 10:07:46.413245 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.412627    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4e80f436-0cf4-45a3-8fdf-53bdae6a7fad-hostroot\") pod \"multus-p2rv8\" (UID: \"4e80f436-0cf4-45a3-8fdf-53bdae6a7fad\") " pod="openshift-multus/multus-p2rv8"
Jul 28 10:07:46.413245 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.412653    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4e80f436-0cf4-45a3-8fdf-53bdae6a7fad-host-run-multus-certs\") pod \"multus-p2rv8\" (UID: \"4e80f436-0cf4-45a3-8fdf-53bdae6a7fad\") " pod="openshift-multus/multus-p2rv8"
Jul 28 10:07:46.413245 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.412696    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2c5a2bd4-8e86-493f-9479-1a183cc866e6-lib-modules\") pod \"tuned-msf7n\" (UID: \"2c5a2bd4-8e86-493f-9479-1a183cc866e6\") " pod="openshift-cluster-node-tuning-operator/tuned-msf7n"
Jul 28 10:07:46.414294 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.412722    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e80f436-0cf4-45a3-8fdf-53bdae6a7fad-multus-cni-dir\") pod \"multus-p2rv8\" (UID: \"4e80f436-0cf4-45a3-8fdf-53bdae6a7fad\") " pod="openshift-multus/multus-p2rv8"
Jul 28 10:07:46.414294 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.412766    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4e80f436-0cf4-45a3-8fdf-53bdae6a7fad-cni-binary-copy\") pod \"multus-p2rv8\" (UID: \"4e80f436-0cf4-45a3-8fdf-53bdae6a7fad\") " pod="openshift-multus/multus-p2rv8"
Jul 28 10:07:46.414294 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.412790    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2c5a2bd4-8e86-493f-9479-1a183cc866e6-etc-sysconfig\") pod \"tuned-msf7n\" (UID: \"2c5a2bd4-8e86-493f-9479-1a183cc866e6\") " pod="openshift-cluster-node-tuning-operator/tuned-msf7n"
Jul 28 10:07:46.414294 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.412815    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c5a2bd4-8e86-493f-9479-1a183cc866e6-var-lib-kubelet\") pod \"tuned-msf7n\" (UID: \"2c5a2bd4-8e86-493f-9479-1a183cc866e6\") " pod="openshift-cluster-node-tuning-operator/tuned-msf7n"
Jul 28 10:07:46.414294 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.412872    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4k7n\" (UniqueName: \"kubernetes.io/projected/2c5a2bd4-8e86-493f-9479-1a183cc866e6-kube-api-access-t4k7n\") pod \"tuned-msf7n\" (UID: \"2c5a2bd4-8e86-493f-9479-1a183cc866e6\") " pod="openshift-cluster-node-tuning-operator/tuned-msf7n"
Jul 28 10:07:46.414294 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.412988    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cblx8\" (UniqueName: \"kubernetes.io/projected/a1a543f9-5413-4260-b48d-f767a66f23c7-kube-api-access-cblx8\") pod \"node-ca-h8krb\" (UID: \"a1a543f9-5413-4260-b48d-f767a66f23c7\") " pod="openshift-image-registry/node-ca-h8krb"
Jul 28 10:07:46.414294 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.413019    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4e80f436-0cf4-45a3-8fdf-53bdae6a7fad-host-run-k8s-cni-cncf-io\") pod \"multus-p2rv8\" (UID: \"4e80f436-0cf4-45a3-8fdf-53bdae6a7fad\") " pod="openshift-multus/multus-p2rv8"
Jul 28 10:07:46.414294 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.413048    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e80f436-0cf4-45a3-8fdf-53bdae6a7fad-multus-conf-dir\") pod \"multus-p2rv8\" (UID: \"4e80f436-0cf4-45a3-8fdf-53bdae6a7fad\") " pod="openshift-multus/multus-p2rv8"
Jul 28 10:07:46.414294 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.413089    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e80f436-0cf4-45a3-8fdf-53bdae6a7fad-etc-kubernetes\") pod \"multus-p2rv8\" (UID: \"4e80f436-0cf4-45a3-8fdf-53bdae6a7fad\") " pod="openshift-multus/multus-p2rv8"
Jul 28 10:07:46.414294 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.413130    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2c5a2bd4-8e86-493f-9479-1a183cc866e6-etc-systemd\") pod \"tuned-msf7n\" (UID: \"2c5a2bd4-8e86-493f-9479-1a183cc866e6\") " pod="openshift-cluster-node-tuning-operator/tuned-msf7n"
Jul 28 10:07:46.414294 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.413155    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e80f436-0cf4-45a3-8fdf-53bdae6a7fad-host-var-lib-kubelet\") pod \"multus-p2rv8\" (UID: \"4e80f436-0cf4-45a3-8fdf-53bdae6a7fad\") " pod="openshift-multus/multus-p2rv8"
Jul 28 10:07:46.414294 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.413199    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lr9x\" (UniqueName: \"kubernetes.io/projected/4e80f436-0cf4-45a3-8fdf-53bdae6a7fad-kube-api-access-5lr9x\") pod \"multus-p2rv8\" (UID: \"4e80f436-0cf4-45a3-8fdf-53bdae6a7fad\") " pod="openshift-multus/multus-p2rv8"
Jul 28 10:07:46.414294 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.413269    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c5a2bd4-8e86-493f-9479-1a183cc866e6-etc-kubernetes\") pod \"tuned-msf7n\" (UID: \"2c5a2bd4-8e86-493f-9479-1a183cc866e6\") " pod="openshift-cluster-node-tuning-operator/tuned-msf7n"
Jul 28 10:07:46.414294 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.413305    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2c5a2bd4-8e86-493f-9479-1a183cc866e6-run\") pod \"tuned-msf7n\" (UID: \"2c5a2bd4-8e86-493f-9479-1a183cc866e6\") " pod="openshift-cluster-node-tuning-operator/tuned-msf7n"
Jul 28 10:07:46.414294 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.413338    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2c5a2bd4-8e86-493f-9479-1a183cc866e6-sys\") pod \"tuned-msf7n\" (UID: \"2c5a2bd4-8e86-493f-9479-1a183cc866e6\") " pod="openshift-cluster-node-tuning-operator/tuned-msf7n"
Jul 28 10:07:46.414294 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.413377    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a1a543f9-5413-4260-b48d-f767a66f23c7-serviceca\") pod \"node-ca-h8krb\" (UID: \"a1a543f9-5413-4260-b48d-f767a66f23c7\") " pod="openshift-image-registry/node-ca-h8krb"
Jul 28 10:07:46.414294 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.413408    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm8hf\" (UniqueName: \"kubernetes.io/projected/58c000f3-06db-4572-8abb-89ca11078c3c-kube-api-access-fm8hf\") pod \"iptables-alerter-vgktv\" (UID: \"58c000f3-06db-4572-8abb-89ca11078c3c\") " pod="openshift-network-operator/iptables-alerter-vgktv"
Jul 28 10:07:46.415272 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.413443    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4e80f436-0cf4-45a3-8fdf-53bdae6a7fad-multus-socket-dir-parent\") pod \"multus-p2rv8\" (UID: \"4e80f436-0cf4-45a3-8fdf-53bdae6a7fad\") " pod="openshift-multus/multus-p2rv8"
Jul 28 10:07:46.415272 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.413486    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e80f436-0cf4-45a3-8fdf-53bdae6a7fad-cnibin\") pod \"multus-p2rv8\" (UID: \"4e80f436-0cf4-45a3-8fdf-53bdae6a7fad\") " pod="openshift-multus/multus-p2rv8"
Jul 28 10:07:46.415272 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.413517    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4e80f436-0cf4-45a3-8fdf-53bdae6a7fad-os-release\") pod \"multus-p2rv8\" (UID: \"4e80f436-0cf4-45a3-8fdf-53bdae6a7fad\") " pod="openshift-multus/multus-p2rv8"
Jul 28 10:07:46.415272 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.413543    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2c5a2bd4-8e86-493f-9479-1a183cc866e6-etc-modprobe-d\") pod \"tuned-msf7n\" (UID: \"2c5a2bd4-8e86-493f-9479-1a183cc866e6\") " pod="openshift-cluster-node-tuning-operator/tuned-msf7n"
Jul 28 10:07:46.416281 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.416192    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5z8xh"
Jul 28 10:07:46.417644 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.417625    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 10:07:46.417930 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.417911    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 10:07:46.418202 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.418179    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-2dxng\""
Jul 28 10:07:46.418482 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.418464    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xbfxp\""
Jul 28 10:07:46.418693 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.418658    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 10:07:46.418806 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.418784    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4wbtr\""
Jul 28 10:07:46.418933 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.418915    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 10:07:46.419003 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.418967    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 10:07:46.419053 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.419020    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 10:07:46.422107 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.421846    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 10:07:46.422227 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.422210    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.426392 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.426010    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 10:07:46.426392 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.426076    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 10:07:46.426392 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.426126    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 10:07:46.426392 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.426224    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xgb8w\""
Jul 28 10:07:46.426392 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.426230    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 10:07:46.427363 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.427339    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 10:07:46.427453 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.427392    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 10:07:46.432999 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.432974    2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 10:07:46.471250 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.471222    2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-qsmgb"
Jul 28 10:07:46.486697 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.486630    2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qsmgb"
Jul 28 10:07:46.500383 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.500353    2557 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 10:07:46.514864 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.514801    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2c5a2bd4-8e86-493f-9479-1a183cc866e6-run\") pod \"tuned-msf7n\" (UID: \"2c5a2bd4-8e86-493f-9479-1a183cc866e6\") " pod="openshift-cluster-node-tuning-operator/tuned-msf7n"
Jul 28 10:07:46.515022 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.514874    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2c5a2bd4-8e86-493f-9479-1a183cc866e6-sys\") pod \"tuned-msf7n\" (UID: \"2c5a2bd4-8e86-493f-9479-1a183cc866e6\") " pod="openshift-cluster-node-tuning-operator/tuned-msf7n"
Jul 28 10:07:46.515022 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.514954    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2c5a2bd4-8e86-493f-9479-1a183cc866e6-sys\") pod \"tuned-msf7n\" (UID: \"2c5a2bd4-8e86-493f-9479-1a183cc866e6\") " pod="openshift-cluster-node-tuning-operator/tuned-msf7n"
Jul 28 10:07:46.515022 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.514962    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2c5a2bd4-8e86-493f-9479-1a183cc866e6-run\") pod \"tuned-msf7n\" (UID: \"2c5a2bd4-8e86-493f-9479-1a183cc866e6\") " pod="openshift-cluster-node-tuning-operator/tuned-msf7n"
Jul 28 10:07:46.515022 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.514988    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a1a543f9-5413-4260-b48d-f767a66f23c7-serviceca\") pod \"node-ca-h8krb\" (UID: \"a1a543f9-5413-4260-b48d-f767a66f23c7\") " pod="openshift-image-registry/node-ca-h8krb"
Jul 28 10:07:46.515247 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.515042    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f9fe54ae-5731-4fdd-b668-c70aca78b5bf-system-cni-dir\") pod \"multus-additional-cni-plugins-gzgcl\" (UID: \"f9fe54ae-5731-4fdd-b668-c70aca78b5bf\") " pod="openshift-multus/multus-additional-cni-plugins-gzgcl"
Jul 28 10:07:46.515247 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.515073    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ceccf442-0ab2-4993-bf49-d0375abe1e62-host-run-ovn-kubernetes\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.515247 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.515099    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ceccf442-0ab2-4993-bf49-d0375abe1e62-host-cni-netd\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.515247 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.515127    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ceccf442-0ab2-4993-bf49-d0375abe1e62-ovnkube-script-lib\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.515247 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.515166    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e80f436-0cf4-45a3-8fdf-53bdae6a7fad-cnibin\") pod \"multus-p2rv8\" (UID: \"4e80f436-0cf4-45a3-8fdf-53bdae6a7fad\") " pod="openshift-multus/multus-p2rv8"
Jul 28 10:07:46.515247 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.515193    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4e80f436-0cf4-45a3-8fdf-53bdae6a7fad-os-release\") pod \"multus-p2rv8\" (UID: \"4e80f436-0cf4-45a3-8fdf-53bdae6a7fad\") " pod="openshift-multus/multus-p2rv8"
Jul 28 10:07:46.515247 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.515218    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2c5a2bd4-8e86-493f-9479-1a183cc866e6-etc-sysctl-d\") pod \"tuned-msf7n\" (UID: \"2c5a2bd4-8e86-493f-9479-1a183cc866e6\") " pod="openshift-cluster-node-tuning-operator/tuned-msf7n"
Jul 28 10:07:46.515247 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.515243    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f9fe54ae-5731-4fdd-b668-c70aca78b5bf-os-release\") pod \"multus-additional-cni-plugins-gzgcl\" (UID: \"f9fe54ae-5731-4fdd-b668-c70aca78b5bf\") " pod="openshift-multus/multus-additional-cni-plugins-gzgcl"
Jul 28 10:07:46.515623 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.515269    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ceccf442-0ab2-4993-bf49-d0375abe1e62-run-ovn\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.515623 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.515272    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e80f436-0cf4-45a3-8fdf-53bdae6a7fad-cnibin\") pod \"multus-p2rv8\" (UID: \"4e80f436-0cf4-45a3-8fdf-53bdae6a7fad\") " pod="openshift-multus/multus-p2rv8"
Jul 28 10:07:46.515623 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.515293    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ceccf442-0ab2-4993-bf49-d0375abe1e62-ovnkube-config\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.515623 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.515320    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh26c\" (UniqueName: \"kubernetes.io/projected/ceccf442-0ab2-4993-bf49-d0375abe1e62-kube-api-access-zh26c\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.515623 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.515345    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2c5a2bd4-8e86-493f-9479-1a183cc866e6-tmp\") pod \"tuned-msf7n\" (UID: \"2c5a2bd4-8e86-493f-9479-1a183cc866e6\") " pod="openshift-cluster-node-tuning-operator/tuned-msf7n"
Jul 28 10:07:46.515623 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.515389    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e80f436-0cf4-45a3-8fdf-53bdae6a7fad-system-cni-dir\") pod \"multus-p2rv8\" (UID: \"4e80f436-0cf4-45a3-8fdf-53bdae6a7fad\") " pod="openshift-multus/multus-p2rv8"
Jul 28 10:07:46.515623 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.515414    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f9fe54ae-5731-4fdd-b668-c70aca78b5bf-cni-binary-copy\") pod \"multus-additional-cni-plugins-gzgcl\" (UID: \"f9fe54ae-5731-4fdd-b668-c70aca78b5bf\") " pod="openshift-multus/multus-additional-cni-plugins-gzgcl"
Jul 28 10:07:46.515623 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.515426    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2c5a2bd4-8e86-493f-9479-1a183cc866e6-etc-sysctl-d\") pod \"tuned-msf7n\" (UID: \"2c5a2bd4-8e86-493f-9479-1a183cc866e6\") " pod="openshift-cluster-node-tuning-operator/tuned-msf7n"
Jul 28 10:07:46.515623 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.515440    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f0967024-b386-49dd-a50f-4b69599548aa-socket-dir\") pod \"aws-ebs-csi-driver-node-hh49h\" (UID: \"f0967024-b386-49dd-a50f-4b69599548aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hh49h"
Jul 28 10:07:46.515623 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.515468    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2c5a2bd4-8e86-493f-9479-1a183cc866e6-host\") pod \"tuned-msf7n\" (UID: \"2c5a2bd4-8e86-493f-9479-1a183cc866e6\") " pod="openshift-cluster-node-tuning-operator/tuned-msf7n"
Jul 28 10:07:46.515623 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.515498    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4e80f436-0cf4-45a3-8fdf-53bdae6a7fad-host-var-lib-cni-bin\") pod \"multus-p2rv8\" (UID: \"4e80f436-0cf4-45a3-8fdf-53bdae6a7fad\") " pod="openshift-multus/multus-p2rv8"
Jul 28 10:07:46.515623 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.515514    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4e80f436-0cf4-45a3-8fdf-53bdae6a7fad-hostroot\") pod \"multus-p2rv8\" (UID: \"4e80f436-0cf4-45a3-8fdf-53bdae6a7fad\") " pod="openshift-multus/multus-p2rv8"
Jul 28 10:07:46.515623 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.515530    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d23b40b0-9299-4a0b-a87c-d2f2cb921fd9-metrics-certs\") pod \"network-metrics-daemon-8fltb\" (UID: \"d23b40b0-9299-4a0b-a87c-d2f2cb921fd9\") " pod="openshift-multus/network-metrics-daemon-8fltb"
Jul 28 10:07:46.515623 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.515546    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f9fe54ae-5731-4fdd-b668-c70aca78b5bf-cnibin\") pod \"multus-additional-cni-plugins-gzgcl\" (UID: \"f9fe54ae-5731-4fdd-b668-c70aca78b5bf\") " pod="openshift-multus/multus-additional-cni-plugins-gzgcl"
Jul 28 10:07:46.515623 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.515562    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f0967024-b386-49dd-a50f-4b69599548aa-sys-fs\") pod \"aws-ebs-csi-driver-node-hh49h\" (UID: \"f0967024-b386-49dd-a50f-4b69599548aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hh49h"
Jul 28 10:07:46.515623 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.515594    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ceccf442-0ab2-4993-bf49-d0375abe1e62-host-slash\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.515623 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.515611    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2c5a2bd4-8e86-493f-9479-1a183cc866e6-lib-modules\") pod \"tuned-msf7n\" (UID: \"2c5a2bd4-8e86-493f-9479-1a183cc866e6\") " pod="openshift-cluster-node-tuning-operator/tuned-msf7n"
Jul 28 10:07:46.516568 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.515636    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e80f436-0cf4-45a3-8fdf-53bdae6a7fad-multus-cni-dir\") pod \"multus-p2rv8\" (UID: \"4e80f436-0cf4-45a3-8fdf-53bdae6a7fad\") " pod="openshift-multus/multus-p2rv8"
Jul 28 10:07:46.516568 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.515663    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f9fe54ae-5731-4fdd-b668-c70aca78b5bf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gzgcl\" (UID: \"f9fe54ae-5731-4fdd-b668-c70aca78b5bf\") " pod="openshift-multus/multus-additional-cni-plugins-gzgcl"
Jul 28 10:07:46.516568 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.515707    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f0967024-b386-49dd-a50f-4b69599548aa-registration-dir\") pod \"aws-ebs-csi-driver-node-hh49h\" (UID: \"f0967024-b386-49dd-a50f-4b69599548aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hh49h"
Jul 28 10:07:46.516568 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.515731    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ceccf442-0ab2-4993-bf49-d0375abe1e62-node-log\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.516568 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.515777    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a1a543f9-5413-4260-b48d-f767a66f23c7-serviceca\") pod \"node-ca-h8krb\" (UID: \"a1a543f9-5413-4260-b48d-f767a66f23c7\") " pod="openshift-image-registry/node-ca-h8krb"
Jul 28 10:07:46.516568 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.515770    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5lph\" (UniqueName: \"kubernetes.io/projected/d23b40b0-9299-4a0b-a87c-d2f2cb921fd9-kube-api-access-t5lph\") pod \"network-metrics-daemon-8fltb\" (UID: \"d23b40b0-9299-4a0b-a87c-d2f2cb921fd9\") " pod="openshift-multus/network-metrics-daemon-8fltb"
Jul 28 10:07:46.516568 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.515833    2557 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 10:07:46.516568 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.515860    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e80f436-0cf4-45a3-8fdf-53bdae6a7fad-system-cni-dir\") pod \"multus-p2rv8\" (UID: \"4e80f436-0cf4-45a3-8fdf-53bdae6a7fad\") " pod="openshift-multus/multus-p2rv8"
Jul 28 10:07:46.516568 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.515861    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2c5a2bd4-8e86-493f-9479-1a183cc866e6-etc-sysconfig\") pod \"tuned-msf7n\" (UID: \"2c5a2bd4-8e86-493f-9479-1a183cc866e6\") " pod="openshift-cluster-node-tuning-operator/tuned-msf7n"
Jul 28 10:07:46.516568 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.515897    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4k7n\" (UniqueName: \"kubernetes.io/projected/2c5a2bd4-8e86-493f-9479-1a183cc866e6-kube-api-access-t4k7n\") pod \"tuned-msf7n\" (UID: \"2c5a2bd4-8e86-493f-9479-1a183cc866e6\") " pod="openshift-cluster-node-tuning-operator/tuned-msf7n"
Jul 28 10:07:46.516568 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.515915    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2c5a2bd4-8e86-493f-9479-1a183cc866e6-etc-sysconfig\") pod \"tuned-msf7n\" (UID: \"2c5a2bd4-8e86-493f-9479-1a183cc866e6\") " pod="openshift-cluster-node-tuning-operator/tuned-msf7n"
Jul 28 10:07:46.516568 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.515924    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb-dbus\") pod \"global-pull-secret-syncer-jhkxw\" (UID: \"ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb\") " pod="kube-system/global-pull-secret-syncer-jhkxw"
Jul 28 10:07:46.516568 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.515973    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pbkb\" (UniqueName: \"kubernetes.io/projected/f8ad89c4-5ef2-477b-ab4c-e2c97b9a68fa-kube-api-access-9pbkb\") pod \"node-resolver-5z8xh\" (UID: \"f8ad89c4-5ef2-477b-ab4c-e2c97b9a68fa\") " pod="openshift-dns/node-resolver-5z8xh"
Jul 28 10:07:46.516568 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.515497    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4e80f436-0cf4-45a3-8fdf-53bdae6a7fad-os-release\") pod \"multus-p2rv8\" (UID: \"4e80f436-0cf4-45a3-8fdf-53bdae6a7fad\") " pod="openshift-multus/multus-p2rv8"
Jul 28 10:07:46.516568 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.516071    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4e80f436-0cf4-45a3-8fdf-53bdae6a7fad-host-var-lib-cni-bin\") pod \"multus-p2rv8\" (UID: \"4e80f436-0cf4-45a3-8fdf-53bdae6a7fad\") " pod="openshift-multus/multus-p2rv8"
Jul 28 10:07:46.516568 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.516131    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2c5a2bd4-8e86-493f-9479-1a183cc866e6-host\") pod \"tuned-msf7n\" (UID: \"2c5a2bd4-8e86-493f-9479-1a183cc866e6\") " pod="openshift-cluster-node-tuning-operator/tuned-msf7n"
Jul 28 10:07:46.516568 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.516164    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e80f436-0cf4-45a3-8fdf-53bdae6a7fad-etc-kubernetes\") pod \"multus-p2rv8\" (UID: \"4e80f436-0cf4-45a3-8fdf-53bdae6a7fad\") " pod="openshift-multus/multus-p2rv8"
Jul 28 10:07:46.516568 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.516177    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4e80f436-0cf4-45a3-8fdf-53bdae6a7fad-hostroot\") pod \"multus-p2rv8\" (UID: \"4e80f436-0cf4-45a3-8fdf-53bdae6a7fad\") " pod="openshift-multus/multus-p2rv8"
Jul 28 10:07:46.517510 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.516194    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8ad89c4-5ef2-477b-ab4c-e2c97b9a68fa-tmp-dir\") pod \"node-resolver-5z8xh\" (UID: \"f8ad89c4-5ef2-477b-ab4c-e2c97b9a68fa\") " pod="openshift-dns/node-resolver-5z8xh"
Jul 28 10:07:46.517510 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.516223    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ceccf442-0ab2-4993-bf49-d0375abe1e62-systemd-units\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.517510 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.516248    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ceccf442-0ab2-4993-bf49-d0375abe1e62-etc-openvswitch\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.517510 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.516295    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e80f436-0cf4-45a3-8fdf-53bdae6a7fad-host-var-lib-kubelet\") pod \"multus-p2rv8\" (UID: \"4e80f436-0cf4-45a3-8fdf-53bdae6a7fad\") " pod="openshift-multus/multus-p2rv8"
Jul 28 10:07:46.517510 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.516299    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2c5a2bd4-8e86-493f-9479-1a183cc866e6-lib-modules\") pod \"tuned-msf7n\" (UID: \"2c5a2bd4-8e86-493f-9479-1a183cc866e6\") " pod="openshift-cluster-node-tuning-operator/tuned-msf7n"
Jul 28 10:07:46.517510 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.516328    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lr9x\" (UniqueName: \"kubernetes.io/projected/4e80f436-0cf4-45a3-8fdf-53bdae6a7fad-kube-api-access-5lr9x\") pod \"multus-p2rv8\" (UID: \"4e80f436-0cf4-45a3-8fdf-53bdae6a7fad\") " pod="openshift-multus/multus-p2rv8"
Jul 28 10:07:46.517510 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.516355    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e80f436-0cf4-45a3-8fdf-53bdae6a7fad-multus-cni-dir\") pod \"multus-p2rv8\" (UID: \"4e80f436-0cf4-45a3-8fdf-53bdae6a7fad\") " pod="openshift-multus/multus-p2rv8"
Jul 28 10:07:46.517510 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.516359    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6d704fd4-c194-4c37-a80e-dbde31ebd184-agent-certs\") pod \"konnectivity-agent-v7gsk\" (UID: \"6d704fd4-c194-4c37-a80e-dbde31ebd184\") " pod="kube-system/konnectivity-agent-v7gsk"
Jul 28 10:07:46.517510 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.516367    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e80f436-0cf4-45a3-8fdf-53bdae6a7fad-host-var-lib-kubelet\") pod \"multus-p2rv8\" (UID: \"4e80f436-0cf4-45a3-8fdf-53bdae6a7fad\") " pod="openshift-multus/multus-p2rv8"
Jul 28 10:07:46.517510 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.516416    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e80f436-0cf4-45a3-8fdf-53bdae6a7fad-etc-kubernetes\") pod \"multus-p2rv8\" (UID: \"4e80f436-0cf4-45a3-8fdf-53bdae6a7fad\") " pod="openshift-multus/multus-p2rv8"
Jul 28 10:07:46.517510 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.516454    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chxfc\" (UniqueName: \"kubernetes.io/projected/f0967024-b386-49dd-a50f-4b69599548aa-kube-api-access-chxfc\") pod \"aws-ebs-csi-driver-node-hh49h\" (UID: \"f0967024-b386-49dd-a50f-4b69599548aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hh49h"
Jul 28 10:07:46.517510 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.516483    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ceccf442-0ab2-4993-bf49-d0375abe1e62-host-kubelet\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.517510 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.516511    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ceccf442-0ab2-4993-bf49-d0375abe1e62-run-openvswitch\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.517510 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.516561    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c5a2bd4-8e86-493f-9479-1a183cc866e6-etc-kubernetes\") pod \"tuned-msf7n\" (UID: \"2c5a2bd4-8e86-493f-9479-1a183cc866e6\") " pod="openshift-cluster-node-tuning-operator/tuned-msf7n"
Jul 28 10:07:46.517510 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.516599    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm8hf\" (UniqueName: \"kubernetes.io/projected/58c000f3-06db-4572-8abb-89ca11078c3c-kube-api-access-fm8hf\") pod \"iptables-alerter-vgktv\" (UID: \"58c000f3-06db-4572-8abb-89ca11078c3c\") " pod="openshift-network-operator/iptables-alerter-vgktv"
Jul 28 10:07:46.517510 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.516664    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c5a2bd4-8e86-493f-9479-1a183cc866e6-etc-kubernetes\") pod \"tuned-msf7n\" (UID: \"2c5a2bd4-8e86-493f-9479-1a183cc866e6\") " pod="openshift-cluster-node-tuning-operator/tuned-msf7n"
Jul 28 10:07:46.517510 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.516775    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4e80f436-0cf4-45a3-8fdf-53bdae6a7fad-multus-socket-dir-parent\") pod \"multus-p2rv8\" (UID: \"4e80f436-0cf4-45a3-8fdf-53bdae6a7fad\") " pod="openshift-multus/multus-p2rv8"
Jul 28 10:07:46.518402 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.516828    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2c5a2bd4-8e86-493f-9479-1a183cc866e6-etc-modprobe-d\") pod \"tuned-msf7n\" (UID: \"2c5a2bd4-8e86-493f-9479-1a183cc866e6\") " pod="openshift-cluster-node-tuning-operator/tuned-msf7n"
Jul 28 10:07:46.518402 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.516854    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a1a543f9-5413-4260-b48d-f767a66f23c7-host\") pod \"node-ca-h8krb\" (UID: \"a1a543f9-5413-4260-b48d-f767a66f23c7\") " pod="openshift-image-registry/node-ca-h8krb"
Jul 28 10:07:46.518402 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.516876    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4e80f436-0cf4-45a3-8fdf-53bdae6a7fad-multus-socket-dir-parent\") pod \"multus-p2rv8\" (UID: \"4e80f436-0cf4-45a3-8fdf-53bdae6a7fad\") " pod="openshift-multus/multus-p2rv8"
Jul 28 10:07:46.518402 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.516899    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/58c000f3-06db-4572-8abb-89ca11078c3c-iptables-alerter-script\") pod \"iptables-alerter-vgktv\" (UID: \"58c000f3-06db-4572-8abb-89ca11078c3c\") " pod="openshift-network-operator/iptables-alerter-vgktv"
Jul 28 10:07:46.518402 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.516924    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4e80f436-0cf4-45a3-8fdf-53bdae6a7fad-host-run-netns\") pod \"multus-p2rv8\" (UID: \"4e80f436-0cf4-45a3-8fdf-53bdae6a7fad\") " pod="openshift-multus/multus-p2rv8"
Jul 28 10:07:46.518402 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.516929    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a1a543f9-5413-4260-b48d-f767a66f23c7-host\") pod \"node-ca-h8krb\" (UID: \"a1a543f9-5413-4260-b48d-f767a66f23c7\") " pod="openshift-image-registry/node-ca-h8krb"
Jul 28 10:07:46.518402 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.516952    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4e80f436-0cf4-45a3-8fdf-53bdae6a7fad-host-var-lib-cni-multus\") pod \"multus-p2rv8\" (UID: \"4e80f436-0cf4-45a3-8fdf-53bdae6a7fad\") " pod="openshift-multus/multus-p2rv8"
Jul 28 10:07:46.518402 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.516984    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f0967024-b386-49dd-a50f-4b69599548aa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hh49h\" (UID: \"f0967024-b386-49dd-a50f-4b69599548aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hh49h"
Jul 28 10:07:46.518402 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.517003    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2c5a2bd4-8e86-493f-9479-1a183cc866e6-etc-modprobe-d\") pod \"tuned-msf7n\" (UID: \"2c5a2bd4-8e86-493f-9479-1a183cc866e6\") " pod="openshift-cluster-node-tuning-operator/tuned-msf7n"
Jul 28 10:07:46.518402 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.517009    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2c5a2bd4-8e86-493f-9479-1a183cc866e6-etc-sysctl-conf\") pod \"tuned-msf7n\" (UID: \"2c5a2bd4-8e86-493f-9479-1a183cc866e6\") " pod="openshift-cluster-node-tuning-operator/tuned-msf7n"
Jul 28 10:07:46.518402 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.517066    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4e80f436-0cf4-45a3-8fdf-53bdae6a7fad-multus-daemon-config\") pod \"multus-p2rv8\" (UID: \"4e80f436-0cf4-45a3-8fdf-53bdae6a7fad\") " pod="openshift-multus/multus-p2rv8"
Jul 28 10:07:46.518402 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.517097    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6mbt\" (UniqueName: \"kubernetes.io/projected/f9fe54ae-5731-4fdd-b668-c70aca78b5bf-kube-api-access-b6mbt\") pod \"multus-additional-cni-plugins-gzgcl\" (UID: \"f9fe54ae-5731-4fdd-b668-c70aca78b5bf\") " pod="openshift-multus/multus-additional-cni-plugins-gzgcl"
Jul 28 10:07:46.518402 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.517135    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ceccf442-0ab2-4993-bf49-d0375abe1e62-host-run-netns\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.518402 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.517161    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ceccf442-0ab2-4993-bf49-d0375abe1e62-run-systemd\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.518402 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.517136    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2c5a2bd4-8e86-493f-9479-1a183cc866e6-etc-sysctl-conf\") pod \"tuned-msf7n\" (UID: \"2c5a2bd4-8e86-493f-9479-1a183cc866e6\") " pod="openshift-cluster-node-tuning-operator/tuned-msf7n"
Jul 28 10:07:46.518402 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.517197    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ceccf442-0ab2-4993-bf49-d0375abe1e62-log-socket\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.518402 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.517228    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2c5a2bd4-8e86-493f-9479-1a183cc866e6-etc-tuned\") pod \"tuned-msf7n\" (UID: \"2c5a2bd4-8e86-493f-9479-1a183cc866e6\") " pod="openshift-cluster-node-tuning-operator/tuned-msf7n"
Jul 28 10:07:46.519269 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.517257    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/58c000f3-06db-4572-8abb-89ca11078c3c-host-slash\") pod \"iptables-alerter-vgktv\" (UID: \"58c000f3-06db-4572-8abb-89ca11078c3c\") " pod="openshift-network-operator/iptables-alerter-vgktv"
Jul 28 10:07:46.519269 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.517283    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4e80f436-0cf4-45a3-8fdf-53bdae6a7fad-host-run-multus-certs\") pod \"multus-p2rv8\" (UID: \"4e80f436-0cf4-45a3-8fdf-53bdae6a7fad\") " pod="openshift-multus/multus-p2rv8"
Jul 28 10:07:46.519269 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.517314    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f9fe54ae-5731-4fdd-b668-c70aca78b5bf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gzgcl\" (UID: \"f9fe54ae-5731-4fdd-b668-c70aca78b5bf\") " pod="openshift-multus/multus-additional-cni-plugins-gzgcl"
Jul 28 10:07:46.519269 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.517406    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phrdk\" (UniqueName: \"kubernetes.io/projected/0bc74d3d-3abf-47f5-8149-7d07a9a34e7c-kube-api-access-phrdk\") pod \"network-check-target-ddrhj\" (UID: \"0bc74d3d-3abf-47f5-8149-7d07a9a34e7c\") " pod="openshift-network-diagnostics/network-check-target-ddrhj"
Jul 28 10:07:46.519269 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.517449    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/58c000f3-06db-4572-8abb-89ca11078c3c-iptables-alerter-script\") pod \"iptables-alerter-vgktv\" (UID: \"58c000f3-06db-4572-8abb-89ca11078c3c\") " pod="openshift-network-operator/iptables-alerter-vgktv"
Jul 28 10:07:46.519269 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.517461    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f8ad89c4-5ef2-477b-ab4c-e2c97b9a68fa-hosts-file\") pod \"node-resolver-5z8xh\" (UID: \"f8ad89c4-5ef2-477b-ab4c-e2c97b9a68fa\") " pod="openshift-dns/node-resolver-5z8xh"
Jul 28 10:07:46.519269 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.517496    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ceccf442-0ab2-4993-bf49-d0375abe1e62-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.519269 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.517524    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ceccf442-0ab2-4993-bf49-d0375abe1e62-ovn-node-metrics-cert\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.519269 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.517550    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f9fe54ae-5731-4fdd-b668-c70aca78b5bf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gzgcl\" (UID: \"f9fe54ae-5731-4fdd-b668-c70aca78b5bf\") " pod="openshift-multus/multus-additional-cni-plugins-gzgcl"
Jul 28 10:07:46.519269 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.517580    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4e80f436-0cf4-45a3-8fdf-53bdae6a7fad-cni-binary-copy\") pod \"multus-p2rv8\" (UID: \"4e80f436-0cf4-45a3-8fdf-53bdae6a7fad\") " pod="openshift-multus/multus-p2rv8"
Jul 28 10:07:46.519269 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.517607    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c5a2bd4-8e86-493f-9479-1a183cc866e6-var-lib-kubelet\") pod \"tuned-msf7n\" (UID: \"2c5a2bd4-8e86-493f-9479-1a183cc866e6\") " pod="openshift-cluster-node-tuning-operator/tuned-msf7n"
Jul 28 10:07:46.519269 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.517635    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cblx8\" (UniqueName: \"kubernetes.io/projected/a1a543f9-5413-4260-b48d-f767a66f23c7-kube-api-access-cblx8\") pod \"node-ca-h8krb\" (UID: \"a1a543f9-5413-4260-b48d-f767a66f23c7\") " pod="openshift-image-registry/node-ca-h8krb"
Jul 28 10:07:46.519269 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.517663    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4e80f436-0cf4-45a3-8fdf-53bdae6a7fad-host-run-k8s-cni-cncf-io\") pod \"multus-p2rv8\" (UID: \"4e80f436-0cf4-45a3-8fdf-53bdae6a7fad\") " pod="openshift-multus/multus-p2rv8"
Jul 28 10:07:46.519269 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.517740    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb-original-pull-secret\") pod \"global-pull-secret-syncer-jhkxw\" (UID: \"ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb\") " pod="kube-system/global-pull-secret-syncer-jhkxw"
Jul 28 10:07:46.519269 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.517819    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4e80f436-0cf4-45a3-8fdf-53bdae6a7fad-host-var-lib-cni-multus\") pod \"multus-p2rv8\" (UID: \"4e80f436-0cf4-45a3-8fdf-53bdae6a7fad\") " pod="openshift-multus/multus-p2rv8"
Jul 28 10:07:46.519269 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.517876    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f0967024-b386-49dd-a50f-4b69599548aa-device-dir\") pod \"aws-ebs-csi-driver-node-hh49h\" (UID: \"f0967024-b386-49dd-a50f-4b69599548aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hh49h"
Jul 28 10:07:46.520202 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.517902    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4e80f436-0cf4-45a3-8fdf-53bdae6a7fad-multus-daemon-config\") pod \"multus-p2rv8\" (UID: \"4e80f436-0cf4-45a3-8fdf-53bdae6a7fad\") " pod="openshift-multus/multus-p2rv8"
Jul 28 10:07:46.520202 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.517906    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ceccf442-0ab2-4993-bf49-d0375abe1e62-var-lib-openvswitch\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.520202 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.517947    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e80f436-0cf4-45a3-8fdf-53bdae6a7fad-multus-conf-dir\") pod \"multus-p2rv8\" (UID: \"4e80f436-0cf4-45a3-8fdf-53bdae6a7fad\") " pod="openshift-multus/multus-p2rv8"
Jul 28 10:07:46.520202 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.517973    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6d704fd4-c194-4c37-a80e-dbde31ebd184-konnectivity-ca\") pod \"konnectivity-agent-v7gsk\" (UID: \"6d704fd4-c194-4c37-a80e-dbde31ebd184\") " pod="kube-system/konnectivity-agent-v7gsk"
Jul 28 10:07:46.520202 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.518024    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4e80f436-0cf4-45a3-8fdf-53bdae6a7fad-host-run-k8s-cni-cncf-io\") pod \"multus-p2rv8\" (UID: \"4e80f436-0cf4-45a3-8fdf-53bdae6a7fad\") " pod="openshift-multus/multus-p2rv8"
Jul 28 10:07:46.520202 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.518051    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ceccf442-0ab2-4993-bf49-d0375abe1e62-host-cni-bin\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.520202 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.518103    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ceccf442-0ab2-4993-bf49-d0375abe1e62-env-overrides\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.520202 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.518174    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2c5a2bd4-8e86-493f-9479-1a183cc866e6-etc-systemd\") pod \"tuned-msf7n\" (UID: \"2c5a2bd4-8e86-493f-9479-1a183cc866e6\") " pod="openshift-cluster-node-tuning-operator/tuned-msf7n"
Jul 28 10:07:46.520202 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.518215    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb-kubelet-config\") pod \"global-pull-secret-syncer-jhkxw\" (UID: \"ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb\") " pod="kube-system/global-pull-secret-syncer-jhkxw"
Jul 28 10:07:46.520202 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.518221    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c5a2bd4-8e86-493f-9479-1a183cc866e6-var-lib-kubelet\") pod \"tuned-msf7n\" (UID: \"2c5a2bd4-8e86-493f-9479-1a183cc866e6\") " pod="openshift-cluster-node-tuning-operator/tuned-msf7n"
Jul 28 10:07:46.520202 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.517944    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/58c000f3-06db-4572-8abb-89ca11078c3c-host-slash\") pod \"iptables-alerter-vgktv\" (UID: \"58c000f3-06db-4572-8abb-89ca11078c3c\") " pod="openshift-network-operator/iptables-alerter-vgktv"
Jul 28 10:07:46.520202 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.517975    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4e80f436-0cf4-45a3-8fdf-53bdae6a7fad-host-run-multus-certs\") pod \"multus-p2rv8\" (UID: \"4e80f436-0cf4-45a3-8fdf-53bdae6a7fad\") " pod="openshift-multus/multus-p2rv8"
Jul 28 10:07:46.520202 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.518264    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4e80f436-0cf4-45a3-8fdf-53bdae6a7fad-host-run-netns\") pod \"multus-p2rv8\" (UID: \"4e80f436-0cf4-45a3-8fdf-53bdae6a7fad\") " pod="openshift-multus/multus-p2rv8"
Jul 28 10:07:46.520202 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.518284    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f0967024-b386-49dd-a50f-4b69599548aa-etc-selinux\") pod \"aws-ebs-csi-driver-node-hh49h\" (UID: \"f0967024-b386-49dd-a50f-4b69599548aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hh49h"
Jul 28 10:07:46.520202 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.518291    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2c5a2bd4-8e86-493f-9479-1a183cc866e6-etc-systemd\") pod \"tuned-msf7n\" (UID: \"2c5a2bd4-8e86-493f-9479-1a183cc866e6\") " pod="openshift-cluster-node-tuning-operator/tuned-msf7n"
Jul 28 10:07:46.520202 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.518339    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e80f436-0cf4-45a3-8fdf-53bdae6a7fad-multus-conf-dir\") pod \"multus-p2rv8\" (UID: \"4e80f436-0cf4-45a3-8fdf-53bdae6a7fad\") " pod="openshift-multus/multus-p2rv8"
Jul 28 10:07:46.520202 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.518343    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4e80f436-0cf4-45a3-8fdf-53bdae6a7fad-cni-binary-copy\") pod \"multus-p2rv8\" (UID: \"4e80f436-0cf4-45a3-8fdf-53bdae6a7fad\") " pod="openshift-multus/multus-p2rv8"
Jul 28 10:07:46.521501 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.521472    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2c5a2bd4-8e86-493f-9479-1a183cc866e6-tmp\") pod \"tuned-msf7n\" (UID: \"2c5a2bd4-8e86-493f-9479-1a183cc866e6\") " pod="openshift-cluster-node-tuning-operator/tuned-msf7n"
Jul 28 10:07:46.523275 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.523251    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2c5a2bd4-8e86-493f-9479-1a183cc866e6-etc-tuned\") pod \"tuned-msf7n\" (UID: \"2c5a2bd4-8e86-493f-9479-1a183cc866e6\") " pod="openshift-cluster-node-tuning-operator/tuned-msf7n"
Jul 28 10:07:46.527984 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.527951    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cblx8\" (UniqueName: \"kubernetes.io/projected/a1a543f9-5413-4260-b48d-f767a66f23c7-kube-api-access-cblx8\") pod \"node-ca-h8krb\" (UID: \"a1a543f9-5413-4260-b48d-f767a66f23c7\") " pod="openshift-image-registry/node-ca-h8krb"
Jul 28 10:07:46.530928 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.530907    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm8hf\" (UniqueName: \"kubernetes.io/projected/58c000f3-06db-4572-8abb-89ca11078c3c-kube-api-access-fm8hf\") pod \"iptables-alerter-vgktv\" (UID: \"58c000f3-06db-4572-8abb-89ca11078c3c\") " pod="openshift-network-operator/iptables-alerter-vgktv"
Jul 28 10:07:46.536945 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.536897    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4k7n\" (UniqueName: \"kubernetes.io/projected/2c5a2bd4-8e86-493f-9479-1a183cc866e6-kube-api-access-t4k7n\") pod \"tuned-msf7n\" (UID: \"2c5a2bd4-8e86-493f-9479-1a183cc866e6\") " pod="openshift-cluster-node-tuning-operator/tuned-msf7n"
Jul 28 10:07:46.539152 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.539125    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lr9x\" (UniqueName: \"kubernetes.io/projected/4e80f436-0cf4-45a3-8fdf-53bdae6a7fad-kube-api-access-5lr9x\") pod \"multus-p2rv8\" (UID: \"4e80f436-0cf4-45a3-8fdf-53bdae6a7fad\") " pod="openshift-multus/multus-p2rv8"
Jul 28 10:07:46.548981 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.548768    2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 10:07:46.581687 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.581594    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-88.ec2.internal" event={"ID":"6ff9d18249fb2db5835b1f8658874702","Type":"ContainerStarted","Data":"14960905c879d472cc750540742c5aa567033a8be31131794e8cb33148896a1b"}
Jul 28 10:07:46.583358 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.583320    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-88.ec2.internal" event={"ID":"ae61811891b0aefce4345602b66dee13","Type":"ContainerStarted","Data":"61f3947314d17a187c67ba33f68b313efbd29431803ab44b29b4f97d77a483b4"}
Jul 28 10:07:46.619183 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.619146    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f9fe54ae-5731-4fdd-b668-c70aca78b5bf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gzgcl\" (UID: \"f9fe54ae-5731-4fdd-b668-c70aca78b5bf\") " pod="openshift-multus/multus-additional-cni-plugins-gzgcl"
Jul 28 10:07:46.619183 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.619196    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb-original-pull-secret\") pod \"global-pull-secret-syncer-jhkxw\" (UID: \"ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb\") " pod="kube-system/global-pull-secret-syncer-jhkxw"
Jul 28 10:07:46.619425 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.619223    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f0967024-b386-49dd-a50f-4b69599548aa-device-dir\") pod \"aws-ebs-csi-driver-node-hh49h\" (UID: \"f0967024-b386-49dd-a50f-4b69599548aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hh49h"
Jul 28 10:07:46.619425 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.619247    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ceccf442-0ab2-4993-bf49-d0375abe1e62-var-lib-openvswitch\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.619425 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.619276    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6d704fd4-c194-4c37-a80e-dbde31ebd184-konnectivity-ca\") pod \"konnectivity-agent-v7gsk\" (UID: \"6d704fd4-c194-4c37-a80e-dbde31ebd184\") " pod="kube-system/konnectivity-agent-v7gsk"
Jul 28 10:07:46.619425 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.619315    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ceccf442-0ab2-4993-bf49-d0375abe1e62-host-cni-bin\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.619425 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.619340    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ceccf442-0ab2-4993-bf49-d0375abe1e62-env-overrides\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.619425 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:46.619353    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 10:07:46.619425 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.619368    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb-kubelet-config\") pod \"global-pull-secret-syncer-jhkxw\" (UID: \"ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb\") " pod="kube-system/global-pull-secret-syncer-jhkxw"
Jul 28 10:07:46.619425 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.619397    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f0967024-b386-49dd-a50f-4b69599548aa-etc-selinux\") pod \"aws-ebs-csi-driver-node-hh49h\" (UID: \"f0967024-b386-49dd-a50f-4b69599548aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hh49h"
Jul 28 10:07:46.619794 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:46.619482    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb-original-pull-secret podName:ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb nodeName:}" failed. No retries permitted until 2026-07-28 10:07:47.119463272 +0000 UTC m=+3.078231630 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb-original-pull-secret") pod "global-pull-secret-syncer-jhkxw" (UID: "ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb") : object "kube-system"/"original-pull-secret" not registered
Jul 28 10:07:46.619794 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.619459    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f9fe54ae-5731-4fdd-b668-c70aca78b5bf-system-cni-dir\") pod \"multus-additional-cni-plugins-gzgcl\" (UID: \"f9fe54ae-5731-4fdd-b668-c70aca78b5bf\") " pod="openshift-multus/multus-additional-cni-plugins-gzgcl"
Jul 28 10:07:46.619794 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.619532    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ceccf442-0ab2-4993-bf49-d0375abe1e62-host-run-ovn-kubernetes\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.619794 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.619576    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ceccf442-0ab2-4993-bf49-d0375abe1e62-host-cni-netd\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.619794 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.619600    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ceccf442-0ab2-4993-bf49-d0375abe1e62-ovnkube-script-lib\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.619794 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.619629    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f9fe54ae-5731-4fdd-b668-c70aca78b5bf-os-release\") pod \"multus-additional-cni-plugins-gzgcl\" (UID: \"f9fe54ae-5731-4fdd-b668-c70aca78b5bf\") " pod="openshift-multus/multus-additional-cni-plugins-gzgcl"
Jul 28 10:07:46.619794 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.619675    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ceccf442-0ab2-4993-bf49-d0375abe1e62-run-ovn\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.619794 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.619700    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ceccf442-0ab2-4993-bf49-d0375abe1e62-ovnkube-config\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.619794 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.619719    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ceccf442-0ab2-4993-bf49-d0375abe1e62-host-cni-bin\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.619794 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.619727    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh26c\" (UniqueName: \"kubernetes.io/projected/ceccf442-0ab2-4993-bf49-d0375abe1e62-kube-api-access-zh26c\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.619794 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.619722    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb-kubelet-config\") pod \"global-pull-secret-syncer-jhkxw\" (UID: \"ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb\") " pod="kube-system/global-pull-secret-syncer-jhkxw"
Jul 28 10:07:46.619794 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.619752    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ceccf442-0ab2-4993-bf49-d0375abe1e62-host-run-ovn-kubernetes\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.619794 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.619767    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f0967024-b386-49dd-a50f-4b69599548aa-device-dir\") pod \"aws-ebs-csi-driver-node-hh49h\" (UID: \"f0967024-b386-49dd-a50f-4b69599548aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hh49h"
Jul 28 10:07:46.619794 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.619794    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f9fe54ae-5731-4fdd-b668-c70aca78b5bf-system-cni-dir\") pod \"multus-additional-cni-plugins-gzgcl\" (UID: \"f9fe54ae-5731-4fdd-b668-c70aca78b5bf\") " pod="openshift-multus/multus-additional-cni-plugins-gzgcl"
Jul 28 10:07:46.619794 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.619792    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f0967024-b386-49dd-a50f-4b69599548aa-etc-selinux\") pod \"aws-ebs-csi-driver-node-hh49h\" (UID: \"f0967024-b386-49dd-a50f-4b69599548aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hh49h"
Jul 28 10:07:46.620445 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.619816    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ceccf442-0ab2-4993-bf49-d0375abe1e62-run-ovn\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.620445 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.619720    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ceccf442-0ab2-4993-bf49-d0375abe1e62-var-lib-openvswitch\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.620445 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.619954    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ceccf442-0ab2-4993-bf49-d0375abe1e62-host-cni-netd\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.620445 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.620008    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f9fe54ae-5731-4fdd-b668-c70aca78b5bf-cni-binary-copy\") pod \"multus-additional-cni-plugins-gzgcl\" (UID: \"f9fe54ae-5731-4fdd-b668-c70aca78b5bf\") " pod="openshift-multus/multus-additional-cni-plugins-gzgcl"
Jul 28 10:07:46.620445 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.619999    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f9fe54ae-5731-4fdd-b668-c70aca78b5bf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gzgcl\" (UID: \"f9fe54ae-5731-4fdd-b668-c70aca78b5bf\") " pod="openshift-multus/multus-additional-cni-plugins-gzgcl"
Jul 28 10:07:46.620445 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.620140    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f0967024-b386-49dd-a50f-4b69599548aa-socket-dir\") pod \"aws-ebs-csi-driver-node-hh49h\" (UID: \"f0967024-b386-49dd-a50f-4b69599548aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hh49h"
Jul 28 10:07:46.620445 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.620325    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ceccf442-0ab2-4993-bf49-d0375abe1e62-env-overrides\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.620747 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.620483    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6d704fd4-c194-4c37-a80e-dbde31ebd184-konnectivity-ca\") pod \"konnectivity-agent-v7gsk\" (UID: \"6d704fd4-c194-4c37-a80e-dbde31ebd184\") " pod="kube-system/konnectivity-agent-v7gsk"
Jul 28 10:07:46.620747 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.620048    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f9fe54ae-5731-4fdd-b668-c70aca78b5bf-os-release\") pod \"multus-additional-cni-plugins-gzgcl\" (UID: \"f9fe54ae-5731-4fdd-b668-c70aca78b5bf\") " pod="openshift-multus/multus-additional-cni-plugins-gzgcl"
Jul 28 10:07:46.620829 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.620776    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ceccf442-0ab2-4993-bf49-d0375abe1e62-ovnkube-config\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.620969 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.620937    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d23b40b0-9299-4a0b-a87c-d2f2cb921fd9-metrics-certs\") pod \"network-metrics-daemon-8fltb\" (UID: \"d23b40b0-9299-4a0b-a87c-d2f2cb921fd9\") " pod="openshift-multus/network-metrics-daemon-8fltb"
Jul 28 10:07:46.620969 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.620937    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ceccf442-0ab2-4993-bf49-d0375abe1e62-ovnkube-script-lib\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.621116 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.620986    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f0967024-b386-49dd-a50f-4b69599548aa-socket-dir\") pod \"aws-ebs-csi-driver-node-hh49h\" (UID: \"f0967024-b386-49dd-a50f-4b69599548aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hh49h"
Jul 28 10:07:46.621116 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.620990    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f9fe54ae-5731-4fdd-b668-c70aca78b5bf-cnibin\") pod \"multus-additional-cni-plugins-gzgcl\" (UID: \"f9fe54ae-5731-4fdd-b668-c70aca78b5bf\") " pod="openshift-multus/multus-additional-cni-plugins-gzgcl"
Jul 28 10:07:46.621116 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.621032    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f9fe54ae-5731-4fdd-b668-c70aca78b5bf-cnibin\") pod \"multus-additional-cni-plugins-gzgcl\" (UID: \"f9fe54ae-5731-4fdd-b668-c70aca78b5bf\") " pod="openshift-multus/multus-additional-cni-plugins-gzgcl"
Jul 28 10:07:46.621116 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:46.621048    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:07:46.621116 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.621067    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f0967024-b386-49dd-a50f-4b69599548aa-sys-fs\") pod \"aws-ebs-csi-driver-node-hh49h\" (UID: \"f0967024-b386-49dd-a50f-4b69599548aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hh49h"
Jul 28 10:07:46.621116 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:46.621097    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d23b40b0-9299-4a0b-a87c-d2f2cb921fd9-metrics-certs podName:d23b40b0-9299-4a0b-a87c-d2f2cb921fd9 nodeName:}" failed. No retries permitted until 2026-07-28 10:07:47.121079598 +0000 UTC m=+3.079847978 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d23b40b0-9299-4a0b-a87c-d2f2cb921fd9-metrics-certs") pod "network-metrics-daemon-8fltb" (UID: "d23b40b0-9299-4a0b-a87c-d2f2cb921fd9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:07:46.621116 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.621116    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ceccf442-0ab2-4993-bf49-d0375abe1e62-host-slash\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.621379 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.621128    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f0967024-b386-49dd-a50f-4b69599548aa-sys-fs\") pod \"aws-ebs-csi-driver-node-hh49h\" (UID: \"f0967024-b386-49dd-a50f-4b69599548aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hh49h"
Jul 28 10:07:46.621379 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.621148    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f9fe54ae-5731-4fdd-b668-c70aca78b5bf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gzgcl\" (UID: \"f9fe54ae-5731-4fdd-b668-c70aca78b5bf\") " pod="openshift-multus/multus-additional-cni-plugins-gzgcl"
Jul 28 10:07:46.621379 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.621172    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ceccf442-0ab2-4993-bf49-d0375abe1e62-host-slash\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.621379 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.621180    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f9fe54ae-5731-4fdd-b668-c70aca78b5bf-cni-binary-copy\") pod \"multus-additional-cni-plugins-gzgcl\" (UID: \"f9fe54ae-5731-4fdd-b668-c70aca78b5bf\") " pod="openshift-multus/multus-additional-cni-plugins-gzgcl"
Jul 28 10:07:46.621379 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.621173    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f0967024-b386-49dd-a50f-4b69599548aa-registration-dir\") pod \"aws-ebs-csi-driver-node-hh49h\" (UID: \"f0967024-b386-49dd-a50f-4b69599548aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hh49h"
Jul 28 10:07:46.621379 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.621224    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ceccf442-0ab2-4993-bf49-d0375abe1e62-node-log\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.621379 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.621242    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5lph\" (UniqueName: \"kubernetes.io/projected/d23b40b0-9299-4a0b-a87c-d2f2cb921fd9-kube-api-access-t5lph\") pod \"network-metrics-daemon-8fltb\" (UID: \"d23b40b0-9299-4a0b-a87c-d2f2cb921fd9\") " pod="openshift-multus/network-metrics-daemon-8fltb"
Jul 28 10:07:46.621379 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.621267    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb-dbus\") pod \"global-pull-secret-syncer-jhkxw\" (UID: \"ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb\") " pod="kube-system/global-pull-secret-syncer-jhkxw"
Jul 28 10:07:46.621379 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.621288    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ceccf442-0ab2-4993-bf49-d0375abe1e62-node-log\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.621379 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.621294    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pbkb\" (UniqueName: \"kubernetes.io/projected/f8ad89c4-5ef2-477b-ab4c-e2c97b9a68fa-kube-api-access-9pbkb\") pod \"node-resolver-5z8xh\" (UID: \"f8ad89c4-5ef2-477b-ab4c-e2c97b9a68fa\") " pod="openshift-dns/node-resolver-5z8xh"
Jul 28 10:07:46.621379 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.621319    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8ad89c4-5ef2-477b-ab4c-e2c97b9a68fa-tmp-dir\") pod \"node-resolver-5z8xh\" (UID: \"f8ad89c4-5ef2-477b-ab4c-e2c97b9a68fa\") " pod="openshift-dns/node-resolver-5z8xh"
Jul 28 10:07:46.621379 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.621341    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ceccf442-0ab2-4993-bf49-d0375abe1e62-systemd-units\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.621379 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.621360    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ceccf442-0ab2-4993-bf49-d0375abe1e62-etc-openvswitch\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.621379 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.621380    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6d704fd4-c194-4c37-a80e-dbde31ebd184-agent-certs\") pod \"konnectivity-agent-v7gsk\" (UID: \"6d704fd4-c194-4c37-a80e-dbde31ebd184\") " pod="kube-system/konnectivity-agent-v7gsk"
Jul 28 10:07:46.622004 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.621597    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb-dbus\") pod \"global-pull-secret-syncer-jhkxw\" (UID: \"ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb\") " pod="kube-system/global-pull-secret-syncer-jhkxw"
Jul 28 10:07:46.622004 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.621878    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chxfc\" (UniqueName: \"kubernetes.io/projected/f0967024-b386-49dd-a50f-4b69599548aa-kube-api-access-chxfc\") pod \"aws-ebs-csi-driver-node-hh49h\" (UID: \"f0967024-b386-49dd-a50f-4b69599548aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hh49h"
Jul 28 10:07:46.622004 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.621944    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ceccf442-0ab2-4993-bf49-d0375abe1e62-systemd-units\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.622004 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.622000    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ceccf442-0ab2-4993-bf49-d0375abe1e62-host-kubelet\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.622216 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.622040    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ceccf442-0ab2-4993-bf49-d0375abe1e62-run-openvswitch\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.622216 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.622084    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f0967024-b386-49dd-a50f-4b69599548aa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hh49h\" (UID: \"f0967024-b386-49dd-a50f-4b69599548aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hh49h"
Jul 28 10:07:46.622216 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.622118    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6mbt\" (UniqueName: \"kubernetes.io/projected/f9fe54ae-5731-4fdd-b668-c70aca78b5bf-kube-api-access-b6mbt\") pod \"multus-additional-cni-plugins-gzgcl\" (UID: \"f9fe54ae-5731-4fdd-b668-c70aca78b5bf\") " pod="openshift-multus/multus-additional-cni-plugins-gzgcl"
Jul 28 10:07:46.622216 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.622155    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ceccf442-0ab2-4993-bf49-d0375abe1e62-host-run-netns\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.622216 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.622175    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f9fe54ae-5731-4fdd-b668-c70aca78b5bf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gzgcl\" (UID: \"f9fe54ae-5731-4fdd-b668-c70aca78b5bf\") " pod="openshift-multus/multus-additional-cni-plugins-gzgcl"
Jul 28 10:07:46.622216 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.622190    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ceccf442-0ab2-4993-bf49-d0375abe1e62-run-systemd\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.622216 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.622207    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8ad89c4-5ef2-477b-ab4c-e2c97b9a68fa-tmp-dir\") pod \"node-resolver-5z8xh\" (UID: \"f8ad89c4-5ef2-477b-ab4c-e2c97b9a68fa\") " pod="openshift-dns/node-resolver-5z8xh"
Jul 28 10:07:46.622589 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.621259    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f0967024-b386-49dd-a50f-4b69599548aa-registration-dir\") pod \"aws-ebs-csi-driver-node-hh49h\" (UID: \"f0967024-b386-49dd-a50f-4b69599548aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hh49h"
Jul 28 10:07:46.622589 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.622250    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ceccf442-0ab2-4993-bf49-d0375abe1e62-run-systemd\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.622589 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.622281    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f0967024-b386-49dd-a50f-4b69599548aa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hh49h\" (UID: \"f0967024-b386-49dd-a50f-4b69599548aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hh49h"
Jul 28 10:07:46.622589 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.622314    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ceccf442-0ab2-4993-bf49-d0375abe1e62-host-kubelet\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.622589 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.622339    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ceccf442-0ab2-4993-bf49-d0375abe1e62-run-openvswitch\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.622589 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.622375    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ceccf442-0ab2-4993-bf49-d0375abe1e62-log-socket\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.622589 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.622394    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ceccf442-0ab2-4993-bf49-d0375abe1e62-host-run-netns\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.622589 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.622438    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ceccf442-0ab2-4993-bf49-d0375abe1e62-etc-openvswitch\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.622589 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.622443    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ceccf442-0ab2-4993-bf49-d0375abe1e62-log-socket\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.622589 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.622484    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f9fe54ae-5731-4fdd-b668-c70aca78b5bf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gzgcl\" (UID: \"f9fe54ae-5731-4fdd-b668-c70aca78b5bf\") " pod="openshift-multus/multus-additional-cni-plugins-gzgcl"
Jul 28 10:07:46.622589 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.622525    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phrdk\" (UniqueName: \"kubernetes.io/projected/0bc74d3d-3abf-47f5-8149-7d07a9a34e7c-kube-api-access-phrdk\") pod \"network-check-target-ddrhj\" (UID: \"0bc74d3d-3abf-47f5-8149-7d07a9a34e7c\") " pod="openshift-network-diagnostics/network-check-target-ddrhj"
Jul 28 10:07:46.622589 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.622561    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f8ad89c4-5ef2-477b-ab4c-e2c97b9a68fa-hosts-file\") pod \"node-resolver-5z8xh\" (UID: \"f8ad89c4-5ef2-477b-ab4c-e2c97b9a68fa\") " pod="openshift-dns/node-resolver-5z8xh"
Jul 28 10:07:46.623187 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.622603    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ceccf442-0ab2-4993-bf49-d0375abe1e62-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.623187 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.622640    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ceccf442-0ab2-4993-bf49-d0375abe1e62-ovn-node-metrics-cert\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.623187 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.622726    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f9fe54ae-5731-4fdd-b668-c70aca78b5bf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gzgcl\" (UID: \"f9fe54ae-5731-4fdd-b668-c70aca78b5bf\") " pod="openshift-multus/multus-additional-cni-plugins-gzgcl"
Jul 28 10:07:46.623187 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.622785    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ceccf442-0ab2-4993-bf49-d0375abe1e62-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.623187 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.622847    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f8ad89c4-5ef2-477b-ab4c-e2c97b9a68fa-hosts-file\") pod \"node-resolver-5z8xh\" (UID: \"f8ad89c4-5ef2-477b-ab4c-e2c97b9a68fa\") " pod="openshift-dns/node-resolver-5z8xh"
Jul 28 10:07:46.625881 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.625004    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6d704fd4-c194-4c37-a80e-dbde31ebd184-agent-certs\") pod \"konnectivity-agent-v7gsk\" (UID: \"6d704fd4-c194-4c37-a80e-dbde31ebd184\") " pod="kube-system/konnectivity-agent-v7gsk"
Jul 28 10:07:46.626549 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.626523    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ceccf442-0ab2-4993-bf49-d0375abe1e62-ovn-node-metrics-cert\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.633225 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.633160    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5lph\" (UniqueName: \"kubernetes.io/projected/d23b40b0-9299-4a0b-a87c-d2f2cb921fd9-kube-api-access-t5lph\") pod \"network-metrics-daemon-8fltb\" (UID: \"d23b40b0-9299-4a0b-a87c-d2f2cb921fd9\") " pod="openshift-multus/network-metrics-daemon-8fltb"
Jul 28 10:07:46.633720 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.633467    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh26c\" (UniqueName: \"kubernetes.io/projected/ceccf442-0ab2-4993-bf49-d0375abe1e62-kube-api-access-zh26c\") pod \"ovnkube-node-hdqls\" (UID: \"ceccf442-0ab2-4993-bf49-d0375abe1e62\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:46.637016 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:46.636948    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 10:07:46.637016 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:46.636977    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 10:07:46.637016 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:46.636992    2557 projected.go:196] Error preparing data for projected volume kube-api-access-phrdk for pod openshift-network-diagnostics/network-check-target-ddrhj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:07:46.637236 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:46.637057    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0bc74d3d-3abf-47f5-8149-7d07a9a34e7c-kube-api-access-phrdk podName:0bc74d3d-3abf-47f5-8149-7d07a9a34e7c nodeName:}" failed. No retries permitted until 2026-07-28 10:07:47.137037933 +0000 UTC m=+3.095806293 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-phrdk" (UniqueName: "kubernetes.io/projected/0bc74d3d-3abf-47f5-8149-7d07a9a34e7c-kube-api-access-phrdk") pod "network-check-target-ddrhj" (UID: "0bc74d3d-3abf-47f5-8149-7d07a9a34e7c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:07:46.638485 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.638454    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6mbt\" (UniqueName: \"kubernetes.io/projected/f9fe54ae-5731-4fdd-b668-c70aca78b5bf-kube-api-access-b6mbt\") pod \"multus-additional-cni-plugins-gzgcl\" (UID: \"f9fe54ae-5731-4fdd-b668-c70aca78b5bf\") " pod="openshift-multus/multus-additional-cni-plugins-gzgcl"
Jul 28 10:07:46.639758 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.639735    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chxfc\" (UniqueName: \"kubernetes.io/projected/f0967024-b386-49dd-a50f-4b69599548aa-kube-api-access-chxfc\") pod \"aws-ebs-csi-driver-node-hh49h\" (UID: \"f0967024-b386-49dd-a50f-4b69599548aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hh49h"
Jul 28 10:07:46.639929 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.639907    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pbkb\" (UniqueName: \"kubernetes.io/projected/f8ad89c4-5ef2-477b-ab4c-e2c97b9a68fa-kube-api-access-9pbkb\") pod \"node-resolver-5z8xh\" (UID: \"f8ad89c4-5ef2-477b-ab4c-e2c97b9a68fa\") " pod="openshift-dns/node-resolver-5z8xh"
Jul 28 10:07:46.711587 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.711527    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vgktv"
Jul 28 10:07:46.724188 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:46.724161    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58c000f3_06db_4572_8abb_89ca11078c3c.slice/crio-16e00fb95d165c3592ed88a7fd8444f38ada4ec3a26bb9a733aa1de85bd95be2 WatchSource:0}: Error finding container 16e00fb95d165c3592ed88a7fd8444f38ada4ec3a26bb9a733aa1de85bd95be2: Status 404 returned error can't find the container with id 16e00fb95d165c3592ed88a7fd8444f38ada4ec3a26bb9a733aa1de85bd95be2
Jul 28 10:07:46.725084 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.725058    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-p2rv8"
Jul 28 10:07:46.733890 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:46.733855    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e80f436_0cf4_45a3_8fdf_53bdae6a7fad.slice/crio-2552cdf1012b93e7c3a4199c231fdda64234654637ee1b611e338ae9ada140a1 WatchSource:0}: Error finding container 2552cdf1012b93e7c3a4199c231fdda64234654637ee1b611e338ae9ada140a1: Status 404 returned error can't find the container with id 2552cdf1012b93e7c3a4199c231fdda64234654637ee1b611e338ae9ada140a1
Jul 28 10:07:46.735035 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.735009    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-msf7n"
Jul 28 10:07:46.740856 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.740828    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-h8krb"
Jul 28 10:07:46.745066 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:46.745034    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c5a2bd4_8e86_493f_9479_1a183cc866e6.slice/crio-b14cc8563f2a116fbd09a0228c202f1d8265a26b6978303eb211b671ca502a0d WatchSource:0}: Error finding container b14cc8563f2a116fbd09a0228c202f1d8265a26b6978303eb211b671ca502a0d: Status 404 returned error can't find the container with id b14cc8563f2a116fbd09a0228c202f1d8265a26b6978303eb211b671ca502a0d
Jul 28 10:07:46.749934 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.749910    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gzgcl"
Jul 28 10:07:46.752153 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:46.752127    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1a543f9_5413_4260_b48d_f767a66f23c7.slice/crio-0068c17f78bc3f3ad509597efda04d1e7a8d1acc8501617dd795e7e71b9b3aff WatchSource:0}: Error finding container 0068c17f78bc3f3ad509597efda04d1e7a8d1acc8501617dd795e7e71b9b3aff: Status 404 returned error can't find the container with id 0068c17f78bc3f3ad509597efda04d1e7a8d1acc8501617dd795e7e71b9b3aff
Jul 28 10:07:46.760443 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.760421    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v7gsk"
Jul 28 10:07:46.761051 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:46.761025    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9fe54ae_5731_4fdd_b668_c70aca78b5bf.slice/crio-122c62ba521fd47557252b0eb34d737d661655295ef576aec67d8eacec7f7f60 WatchSource:0}: Error finding container 122c62ba521fd47557252b0eb34d737d661655295ef576aec67d8eacec7f7f60: Status 404 returned error can't find the container with id 122c62ba521fd47557252b0eb34d737d661655295ef576aec67d8eacec7f7f60
Jul 28 10:07:46.770631 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.770605    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hh49h"
Jul 28 10:07:46.774538 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:46.774511    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d704fd4_c194_4c37_a80e_dbde31ebd184.slice/crio-7774b94ed326fcde59acd393c1cb942a40ceb9fecf959b906dd2697fa65089e1 WatchSource:0}: Error finding container 7774b94ed326fcde59acd393c1cb942a40ceb9fecf959b906dd2697fa65089e1: Status 404 returned error can't find the container with id 7774b94ed326fcde59acd393c1cb942a40ceb9fecf959b906dd2697fa65089e1
Jul 28 10:07:46.777944 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.777923    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5z8xh"
Jul 28 10:07:46.785657 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:46.785635    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:07:47.076681 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:47.076653    2557 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 10:07:47.128146 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:47.128102    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb-original-pull-secret\") pod \"global-pull-secret-syncer-jhkxw\" (UID: \"ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb\") " pod="kube-system/global-pull-secret-syncer-jhkxw"
Jul 28 10:07:47.128300 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:47.128267    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 10:07:47.128300 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:47.128290    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d23b40b0-9299-4a0b-a87c-d2f2cb921fd9-metrics-certs\") pod \"network-metrics-daemon-8fltb\" (UID: \"d23b40b0-9299-4a0b-a87c-d2f2cb921fd9\") " pod="openshift-multus/network-metrics-daemon-8fltb"
Jul 28 10:07:47.128387 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:47.128338    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb-original-pull-secret podName:ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb nodeName:}" failed. No retries permitted until 2026-07-28 10:07:48.128320698 +0000 UTC m=+4.087089057 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb-original-pull-secret") pod "global-pull-secret-syncer-jhkxw" (UID: "ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb") : object "kube-system"/"original-pull-secret" not registered
Jul 28 10:07:47.128442 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:47.128399    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:07:47.128475 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:47.128455    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d23b40b0-9299-4a0b-a87c-d2f2cb921fd9-metrics-certs podName:d23b40b0-9299-4a0b-a87c-d2f2cb921fd9 nodeName:}" failed. No retries permitted until 2026-07-28 10:07:48.128442423 +0000 UTC m=+4.087210782 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d23b40b0-9299-4a0b-a87c-d2f2cb921fd9-metrics-certs") pod "network-metrics-daemon-8fltb" (UID: "d23b40b0-9299-4a0b-a87c-d2f2cb921fd9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:07:47.229026 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:47.228986    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phrdk\" (UniqueName: \"kubernetes.io/projected/0bc74d3d-3abf-47f5-8149-7d07a9a34e7c-kube-api-access-phrdk\") pod \"network-check-target-ddrhj\" (UID: \"0bc74d3d-3abf-47f5-8149-7d07a9a34e7c\") " pod="openshift-network-diagnostics/network-check-target-ddrhj"
Jul 28 10:07:47.229222 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:47.229198    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 10:07:47.229278 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:47.229231    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 10:07:47.229278 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:47.229248    2557 projected.go:196] Error preparing data for projected volume kube-api-access-phrdk for pod openshift-network-diagnostics/network-check-target-ddrhj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:07:47.229359 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:47.229314    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0bc74d3d-3abf-47f5-8149-7d07a9a34e7c-kube-api-access-phrdk podName:0bc74d3d-3abf-47f5-8149-7d07a9a34e7c nodeName:}" failed. No retries permitted until 2026-07-28 10:07:48.229293841 +0000 UTC m=+4.188062219 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-phrdk" (UniqueName: "kubernetes.io/projected/0bc74d3d-3abf-47f5-8149-7d07a9a34e7c-kube-api-access-phrdk") pod "network-check-target-ddrhj" (UID: "0bc74d3d-3abf-47f5-8149-7d07a9a34e7c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:07:47.489124 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:47.489085    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 10:02:46 +0000 UTC" deadline="2027-12-31 10:19:14.454548029 +0000 UTC"
Jul 28 10:07:47.489124 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:47.489120    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12504h11m26.965431686s"
Jul 28 10:07:47.574866 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:47.574788    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddrhj"
Jul 28 10:07:47.575056 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:47.574921    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddrhj" podUID="0bc74d3d-3abf-47f5-8149-7d07a9a34e7c"
Jul 28 10:07:47.575056 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:47.574796    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jhkxw"
Jul 28 10:07:47.575056 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:47.574984    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8fltb"
Jul 28 10:07:47.575244 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:47.575057    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jhkxw" podUID="ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb"
Jul 28 10:07:47.575481 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:47.575429    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8fltb" podUID="d23b40b0-9299-4a0b-a87c-d2f2cb921fd9"
Jul 28 10:07:47.588557 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:47.587464    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vgktv" event={"ID":"58c000f3-06db-4572-8abb-89ca11078c3c","Type":"ContainerStarted","Data":"16e00fb95d165c3592ed88a7fd8444f38ada4ec3a26bb9a733aa1de85bd95be2"}
Jul 28 10:07:47.589194 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:47.589162    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v7gsk" event={"ID":"6d704fd4-c194-4c37-a80e-dbde31ebd184","Type":"ContainerStarted","Data":"7774b94ed326fcde59acd393c1cb942a40ceb9fecf959b906dd2697fa65089e1"}
Jul 28 10:07:47.591949 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:47.591876    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gzgcl" event={"ID":"f9fe54ae-5731-4fdd-b668-c70aca78b5bf","Type":"ContainerStarted","Data":"122c62ba521fd47557252b0eb34d737d661655295ef576aec67d8eacec7f7f60"}
Jul 28 10:07:47.595437 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:47.595371    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-h8krb" event={"ID":"a1a543f9-5413-4260-b48d-f767a66f23c7","Type":"ContainerStarted","Data":"0068c17f78bc3f3ad509597efda04d1e7a8d1acc8501617dd795e7e71b9b3aff"}
Jul 28 10:07:47.597709 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:47.597659    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-msf7n" event={"ID":"2c5a2bd4-8e86-493f-9479-1a183cc866e6","Type":"ContainerStarted","Data":"b14cc8563f2a116fbd09a0228c202f1d8265a26b6978303eb211b671ca502a0d"}
Jul 28 10:07:47.599364 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:47.599338    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-p2rv8" event={"ID":"4e80f436-0cf4-45a3-8fdf-53bdae6a7fad","Type":"ContainerStarted","Data":"2552cdf1012b93e7c3a4199c231fdda64234654637ee1b611e338ae9ada140a1"}
Jul 28 10:07:48.124558 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:48.124182    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0967024_b386_49dd_a50f_4b69599548aa.slice/crio-6001f55d379a8b753dc95d1917fc2630f7cd9018b33cd184953be21c9edb6fe9 WatchSource:0}: Error finding container 6001f55d379a8b753dc95d1917fc2630f7cd9018b33cd184953be21c9edb6fe9: Status 404 returned error can't find the container with id 6001f55d379a8b753dc95d1917fc2630f7cd9018b33cd184953be21c9edb6fe9
Jul 28 10:07:48.132247 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:48.132008    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8ad89c4_5ef2_477b_ab4c_e2c97b9a68fa.slice/crio-d1ea50587e2c7a67f9f7b40711b6843d4630e95ab46bb1c7b868ac50895c15c5 WatchSource:0}: Error finding container d1ea50587e2c7a67f9f7b40711b6843d4630e95ab46bb1c7b868ac50895c15c5: Status 404 returned error can't find the container with id d1ea50587e2c7a67f9f7b40711b6843d4630e95ab46bb1c7b868ac50895c15c5
Jul 28 10:07:48.137943 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:48.137915    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb-original-pull-secret\") pod \"global-pull-secret-syncer-jhkxw\" (UID: \"ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb\") " pod="kube-system/global-pull-secret-syncer-jhkxw"
Jul 28 10:07:48.138050 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:48.137978    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d23b40b0-9299-4a0b-a87c-d2f2cb921fd9-metrics-certs\") pod \"network-metrics-daemon-8fltb\" (UID: \"d23b40b0-9299-4a0b-a87c-d2f2cb921fd9\") " pod="openshift-multus/network-metrics-daemon-8fltb"
Jul 28 10:07:48.138153 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:48.138137    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:07:48.138224 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:48.138202    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d23b40b0-9299-4a0b-a87c-d2f2cb921fd9-metrics-certs podName:d23b40b0-9299-4a0b-a87c-d2f2cb921fd9 nodeName:}" failed. No retries permitted until 2026-07-28 10:07:50.138181125 +0000 UTC m=+6.096949486 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d23b40b0-9299-4a0b-a87c-d2f2cb921fd9-metrics-certs") pod "network-metrics-daemon-8fltb" (UID: "d23b40b0-9299-4a0b-a87c-d2f2cb921fd9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:07:48.138296 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:48.138267    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 10:07:48.138360 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:48.138324    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb-original-pull-secret podName:ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb nodeName:}" failed. No retries permitted until 2026-07-28 10:07:50.138307023 +0000 UTC m=+6.097075385 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb-original-pull-secret") pod "global-pull-secret-syncer-jhkxw" (UID: "ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb") : object "kube-system"/"original-pull-secret" not registered
Jul 28 10:07:48.163732 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:07:48.163696    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceccf442_0ab2_4993_bf49_d0375abe1e62.slice/crio-9250f38548f4fb0d9157aa02de819af6d887589a2ad2e4f35f56deb292699e5f WatchSource:0}: Error finding container 9250f38548f4fb0d9157aa02de819af6d887589a2ad2e4f35f56deb292699e5f: Status 404 returned error can't find the container with id 9250f38548f4fb0d9157aa02de819af6d887589a2ad2e4f35f56deb292699e5f
Jul 28 10:07:48.239741 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:48.239651    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phrdk\" (UniqueName: \"kubernetes.io/projected/0bc74d3d-3abf-47f5-8149-7d07a9a34e7c-kube-api-access-phrdk\") pod \"network-check-target-ddrhj\" (UID: \"0bc74d3d-3abf-47f5-8149-7d07a9a34e7c\") " pod="openshift-network-diagnostics/network-check-target-ddrhj"
Jul 28 10:07:48.244314 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:48.244272    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 10:07:48.244314 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:48.244319    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 10:07:48.244554 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:48.244335    2557 projected.go:196] Error preparing data for projected volume kube-api-access-phrdk for pod openshift-network-diagnostics/network-check-target-ddrhj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:07:48.244554 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:48.244425    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0bc74d3d-3abf-47f5-8149-7d07a9a34e7c-kube-api-access-phrdk podName:0bc74d3d-3abf-47f5-8149-7d07a9a34e7c nodeName:}" failed. No retries permitted until 2026-07-28 10:07:50.244403604 +0000 UTC m=+6.203171977 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-phrdk" (UniqueName: "kubernetes.io/projected/0bc74d3d-3abf-47f5-8149-7d07a9a34e7c-kube-api-access-phrdk") pod "network-check-target-ddrhj" (UID: "0bc74d3d-3abf-47f5-8149-7d07a9a34e7c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:07:48.489784 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:48.489652    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 10:02:46 +0000 UTC" deadline="2027-12-24 02:36:50.368942823 +0000 UTC"
Jul 28 10:07:48.489784 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:48.489717    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12328h29m1.87923111s"
Jul 28 10:07:48.607074 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:48.606988    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5z8xh" event={"ID":"f8ad89c4-5ef2-477b-ab4c-e2c97b9a68fa","Type":"ContainerStarted","Data":"d1ea50587e2c7a67f9f7b40711b6843d4630e95ab46bb1c7b868ac50895c15c5"}
Jul 28 10:07:48.611148 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:48.611003    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hdqls" event={"ID":"ceccf442-0ab2-4993-bf49-d0375abe1e62","Type":"ContainerStarted","Data":"9250f38548f4fb0d9157aa02de819af6d887589a2ad2e4f35f56deb292699e5f"}
Jul 28 10:07:48.613130 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:48.613108    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hh49h" event={"ID":"f0967024-b386-49dd-a50f-4b69599548aa","Type":"ContainerStarted","Data":"6001f55d379a8b753dc95d1917fc2630f7cd9018b33cd184953be21c9edb6fe9"}
Jul 28 10:07:48.619616 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:48.619563    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-88.ec2.internal" event={"ID":"6ff9d18249fb2db5835b1f8658874702","Type":"ContainerStarted","Data":"34c7b37cb8243828daa015f6b3c7d7784513899fefdc59c24d57a8337712f66f"}
Jul 28 10:07:48.637444 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:48.637389    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-88.ec2.internal" podStartSLOduration=2.637369074 podStartE2EDuration="2.637369074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:07:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:07:48.636762957 +0000 UTC m=+4.595531353" watchObservedRunningTime="2026-07-28 10:07:48.637369074 +0000 UTC m=+4.596137458"
Jul 28 10:07:49.575621 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:49.574698    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8fltb"
Jul 28 10:07:49.575621 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:49.574868    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8fltb" podUID="d23b40b0-9299-4a0b-a87c-d2f2cb921fd9"
Jul 28 10:07:49.575621 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:49.575283    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddrhj"
Jul 28 10:07:49.575621 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:49.575379    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddrhj" podUID="0bc74d3d-3abf-47f5-8149-7d07a9a34e7c"
Jul 28 10:07:49.575621 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:49.575439    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jhkxw"
Jul 28 10:07:49.575621 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:49.575522    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jhkxw" podUID="ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb"
Jul 28 10:07:49.624777 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:49.624736    2557 generic.go:356] "Generic (PLEG): container finished" podID="ae61811891b0aefce4345602b66dee13" containerID="60ea7e67bc0d011238d2df780f3aa8e3d413392bfe089b1565be83cb70f2648e" exitCode=0
Jul 28 10:07:49.625346 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:49.625319    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-88.ec2.internal" event={"ID":"ae61811891b0aefce4345602b66dee13","Type":"ContainerDied","Data":"60ea7e67bc0d011238d2df780f3aa8e3d413392bfe089b1565be83cb70f2648e"}
Jul 28 10:07:50.157001 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:50.156956    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb-original-pull-secret\") pod \"global-pull-secret-syncer-jhkxw\" (UID: \"ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb\") " pod="kube-system/global-pull-secret-syncer-jhkxw"
Jul 28 10:07:50.157220 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:50.157040    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d23b40b0-9299-4a0b-a87c-d2f2cb921fd9-metrics-certs\") pod \"network-metrics-daemon-8fltb\" (UID: \"d23b40b0-9299-4a0b-a87c-d2f2cb921fd9\") " pod="openshift-multus/network-metrics-daemon-8fltb"
Jul 28 10:07:50.157220 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:50.157204    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:07:50.157347 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:50.157278    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d23b40b0-9299-4a0b-a87c-d2f2cb921fd9-metrics-certs podName:d23b40b0-9299-4a0b-a87c-d2f2cb921fd9 nodeName:}" failed. No retries permitted until 2026-07-28 10:07:54.157257646 +0000 UTC m=+10.116026008 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d23b40b0-9299-4a0b-a87c-d2f2cb921fd9-metrics-certs") pod "network-metrics-daemon-8fltb" (UID: "d23b40b0-9299-4a0b-a87c-d2f2cb921fd9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:07:50.157775 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:50.157718    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 10:07:50.157887 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:50.157777    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb-original-pull-secret podName:ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb nodeName:}" failed. No retries permitted until 2026-07-28 10:07:54.157761138 +0000 UTC m=+10.116529497 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb-original-pull-secret") pod "global-pull-secret-syncer-jhkxw" (UID: "ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb") : object "kube-system"/"original-pull-secret" not registered
Jul 28 10:07:50.258818 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:50.258145    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phrdk\" (UniqueName: \"kubernetes.io/projected/0bc74d3d-3abf-47f5-8149-7d07a9a34e7c-kube-api-access-phrdk\") pod \"network-check-target-ddrhj\" (UID: \"0bc74d3d-3abf-47f5-8149-7d07a9a34e7c\") " pod="openshift-network-diagnostics/network-check-target-ddrhj"
Jul 28 10:07:50.258818 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:50.258331    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 10:07:50.258818 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:50.258358    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 10:07:50.258818 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:50.258371    2557 projected.go:196] Error preparing data for projected volume kube-api-access-phrdk for pod openshift-network-diagnostics/network-check-target-ddrhj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:07:50.258818 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:50.258434    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0bc74d3d-3abf-47f5-8149-7d07a9a34e7c-kube-api-access-phrdk podName:0bc74d3d-3abf-47f5-8149-7d07a9a34e7c nodeName:}" failed. No retries permitted until 2026-07-28 10:07:54.25841326 +0000 UTC m=+10.217181621 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-phrdk" (UniqueName: "kubernetes.io/projected/0bc74d3d-3abf-47f5-8149-7d07a9a34e7c-kube-api-access-phrdk") pod "network-check-target-ddrhj" (UID: "0bc74d3d-3abf-47f5-8149-7d07a9a34e7c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:07:51.574958 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:51.574924    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddrhj"
Jul 28 10:07:51.575463 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:51.575051    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddrhj" podUID="0bc74d3d-3abf-47f5-8149-7d07a9a34e7c"
Jul 28 10:07:51.575463 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:51.575431    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jhkxw"
Jul 28 10:07:51.575580 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:51.575528    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jhkxw" podUID="ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb"
Jul 28 10:07:51.575631 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:51.575588    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8fltb"
Jul 28 10:07:51.575716 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:51.575657    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8fltb" podUID="d23b40b0-9299-4a0b-a87c-d2f2cb921fd9"
Jul 28 10:07:53.574570 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:53.574530    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8fltb"
Jul 28 10:07:53.575050 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:53.574716    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8fltb" podUID="d23b40b0-9299-4a0b-a87c-d2f2cb921fd9"
Jul 28 10:07:53.575128 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:53.575114    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddrhj"
Jul 28 10:07:53.575211 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:53.575189    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddrhj" podUID="0bc74d3d-3abf-47f5-8149-7d07a9a34e7c"
Jul 28 10:07:53.575272 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:53.575263    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jhkxw"
Jul 28 10:07:53.575330 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:53.575316    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jhkxw" podUID="ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb"
Jul 28 10:07:54.193240 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:54.192497    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d23b40b0-9299-4a0b-a87c-d2f2cb921fd9-metrics-certs\") pod \"network-metrics-daemon-8fltb\" (UID: \"d23b40b0-9299-4a0b-a87c-d2f2cb921fd9\") " pod="openshift-multus/network-metrics-daemon-8fltb"
Jul 28 10:07:54.193240 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:54.192596    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb-original-pull-secret\") pod \"global-pull-secret-syncer-jhkxw\" (UID: \"ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb\") " pod="kube-system/global-pull-secret-syncer-jhkxw"
Jul 28 10:07:54.193240 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:54.192745    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 10:07:54.193240 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:54.192808    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb-original-pull-secret podName:ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb nodeName:}" failed. No retries permitted until 2026-07-28 10:08:02.192790525 +0000 UTC m=+18.151558896 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb-original-pull-secret") pod "global-pull-secret-syncer-jhkxw" (UID: "ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb") : object "kube-system"/"original-pull-secret" not registered
Jul 28 10:07:54.193240 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:54.193208    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:07:54.193544 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:54.193258    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d23b40b0-9299-4a0b-a87c-d2f2cb921fd9-metrics-certs podName:d23b40b0-9299-4a0b-a87c-d2f2cb921fd9 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:02.193244102 +0000 UTC m=+18.152012472 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d23b40b0-9299-4a0b-a87c-d2f2cb921fd9-metrics-certs") pod "network-metrics-daemon-8fltb" (UID: "d23b40b0-9299-4a0b-a87c-d2f2cb921fd9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:07:54.294071 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:54.293992    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phrdk\" (UniqueName: \"kubernetes.io/projected/0bc74d3d-3abf-47f5-8149-7d07a9a34e7c-kube-api-access-phrdk\") pod \"network-check-target-ddrhj\" (UID: \"0bc74d3d-3abf-47f5-8149-7d07a9a34e7c\") " pod="openshift-network-diagnostics/network-check-target-ddrhj"
Jul 28 10:07:54.294258 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:54.294191    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 10:07:54.294258 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:54.294221    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 10:07:54.294258 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:54.294234    2557 projected.go:196] Error preparing data for projected volume kube-api-access-phrdk for pod openshift-network-diagnostics/network-check-target-ddrhj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:07:54.294411 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:54.294299    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0bc74d3d-3abf-47f5-8149-7d07a9a34e7c-kube-api-access-phrdk podName:0bc74d3d-3abf-47f5-8149-7d07a9a34e7c nodeName:}" failed. No retries permitted until 2026-07-28 10:08:02.294278392 +0000 UTC m=+18.253046772 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-phrdk" (UniqueName: "kubernetes.io/projected/0bc74d3d-3abf-47f5-8149-7d07a9a34e7c-kube-api-access-phrdk") pod "network-check-target-ddrhj" (UID: "0bc74d3d-3abf-47f5-8149-7d07a9a34e7c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:07:55.575519 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:55.574825    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddrhj"
Jul 28 10:07:55.575519 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:55.574937    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddrhj" podUID="0bc74d3d-3abf-47f5-8149-7d07a9a34e7c"
Jul 28 10:07:55.575519 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:55.575298    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jhkxw"
Jul 28 10:07:55.575519 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:55.575379    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jhkxw" podUID="ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb"
Jul 28 10:07:55.575519 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:55.575419    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8fltb"
Jul 28 10:07:55.575519 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:55.575484    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8fltb" podUID="d23b40b0-9299-4a0b-a87c-d2f2cb921fd9"
Jul 28 10:07:57.574243 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:57.574210    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8fltb"
Jul 28 10:07:57.574734 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:57.574213    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddrhj"
Jul 28 10:07:57.574734 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:57.574324    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8fltb" podUID="d23b40b0-9299-4a0b-a87c-d2f2cb921fd9"
Jul 28 10:07:57.574734 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:57.574419    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddrhj" podUID="0bc74d3d-3abf-47f5-8149-7d07a9a34e7c"
Jul 28 10:07:57.574734 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:57.574225    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jhkxw"
Jul 28 10:07:57.574734 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:57.574525    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jhkxw" podUID="ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb"
Jul 28 10:07:59.574368 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:59.574331    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jhkxw"
Jul 28 10:07:59.574782 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:59.574331    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8fltb"
Jul 28 10:07:59.574782 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:59.574459    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jhkxw" podUID="ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb"
Jul 28 10:07:59.574782 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:07:59.574331    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddrhj"
Jul 28 10:07:59.574782 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:59.574522    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8fltb" podUID="d23b40b0-9299-4a0b-a87c-d2f2cb921fd9"
Jul 28 10:07:59.574782 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:07:59.574579    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddrhj" podUID="0bc74d3d-3abf-47f5-8149-7d07a9a34e7c"
Jul 28 10:08:01.574385 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:01.574294    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddrhj"
Jul 28 10:08:01.574385 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:01.574347    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8fltb"
Jul 28 10:08:01.574816 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:01.574303    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jhkxw"
Jul 28 10:08:01.574816 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:01.574440    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddrhj" podUID="0bc74d3d-3abf-47f5-8149-7d07a9a34e7c"
Jul 28 10:08:01.574816 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:01.574515    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jhkxw" podUID="ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb"
Jul 28 10:08:01.574816 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:01.574592    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8fltb" podUID="d23b40b0-9299-4a0b-a87c-d2f2cb921fd9"
Jul 28 10:08:02.256224 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:02.256188    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d23b40b0-9299-4a0b-a87c-d2f2cb921fd9-metrics-certs\") pod \"network-metrics-daemon-8fltb\" (UID: \"d23b40b0-9299-4a0b-a87c-d2f2cb921fd9\") " pod="openshift-multus/network-metrics-daemon-8fltb"
Jul 28 10:08:02.256410 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:02.256258    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb-original-pull-secret\") pod \"global-pull-secret-syncer-jhkxw\" (UID: \"ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb\") " pod="kube-system/global-pull-secret-syncer-jhkxw"
Jul 28 10:08:02.256410 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:02.256360    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:08:02.256511 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:02.256431    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d23b40b0-9299-4a0b-a87c-d2f2cb921fd9-metrics-certs podName:d23b40b0-9299-4a0b-a87c-d2f2cb921fd9 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:18.256413922 +0000 UTC m=+34.215182279 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d23b40b0-9299-4a0b-a87c-d2f2cb921fd9-metrics-certs") pod "network-metrics-daemon-8fltb" (UID: "d23b40b0-9299-4a0b-a87c-d2f2cb921fd9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:08:02.256511 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:02.256364    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 10:08:02.256511 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:02.256503    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb-original-pull-secret podName:ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb nodeName:}" failed. No retries permitted until 2026-07-28 10:08:18.256490997 +0000 UTC m=+34.215259356 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb-original-pull-secret") pod "global-pull-secret-syncer-jhkxw" (UID: "ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb") : object "kube-system"/"original-pull-secret" not registered
Jul 28 10:08:02.357580 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:02.357544    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phrdk\" (UniqueName: \"kubernetes.io/projected/0bc74d3d-3abf-47f5-8149-7d07a9a34e7c-kube-api-access-phrdk\") pod \"network-check-target-ddrhj\" (UID: \"0bc74d3d-3abf-47f5-8149-7d07a9a34e7c\") " pod="openshift-network-diagnostics/network-check-target-ddrhj"
Jul 28 10:08:02.357754 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:02.357737    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 10:08:02.357810 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:02.357760    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 10:08:02.357810 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:02.357772    2557 projected.go:196] Error preparing data for projected volume kube-api-access-phrdk for pod openshift-network-diagnostics/network-check-target-ddrhj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:08:02.357894 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:02.357826    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0bc74d3d-3abf-47f5-8149-7d07a9a34e7c-kube-api-access-phrdk podName:0bc74d3d-3abf-47f5-8149-7d07a9a34e7c nodeName:}" failed. No retries permitted until 2026-07-28 10:08:18.357812004 +0000 UTC m=+34.316580361 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-phrdk" (UniqueName: "kubernetes.io/projected/0bc74d3d-3abf-47f5-8149-7d07a9a34e7c-kube-api-access-phrdk") pod "network-check-target-ddrhj" (UID: "0bc74d3d-3abf-47f5-8149-7d07a9a34e7c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:08:03.574161 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:03.574119    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddrhj"
Jul 28 10:08:03.574581 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:03.574119    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jhkxw"
Jul 28 10:08:03.574581 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:03.574229    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddrhj" podUID="0bc74d3d-3abf-47f5-8149-7d07a9a34e7c"
Jul 28 10:08:03.574581 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:03.574239    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8fltb"
Jul 28 10:08:03.574581 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:03.574310    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jhkxw" podUID="ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb"
Jul 28 10:08:03.574581 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:03.574391    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8fltb" podUID="d23b40b0-9299-4a0b-a87c-d2f2cb921fd9"
Jul 28 10:08:05.574406 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:05.573992    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddrhj"
Jul 28 10:08:05.575065 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:05.574006    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8fltb"
Jul 28 10:08:05.575065 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:05.574529    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddrhj" podUID="0bc74d3d-3abf-47f5-8149-7d07a9a34e7c"
Jul 28 10:08:05.575065 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:05.574593    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8fltb" podUID="d23b40b0-9299-4a0b-a87c-d2f2cb921fd9"
Jul 28 10:08:05.575065 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:05.574006    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jhkxw"
Jul 28 10:08:05.575065 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:05.574714    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jhkxw" podUID="ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb"
Jul 28 10:08:05.658448 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:05.658417    2557 generic.go:356] "Generic (PLEG): container finished" podID="f9fe54ae-5731-4fdd-b668-c70aca78b5bf" containerID="f245e6502eb9f9302e73e3296597c23d10774b34967a55c9d1b93676201520b4" exitCode=0
Jul 28 10:08:05.658690 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:05.658497    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gzgcl" event={"ID":"f9fe54ae-5731-4fdd-b668-c70aca78b5bf","Type":"ContainerDied","Data":"f245e6502eb9f9302e73e3296597c23d10774b34967a55c9d1b93676201520b4"}
Jul 28 10:08:05.663041 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:05.662994    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-h8krb" event={"ID":"a1a543f9-5413-4260-b48d-f767a66f23c7","Type":"ContainerStarted","Data":"cd2947c0f06bfe3e937d684a5f807665ab616d2774e94351e975aa5246a303c9"}
Jul 28 10:08:05.665722 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:05.665696    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-p2rv8" event={"ID":"4e80f436-0cf4-45a3-8fdf-53bdae6a7fad","Type":"ContainerStarted","Data":"157f5e6e5205b7a1bf137de6a40b1f800c58b354a583931db41a28ae420d0fb2"}
Jul 28 10:08:05.667302 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:05.667276    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5z8xh" event={"ID":"f8ad89c4-5ef2-477b-ab4c-e2c97b9a68fa","Type":"ContainerStarted","Data":"f9f541cb09d00b76e0157a23c4c1034b33c1a143854c1f092ba4b12a58727b9e"}
Jul 28 10:08:05.670248 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:05.670224    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hdqls" event={"ID":"ceccf442-0ab2-4993-bf49-d0375abe1e62","Type":"ContainerStarted","Data":"a3baaada8ee2e191d3968d2a6c608eb8642581792f0763f0de6405aa554e7de4"}
Jul 28 10:08:05.670337 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:05.670284    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hdqls" event={"ID":"ceccf442-0ab2-4993-bf49-d0375abe1e62","Type":"ContainerStarted","Data":"345f9e7b879f5b5ea2bd31973a36acc3ec42919df84c48a28e59f580b7a5e184"}
Jul 28 10:08:05.670337 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:05.670327    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hdqls" event={"ID":"ceccf442-0ab2-4993-bf49-d0375abe1e62","Type":"ContainerStarted","Data":"1902726d552ff29ebe28fef4fe588b9dc84074691b6e8e16896f4716ee9f7b87"}
Jul 28 10:08:05.670446 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:05.670375    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hdqls" event={"ID":"ceccf442-0ab2-4993-bf49-d0375abe1e62","Type":"ContainerStarted","Data":"01e14e20f00d4c81cf3e59b7c541499a54b6bf69eada80fc098224a81172ca47"}
Jul 28 10:08:05.673924 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:05.673898    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-88.ec2.internal" event={"ID":"ae61811891b0aefce4345602b66dee13","Type":"ContainerStarted","Data":"42763336f33799b3cfd466cfde80dcdc424be01c0366160788855d39036c3280"}
Jul 28 10:08:05.675260 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:05.675228    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v7gsk" event={"ID":"6d704fd4-c194-4c37-a80e-dbde31ebd184","Type":"ContainerStarted","Data":"6bd80a3a78376cc3fa232862215bb1b37ab64dc3f8fffd5d98aa1dc91f425326"}
Jul 28 10:08:05.676709 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:05.676687    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-msf7n" event={"ID":"2c5a2bd4-8e86-493f-9479-1a183cc866e6","Type":"ContainerStarted","Data":"9cf1c9cc5f3598c4cc9c7afe899659e075ad25aa8733532bfe77144786e82ac9"}
Jul 28 10:08:05.678212 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:05.678176    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hh49h" event={"ID":"f0967024-b386-49dd-a50f-4b69599548aa","Type":"ContainerStarted","Data":"2b387f892b09397759b847020cf7ab75d7510535c0f188108c97809204bf3f38"}
Jul 28 10:08:05.696740 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:05.696697    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-88.ec2.internal" podStartSLOduration=19.696683188 podStartE2EDuration="19.696683188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:07:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:05.696518017 +0000 UTC m=+21.655286396" watchObservedRunningTime="2026-07-28 10:08:05.696683188 +0000 UTC m=+21.655451567"
Jul 28 10:08:05.712103 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:05.712042    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-h8krb" podStartSLOduration=2.297194243 podStartE2EDuration="20.71201862s" totalImagesPullingTime="18.414824377s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:07:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:05.710962605 +0000 UTC m=+21.669730984" watchObservedRunningTime="2026-07-28 10:08:05.71201862 +0000 UTC m=+21.670787000"
Jul 28 10:08:05.725822 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:05.725781    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-5z8xh" podStartSLOduration=3.687155447 podStartE2EDuration="20.725765657s" totalImagesPullingTime="17.03861021s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:07:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:05.725637005 +0000 UTC m=+21.684405381" watchObservedRunningTime="2026-07-28 10:08:05.725765657 +0000 UTC m=+21.684534035"
Jul 28 10:08:05.743430 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:05.743352    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-p2rv8" podStartSLOduration=2.306700208 podStartE2EDuration="20.743329556s" totalImagesPullingTime="18.436629348s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:07:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:05.743128674 +0000 UTC m=+21.701897055" watchObservedRunningTime="2026-07-28 10:08:05.743329556 +0000 UTC m=+21.702097936"
Jul 28 10:08:05.762846 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:05.762791    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-msf7n" podStartSLOduration=2.338071517 podStartE2EDuration="20.762774857s" totalImagesPullingTime="18.42470334s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:07:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:05.762586317 +0000 UTC m=+21.721354696" watchObservedRunningTime="2026-07-28 10:08:05.762774857 +0000 UTC m=+21.721543239"
Jul 28 10:08:05.776766 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:05.776711    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-v7gsk" podStartSLOduration=11.102364908 podStartE2EDuration="20.776694888s" totalImagesPullingTime="9.67432998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:07:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:05.776348701 +0000 UTC m=+21.735117080" watchObservedRunningTime="2026-07-28 10:08:05.776694888 +0000 UTC m=+21.735463267"
Jul 28 10:08:06.401969 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:06.401926    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-v7gsk"
Jul 28 10:08:06.402879 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:06.402787    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-v7gsk"
Jul 28 10:08:06.404418 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:06.404398    2557 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 10:08:06.499355 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:06.499082    2557 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T10:08:06.404413473Z","UUID":"f246084d-1ec4-4dd3-8651-25b38d0dda97","Handler":null,"Name":"","Endpoint":""}
Jul 28 10:08:06.500806 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:06.500784    2557 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 10:08:06.500920 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:06.500829    2557 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 10:08:06.682741 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:06.682695    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vgktv" event={"ID":"58c000f3-06db-4572-8abb-89ca11078c3c","Type":"ContainerStarted","Data":"01d8fc6bc41035971d3bd886e8de318bf977301f4f65914b91ec2991ce2f8f9c"}
Jul 28 10:08:06.686346 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:06.686318    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hdqls" event={"ID":"ceccf442-0ab2-4993-bf49-d0375abe1e62","Type":"ContainerStarted","Data":"0dec39604333182bd7e907e5b468ebbe8dd9ee0dc6f12e46571dd0bdd2d3a586"}
Jul 28 10:08:06.686479 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:06.686399    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hdqls" event={"ID":"ceccf442-0ab2-4993-bf49-d0375abe1e62","Type":"ContainerStarted","Data":"caee19a99c462be667150bbaa3028b51dbdb49c66e1aec072c74c77f36c38248"}
Jul 28 10:08:06.688352 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:06.688245    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hh49h" event={"ID":"f0967024-b386-49dd-a50f-4b69599548aa","Type":"ContainerStarted","Data":"aa654808a90fe677eafbaedc9ad303b9684940900671d5f50d4c424a9fee28fb"}
Jul 28 10:08:06.689190 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:06.689168    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-v7gsk"
Jul 28 10:08:06.689657 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:06.689639    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-v7gsk"
Jul 28 10:08:06.699206 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:06.699168    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vgktv" podStartSLOduration=8.174472197 podStartE2EDuration="21.699156612s" totalImagesPullingTime="13.524684415s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:07:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:06.699006056 +0000 UTC m=+22.657774435" watchObservedRunningTime="2026-07-28 10:08:06.699156612 +0000 UTC m=+22.657924991"
Jul 28 10:08:07.574293 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:07.574256    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddrhj"
Jul 28 10:08:07.574693 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:07.574399    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddrhj" podUID="0bc74d3d-3abf-47f5-8149-7d07a9a34e7c"
Jul 28 10:08:07.574693 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:07.574431    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jhkxw"
Jul 28 10:08:07.574693 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:07.574490    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8fltb"
Jul 28 10:08:07.574693 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:07.574595    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jhkxw" podUID="ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb"
Jul 28 10:08:07.574962 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:07.574717    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8fltb" podUID="d23b40b0-9299-4a0b-a87c-d2f2cb921fd9"
Jul 28 10:08:08.697403 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:08.697155    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hdqls" event={"ID":"ceccf442-0ab2-4993-bf49-d0375abe1e62","Type":"ContainerStarted","Data":"515097ce996388a5b3c356c5773a58c424ea0d5e872e279c49a2a403140ebfa2"}
Jul 28 10:08:08.699413 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:08.699381    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hh49h" event={"ID":"f0967024-b386-49dd-a50f-4b69599548aa","Type":"ContainerStarted","Data":"aa7a27cf9de4cf4dfc720f93a26c7f0124c464357b271bf2970df39037eccf15"}
Jul 28 10:08:08.721835 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:08.721781    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hh49h" podStartSLOduration=4.580137317 podStartE2EDuration="23.721761736s" totalImagesPullingTime="19.141624419s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:07:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:08.721023545 +0000 UTC m=+24.679791945" watchObservedRunningTime="2026-07-28 10:08:08.721761736 +0000 UTC m=+24.680530115"
Jul 28 10:08:09.574849 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:09.574810    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddrhj"
Jul 28 10:08:09.575077 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:09.574945    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jhkxw"
Jul 28 10:08:09.575077 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:09.574945    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddrhj" podUID="0bc74d3d-3abf-47f5-8149-7d07a9a34e7c"
Jul 28 10:08:09.575077 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:09.574950    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8fltb"
Jul 28 10:08:09.575077 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:09.575055    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jhkxw" podUID="ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb"
Jul 28 10:08:09.575279 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:09.575128    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8fltb" podUID="d23b40b0-9299-4a0b-a87c-d2f2cb921fd9"
Jul 28 10:08:10.706536 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:10.706336    2557 generic.go:356] "Generic (PLEG): container finished" podID="f9fe54ae-5731-4fdd-b668-c70aca78b5bf" containerID="91cbba8526807708dbd95c6da95cd4a6fa67312f5e1a9afdb1420d8e99aea6be" exitCode=0
Jul 28 10:08:10.707365 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:10.706417    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gzgcl" event={"ID":"f9fe54ae-5731-4fdd-b668-c70aca78b5bf","Type":"ContainerDied","Data":"91cbba8526807708dbd95c6da95cd4a6fa67312f5e1a9afdb1420d8e99aea6be"}
Jul 28 10:08:10.710494 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:10.710467    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hdqls" event={"ID":"ceccf442-0ab2-4993-bf49-d0375abe1e62","Type":"ContainerStarted","Data":"0069ecc123f8a52065d5135c8ebbf657373b468d8c81e169e60ed27f709ef13b"}
Jul 28 10:08:10.710797 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:10.710775    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:08:10.710890 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:10.710844    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:08:10.710890 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:10.710887    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:08:10.726658 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:10.726633    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:08:10.726847 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:10.726830    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:08:10.764740 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:10.764690    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hdqls" podStartSLOduration=8.65001629 podStartE2EDuration="25.764674065s" totalImagesPullingTime="17.114657775s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:07:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:10.764275565 +0000 UTC m=+26.723043943" watchObservedRunningTime="2026-07-28 10:08:10.764674065 +0000 UTC m=+26.723442443"
Jul 28 10:08:11.574542 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:11.574512    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddrhj"
Jul 28 10:08:11.574724 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:11.574553    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8fltb"
Jul 28 10:08:11.574724 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:11.574554    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jhkxw"
Jul 28 10:08:11.574724 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:11.574668    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddrhj" podUID="0bc74d3d-3abf-47f5-8149-7d07a9a34e7c"
Jul 28 10:08:11.574836 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:11.574786    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8fltb" podUID="d23b40b0-9299-4a0b-a87c-d2f2cb921fd9"
Jul 28 10:08:11.574932 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:11.574869    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jhkxw" podUID="ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb"
Jul 28 10:08:11.714902 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:11.714865    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gzgcl" event={"ID":"f9fe54ae-5731-4fdd-b668-c70aca78b5bf","Type":"ContainerStarted","Data":"42dfedda0b863b88ba8e45a69c10ed9bf1dc89d83b547e1e78cee88cac2bd71b"}
Jul 28 10:08:11.884537 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:11.884454    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jhkxw"]
Jul 28 10:08:11.884684 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:11.884567    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jhkxw"
Jul 28 10:08:11.884684 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:11.884664    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jhkxw" podUID="ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb"
Jul 28 10:08:11.888467 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:11.888440    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ddrhj"]
Jul 28 10:08:11.888629 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:11.888563    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddrhj"
Jul 28 10:08:11.888701 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:11.888678    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddrhj" podUID="0bc74d3d-3abf-47f5-8149-7d07a9a34e7c"
Jul 28 10:08:11.889118 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:11.889097    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8fltb"]
Jul 28 10:08:11.889199 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:11.889190    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8fltb"
Jul 28 10:08:11.889279 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:11.889260    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8fltb" podUID="d23b40b0-9299-4a0b-a87c-d2f2cb921fd9"
Jul 28 10:08:12.718982 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:12.718729    2557 generic.go:356] "Generic (PLEG): container finished" podID="f9fe54ae-5731-4fdd-b668-c70aca78b5bf" containerID="42dfedda0b863b88ba8e45a69c10ed9bf1dc89d83b547e1e78cee88cac2bd71b" exitCode=0
Jul 28 10:08:12.718982 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:12.718815    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gzgcl" event={"ID":"f9fe54ae-5731-4fdd-b668-c70aca78b5bf","Type":"ContainerDied","Data":"42dfedda0b863b88ba8e45a69c10ed9bf1dc89d83b547e1e78cee88cac2bd71b"}
Jul 28 10:08:13.574731 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:13.574706    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddrhj"
Jul 28 10:08:13.574837 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:13.574734    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jhkxw"
Jul 28 10:08:13.574837 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:13.574810    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8fltb"
Jul 28 10:08:13.574837 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:13.574817    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddrhj" podUID="0bc74d3d-3abf-47f5-8149-7d07a9a34e7c"
Jul 28 10:08:13.574954 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:13.574878    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jhkxw" podUID="ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb"
Jul 28 10:08:13.574995 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:13.574966    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8fltb" podUID="d23b40b0-9299-4a0b-a87c-d2f2cb921fd9"
Jul 28 10:08:13.722887 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:13.722851    2557 generic.go:356] "Generic (PLEG): container finished" podID="f9fe54ae-5731-4fdd-b668-c70aca78b5bf" containerID="71864ac48366257f5109e4891426c0106498ca1c85acda040843ee642bfaffc7" exitCode=0
Jul 28 10:08:13.723240 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:13.722905    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gzgcl" event={"ID":"f9fe54ae-5731-4fdd-b668-c70aca78b5bf","Type":"ContainerDied","Data":"71864ac48366257f5109e4891426c0106498ca1c85acda040843ee642bfaffc7"}
Jul 28 10:08:15.573965 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:15.573932    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddrhj"
Jul 28 10:08:15.574629 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:15.573935    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8fltb"
Jul 28 10:08:15.574629 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:15.574056    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddrhj" podUID="0bc74d3d-3abf-47f5-8149-7d07a9a34e7c"
Jul 28 10:08:15.574629 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:15.573938    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jhkxw"
Jul 28 10:08:15.574629 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:15.574157    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8fltb" podUID="d23b40b0-9299-4a0b-a87c-d2f2cb921fd9"
Jul 28 10:08:15.574629 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:15.574196    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jhkxw" podUID="ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb"
Jul 28 10:08:17.574778 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:17.574741    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jhkxw"
Jul 28 10:08:17.575174 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:17.574741    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8fltb"
Jul 28 10:08:17.575174 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:17.574856    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jhkxw" podUID="ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb"
Jul 28 10:08:17.575174 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:17.574741    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddrhj"
Jul 28 10:08:17.575174 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:17.574924    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8fltb" podUID="d23b40b0-9299-4a0b-a87c-d2f2cb921fd9"
Jul 28 10:08:17.575174 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:17.575007    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddrhj" podUID="0bc74d3d-3abf-47f5-8149-7d07a9a34e7c"
Jul 28 10:08:17.886011 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:17.885985    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-88.ec2.internal" event="NodeReady"
Jul 28 10:08:17.886179 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:17.886119    2557 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 10:08:17.925682 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:17.925653    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-69b959dbcb-z2hmw"]
Jul 28 10:08:17.934110 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:17.934088    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69b959dbcb-z2hmw"
Jul 28 10:08:17.936420 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:17.936384    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 28 10:08:17.936532 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:17.936424    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 28 10:08:17.937426 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:17.937403    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 28 10:08:17.937845 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:17.937802    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-fn5hp\""
Jul 28 10:08:17.937956 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:17.937855    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ljlnk"]
Jul 28 10:08:17.943543 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:17.943515    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 28 10:08:17.952449 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:17.952419    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-nsq8v"]
Jul 28 10:08:17.952578 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:17.952544    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ljlnk"
Jul 28 10:08:17.955448 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:17.954868    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dfkl4\""
Jul 28 10:08:17.955448 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:17.954973    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 10:08:17.955448 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:17.954877    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 10:08:17.955448 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:17.954972    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 10:08:17.962676 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:17.962650    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69b959dbcb-z2hmw"]
Jul 28 10:08:17.962676 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:17.962681    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ljlnk"]
Jul 28 10:08:17.962887 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:17.962688    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nsq8v"
Jul 28 10:08:17.962887 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:17.962698    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nsq8v"]
Jul 28 10:08:17.965853 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:17.965821    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dv6fr\""
Jul 28 10:08:17.965978 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:17.965855    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 10:08:17.965978 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:17.965902    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 10:08:18.072773 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:18.072743    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e43afa04-2c0d-445d-a2c8-86512466e05b-trusted-ca\") pod \"image-registry-69b959dbcb-z2hmw\" (UID: \"e43afa04-2c0d-445d-a2c8-86512466e05b\") " pod="openshift-image-registry/image-registry-69b959dbcb-z2hmw"
Jul 28 10:08:18.072918 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:18.072796    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dznwt\" (UniqueName: \"kubernetes.io/projected/c3fe785d-4bae-4988-85f0-2a02259913b1-kube-api-access-dznwt\") pod \"dns-default-nsq8v\" (UID: \"c3fe785d-4bae-4988-85f0-2a02259913b1\") " pod="openshift-dns/dns-default-nsq8v"
Jul 28 10:08:18.072918 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:18.072814    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/74cc1682-9675-406b-8fe7-78713ec81657-cert\") pod \"ingress-canary-ljlnk\" (UID: \"74cc1682-9675-406b-8fe7-78713ec81657\") " pod="openshift-ingress-canary/ingress-canary-ljlnk"
Jul 28 10:08:18.072918 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:18.072872    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e43afa04-2c0d-445d-a2c8-86512466e05b-registry-certificates\") pod \"image-registry-69b959dbcb-z2hmw\" (UID: \"e43afa04-2c0d-445d-a2c8-86512466e05b\") " pod="openshift-image-registry/image-registry-69b959dbcb-z2hmw"
Jul 28 10:08:18.072918 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:18.072902    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e43afa04-2c0d-445d-a2c8-86512466e05b-bound-sa-token\") pod \"image-registry-69b959dbcb-z2hmw\" (UID: \"e43afa04-2c0d-445d-a2c8-86512466e05b\") " pod="openshift-image-registry/image-registry-69b959dbcb-z2hmw"
Jul 28 10:08:18.073078 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:18.072970    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3fe785d-4bae-4988-85f0-2a02259913b1-tmp-dir\") pod \"dns-default-nsq8v\" (UID: \"c3fe785d-4bae-4988-85f0-2a02259913b1\") " pod="openshift-dns/dns-default-nsq8v"
Jul 28 10:08:18.073078 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:18.073002    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e43afa04-2c0d-445d-a2c8-86512466e05b-registry-tls\") pod \"image-registry-69b959dbcb-z2hmw\" (UID: \"e43afa04-2c0d-445d-a2c8-86512466e05b\") " pod="openshift-image-registry/image-registry-69b959dbcb-z2hmw"
Jul 28 10:08:18.073078 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:18.073019    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzql9\" (UniqueName: \"kubernetes.io/projected/74cc1682-9675-406b-8fe7-78713ec81657-kube-api-access-bzql9\") pod \"ingress-canary-ljlnk\" (UID: \"74cc1682-9675-406b-8fe7-78713ec81657\") " pod="openshift-ingress-canary/ingress-canary-ljlnk"
Jul 28 10:08:18.073078 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:18.073037    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e43afa04-2c0d-445d-a2c8-86512466e05b-installation-pull-secrets\") pod \"image-registry-69b959dbcb-z2hmw\" (UID: \"e43afa04-2c0d-445d-a2c8-86512466e05b\") " pod="openshift-image-registry/image-registry-69b959dbcb-z2hmw"
Jul 28 10:08:18.073259 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:18.073087    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e43afa04-2c0d-445d-a2c8-86512466e05b-ca-trust-extracted\") pod \"image-registry-69b959dbcb-z2hmw\" (UID: \"e43afa04-2c0d-445d-a2c8-86512466e05b\") " pod="openshift-image-registry/image-registry-69b959dbcb-z2hmw"
Jul 28 10:08:18.073259 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:18.073129    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e43afa04-2c0d-445d-a2c8-86512466e05b-image-registry-private-configuration\") pod \"image-registry-69b959dbcb-z2hmw\" (UID: \"e43afa04-2c0d-445d-a2c8-86512466e05b\") " pod="openshift-image-registry/image-registry-69b959dbcb-z2hmw"
Jul 28 10:08:18.073259 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:18.073152    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg72v\" (UniqueName: \"kubernetes.io/projected/e43afa04-2c0d-445d-a2c8-86512466e05b-kube-api-access-zg72v\") pod \"image-registry-69b959dbcb-z2hmw\" (UID: \"e43afa04-2c0d-445d-a2c8-86512466e05b\") " pod="openshift-image-registry/image-registry-69b959dbcb-z2hmw"
Jul 28 10:08:18.073259 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:18.073171    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c3fe785d-4bae-4988-85f0-2a02259913b1-config-volume\") pod \"dns-default-nsq8v\" (UID: \"c3fe785d-4bae-4988-85f0-2a02259913b1\") " pod="openshift-dns/dns-default-nsq8v"
Jul 28 10:08:18.073259 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:18.073191    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c3fe785d-4bae-4988-85f0-2a02259913b1-metrics-tls\") pod \"dns-default-nsq8v\" (UID: \"c3fe785d-4bae-4988-85f0-2a02259913b1\") " pod="openshift-dns/dns-default-nsq8v"
Jul 28 10:08:18.174105 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:18.174024    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c3fe785d-4bae-4988-85f0-2a02259913b1-config-volume\") pod \"dns-default-nsq8v\" (UID: \"c3fe785d-4bae-4988-85f0-2a02259913b1\") " pod="openshift-dns/dns-default-nsq8v"
Jul 28 10:08:18.174105 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:18.174058    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c3fe785d-4bae-4988-85f0-2a02259913b1-metrics-tls\") pod \"dns-default-nsq8v\" (UID: \"c3fe785d-4bae-4988-85f0-2a02259913b1\") " pod="openshift-dns/dns-default-nsq8v"
Jul 28 10:08:18.174105 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:18.174078    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e43afa04-2c0d-445d-a2c8-86512466e05b-trusted-ca\") pod \"image-registry-69b959dbcb-z2hmw\" (UID: \"e43afa04-2c0d-445d-a2c8-86512466e05b\") " pod="openshift-image-registry/image-registry-69b959dbcb-z2hmw"
Jul 28 10:08:18.174105 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:18.174107    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dznwt\" (UniqueName: \"kubernetes.io/projected/c3fe785d-4bae-4988-85f0-2a02259913b1-kube-api-access-dznwt\") pod \"dns-default-nsq8v\" (UID: \"c3fe785d-4bae-4988-85f0-2a02259913b1\") " pod="openshift-dns/dns-default-nsq8v"
Jul 28 10:08:18.174404 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:18.174124    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/74cc1682-9675-406b-8fe7-78713ec81657-cert\") pod \"ingress-canary-ljlnk\" (UID: \"74cc1682-9675-406b-8fe7-78713ec81657\") " pod="openshift-ingress-canary/ingress-canary-ljlnk"
Jul 28 10:08:18.174404 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:18.174141    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e43afa04-2c0d-445d-a2c8-86512466e05b-registry-certificates\") pod \"image-registry-69b959dbcb-z2hmw\" (UID: \"e43afa04-2c0d-445d-a2c8-86512466e05b\") " pod="openshift-image-registry/image-registry-69b959dbcb-z2hmw"
Jul 28 10:08:18.174404 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:18.174158    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e43afa04-2c0d-445d-a2c8-86512466e05b-bound-sa-token\") pod \"image-registry-69b959dbcb-z2hmw\" (UID: \"e43afa04-2c0d-445d-a2c8-86512466e05b\") " pod="openshift-image-registry/image-registry-69b959dbcb-z2hmw"
Jul 28 10:08:18.174404 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:18.174177    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 10:08:18.174404 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:18.174205    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3fe785d-4bae-4988-85f0-2a02259913b1-tmp-dir\") pod \"dns-default-nsq8v\" (UID: \"c3fe785d-4bae-4988-85f0-2a02259913b1\") " pod="openshift-dns/dns-default-nsq8v"
Jul 28 10:08:18.174404 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:18.174263    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3fe785d-4bae-4988-85f0-2a02259913b1-metrics-tls podName:c3fe785d-4bae-4988-85f0-2a02259913b1 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:18.674236385 +0000 UTC m=+34.633004762 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c3fe785d-4bae-4988-85f0-2a02259913b1-metrics-tls") pod "dns-default-nsq8v" (UID: "c3fe785d-4bae-4988-85f0-2a02259913b1") : secret "dns-default-metrics-tls" not found
Jul 28 10:08:18.174404 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:18.174294    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e43afa04-2c0d-445d-a2c8-86512466e05b-registry-tls\") pod \"image-registry-69b959dbcb-z2hmw\" (UID: \"e43afa04-2c0d-445d-a2c8-86512466e05b\") " pod="openshift-image-registry/image-registry-69b959dbcb-z2hmw"
Jul 28 10:08:18.174404 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:18.174332    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzql9\" (UniqueName: \"kubernetes.io/projected/74cc1682-9675-406b-8fe7-78713ec81657-kube-api-access-bzql9\") pod \"ingress-canary-ljlnk\" (UID: \"74cc1682-9675-406b-8fe7-78713ec81657\") " pod="openshift-ingress-canary/ingress-canary-ljlnk"
Jul 28 10:08:18.174404 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:18.174368    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e43afa04-2c0d-445d-a2c8-86512466e05b-installation-pull-secrets\") pod \"image-registry-69b959dbcb-z2hmw\" (UID: \"e43afa04-2c0d-445d-a2c8-86512466e05b\") " pod="openshift-image-registry/image-registry-69b959dbcb-z2hmw"
Jul 28 10:08:18.174404 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:18.174405    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e43afa04-2c0d-445d-a2c8-86512466e05b-ca-trust-extracted\") pod \"image-registry-69b959dbcb-z2hmw\" (UID: \"e43afa04-2c0d-445d-a2c8-86512466e05b\") " pod="openshift-image-registry/image-registry-69b959dbcb-z2hmw"
Jul 28 10:08:18.174940 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:18.174441    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e43afa04-2c0d-445d-a2c8-86512466e05b-image-registry-private-configuration\") pod \"image-registry-69b959dbcb-z2hmw\" (UID: \"e43afa04-2c0d-445d-a2c8-86512466e05b\") " pod="openshift-image-registry/image-registry-69b959dbcb-z2hmw"
Jul 28 10:08:18.174940 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:18.174473    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg72v\" (UniqueName: \"kubernetes.io/projected/e43afa04-2c0d-445d-a2c8-86512466e05b-kube-api-access-zg72v\") pod \"image-registry-69b959dbcb-z2hmw\" (UID: \"e43afa04-2c0d-445d-a2c8-86512466e05b\") " pod="openshift-image-registry/image-registry-69b959dbcb-z2hmw"
Jul 28 10:08:18.174940 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:18.174531    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3fe785d-4bae-4988-85f0-2a02259913b1-tmp-dir\") pod \"dns-default-nsq8v\" (UID: \"c3fe785d-4bae-4988-85f0-2a02259913b1\") " pod="openshift-dns/dns-default-nsq8v"
Jul 28 10:08:18.174940 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:18.174851    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c3fe785d-4bae-4988-85f0-2a02259913b1-config-volume\") pod \"dns-default-nsq8v\" (UID: \"c3fe785d-4bae-4988-85f0-2a02259913b1\") " pod="openshift-dns/dns-default-nsq8v"
Jul 28 10:08:18.174940 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:18.174879    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 10:08:18.174940 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:18.174895    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69b959dbcb-z2hmw: secret "image-registry-tls" not found
Jul 28 10:08:18.174940 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:18.174941    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e43afa04-2c0d-445d-a2c8-86512466e05b-registry-tls podName:e43afa04-2c0d-445d-a2c8-86512466e05b nodeName:}" failed. No retries permitted until 2026-07-28 10:08:18.674924966 +0000 UTC m=+34.633693344 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e43afa04-2c0d-445d-a2c8-86512466e05b-registry-tls") pod "image-registry-69b959dbcb-z2hmw" (UID: "e43afa04-2c0d-445d-a2c8-86512466e05b") : secret "image-registry-tls" not found
Jul 28 10:08:18.175221 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:18.174966    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e43afa04-2c0d-445d-a2c8-86512466e05b-registry-certificates\") pod \"image-registry-69b959dbcb-z2hmw\" (UID: \"e43afa04-2c0d-445d-a2c8-86512466e05b\") " pod="openshift-image-registry/image-registry-69b959dbcb-z2hmw"
Jul 28 10:08:18.175221 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:18.175164    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e43afa04-2c0d-445d-a2c8-86512466e05b-ca-trust-extracted\") pod \"image-registry-69b959dbcb-z2hmw\" (UID: \"e43afa04-2c0d-445d-a2c8-86512466e05b\") " pod="openshift-image-registry/image-registry-69b959dbcb-z2hmw"
Jul 28 10:08:18.175615 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:18.175574    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e43afa04-2c0d-445d-a2c8-86512466e05b-trusted-ca\") pod \"image-registry-69b959dbcb-z2hmw\" (UID: \"e43afa04-2c0d-445d-a2c8-86512466e05b\") " pod="openshift-image-registry/image-registry-69b959dbcb-z2hmw"
Jul 28 10:08:18.178466 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:18.178444    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e43afa04-2c0d-445d-a2c8-86512466e05b-image-registry-private-configuration\") pod \"image-registry-69b959dbcb-z2hmw\" (UID: \"e43afa04-2c0d-445d-a2c8-86512466e05b\") " pod="openshift-image-registry/image-registry-69b959dbcb-z2hmw"
Jul 28 10:08:18.178563 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:18.178483    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e43afa04-2c0d-445d-a2c8-86512466e05b-installation-pull-secrets\") pod \"image-registry-69b959dbcb-z2hmw\" (UID: \"e43afa04-2c0d-445d-a2c8-86512466e05b\") " pod="openshift-image-registry/image-registry-69b959dbcb-z2hmw"
Jul 28 10:08:18.178563 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:18.178523    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/74cc1682-9675-406b-8fe7-78713ec81657-cert\") pod \"ingress-canary-ljlnk\" (UID: \"74cc1682-9675-406b-8fe7-78713ec81657\") " pod="openshift-ingress-canary/ingress-canary-ljlnk"
Jul 28 10:08:18.182811 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:18.182783    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dznwt\" (UniqueName: \"kubernetes.io/projected/c3fe785d-4bae-4988-85f0-2a02259913b1-kube-api-access-dznwt\") pod \"dns-default-nsq8v\" (UID: \"c3fe785d-4bae-4988-85f0-2a02259913b1\") " pod="openshift-dns/dns-default-nsq8v"
Jul 28 10:08:18.183165 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:18.183122    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg72v\" (UniqueName: \"kubernetes.io/projected/e43afa04-2c0d-445d-a2c8-86512466e05b-kube-api-access-zg72v\") pod \"image-registry-69b959dbcb-z2hmw\" (UID: \"e43afa04-2c0d-445d-a2c8-86512466e05b\") " pod="openshift-image-registry/image-registry-69b959dbcb-z2hmw"
Jul 28 10:08:18.183512 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:18.183485    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e43afa04-2c0d-445d-a2c8-86512466e05b-bound-sa-token\") pod \"image-registry-69b959dbcb-z2hmw\" (UID: \"e43afa04-2c0d-445d-a2c8-86512466e05b\") " pod="openshift-image-registry/image-registry-69b959dbcb-z2hmw"
Jul 28 10:08:18.184033 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:18.184014    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzql9\" (UniqueName: \"kubernetes.io/projected/74cc1682-9675-406b-8fe7-78713ec81657-kube-api-access-bzql9\") pod \"ingress-canary-ljlnk\" (UID: \"74cc1682-9675-406b-8fe7-78713ec81657\") " pod="openshift-ingress-canary/ingress-canary-ljlnk"
Jul 28 10:08:18.262512 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:18.262469    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ljlnk"
Jul 28 10:08:18.274818 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:18.274778    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb-original-pull-secret\") pod \"global-pull-secret-syncer-jhkxw\" (UID: \"ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb\") " pod="kube-system/global-pull-secret-syncer-jhkxw"
Jul 28 10:08:18.274971 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:18.274827    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d23b40b0-9299-4a0b-a87c-d2f2cb921fd9-metrics-certs\") pod \"network-metrics-daemon-8fltb\" (UID: \"d23b40b0-9299-4a0b-a87c-d2f2cb921fd9\") " pod="openshift-multus/network-metrics-daemon-8fltb"
Jul 28 10:08:18.274971 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:18.274927    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 10:08:18.274971 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:18.274937    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:08:18.275075 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:18.274993    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d23b40b0-9299-4a0b-a87c-d2f2cb921fd9-metrics-certs podName:d23b40b0-9299-4a0b-a87c-d2f2cb921fd9 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:50.274976507 +0000 UTC m=+66.233744864 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d23b40b0-9299-4a0b-a87c-d2f2cb921fd9-metrics-certs") pod "network-metrics-daemon-8fltb" (UID: "d23b40b0-9299-4a0b-a87c-d2f2cb921fd9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:08:18.275075 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:18.275017    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb-original-pull-secret podName:ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb nodeName:}" failed. No retries permitted until 2026-07-28 10:08:50.275002491 +0000 UTC m=+66.233770863 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb-original-pull-secret") pod "global-pull-secret-syncer-jhkxw" (UID: "ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb") : object "kube-system"/"original-pull-secret" not registered
Jul 28 10:08:18.375548 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:18.375376    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phrdk\" (UniqueName: \"kubernetes.io/projected/0bc74d3d-3abf-47f5-8149-7d07a9a34e7c-kube-api-access-phrdk\") pod \"network-check-target-ddrhj\" (UID: \"0bc74d3d-3abf-47f5-8149-7d07a9a34e7c\") " pod="openshift-network-diagnostics/network-check-target-ddrhj"
Jul 28 10:08:18.375548 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:18.375537    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 10:08:18.375548 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:18.375557    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 10:08:18.375803 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:18.375568    2557 projected.go:196] Error preparing data for projected volume kube-api-access-phrdk for pod openshift-network-diagnostics/network-check-target-ddrhj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:08:18.375803 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:18.375645    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0bc74d3d-3abf-47f5-8149-7d07a9a34e7c-kube-api-access-phrdk podName:0bc74d3d-3abf-47f5-8149-7d07a9a34e7c nodeName:}" failed. No retries permitted until 2026-07-28 10:08:50.375630071 +0000 UTC m=+66.334398428 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-phrdk" (UniqueName: "kubernetes.io/projected/0bc74d3d-3abf-47f5-8149-7d07a9a34e7c-kube-api-access-phrdk") pod "network-check-target-ddrhj" (UID: "0bc74d3d-3abf-47f5-8149-7d07a9a34e7c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:08:18.432092 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:18.431972    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ljlnk"]
Jul 28 10:08:18.436623 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:08:18.436575    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74cc1682_9675_406b_8fe7_78713ec81657.slice/crio-1ba6893e551d1946af9a5f0785a709cea72d03f76fcd7c0f9da4bf0887c5a0a0 WatchSource:0}: Error finding container 1ba6893e551d1946af9a5f0785a709cea72d03f76fcd7c0f9da4bf0887c5a0a0: Status 404 returned error can't find the container with id 1ba6893e551d1946af9a5f0785a709cea72d03f76fcd7c0f9da4bf0887c5a0a0
Jul 28 10:08:18.677092 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:18.677059    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e43afa04-2c0d-445d-a2c8-86512466e05b-registry-tls\") pod \"image-registry-69b959dbcb-z2hmw\" (UID: \"e43afa04-2c0d-445d-a2c8-86512466e05b\") " pod="openshift-image-registry/image-registry-69b959dbcb-z2hmw"
Jul 28 10:08:18.677647 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:18.677103    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c3fe785d-4bae-4988-85f0-2a02259913b1-metrics-tls\") pod \"dns-default-nsq8v\" (UID: \"c3fe785d-4bae-4988-85f0-2a02259913b1\") " pod="openshift-dns/dns-default-nsq8v"
Jul 28 10:08:18.677647 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:18.677196    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 10:08:18.677647 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:18.677201    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 10:08:18.677647 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:18.677217    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69b959dbcb-z2hmw: secret "image-registry-tls" not found
Jul 28 10:08:18.677647 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:18.677247    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3fe785d-4bae-4988-85f0-2a02259913b1-metrics-tls podName:c3fe785d-4bae-4988-85f0-2a02259913b1 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:19.677234551 +0000 UTC m=+35.636002907 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c3fe785d-4bae-4988-85f0-2a02259913b1-metrics-tls") pod "dns-default-nsq8v" (UID: "c3fe785d-4bae-4988-85f0-2a02259913b1") : secret "dns-default-metrics-tls" not found
Jul 28 10:08:18.677647 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:18.677259    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e43afa04-2c0d-445d-a2c8-86512466e05b-registry-tls podName:e43afa04-2c0d-445d-a2c8-86512466e05b nodeName:}" failed. No retries permitted until 2026-07-28 10:08:19.677253391 +0000 UTC m=+35.636021748 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e43afa04-2c0d-445d-a2c8-86512466e05b-registry-tls") pod "image-registry-69b959dbcb-z2hmw" (UID: "e43afa04-2c0d-445d-a2c8-86512466e05b") : secret "image-registry-tls" not found
Jul 28 10:08:18.735412 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:18.735323    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ljlnk" event={"ID":"74cc1682-9675-406b-8fe7-78713ec81657","Type":"ContainerStarted","Data":"1ba6893e551d1946af9a5f0785a709cea72d03f76fcd7c0f9da4bf0887c5a0a0"}
Jul 28 10:08:18.737908 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:18.737880    2557 generic.go:356] "Generic (PLEG): container finished" podID="f9fe54ae-5731-4fdd-b668-c70aca78b5bf" containerID="52cd2da734332a46131bad9f8267d969518748b23bc692365eb2c3fc06870557" exitCode=0
Jul 28 10:08:18.737908 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:18.737917    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gzgcl" event={"ID":"f9fe54ae-5731-4fdd-b668-c70aca78b5bf","Type":"ContainerDied","Data":"52cd2da734332a46131bad9f8267d969518748b23bc692365eb2c3fc06870557"}
Jul 28 10:08:19.065852 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:19.065815    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-m4n6h"]
Jul 28 10:08:19.075894 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:19.075865    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-m4n6h"
Jul 28 10:08:19.079168 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:19.078893    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 28 10:08:19.079168 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:19.078935    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-g45vf\""
Jul 28 10:08:19.079168 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:19.079111    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-m4n6h"]
Jul 28 10:08:19.079766 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:19.079741    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 28 10:08:19.181140 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:19.181100    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/30e72787-1717-4a7a-b2e2-43e955527f77-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-m4n6h\" (UID: \"30e72787-1717-4a7a-b2e2-43e955527f77\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-m4n6h"
Jul 28 10:08:19.181319 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:19.181224    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/30e72787-1717-4a7a-b2e2-43e955527f77-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-m4n6h\" (UID: \"30e72787-1717-4a7a-b2e2-43e955527f77\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-m4n6h"
Jul 28 10:08:19.282390 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:19.282352    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/30e72787-1717-4a7a-b2e2-43e955527f77-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-m4n6h\" (UID: \"30e72787-1717-4a7a-b2e2-43e955527f77\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-m4n6h"
Jul 28 10:08:19.282589 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:19.282443    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/30e72787-1717-4a7a-b2e2-43e955527f77-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-m4n6h\" (UID: \"30e72787-1717-4a7a-b2e2-43e955527f77\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-m4n6h"
Jul 28 10:08:19.282686 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:19.282664    2557 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 10:08:19.282763 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:19.282751    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30e72787-1717-4a7a-b2e2-43e955527f77-networking-console-plugin-cert podName:30e72787-1717-4a7a-b2e2-43e955527f77 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:19.782727551 +0000 UTC m=+35.741495922 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/30e72787-1717-4a7a-b2e2-43e955527f77-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-m4n6h" (UID: "30e72787-1717-4a7a-b2e2-43e955527f77") : secret "networking-console-plugin-cert" not found
Jul 28 10:08:19.283089 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:19.283068    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/30e72787-1717-4a7a-b2e2-43e955527f77-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-m4n6h\" (UID: \"30e72787-1717-4a7a-b2e2-43e955527f77\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-m4n6h"
Jul 28 10:08:19.574715 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:19.574683    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jhkxw"
Jul 28 10:08:19.574895 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:19.574687    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddrhj"
Jul 28 10:08:19.574895 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:19.574695    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8fltb"
Jul 28 10:08:19.577439 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:19.577415    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 10:08:19.577565 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:19.577448    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 10:08:19.577565 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:19.577476    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l65kq\""
Jul 28 10:08:19.577565 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:19.577502    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 10:08:19.577565 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:19.577536    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xsq5w\""
Jul 28 10:08:19.578000 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:19.577816    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 10:08:19.686866 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:19.686830    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e43afa04-2c0d-445d-a2c8-86512466e05b-registry-tls\") pod \"image-registry-69b959dbcb-z2hmw\" (UID: \"e43afa04-2c0d-445d-a2c8-86512466e05b\") " pod="openshift-image-registry/image-registry-69b959dbcb-z2hmw"
Jul 28 10:08:19.687335 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:19.686922    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c3fe785d-4bae-4988-85f0-2a02259913b1-metrics-tls\") pod \"dns-default-nsq8v\" (UID: \"c3fe785d-4bae-4988-85f0-2a02259913b1\") " pod="openshift-dns/dns-default-nsq8v"
Jul 28 10:08:19.687335 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:19.687009    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 10:08:19.687335 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:19.687035    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69b959dbcb-z2hmw: secret "image-registry-tls" not found
Jul 28 10:08:19.687335 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:19.687099    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 10:08:19.687335 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:19.687109    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e43afa04-2c0d-445d-a2c8-86512466e05b-registry-tls podName:e43afa04-2c0d-445d-a2c8-86512466e05b nodeName:}" failed. No retries permitted until 2026-07-28 10:08:21.68708486 +0000 UTC m=+37.645853218 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e43afa04-2c0d-445d-a2c8-86512466e05b-registry-tls") pod "image-registry-69b959dbcb-z2hmw" (UID: "e43afa04-2c0d-445d-a2c8-86512466e05b") : secret "image-registry-tls" not found
Jul 28 10:08:19.687335 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:19.687209    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3fe785d-4bae-4988-85f0-2a02259913b1-metrics-tls podName:c3fe785d-4bae-4988-85f0-2a02259913b1 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:21.687190927 +0000 UTC m=+37.645959319 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c3fe785d-4bae-4988-85f0-2a02259913b1-metrics-tls") pod "dns-default-nsq8v" (UID: "c3fe785d-4bae-4988-85f0-2a02259913b1") : secret "dns-default-metrics-tls" not found
Jul 28 10:08:19.742990 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:19.742956    2557 generic.go:356] "Generic (PLEG): container finished" podID="f9fe54ae-5731-4fdd-b668-c70aca78b5bf" containerID="453ba6eaf465b35a2070872819ae068b7f55a702e22557417bbc090fc317d6e1" exitCode=0
Jul 28 10:08:19.743158 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:19.743015    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gzgcl" event={"ID":"f9fe54ae-5731-4fdd-b668-c70aca78b5bf","Type":"ContainerDied","Data":"453ba6eaf465b35a2070872819ae068b7f55a702e22557417bbc090fc317d6e1"}
Jul 28 10:08:19.787814 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:19.787782    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/30e72787-1717-4a7a-b2e2-43e955527f77-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-m4n6h\" (UID: \"30e72787-1717-4a7a-b2e2-43e955527f77\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-m4n6h"
Jul 28 10:08:19.787951 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:19.787885    2557 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 10:08:19.787951 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:19.787933    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30e72787-1717-4a7a-b2e2-43e955527f77-networking-console-plugin-cert podName:30e72787-1717-4a7a-b2e2-43e955527f77 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:20.787920489 +0000 UTC m=+36.746688847 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/30e72787-1717-4a7a-b2e2-43e955527f77-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-m4n6h" (UID: "30e72787-1717-4a7a-b2e2-43e955527f77") : secret "networking-console-plugin-cert" not found
Jul 28 10:08:20.749163 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:20.748969    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gzgcl" event={"ID":"f9fe54ae-5731-4fdd-b668-c70aca78b5bf","Type":"ContainerStarted","Data":"82b8e8a6ff6893132efd00abefa471664f403795dc9387aa2acf3618dfa36862"}
Jul 28 10:08:20.776731 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:20.776685    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-gzgcl" podStartSLOduration=7.439879172 podStartE2EDuration="35.776669939s" totalImagesPullingTime="28.336790767s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:07:45 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:20.775354657 +0000 UTC m=+36.734123036" watchObservedRunningTime="2026-07-28 10:08:20.776669939 +0000 UTC m=+36.735438318"
Jul 28 10:08:20.794464 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:20.794431    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/30e72787-1717-4a7a-b2e2-43e955527f77-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-m4n6h\" (UID: \"30e72787-1717-4a7a-b2e2-43e955527f77\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-m4n6h"
Jul 28 10:08:20.794623 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:20.794561    2557 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 10:08:20.794669 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:20.794629    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30e72787-1717-4a7a-b2e2-43e955527f77-networking-console-plugin-cert podName:30e72787-1717-4a7a-b2e2-43e955527f77 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:22.794614682 +0000 UTC m=+38.753383056 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/30e72787-1717-4a7a-b2e2-43e955527f77-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-m4n6h" (UID: "30e72787-1717-4a7a-b2e2-43e955527f77") : secret "networking-console-plugin-cert" not found
Jul 28 10:08:21.703138 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:21.703057    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c3fe785d-4bae-4988-85f0-2a02259913b1-metrics-tls\") pod \"dns-default-nsq8v\" (UID: \"c3fe785d-4bae-4988-85f0-2a02259913b1\") " pod="openshift-dns/dns-default-nsq8v"
Jul 28 10:08:21.703274 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:21.703146    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e43afa04-2c0d-445d-a2c8-86512466e05b-registry-tls\") pod \"image-registry-69b959dbcb-z2hmw\" (UID: \"e43afa04-2c0d-445d-a2c8-86512466e05b\") " pod="openshift-image-registry/image-registry-69b959dbcb-z2hmw"
Jul 28 10:08:21.703274 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:21.703228    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 10:08:21.703274 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:21.703246    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 10:08:21.703274 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:21.703257    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69b959dbcb-z2hmw: secret "image-registry-tls" not found
Jul 28 10:08:21.703396 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:21.703297    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3fe785d-4bae-4988-85f0-2a02259913b1-metrics-tls podName:c3fe785d-4bae-4988-85f0-2a02259913b1 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:25.703280045 +0000 UTC m=+41.662048402 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c3fe785d-4bae-4988-85f0-2a02259913b1-metrics-tls") pod "dns-default-nsq8v" (UID: "c3fe785d-4bae-4988-85f0-2a02259913b1") : secret "dns-default-metrics-tls" not found
Jul 28 10:08:21.703396 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:21.703313    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e43afa04-2c0d-445d-a2c8-86512466e05b-registry-tls podName:e43afa04-2c0d-445d-a2c8-86512466e05b nodeName:}" failed. No retries permitted until 2026-07-28 10:08:25.703306339 +0000 UTC m=+41.662074696 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e43afa04-2c0d-445d-a2c8-86512466e05b-registry-tls") pod "image-registry-69b959dbcb-z2hmw" (UID: "e43afa04-2c0d-445d-a2c8-86512466e05b") : secret "image-registry-tls" not found
Jul 28 10:08:21.752740 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:21.752706    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ljlnk" event={"ID":"74cc1682-9675-406b-8fe7-78713ec81657","Type":"ContainerStarted","Data":"273c057554ba55fae2f4eb6507f0adf7a50f54461ce5f0491d40db5fa6545551"}
Jul 28 10:08:21.790933 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:21.790879    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ljlnk" podStartSLOduration=2.378587485 podStartE2EDuration="4.790864873s" totalImagesPullingTime="2.412277388s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:08:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:21.790388528 +0000 UTC m=+37.749156906" watchObservedRunningTime="2026-07-28 10:08:21.790864873 +0000 UTC m=+37.749633281"
Jul 28 10:08:22.680179 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:22.680147    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-mpcmc"]
Jul 28 10:08:22.753204 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:22.753175    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mpcmc"]
Jul 28 10:08:22.753626 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:22.753295    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mpcmc"
Jul 28 10:08:22.759458 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:22.759434    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-q8lkz\""
Jul 28 10:08:22.759636 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:22.759584    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 10:08:22.759636 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:22.759626    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 10:08:22.760044 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:22.759971    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 10:08:22.774214 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:22.774182    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 10:08:22.812456 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:22.812424    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/30e72787-1717-4a7a-b2e2-43e955527f77-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-m4n6h\" (UID: \"30e72787-1717-4a7a-b2e2-43e955527f77\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-m4n6h"
Jul 28 10:08:22.812585 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:22.812567    2557 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 10:08:22.812658 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:22.812646    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30e72787-1717-4a7a-b2e2-43e955527f77-networking-console-plugin-cert podName:30e72787-1717-4a7a-b2e2-43e955527f77 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:26.812631705 +0000 UTC m=+42.771400065 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/30e72787-1717-4a7a-b2e2-43e955527f77-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-m4n6h" (UID: "30e72787-1717-4a7a-b2e2-43e955527f77") : secret "networking-console-plugin-cert" not found
Jul 28 10:08:22.913469 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:22.913432    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52n4g\" (UniqueName: \"kubernetes.io/projected/a3e0525c-3c2b-4d2b-ba59-e815dfa07d51-kube-api-access-52n4g\") pod \"insights-runtime-extractor-mpcmc\" (UID: \"a3e0525c-3c2b-4d2b-ba59-e815dfa07d51\") " pod="openshift-insights/insights-runtime-extractor-mpcmc"
Jul 28 10:08:22.913673 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:22.913510    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a3e0525c-3c2b-4d2b-ba59-e815dfa07d51-crio-socket\") pod \"insights-runtime-extractor-mpcmc\" (UID: \"a3e0525c-3c2b-4d2b-ba59-e815dfa07d51\") " pod="openshift-insights/insights-runtime-extractor-mpcmc"
Jul 28 10:08:22.913673 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:22.913568    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a3e0525c-3c2b-4d2b-ba59-e815dfa07d51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mpcmc\" (UID: \"a3e0525c-3c2b-4d2b-ba59-e815dfa07d51\") " pod="openshift-insights/insights-runtime-extractor-mpcmc"
Jul 28 10:08:22.913673 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:22.913627    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a3e0525c-3c2b-4d2b-ba59-e815dfa07d51-data-volume\") pod \"insights-runtime-extractor-mpcmc\" (UID: \"a3e0525c-3c2b-4d2b-ba59-e815dfa07d51\") " pod="openshift-insights/insights-runtime-extractor-mpcmc"
Jul 28 10:08:22.913816 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:22.913697    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a3e0525c-3c2b-4d2b-ba59-e815dfa07d51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mpcmc\" (UID: \"a3e0525c-3c2b-4d2b-ba59-e815dfa07d51\") " pod="openshift-insights/insights-runtime-extractor-mpcmc"
Jul 28 10:08:23.014949 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:23.014916    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a3e0525c-3c2b-4d2b-ba59-e815dfa07d51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mpcmc\" (UID: \"a3e0525c-3c2b-4d2b-ba59-e815dfa07d51\") " pod="openshift-insights/insights-runtime-extractor-mpcmc"
Jul 28 10:08:23.015100 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:23.014960    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a3e0525c-3c2b-4d2b-ba59-e815dfa07d51-data-volume\") pod \"insights-runtime-extractor-mpcmc\" (UID: \"a3e0525c-3c2b-4d2b-ba59-e815dfa07d51\") " pod="openshift-insights/insights-runtime-extractor-mpcmc"
Jul 28 10:08:23.015100 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:23.015087    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 10:08:23.015174 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:23.015153    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 10:08:23.015174 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:23.015163    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3e0525c-3c2b-4d2b-ba59-e815dfa07d51-insights-runtime-extractor-tls podName:a3e0525c-3c2b-4d2b-ba59-e815dfa07d51 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:23.515141179 +0000 UTC m=+39.473909559 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a3e0525c-3c2b-4d2b-ba59-e815dfa07d51-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mpcmc" (UID: "a3e0525c-3c2b-4d2b-ba59-e815dfa07d51") : secret "insights-runtime-extractor-tls" not found
Jul 28 10:08:23.015242 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:23.015201    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a3e0525c-3c2b-4d2b-ba59-e815dfa07d51-kube-rbac-proxy-cm podName:a3e0525c-3c2b-4d2b-ba59-e815dfa07d51 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:23.515190411 +0000 UTC m=+39.473958783 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a3e0525c-3c2b-4d2b-ba59-e815dfa07d51-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mpcmc" (UID: "a3e0525c-3c2b-4d2b-ba59-e815dfa07d51") : configmap "kube-rbac-proxy" not found
Jul 28 10:08:23.015242 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:23.015092    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a3e0525c-3c2b-4d2b-ba59-e815dfa07d51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mpcmc\" (UID: \"a3e0525c-3c2b-4d2b-ba59-e815dfa07d51\") " pod="openshift-insights/insights-runtime-extractor-mpcmc"
Jul 28 10:08:23.015311 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:23.015244    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52n4g\" (UniqueName: \"kubernetes.io/projected/a3e0525c-3c2b-4d2b-ba59-e815dfa07d51-kube-api-access-52n4g\") pod \"insights-runtime-extractor-mpcmc\" (UID: \"a3e0525c-3c2b-4d2b-ba59-e815dfa07d51\") " pod="openshift-insights/insights-runtime-extractor-mpcmc"
Jul 28 10:08:23.015311 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:23.015271    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a3e0525c-3c2b-4d2b-ba59-e815dfa07d51-data-volume\") pod \"insights-runtime-extractor-mpcmc\" (UID: \"a3e0525c-3c2b-4d2b-ba59-e815dfa07d51\") " pod="openshift-insights/insights-runtime-extractor-mpcmc"
Jul 28 10:08:23.015371 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:23.015330    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a3e0525c-3c2b-4d2b-ba59-e815dfa07d51-crio-socket\") pod \"insights-runtime-extractor-mpcmc\" (UID: \"a3e0525c-3c2b-4d2b-ba59-e815dfa07d51\") " pod="openshift-insights/insights-runtime-extractor-mpcmc"
Jul 28 10:08:23.015526 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:23.015513    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a3e0525c-3c2b-4d2b-ba59-e815dfa07d51-crio-socket\") pod \"insights-runtime-extractor-mpcmc\" (UID: \"a3e0525c-3c2b-4d2b-ba59-e815dfa07d51\") " pod="openshift-insights/insights-runtime-extractor-mpcmc"
Jul 28 10:08:23.028638 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:23.028592    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52n4g\" (UniqueName: \"kubernetes.io/projected/a3e0525c-3c2b-4d2b-ba59-e815dfa07d51-kube-api-access-52n4g\") pod \"insights-runtime-extractor-mpcmc\" (UID: \"a3e0525c-3c2b-4d2b-ba59-e815dfa07d51\") " pod="openshift-insights/insights-runtime-extractor-mpcmc"
Jul 28 10:08:23.517875 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:23.517845    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a3e0525c-3c2b-4d2b-ba59-e815dfa07d51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mpcmc\" (UID: \"a3e0525c-3c2b-4d2b-ba59-e815dfa07d51\") " pod="openshift-insights/insights-runtime-extractor-mpcmc"
Jul 28 10:08:23.518028 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:23.517934    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a3e0525c-3c2b-4d2b-ba59-e815dfa07d51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mpcmc\" (UID: \"a3e0525c-3c2b-4d2b-ba59-e815dfa07d51\") " pod="openshift-insights/insights-runtime-extractor-mpcmc"
Jul 28 10:08:23.518028 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:23.517969    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 10:08:23.518094 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:23.518036    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a3e0525c-3c2b-4d2b-ba59-e815dfa07d51-kube-rbac-proxy-cm podName:a3e0525c-3c2b-4d2b-ba59-e815dfa07d51 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:24.518020713 +0000 UTC m=+40.476789095 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a3e0525c-3c2b-4d2b-ba59-e815dfa07d51-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mpcmc" (UID: "a3e0525c-3c2b-4d2b-ba59-e815dfa07d51") : configmap "kube-rbac-proxy" not found
Jul 28 10:08:23.518094 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:23.518041    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 10:08:23.518094 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:23.518080    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3e0525c-3c2b-4d2b-ba59-e815dfa07d51-insights-runtime-extractor-tls podName:a3e0525c-3c2b-4d2b-ba59-e815dfa07d51 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:24.518069379 +0000 UTC m=+40.476837736 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a3e0525c-3c2b-4d2b-ba59-e815dfa07d51-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mpcmc" (UID: "a3e0525c-3c2b-4d2b-ba59-e815dfa07d51") : secret "insights-runtime-extractor-tls" not found
Jul 28 10:08:24.526638 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:24.526591    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a3e0525c-3c2b-4d2b-ba59-e815dfa07d51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mpcmc\" (UID: \"a3e0525c-3c2b-4d2b-ba59-e815dfa07d51\") " pod="openshift-insights/insights-runtime-extractor-mpcmc"
Jul 28 10:08:24.527185 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:24.526743    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a3e0525c-3c2b-4d2b-ba59-e815dfa07d51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mpcmc\" (UID: \"a3e0525c-3c2b-4d2b-ba59-e815dfa07d51\") " pod="openshift-insights/insights-runtime-extractor-mpcmc"
Jul 28 10:08:24.527185 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:24.526747    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 10:08:24.527185 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:24.526816    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3e0525c-3c2b-4d2b-ba59-e815dfa07d51-insights-runtime-extractor-tls podName:a3e0525c-3c2b-4d2b-ba59-e815dfa07d51 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:26.526795305 +0000 UTC m=+42.485563662 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a3e0525c-3c2b-4d2b-ba59-e815dfa07d51-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mpcmc" (UID: "a3e0525c-3c2b-4d2b-ba59-e815dfa07d51") : secret "insights-runtime-extractor-tls" not found
Jul 28 10:08:24.527185 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:24.526823    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 10:08:24.527185 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:24.526862    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a3e0525c-3c2b-4d2b-ba59-e815dfa07d51-kube-rbac-proxy-cm podName:a3e0525c-3c2b-4d2b-ba59-e815dfa07d51 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:26.526851647 +0000 UTC m=+42.485620005 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a3e0525c-3c2b-4d2b-ba59-e815dfa07d51-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mpcmc" (UID: "a3e0525c-3c2b-4d2b-ba59-e815dfa07d51") : configmap "kube-rbac-proxy" not found
Jul 28 10:08:25.298529 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:25.298498    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-8727m"]
Jul 28 10:08:25.330331 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:25.330299    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-8727m"]
Jul 28 10:08:25.330492 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:25.330399    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-8727m"
Jul 28 10:08:25.338809 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:25.338777    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 28 10:08:25.338956 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:25.338823    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 28 10:08:25.339405 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:25.339390    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 28 10:08:25.339470 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:25.339395    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-kh4gm\""
Jul 28 10:08:25.339470 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:25.339417    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 28 10:08:25.432596 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:25.432558    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8960453d-6d8c-4a5a-8b33-c4ebf55e37d8-signing-cabundle\") pod \"service-ca-658d7b7c78-8727m\" (UID: \"8960453d-6d8c-4a5a-8b33-c4ebf55e37d8\") " pod="openshift-service-ca/service-ca-658d7b7c78-8727m"
Jul 28 10:08:25.432785 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:25.432693    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2s7x\" (UniqueName: \"kubernetes.io/projected/8960453d-6d8c-4a5a-8b33-c4ebf55e37d8-kube-api-access-h2s7x\") pod \"service-ca-658d7b7c78-8727m\" (UID: \"8960453d-6d8c-4a5a-8b33-c4ebf55e37d8\") " pod="openshift-service-ca/service-ca-658d7b7c78-8727m"
Jul 28 10:08:25.432785 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:25.432769    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8960453d-6d8c-4a5a-8b33-c4ebf55e37d8-signing-key\") pod \"service-ca-658d7b7c78-8727m\" (UID: \"8960453d-6d8c-4a5a-8b33-c4ebf55e37d8\") " pod="openshift-service-ca/service-ca-658d7b7c78-8727m"
Jul 28 10:08:25.472347 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:25.472327    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5z8xh_f8ad89c4-5ef2-477b-ab4c-e2c97b9a68fa/dns-node-resolver/0.log"
Jul 28 10:08:25.533843 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:25.533799    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8960453d-6d8c-4a5a-8b33-c4ebf55e37d8-signing-key\") pod \"service-ca-658d7b7c78-8727m\" (UID: \"8960453d-6d8c-4a5a-8b33-c4ebf55e37d8\") " pod="openshift-service-ca/service-ca-658d7b7c78-8727m"
Jul 28 10:08:25.534205 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:25.533900    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8960453d-6d8c-4a5a-8b33-c4ebf55e37d8-signing-cabundle\") pod \"service-ca-658d7b7c78-8727m\" (UID: \"8960453d-6d8c-4a5a-8b33-c4ebf55e37d8\") " pod="openshift-service-ca/service-ca-658d7b7c78-8727m"
Jul 28 10:08:25.534205 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:25.533951    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2s7x\" (UniqueName: \"kubernetes.io/projected/8960453d-6d8c-4a5a-8b33-c4ebf55e37d8-kube-api-access-h2s7x\") pod \"service-ca-658d7b7c78-8727m\" (UID: \"8960453d-6d8c-4a5a-8b33-c4ebf55e37d8\") " pod="openshift-service-ca/service-ca-658d7b7c78-8727m"
Jul 28 10:08:25.534578 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:25.534559    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8960453d-6d8c-4a5a-8b33-c4ebf55e37d8-signing-cabundle\") pod \"service-ca-658d7b7c78-8727m\" (UID: \"8960453d-6d8c-4a5a-8b33-c4ebf55e37d8\") " pod="openshift-service-ca/service-ca-658d7b7c78-8727m"
Jul 28 10:08:25.536200 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:25.536174    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8960453d-6d8c-4a5a-8b33-c4ebf55e37d8-signing-key\") pod \"service-ca-658d7b7c78-8727m\" (UID: \"8960453d-6d8c-4a5a-8b33-c4ebf55e37d8\") " pod="openshift-service-ca/service-ca-658d7b7c78-8727m"
Jul 28 10:08:25.545798 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:25.545775    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2s7x\" (UniqueName: \"kubernetes.io/projected/8960453d-6d8c-4a5a-8b33-c4ebf55e37d8-kube-api-access-h2s7x\") pod \"service-ca-658d7b7c78-8727m\" (UID: \"8960453d-6d8c-4a5a-8b33-c4ebf55e37d8\") " pod="openshift-service-ca/service-ca-658d7b7c78-8727m"
Jul 28 10:08:25.639453 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:25.639371    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-8727m"
Jul 28 10:08:25.735391 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:25.735309    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e43afa04-2c0d-445d-a2c8-86512466e05b-registry-tls\") pod \"image-registry-69b959dbcb-z2hmw\" (UID: \"e43afa04-2c0d-445d-a2c8-86512466e05b\") " pod="openshift-image-registry/image-registry-69b959dbcb-z2hmw"
Jul 28 10:08:25.735562 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:25.735483    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 10:08:25.735562 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:25.735513    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69b959dbcb-z2hmw: secret "image-registry-tls" not found
Jul 28 10:08:25.735702 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:25.735595    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e43afa04-2c0d-445d-a2c8-86512466e05b-registry-tls podName:e43afa04-2c0d-445d-a2c8-86512466e05b nodeName:}" failed. No retries permitted until 2026-07-28 10:08:33.73556913 +0000 UTC m=+49.694337510 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e43afa04-2c0d-445d-a2c8-86512466e05b-registry-tls") pod "image-registry-69b959dbcb-z2hmw" (UID: "e43afa04-2c0d-445d-a2c8-86512466e05b") : secret "image-registry-tls" not found
Jul 28 10:08:25.736392 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:25.736053    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c3fe785d-4bae-4988-85f0-2a02259913b1-metrics-tls\") pod \"dns-default-nsq8v\" (UID: \"c3fe785d-4bae-4988-85f0-2a02259913b1\") " pod="openshift-dns/dns-default-nsq8v"
Jul 28 10:08:25.736392 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:25.736213    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 10:08:25.736392 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:25.736265    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3fe785d-4bae-4988-85f0-2a02259913b1-metrics-tls podName:c3fe785d-4bae-4988-85f0-2a02259913b1 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:33.736247226 +0000 UTC m=+49.695015583 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c3fe785d-4bae-4988-85f0-2a02259913b1-metrics-tls") pod "dns-default-nsq8v" (UID: "c3fe785d-4bae-4988-85f0-2a02259913b1") : secret "dns-default-metrics-tls" not found
Jul 28 10:08:25.785687 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:25.785653    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-8727m"]
Jul 28 10:08:25.789780 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:08:25.789756    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8960453d_6d8c_4a5a_8b33_c4ebf55e37d8.slice/crio-329462670ef767a590b7b54cff6de02b74f96b81028f7f6e87bb8c55d005f60f WatchSource:0}: Error finding container 329462670ef767a590b7b54cff6de02b74f96b81028f7f6e87bb8c55d005f60f: Status 404 returned error can't find the container with id 329462670ef767a590b7b54cff6de02b74f96b81028f7f6e87bb8c55d005f60f
Jul 28 10:08:26.466472 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:26.466446    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-h8krb_a1a543f9-5413-4260-b48d-f767a66f23c7/node-ca/0.log"
Jul 28 10:08:26.543067 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:26.543032    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a3e0525c-3c2b-4d2b-ba59-e815dfa07d51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mpcmc\" (UID: \"a3e0525c-3c2b-4d2b-ba59-e815dfa07d51\") " pod="openshift-insights/insights-runtime-extractor-mpcmc"
Jul 28 10:08:26.543548 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:26.543156    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 10:08:26.543548 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:26.543158    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a3e0525c-3c2b-4d2b-ba59-e815dfa07d51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mpcmc\" (UID: \"a3e0525c-3c2b-4d2b-ba59-e815dfa07d51\") " pod="openshift-insights/insights-runtime-extractor-mpcmc"
Jul 28 10:08:26.543548 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:26.543225    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a3e0525c-3c2b-4d2b-ba59-e815dfa07d51-kube-rbac-proxy-cm podName:a3e0525c-3c2b-4d2b-ba59-e815dfa07d51 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:30.543204009 +0000 UTC m=+46.501972387 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a3e0525c-3c2b-4d2b-ba59-e815dfa07d51-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mpcmc" (UID: "a3e0525c-3c2b-4d2b-ba59-e815dfa07d51") : configmap "kube-rbac-proxy" not found
Jul 28 10:08:26.543548 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:26.543343    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 10:08:26.543548 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:26.543394    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3e0525c-3c2b-4d2b-ba59-e815dfa07d51-insights-runtime-extractor-tls podName:a3e0525c-3c2b-4d2b-ba59-e815dfa07d51 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:30.543380889 +0000 UTC m=+46.502149266 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a3e0525c-3c2b-4d2b-ba59-e815dfa07d51-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mpcmc" (UID: "a3e0525c-3c2b-4d2b-ba59-e815dfa07d51") : secret "insights-runtime-extractor-tls" not found
Jul 28 10:08:26.765946 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:26.765903    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-8727m" event={"ID":"8960453d-6d8c-4a5a-8b33-c4ebf55e37d8","Type":"ContainerStarted","Data":"329462670ef767a590b7b54cff6de02b74f96b81028f7f6e87bb8c55d005f60f"}
Jul 28 10:08:26.845431 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:26.845394    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/30e72787-1717-4a7a-b2e2-43e955527f77-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-m4n6h\" (UID: \"30e72787-1717-4a7a-b2e2-43e955527f77\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-m4n6h"
Jul 28 10:08:26.845637 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:26.845593    2557 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 10:08:26.845706 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:26.845669    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30e72787-1717-4a7a-b2e2-43e955527f77-networking-console-plugin-cert podName:30e72787-1717-4a7a-b2e2-43e955527f77 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:34.845652859 +0000 UTC m=+50.804421216 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/30e72787-1717-4a7a-b2e2-43e955527f77-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-m4n6h" (UID: "30e72787-1717-4a7a-b2e2-43e955527f77") : secret "networking-console-plugin-cert" not found
Jul 28 10:08:27.465758 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:27.465727    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ljlnk_74cc1682-9675-406b-8fe7-78713ec81657/serve-healthcheck-canary/0.log"
Jul 28 10:08:28.772007 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:28.771970    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-8727m" event={"ID":"8960453d-6d8c-4a5a-8b33-c4ebf55e37d8","Type":"ContainerStarted","Data":"311179e108a3e922a1a959d56e057f93ea8ba6fea286293dbd4f0627296d7ed5"}
Jul 28 10:08:30.574668 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:30.574626    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a3e0525c-3c2b-4d2b-ba59-e815dfa07d51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mpcmc\" (UID: \"a3e0525c-3c2b-4d2b-ba59-e815dfa07d51\") " pod="openshift-insights/insights-runtime-extractor-mpcmc"
Jul 28 10:08:30.575154 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:30.574868    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a3e0525c-3c2b-4d2b-ba59-e815dfa07d51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mpcmc\" (UID: \"a3e0525c-3c2b-4d2b-ba59-e815dfa07d51\") " pod="openshift-insights/insights-runtime-extractor-mpcmc"
Jul 28 10:08:30.575154 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:30.575033    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 10:08:30.575154 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:30.575098    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3e0525c-3c2b-4d2b-ba59-e815dfa07d51-insights-runtime-extractor-tls podName:a3e0525c-3c2b-4d2b-ba59-e815dfa07d51 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:38.57508144 +0000 UTC m=+54.533849797 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a3e0525c-3c2b-4d2b-ba59-e815dfa07d51-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mpcmc" (UID: "a3e0525c-3c2b-4d2b-ba59-e815dfa07d51") : secret "insights-runtime-extractor-tls" not found
Jul 28 10:08:30.575380 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:30.575254    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a3e0525c-3c2b-4d2b-ba59-e815dfa07d51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mpcmc\" (UID: \"a3e0525c-3c2b-4d2b-ba59-e815dfa07d51\") " pod="openshift-insights/insights-runtime-extractor-mpcmc"
Jul 28 10:08:33.797615 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:33.797576    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c3fe785d-4bae-4988-85f0-2a02259913b1-metrics-tls\") pod \"dns-default-nsq8v\" (UID: \"c3fe785d-4bae-4988-85f0-2a02259913b1\") " pod="openshift-dns/dns-default-nsq8v"
Jul 28 10:08:33.797988 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:33.797693    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e43afa04-2c0d-445d-a2c8-86512466e05b-registry-tls\") pod \"image-registry-69b959dbcb-z2hmw\" (UID: \"e43afa04-2c0d-445d-a2c8-86512466e05b\") " pod="openshift-image-registry/image-registry-69b959dbcb-z2hmw"
Jul 28 10:08:33.800095 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:33.800063    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c3fe785d-4bae-4988-85f0-2a02259913b1-metrics-tls\") pod \"dns-default-nsq8v\" (UID: \"c3fe785d-4bae-4988-85f0-2a02259913b1\") " pod="openshift-dns/dns-default-nsq8v"
Jul 28 10:08:33.800221 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:33.800107    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e43afa04-2c0d-445d-a2c8-86512466e05b-registry-tls\") pod \"image-registry-69b959dbcb-z2hmw\" (UID: \"e43afa04-2c0d-445d-a2c8-86512466e05b\") " pod="openshift-image-registry/image-registry-69b959dbcb-z2hmw"
Jul 28 10:08:33.847175 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:33.847133    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69b959dbcb-z2hmw"
Jul 28 10:08:33.870156 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:33.870129    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nsq8v"
Jul 28 10:08:34.031241 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:34.031183    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-8727m" podStartSLOduration=6.947943704 podStartE2EDuration="9.031165208s" totalImagesPullingTime="2.083221504s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:08:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:28.795428705 +0000 UTC m=+44.754197089" watchObservedRunningTime="2026-07-28 10:08:34.031165208 +0000 UTC m=+49.989933587"
Jul 28 10:08:34.032183 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:34.032156    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69b959dbcb-z2hmw"]
Jul 28 10:08:34.036267 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:08:34.036240    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode43afa04_2c0d_445d_a2c8_86512466e05b.slice/crio-6d992082c109b5ed55c22eb41218984864a024ece25fa0d076d8952f2e3651d9 WatchSource:0}: Error finding container 6d992082c109b5ed55c22eb41218984864a024ece25fa0d076d8952f2e3651d9: Status 404 returned error can't find the container with id 6d992082c109b5ed55c22eb41218984864a024ece25fa0d076d8952f2e3651d9
Jul 28 10:08:34.050589 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:34.050524    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nsq8v"]
Jul 28 10:08:34.053330 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:08:34.053306    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3fe785d_4bae_4988_85f0_2a02259913b1.slice/crio-1781955066c22c8b8d96bc8ab4ae0ae95770b398bd63c3d32a325d8c357308e6 WatchSource:0}: Error finding container 1781955066c22c8b8d96bc8ab4ae0ae95770b398bd63c3d32a325d8c357308e6: Status 404 returned error can't find the container with id 1781955066c22c8b8d96bc8ab4ae0ae95770b398bd63c3d32a325d8c357308e6
Jul 28 10:08:34.789254 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:34.789215    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69b959dbcb-z2hmw" event={"ID":"e43afa04-2c0d-445d-a2c8-86512466e05b","Type":"ContainerStarted","Data":"97f0a69fad9f3328639d9b84ef05994c4cb180f70bb4a4a16badb3d4ed826a9f"}
Jul 28 10:08:34.789539 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:34.789275    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69b959dbcb-z2hmw" event={"ID":"e43afa04-2c0d-445d-a2c8-86512466e05b","Type":"ContainerStarted","Data":"6d992082c109b5ed55c22eb41218984864a024ece25fa0d076d8952f2e3651d9"}
Jul 28 10:08:34.789539 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:34.789310    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-69b959dbcb-z2hmw"
Jul 28 10:08:34.790742 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:34.790716    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nsq8v" event={"ID":"c3fe785d-4bae-4988-85f0-2a02259913b1","Type":"ContainerStarted","Data":"1781955066c22c8b8d96bc8ab4ae0ae95770b398bd63c3d32a325d8c357308e6"}
Jul 28 10:08:34.844475 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:34.844404    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-69b959dbcb-z2hmw" podStartSLOduration=20.844382204 podStartE2EDuration="20.844382204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:08:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:34.843375173 +0000 UTC m=+50.802143553" watchObservedRunningTime="2026-07-28 10:08:34.844382204 +0000 UTC m=+50.803150585"
Jul 28 10:08:34.908822 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:34.908776    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/30e72787-1717-4a7a-b2e2-43e955527f77-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-m4n6h\" (UID: \"30e72787-1717-4a7a-b2e2-43e955527f77\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-m4n6h"
Jul 28 10:08:34.909004 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:34.908930    2557 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 10:08:34.909052 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:34.909006    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30e72787-1717-4a7a-b2e2-43e955527f77-networking-console-plugin-cert podName:30e72787-1717-4a7a-b2e2-43e955527f77 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:50.908985341 +0000 UTC m=+66.867753718 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/30e72787-1717-4a7a-b2e2-43e955527f77-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-m4n6h" (UID: "30e72787-1717-4a7a-b2e2-43e955527f77") : secret "networking-console-plugin-cert" not found
Jul 28 10:08:35.794988 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:35.794953    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nsq8v" event={"ID":"c3fe785d-4bae-4988-85f0-2a02259913b1","Type":"ContainerStarted","Data":"7b65c07d1b60d686c7e1791634a9de191af3cd12647ba9b0bda9e2f045a97a45"}
Jul 28 10:08:35.795201 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:35.795008    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nsq8v" event={"ID":"c3fe785d-4bae-4988-85f0-2a02259913b1","Type":"ContainerStarted","Data":"c104db0beb59ac89e5ce0e66625748261bf4024e0eee37263ba04db87bb6cefa"}
Jul 28 10:08:35.795261 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:35.795211    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-nsq8v"
Jul 28 10:08:38.640836 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:38.640794    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a3e0525c-3c2b-4d2b-ba59-e815dfa07d51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mpcmc\" (UID: \"a3e0525c-3c2b-4d2b-ba59-e815dfa07d51\") " pod="openshift-insights/insights-runtime-extractor-mpcmc"
Jul 28 10:08:38.643056 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:38.643033    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a3e0525c-3c2b-4d2b-ba59-e815dfa07d51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mpcmc\" (UID: \"a3e0525c-3c2b-4d2b-ba59-e815dfa07d51\") " pod="openshift-insights/insights-runtime-extractor-mpcmc"
Jul 28 10:08:38.662088 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:38.662049    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mpcmc"
Jul 28 10:08:38.785976 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:38.785926    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-nsq8v" podStartSLOduration=20.356094341 podStartE2EDuration="21.785908403s" totalImagesPullingTime="1.429814062s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:08:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:35.820696863 +0000 UTC m=+51.779465241" watchObservedRunningTime="2026-07-28 10:08:38.785908403 +0000 UTC m=+54.744676781"
Jul 28 10:08:38.786129 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:38.786099    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mpcmc"]
Jul 28 10:08:38.789268 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:08:38.789243    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3e0525c_3c2b_4d2b_ba59_e815dfa07d51.slice/crio-65b63245fe7cfd0a1e62f77a37e456f45133b994b517aad55124e35b60721c7b WatchSource:0}: Error finding container 65b63245fe7cfd0a1e62f77a37e456f45133b994b517aad55124e35b60721c7b: Status 404 returned error can't find the container with id 65b63245fe7cfd0a1e62f77a37e456f45133b994b517aad55124e35b60721c7b
Jul 28 10:08:38.802998 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:38.802966    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mpcmc" event={"ID":"a3e0525c-3c2b-4d2b-ba59-e815dfa07d51","Type":"ContainerStarted","Data":"65b63245fe7cfd0a1e62f77a37e456f45133b994b517aad55124e35b60721c7b"}
Jul 28 10:08:42.730495 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:42.730463    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hdqls"
Jul 28 10:08:44.822974 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:44.822852    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mpcmc" event={"ID":"a3e0525c-3c2b-4d2b-ba59-e815dfa07d51","Type":"ContainerStarted","Data":"6e3368b4b85128db29a461516600af0738872fafb158e9f0bb53a398cb250dbd"}
Jul 28 10:08:45.799888 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:45.799854    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-nsq8v"
Jul 28 10:08:47.834674 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:47.834638    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mpcmc" event={"ID":"a3e0525c-3c2b-4d2b-ba59-e815dfa07d51","Type":"ContainerStarted","Data":"53bf9c291fe0b523c0761c8ba9e7d94d779451e9bc9043d68ec136d8ab3b2e2a"}
Jul 28 10:08:47.904293 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:47.904255    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-69b959dbcb-z2hmw"]
Jul 28 10:08:47.924814 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:47.924784    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7cfb675d-7ghrv"]
Jul 28 10:08:47.954981 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:47.954953    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7cfb675d-7ghrv"
Jul 28 10:08:47.971760 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:47.971737    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 28 10:08:48.004479 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.004435    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 28 10:08:48.004671 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.004522    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 28 10:08:48.004671 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.004589    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 28 10:08:48.005548 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.005518    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7cfb675d-7ghrv"]
Jul 28 10:08:48.026106 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.026067    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01f4d561-3325-4ca2-a4ea-40f66f5c2824-tmp\") pod \"klusterlet-addon-workmgr-5b7cfb675d-7ghrv\" (UID: \"01f4d561-3325-4ca2-a4ea-40f66f5c2824\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7cfb675d-7ghrv"
Jul 28 10:08:48.026106 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.026108    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqpcv\" (UniqueName: \"kubernetes.io/projected/01f4d561-3325-4ca2-a4ea-40f66f5c2824-kube-api-access-pqpcv\") pod \"klusterlet-addon-workmgr-5b7cfb675d-7ghrv\" (UID: \"01f4d561-3325-4ca2-a4ea-40f66f5c2824\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7cfb675d-7ghrv"
Jul 28 10:08:48.026315 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.026217    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/01f4d561-3325-4ca2-a4ea-40f66f5c2824-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b7cfb675d-7ghrv\" (UID: \"01f4d561-3325-4ca2-a4ea-40f66f5c2824\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7cfb675d-7ghrv"
Jul 28 10:08:48.030859 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.030828    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-xktrq"]
Jul 28 10:08:48.047350 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.047317    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-64cd7c8686-n6ktg"]
Jul 28 10:08:48.047535 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.047481    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-xktrq"
Jul 28 10:08:48.051167 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.051142    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-pxw8l\""
Jul 28 10:08:48.051433 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.051417    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 28 10:08:48.051623 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.051594    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 28 10:08:48.061330 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.061307    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-xktrq"]
Jul 28 10:08:48.061467 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.061451    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64cd7c8686-n6ktg"
Jul 28 10:08:48.091428 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.091355    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64cd7c8686-n6ktg"]
Jul 28 10:08:48.130500 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.130450    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dee5fe51-f82d-411e-8f64-9b36bd2623a3-bound-sa-token\") pod \"image-registry-64cd7c8686-n6ktg\" (UID: \"dee5fe51-f82d-411e-8f64-9b36bd2623a3\") " pod="openshift-image-registry/image-registry-64cd7c8686-n6ktg"
Jul 28 10:08:48.130713 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.130518    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dee5fe51-f82d-411e-8f64-9b36bd2623a3-installation-pull-secrets\") pod \"image-registry-64cd7c8686-n6ktg\" (UID: \"dee5fe51-f82d-411e-8f64-9b36bd2623a3\") " pod="openshift-image-registry/image-registry-64cd7c8686-n6ktg"
Jul 28 10:08:48.130713 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.130620    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fvjb\" (UniqueName: \"kubernetes.io/projected/dee5fe51-f82d-411e-8f64-9b36bd2623a3-kube-api-access-8fvjb\") pod \"image-registry-64cd7c8686-n6ktg\" (UID: \"dee5fe51-f82d-411e-8f64-9b36bd2623a3\") " pod="openshift-image-registry/image-registry-64cd7c8686-n6ktg"
Jul 28 10:08:48.130713 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.130673    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dee5fe51-f82d-411e-8f64-9b36bd2623a3-ca-trust-extracted\") pod \"image-registry-64cd7c8686-n6ktg\" (UID: \"dee5fe51-f82d-411e-8f64-9b36bd2623a3\") " pod="openshift-image-registry/image-registry-64cd7c8686-n6ktg"
Jul 28 10:08:48.130713 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.130710    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/01f4d561-3325-4ca2-a4ea-40f66f5c2824-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b7cfb675d-7ghrv\" (UID: \"01f4d561-3325-4ca2-a4ea-40f66f5c2824\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7cfb675d-7ghrv"
Jul 28 10:08:48.130914 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.130758    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj488\" (UniqueName: \"kubernetes.io/projected/a49e6876-498a-40b4-9ccf-94cf1d02bd97-kube-api-access-zj488\") pod \"downloads-7c97dc4555-xktrq\" (UID: \"a49e6876-498a-40b4-9ccf-94cf1d02bd97\") " pod="openshift-console/downloads-7c97dc4555-xktrq"
Jul 28 10:08:48.130914 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.130792    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dee5fe51-f82d-411e-8f64-9b36bd2623a3-trusted-ca\") pod \"image-registry-64cd7c8686-n6ktg\" (UID: \"dee5fe51-f82d-411e-8f64-9b36bd2623a3\") " pod="openshift-image-registry/image-registry-64cd7c8686-n6ktg"
Jul 28 10:08:48.130914 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.130834    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01f4d561-3325-4ca2-a4ea-40f66f5c2824-tmp\") pod \"klusterlet-addon-workmgr-5b7cfb675d-7ghrv\" (UID: \"01f4d561-3325-4ca2-a4ea-40f66f5c2824\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7cfb675d-7ghrv"
Jul 28 10:08:48.130914 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.130876    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqpcv\" (UniqueName: \"kubernetes.io/projected/01f4d561-3325-4ca2-a4ea-40f66f5c2824-kube-api-access-pqpcv\") pod \"klusterlet-addon-workmgr-5b7cfb675d-7ghrv\" (UID: \"01f4d561-3325-4ca2-a4ea-40f66f5c2824\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7cfb675d-7ghrv"
Jul 28 10:08:48.130914 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.130900    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dee5fe51-f82d-411e-8f64-9b36bd2623a3-registry-tls\") pod \"image-registry-64cd7c8686-n6ktg\" (UID: \"dee5fe51-f82d-411e-8f64-9b36bd2623a3\") " pod="openshift-image-registry/image-registry-64cd7c8686-n6ktg"
Jul 28 10:08:48.131150 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.130935    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dee5fe51-f82d-411e-8f64-9b36bd2623a3-image-registry-private-configuration\") pod \"image-registry-64cd7c8686-n6ktg\" (UID: \"dee5fe51-f82d-411e-8f64-9b36bd2623a3\") " pod="openshift-image-registry/image-registry-64cd7c8686-n6ktg"
Jul 28 10:08:48.131150 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.130972    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dee5fe51-f82d-411e-8f64-9b36bd2623a3-registry-certificates\") pod \"image-registry-64cd7c8686-n6ktg\" (UID: \"dee5fe51-f82d-411e-8f64-9b36bd2623a3\") " pod="openshift-image-registry/image-registry-64cd7c8686-n6ktg"
Jul 28 10:08:48.131875 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.131849    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01f4d561-3325-4ca2-a4ea-40f66f5c2824-tmp\") pod \"klusterlet-addon-workmgr-5b7cfb675d-7ghrv\" (UID: \"01f4d561-3325-4ca2-a4ea-40f66f5c2824\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7cfb675d-7ghrv"
Jul 28 10:08:48.134496 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.134469    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/01f4d561-3325-4ca2-a4ea-40f66f5c2824-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b7cfb675d-7ghrv\" (UID: \"01f4d561-3325-4ca2-a4ea-40f66f5c2824\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7cfb675d-7ghrv"
Jul 28 10:08:48.151867 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.151830    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqpcv\" (UniqueName: \"kubernetes.io/projected/01f4d561-3325-4ca2-a4ea-40f66f5c2824-kube-api-access-pqpcv\") pod \"klusterlet-addon-workmgr-5b7cfb675d-7ghrv\" (UID: \"01f4d561-3325-4ca2-a4ea-40f66f5c2824\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7cfb675d-7ghrv"
Jul 28 10:08:48.231354 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.231316    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dee5fe51-f82d-411e-8f64-9b36bd2623a3-installation-pull-secrets\") pod \"image-registry-64cd7c8686-n6ktg\" (UID: \"dee5fe51-f82d-411e-8f64-9b36bd2623a3\") " pod="openshift-image-registry/image-registry-64cd7c8686-n6ktg"
Jul 28 10:08:48.231545 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.231393    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fvjb\" (UniqueName: \"kubernetes.io/projected/dee5fe51-f82d-411e-8f64-9b36bd2623a3-kube-api-access-8fvjb\") pod \"image-registry-64cd7c8686-n6ktg\" (UID: \"dee5fe51-f82d-411e-8f64-9b36bd2623a3\") " pod="openshift-image-registry/image-registry-64cd7c8686-n6ktg"
Jul 28 10:08:48.231545 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.231454    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dee5fe51-f82d-411e-8f64-9b36bd2623a3-ca-trust-extracted\") pod \"image-registry-64cd7c8686-n6ktg\" (UID: \"dee5fe51-f82d-411e-8f64-9b36bd2623a3\") " pod="openshift-image-registry/image-registry-64cd7c8686-n6ktg"
Jul 28 10:08:48.231545 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.231500    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj488\" (UniqueName: \"kubernetes.io/projected/a49e6876-498a-40b4-9ccf-94cf1d02bd97-kube-api-access-zj488\") pod \"downloads-7c97dc4555-xktrq\" (UID: \"a49e6876-498a-40b4-9ccf-94cf1d02bd97\") " pod="openshift-console/downloads-7c97dc4555-xktrq"
Jul 28 10:08:48.231545 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.231525    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dee5fe51-f82d-411e-8f64-9b36bd2623a3-trusted-ca\") pod \"image-registry-64cd7c8686-n6ktg\" (UID: \"dee5fe51-f82d-411e-8f64-9b36bd2623a3\") " pod="openshift-image-registry/image-registry-64cd7c8686-n6ktg"
Jul 28 10:08:48.231800 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.231567    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dee5fe51-f82d-411e-8f64-9b36bd2623a3-registry-tls\") pod \"image-registry-64cd7c8686-n6ktg\" (UID: \"dee5fe51-f82d-411e-8f64-9b36bd2623a3\") " pod="openshift-image-registry/image-registry-64cd7c8686-n6ktg"
Jul 28 10:08:48.231800 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.231594    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dee5fe51-f82d-411e-8f64-9b36bd2623a3-image-registry-private-configuration\") pod \"image-registry-64cd7c8686-n6ktg\" (UID: \"dee5fe51-f82d-411e-8f64-9b36bd2623a3\") " pod="openshift-image-registry/image-registry-64cd7c8686-n6ktg"
Jul 28 10:08:48.231800 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.231640    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dee5fe51-f82d-411e-8f64-9b36bd2623a3-registry-certificates\") pod \"image-registry-64cd7c8686-n6ktg\" (UID: \"dee5fe51-f82d-411e-8f64-9b36bd2623a3\") " pod="openshift-image-registry/image-registry-64cd7c8686-n6ktg"
Jul 28 10:08:48.231800 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.231683    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dee5fe51-f82d-411e-8f64-9b36bd2623a3-bound-sa-token\") pod \"image-registry-64cd7c8686-n6ktg\" (UID: \"dee5fe51-f82d-411e-8f64-9b36bd2623a3\") " pod="openshift-image-registry/image-registry-64cd7c8686-n6ktg"
Jul 28 10:08:48.232037 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.232012    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dee5fe51-f82d-411e-8f64-9b36bd2623a3-ca-trust-extracted\") pod \"image-registry-64cd7c8686-n6ktg\" (UID: \"dee5fe51-f82d-411e-8f64-9b36bd2623a3\") " pod="openshift-image-registry/image-registry-64cd7c8686-n6ktg"
Jul 28 10:08:48.232643 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.232614    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dee5fe51-f82d-411e-8f64-9b36bd2623a3-trusted-ca\") pod \"image-registry-64cd7c8686-n6ktg\" (UID: \"dee5fe51-f82d-411e-8f64-9b36bd2623a3\") " pod="openshift-image-registry/image-registry-64cd7c8686-n6ktg"
Jul 28 10:08:48.232740 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.232699    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dee5fe51-f82d-411e-8f64-9b36bd2623a3-registry-certificates\") pod \"image-registry-64cd7c8686-n6ktg\" (UID: \"dee5fe51-f82d-411e-8f64-9b36bd2623a3\") " pod="openshift-image-registry/image-registry-64cd7c8686-n6ktg"
Jul 28 10:08:48.234510 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.234483    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dee5fe51-f82d-411e-8f64-9b36bd2623a3-installation-pull-secrets\") pod \"image-registry-64cd7c8686-n6ktg\" (UID: \"dee5fe51-f82d-411e-8f64-9b36bd2623a3\") " pod="openshift-image-registry/image-registry-64cd7c8686-n6ktg"
Jul 28 10:08:48.234652 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.234625    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dee5fe51-f82d-411e-8f64-9b36bd2623a3-image-registry-private-configuration\") pod \"image-registry-64cd7c8686-n6ktg\" (UID: \"dee5fe51-f82d-411e-8f64-9b36bd2623a3\") " pod="openshift-image-registry/image-registry-64cd7c8686-n6ktg"
Jul 28 10:08:48.234807 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.234785    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dee5fe51-f82d-411e-8f64-9b36bd2623a3-registry-tls\") pod \"image-registry-64cd7c8686-n6ktg\" (UID: \"dee5fe51-f82d-411e-8f64-9b36bd2623a3\") " pod="openshift-image-registry/image-registry-64cd7c8686-n6ktg"
Jul 28 10:08:48.251412 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.251375    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dee5fe51-f82d-411e-8f64-9b36bd2623a3-bound-sa-token\") pod \"image-registry-64cd7c8686-n6ktg\" (UID: \"dee5fe51-f82d-411e-8f64-9b36bd2623a3\") " pod="openshift-image-registry/image-registry-64cd7c8686-n6ktg"
Jul 28 10:08:48.253958 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.253928    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fvjb\" (UniqueName: \"kubernetes.io/projected/dee5fe51-f82d-411e-8f64-9b36bd2623a3-kube-api-access-8fvjb\") pod \"image-registry-64cd7c8686-n6ktg\" (UID: \"dee5fe51-f82d-411e-8f64-9b36bd2623a3\") " pod="openshift-image-registry/image-registry-64cd7c8686-n6ktg"
Jul 28 10:08:48.254116 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.254086    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj488\" (UniqueName: \"kubernetes.io/projected/a49e6876-498a-40b4-9ccf-94cf1d02bd97-kube-api-access-zj488\") pod \"downloads-7c97dc4555-xktrq\" (UID: \"a49e6876-498a-40b4-9ccf-94cf1d02bd97\") " pod="openshift-console/downloads-7c97dc4555-xktrq"
Jul 28 10:08:48.267441 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.267410    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7cfb675d-7ghrv"
Jul 28 10:08:48.359391 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.359183    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-xktrq"
Jul 28 10:08:48.372762 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.372731    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64cd7c8686-n6ktg"
Jul 28 10:08:48.401373 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.400175    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5796468f64-jtc5k"]
Jul 28 10:08:48.414383 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.414360    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5796468f64-jtc5k"
Jul 28 10:08:48.423283 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.423248    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5796468f64-jtc5k"]
Jul 28 10:08:48.426274 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.426247    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 28 10:08:48.426864 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.426684    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 28 10:08:48.426864 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.426729    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fhl2z\""
Jul 28 10:08:48.434557 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.434071    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 28 10:08:48.434557 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.434408    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 28 10:08:48.438400 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.438315    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 28 10:08:48.499283 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.499189    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7cfb675d-7ghrv"]
Jul 28 10:08:48.502426 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:08:48.502389    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01f4d561_3325_4ca2_a4ea_40f66f5c2824.slice/crio-b5421c56cd4965a121abf0839674dcfb6153dcae30b8020392800547ba23b3a1 WatchSource:0}: Error finding container b5421c56cd4965a121abf0839674dcfb6153dcae30b8020392800547ba23b3a1: Status 404 returned error can't find the container with id b5421c56cd4965a121abf0839674dcfb6153dcae30b8020392800547ba23b3a1
Jul 28 10:08:48.533881 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.533845    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ebd56b7c-8767-4f6b-9189-1019cdef269b-console-serving-cert\") pod \"console-5796468f64-jtc5k\" (UID: \"ebd56b7c-8767-4f6b-9189-1019cdef269b\") " pod="openshift-console/console-5796468f64-jtc5k"
Jul 28 10:08:48.534041 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.533893    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ebd56b7c-8767-4f6b-9189-1019cdef269b-console-oauth-config\") pod \"console-5796468f64-jtc5k\" (UID: \"ebd56b7c-8767-4f6b-9189-1019cdef269b\") " pod="openshift-console/console-5796468f64-jtc5k"
Jul 28 10:08:48.534091 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.534063    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ebd56b7c-8767-4f6b-9189-1019cdef269b-service-ca\") pod \"console-5796468f64-jtc5k\" (UID: \"ebd56b7c-8767-4f6b-9189-1019cdef269b\") " pod="openshift-console/console-5796468f64-jtc5k"
Jul 28 10:08:48.534134 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.534110    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ebd56b7c-8767-4f6b-9189-1019cdef269b-oauth-serving-cert\") pod \"console-5796468f64-jtc5k\" (UID: \"ebd56b7c-8767-4f6b-9189-1019cdef269b\") " pod="openshift-console/console-5796468f64-jtc5k"
Jul 28 10:08:48.534187 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.534171    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45c94\" (UniqueName: \"kubernetes.io/projected/ebd56b7c-8767-4f6b-9189-1019cdef269b-kube-api-access-45c94\") pod \"console-5796468f64-jtc5k\" (UID: \"ebd56b7c-8767-4f6b-9189-1019cdef269b\") " pod="openshift-console/console-5796468f64-jtc5k"
Jul 28 10:08:48.534233 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.534204    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ebd56b7c-8767-4f6b-9189-1019cdef269b-console-config\") pod \"console-5796468f64-jtc5k\" (UID: \"ebd56b7c-8767-4f6b-9189-1019cdef269b\") " pod="openshift-console/console-5796468f64-jtc5k"
Jul 28 10:08:48.557794 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.557759    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-xktrq"]
Jul 28 10:08:48.562504 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:08:48.562474    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda49e6876_498a_40b4_9ccf_94cf1d02bd97.slice/crio-6bf2a71c61728350a80a6a5a09005678190b8c5c0a2cf5dac5dc54f65414287b WatchSource:0}: Error finding container 6bf2a71c61728350a80a6a5a09005678190b8c5c0a2cf5dac5dc54f65414287b: Status 404 returned error can't find the container with id 6bf2a71c61728350a80a6a5a09005678190b8c5c0a2cf5dac5dc54f65414287b
Jul 28 10:08:48.599743 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.599668    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64cd7c8686-n6ktg"]
Jul 28 10:08:48.604881 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:08:48.604850    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddee5fe51_f82d_411e_8f64_9b36bd2623a3.slice/crio-4bef96d0c6f93835cf517941a55f8bcc95424c33b38342cfd095ea2bd6063fae WatchSource:0}: Error finding container 4bef96d0c6f93835cf517941a55f8bcc95424c33b38342cfd095ea2bd6063fae: Status 404 returned error can't find the container with id 4bef96d0c6f93835cf517941a55f8bcc95424c33b38342cfd095ea2bd6063fae
Jul 28 10:08:48.634799 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.634767    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ebd56b7c-8767-4f6b-9189-1019cdef269b-console-serving-cert\") pod \"console-5796468f64-jtc5k\" (UID: \"ebd56b7c-8767-4f6b-9189-1019cdef269b\") " pod="openshift-console/console-5796468f64-jtc5k"
Jul 28 10:08:48.634991 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.634809    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ebd56b7c-8767-4f6b-9189-1019cdef269b-console-oauth-config\") pod \"console-5796468f64-jtc5k\" (UID: \"ebd56b7c-8767-4f6b-9189-1019cdef269b\") " pod="openshift-console/console-5796468f64-jtc5k"
Jul 28 10:08:48.634991 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.634913    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ebd56b7c-8767-4f6b-9189-1019cdef269b-service-ca\") pod \"console-5796468f64-jtc5k\" (UID: \"ebd56b7c-8767-4f6b-9189-1019cdef269b\") " pod="openshift-console/console-5796468f64-jtc5k"
Jul 28 10:08:48.634991 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.634943    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ebd56b7c-8767-4f6b-9189-1019cdef269b-oauth-serving-cert\") pod \"console-5796468f64-jtc5k\" (UID: \"ebd56b7c-8767-4f6b-9189-1019cdef269b\") " pod="openshift-console/console-5796468f64-jtc5k"
Jul 28 10:08:48.634991 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.634975    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45c94\" (UniqueName: \"kubernetes.io/projected/ebd56b7c-8767-4f6b-9189-1019cdef269b-kube-api-access-45c94\") pod \"console-5796468f64-jtc5k\" (UID: \"ebd56b7c-8767-4f6b-9189-1019cdef269b\") " pod="openshift-console/console-5796468f64-jtc5k"
Jul 28 10:08:48.635211 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.635001    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ebd56b7c-8767-4f6b-9189-1019cdef269b-console-config\") pod \"console-5796468f64-jtc5k\" (UID: \"ebd56b7c-8767-4f6b-9189-1019cdef269b\") " pod="openshift-console/console-5796468f64-jtc5k"
Jul 28 10:08:48.636287 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.635704    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ebd56b7c-8767-4f6b-9189-1019cdef269b-console-config\") pod \"console-5796468f64-jtc5k\" (UID: \"ebd56b7c-8767-4f6b-9189-1019cdef269b\") " pod="openshift-console/console-5796468f64-jtc5k"
Jul 28 10:08:48.636287 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.635813    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ebd56b7c-8767-4f6b-9189-1019cdef269b-service-ca\") pod \"console-5796468f64-jtc5k\" (UID: \"ebd56b7c-8767-4f6b-9189-1019cdef269b\") " pod="openshift-console/console-5796468f64-jtc5k"
Jul 28 10:08:48.636287 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.636247    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ebd56b7c-8767-4f6b-9189-1019cdef269b-oauth-serving-cert\") pod \"console-5796468f64-jtc5k\" (UID: \"ebd56b7c-8767-4f6b-9189-1019cdef269b\") " pod="openshift-console/console-5796468f64-jtc5k"
Jul 28 10:08:48.637734 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.637709    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ebd56b7c-8767-4f6b-9189-1019cdef269b-console-serving-cert\") pod \"console-5796468f64-jtc5k\" (UID: \"ebd56b7c-8767-4f6b-9189-1019cdef269b\") " pod="openshift-console/console-5796468f64-jtc5k"
Jul 28 10:08:48.638306 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.638285    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ebd56b7c-8767-4f6b-9189-1019cdef269b-console-oauth-config\") pod \"console-5796468f64-jtc5k\" (UID: \"ebd56b7c-8767-4f6b-9189-1019cdef269b\") " pod="openshift-console/console-5796468f64-jtc5k"
Jul 28 10:08:48.654325 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.654303    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45c94\" (UniqueName: \"kubernetes.io/projected/ebd56b7c-8767-4f6b-9189-1019cdef269b-kube-api-access-45c94\") pod \"console-5796468f64-jtc5k\" (UID: \"ebd56b7c-8767-4f6b-9189-1019cdef269b\") " pod="openshift-console/console-5796468f64-jtc5k"
Jul 28 10:08:48.729120 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.729090    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5796468f64-jtc5k"
Jul 28 10:08:48.843016 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.842488    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64cd7c8686-n6ktg" event={"ID":"dee5fe51-f82d-411e-8f64-9b36bd2623a3","Type":"ContainerStarted","Data":"ebc7176f39b838fdf9b91b7648a280d3835d235642cb39d5e7acea3fba098288"}
Jul 28 10:08:48.843016 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.842552    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64cd7c8686-n6ktg" event={"ID":"dee5fe51-f82d-411e-8f64-9b36bd2623a3","Type":"ContainerStarted","Data":"4bef96d0c6f93835cf517941a55f8bcc95424c33b38342cfd095ea2bd6063fae"}
Jul 28 10:08:48.843016 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.842709    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-64cd7c8686-n6ktg"
Jul 28 10:08:48.844149 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.844124    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7cfb675d-7ghrv" event={"ID":"01f4d561-3325-4ca2-a4ea-40f66f5c2824","Type":"ContainerStarted","Data":"b5421c56cd4965a121abf0839674dcfb6153dcae30b8020392800547ba23b3a1"}
Jul 28 10:08:48.846104 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.846043    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-xktrq" event={"ID":"a49e6876-498a-40b4-9ccf-94cf1d02bd97","Type":"ContainerStarted","Data":"6bf2a71c61728350a80a6a5a09005678190b8c5c0a2cf5dac5dc54f65414287b"}
Jul 28 10:08:48.864793 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.864693    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-64cd7c8686-n6ktg" podStartSLOduration=1.8646696170000001 podStartE2EDuration="1.864669617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:08:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:48.863422493 +0000 UTC m=+64.822190877" watchObservedRunningTime="2026-07-28 10:08:48.864669617 +0000 UTC m=+64.823437996"
Jul 28 10:08:48.896460 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:48.896433    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5796468f64-jtc5k"]
Jul 28 10:08:48.899835 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:08:48.899806    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebd56b7c_8767_4f6b_9189_1019cdef269b.slice/crio-4ce7a1d5160912af821251dd14b9550b34539374336201e9e7ce158730353451 WatchSource:0}: Error finding container 4ce7a1d5160912af821251dd14b9550b34539374336201e9e7ce158730353451: Status 404 returned error can't find the container with id 4ce7a1d5160912af821251dd14b9550b34539374336201e9e7ce158730353451
Jul 28 10:08:49.856230 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:49.856191    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5796468f64-jtc5k" event={"ID":"ebd56b7c-8767-4f6b-9189-1019cdef269b","Type":"ContainerStarted","Data":"4ce7a1d5160912af821251dd14b9550b34539374336201e9e7ce158730353451"}
Jul 28 10:08:50.354854 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:50.354123    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d23b40b0-9299-4a0b-a87c-d2f2cb921fd9-metrics-certs\") pod \"network-metrics-daemon-8fltb\" (UID: \"d23b40b0-9299-4a0b-a87c-d2f2cb921fd9\") " pod="openshift-multus/network-metrics-daemon-8fltb"
Jul 28 10:08:50.354854 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:50.354244    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb-original-pull-secret\") pod \"global-pull-secret-syncer-jhkxw\" (UID: \"ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb\") " pod="kube-system/global-pull-secret-syncer-jhkxw"
Jul 28 10:08:50.356977 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:50.356768    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 10:08:50.357297 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:50.357277    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 10:08:50.385842 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:50.385727    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb-original-pull-secret\") pod \"global-pull-secret-syncer-jhkxw\" (UID: \"ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb\") " pod="kube-system/global-pull-secret-syncer-jhkxw"
Jul 28 10:08:50.385842 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:50.385771    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d23b40b0-9299-4a0b-a87c-d2f2cb921fd9-metrics-certs\") pod \"network-metrics-daemon-8fltb\" (UID: \"d23b40b0-9299-4a0b-a87c-d2f2cb921fd9\") " pod="openshift-multus/network-metrics-daemon-8fltb"
Jul 28 10:08:50.456182 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:50.455749    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phrdk\" (UniqueName: \"kubernetes.io/projected/0bc74d3d-3abf-47f5-8149-7d07a9a34e7c-kube-api-access-phrdk\") pod \"network-check-target-ddrhj\" (UID: \"0bc74d3d-3abf-47f5-8149-7d07a9a34e7c\") " pod="openshift-network-diagnostics/network-check-target-ddrhj"
Jul 28 10:08:50.458787 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:50.458744    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 10:08:50.471685 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:50.471653    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 10:08:50.486153 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:50.486121    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jhkxw"
Jul 28 10:08:50.489364 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:50.489292    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phrdk\" (UniqueName: \"kubernetes.io/projected/0bc74d3d-3abf-47f5-8149-7d07a9a34e7c-kube-api-access-phrdk\") pod \"network-check-target-ddrhj\" (UID: \"0bc74d3d-3abf-47f5-8149-7d07a9a34e7c\") " pod="openshift-network-diagnostics/network-check-target-ddrhj"
Jul 28 10:08:50.495409 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:50.495383    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xsq5w\""
Jul 28 10:08:50.501006 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:50.500822    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l65kq\""
Jul 28 10:08:50.504201 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:50.503896    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddrhj"
Jul 28 10:08:50.509066 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:50.508800    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8fltb"
Jul 28 10:08:50.963184 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:50.962476    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/30e72787-1717-4a7a-b2e2-43e955527f77-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-m4n6h\" (UID: \"30e72787-1717-4a7a-b2e2-43e955527f77\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-m4n6h"
Jul 28 10:08:50.968055 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:50.967951    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/30e72787-1717-4a7a-b2e2-43e955527f77-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-m4n6h\" (UID: \"30e72787-1717-4a7a-b2e2-43e955527f77\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-m4n6h"
Jul 28 10:08:51.189887 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:51.189683    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-g45vf\""
Jul 28 10:08:51.198008 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:51.197675    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-m4n6h"
Jul 28 10:08:56.462750 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:56.462713    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8fltb"]
Jul 28 10:08:56.472301 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:08:56.472257    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd23b40b0_9299_4a0b_a87c_d2f2cb921fd9.slice/crio-ab7b79e7fefb36552be8c817c29d66763577e493820b06484adc0f1888c3b084 WatchSource:0}: Error finding container ab7b79e7fefb36552be8c817c29d66763577e493820b06484adc0f1888c3b084: Status 404 returned error can't find the container with id ab7b79e7fefb36552be8c817c29d66763577e493820b06484adc0f1888c3b084
Jul 28 10:08:56.676409 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:56.676377    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6dfd5d7545-ckkmc"]
Jul 28 10:08:56.678558 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:56.678528    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dfd5d7545-ckkmc"
Jul 28 10:08:56.683622 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:56.683572    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jhkxw"]
Jul 28 10:08:56.686049 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:56.685862    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-m4n6h"]
Jul 28 10:08:56.688173 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:56.687952    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 28 10:08:56.688364 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:08:56.688338    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded2b30c5_aea1_46f5_a28a_1aff6f3d80bb.slice/crio-3f25f2697f5d19bf8bb851c6218d54a099a841833dc4ef26cbb0c4c1f30a7172 WatchSource:0}: Error finding container 3f25f2697f5d19bf8bb851c6218d54a099a841833dc4ef26cbb0c4c1f30a7172: Status 404 returned error can't find the container with id 3f25f2697f5d19bf8bb851c6218d54a099a841833dc4ef26cbb0c4c1f30a7172
Jul 28 10:08:56.690550 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:08:56.690514    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30e72787_1717_4a7a_b2e2_43e955527f77.slice/crio-84bf3700bb9df0765497c8cf0f7fd2703e0bf4f97e6392f246314f1f2a24147b WatchSource:0}: Error finding container 84bf3700bb9df0765497c8cf0f7fd2703e0bf4f97e6392f246314f1f2a24147b: Status 404 returned error can't find the container with id 84bf3700bb9df0765497c8cf0f7fd2703e0bf4f97e6392f246314f1f2a24147b
Jul 28 10:08:56.695509 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:56.695465    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6dfd5d7545-ckkmc"]
Jul 28 10:08:56.696725 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:56.696702    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ddrhj"]
Jul 28 10:08:56.699874 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:08:56.699850    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bc74d3d_3abf_47f5_8149_7d07a9a34e7c.slice/crio-2619ef34dff5383d739b52f65948d5952edcdcc06b857791a58310bed68dd69a WatchSource:0}: Error finding container 2619ef34dff5383d739b52f65948d5952edcdcc06b857791a58310bed68dd69a: Status 404 returned error can't find the container with id 2619ef34dff5383d739b52f65948d5952edcdcc06b857791a58310bed68dd69a
Jul 28 10:08:56.815818 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:56.815772    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b3c9632-d219-4f7a-8094-36428c83afa0-trusted-ca-bundle\") pod \"console-6dfd5d7545-ckkmc\" (UID: \"1b3c9632-d219-4f7a-8094-36428c83afa0\") " pod="openshift-console/console-6dfd5d7545-ckkmc"
Jul 28 10:08:56.815982 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:56.815845    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bjjv\" (UniqueName: \"kubernetes.io/projected/1b3c9632-d219-4f7a-8094-36428c83afa0-kube-api-access-4bjjv\") pod \"console-6dfd5d7545-ckkmc\" (UID: \"1b3c9632-d219-4f7a-8094-36428c83afa0\") " pod="openshift-console/console-6dfd5d7545-ckkmc"
Jul 28 10:08:56.815982 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:56.815881    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b3c9632-d219-4f7a-8094-36428c83afa0-console-config\") pod \"console-6dfd5d7545-ckkmc\" (UID: \"1b3c9632-d219-4f7a-8094-36428c83afa0\") " pod="openshift-console/console-6dfd5d7545-ckkmc"
Jul 28 10:08:56.815982 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:56.815906    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b3c9632-d219-4f7a-8094-36428c83afa0-oauth-serving-cert\") pod \"console-6dfd5d7545-ckkmc\" (UID: \"1b3c9632-d219-4f7a-8094-36428c83afa0\") " pod="openshift-console/console-6dfd5d7545-ckkmc"
Jul 28 10:08:56.815982 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:56.815934    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b3c9632-d219-4f7a-8094-36428c83afa0-service-ca\") pod \"console-6dfd5d7545-ckkmc\" (UID: \"1b3c9632-d219-4f7a-8094-36428c83afa0\") " pod="openshift-console/console-6dfd5d7545-ckkmc"
Jul 28 10:08:56.816165 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:56.816001    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b3c9632-d219-4f7a-8094-36428c83afa0-console-serving-cert\") pod \"console-6dfd5d7545-ckkmc\" (UID: \"1b3c9632-d219-4f7a-8094-36428c83afa0\") " pod="openshift-console/console-6dfd5d7545-ckkmc"
Jul 28 10:08:56.816165 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:56.816026    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b3c9632-d219-4f7a-8094-36428c83afa0-console-oauth-config\") pod \"console-6dfd5d7545-ckkmc\" (UID: \"1b3c9632-d219-4f7a-8094-36428c83afa0\") " pod="openshift-console/console-6dfd5d7545-ckkmc"
Jul 28 10:08:56.887483 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:56.887444    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7cfb675d-7ghrv" event={"ID":"01f4d561-3325-4ca2-a4ea-40f66f5c2824","Type":"ContainerStarted","Data":"379962eb0c104b839b55b595eee776fee83592056ebec8cea5257170a4664146"}
Jul 28 10:08:56.887686 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:56.887635    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7cfb675d-7ghrv"
Jul 28 10:08:56.889008 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:56.888971    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ddrhj" event={"ID":"0bc74d3d-3abf-47f5-8149-7d07a9a34e7c","Type":"ContainerStarted","Data":"2619ef34dff5383d739b52f65948d5952edcdcc06b857791a58310bed68dd69a"}
Jul 28 10:08:56.889819 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:56.889796    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7cfb675d-7ghrv"
Jul 28 10:08:56.891311 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:56.891285    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5796468f64-jtc5k" event={"ID":"ebd56b7c-8767-4f6b-9189-1019cdef269b","Type":"ContainerStarted","Data":"5d38b6c991aa8b5f9fd16b4103e148fdf03bc0efec1bf98f3db4807daf6269ed"}
Jul 28 10:08:56.893006 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:56.892950    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-m4n6h" event={"ID":"30e72787-1717-4a7a-b2e2-43e955527f77","Type":"ContainerStarted","Data":"84bf3700bb9df0765497c8cf0f7fd2703e0bf4f97e6392f246314f1f2a24147b"}
Jul 28 10:08:56.895750 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:56.895691    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8fltb" event={"ID":"d23b40b0-9299-4a0b-a87c-d2f2cb921fd9","Type":"ContainerStarted","Data":"ab7b79e7fefb36552be8c817c29d66763577e493820b06484adc0f1888c3b084"}
Jul 28 10:08:56.898268 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:56.898242    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mpcmc" event={"ID":"a3e0525c-3c2b-4d2b-ba59-e815dfa07d51","Type":"ContainerStarted","Data":"f0edb18461061924fe177080799d9038ac3cfe42e1f73d763f95f0683d3fe128"}
Jul 28 10:08:56.899643 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:56.899592    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jhkxw" event={"ID":"ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb","Type":"ContainerStarted","Data":"3f25f2697f5d19bf8bb851c6218d54a099a841833dc4ef26cbb0c4c1f30a7172"}
Jul 28 10:08:56.912573 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:56.912521    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7cfb675d-7ghrv" podStartSLOduration=2.108826909 podStartE2EDuration="9.912500675s" totalImagesPullingTime="7.803673766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:08:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:56.912081807 +0000 UTC m=+72.870850186" watchObservedRunningTime="2026-07-28 10:08:56.912500675 +0000 UTC m=+72.871269057"
Jul 28 10:08:56.916413 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:56.916382    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bjjv\" (UniqueName: \"kubernetes.io/projected/1b3c9632-d219-4f7a-8094-36428c83afa0-kube-api-access-4bjjv\") pod \"console-6dfd5d7545-ckkmc\" (UID: \"1b3c9632-d219-4f7a-8094-36428c83afa0\") " pod="openshift-console/console-6dfd5d7545-ckkmc"
Jul 28 10:08:56.916571 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:56.916430    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b3c9632-d219-4f7a-8094-36428c83afa0-console-config\") pod \"console-6dfd5d7545-ckkmc\" (UID: \"1b3c9632-d219-4f7a-8094-36428c83afa0\") " pod="openshift-console/console-6dfd5d7545-ckkmc"
Jul 28 10:08:56.916571 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:56.916448    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b3c9632-d219-4f7a-8094-36428c83afa0-oauth-serving-cert\") pod \"console-6dfd5d7545-ckkmc\" (UID: \"1b3c9632-d219-4f7a-8094-36428c83afa0\") " pod="openshift-console/console-6dfd5d7545-ckkmc"
Jul 28 10:08:56.916571 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:56.916474    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b3c9632-d219-4f7a-8094-36428c83afa0-service-ca\") pod \"console-6dfd5d7545-ckkmc\" (UID: \"1b3c9632-d219-4f7a-8094-36428c83afa0\") " pod="openshift-console/console-6dfd5d7545-ckkmc"
Jul 28 10:08:56.916571 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:56.916500    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b3c9632-d219-4f7a-8094-36428c83afa0-console-serving-cert\") pod \"console-6dfd5d7545-ckkmc\" (UID: \"1b3c9632-d219-4f7a-8094-36428c83afa0\") " pod="openshift-console/console-6dfd5d7545-ckkmc"
Jul 28 10:08:56.916571 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:56.916524    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b3c9632-d219-4f7a-8094-36428c83afa0-console-oauth-config\") pod \"console-6dfd5d7545-ckkmc\" (UID: \"1b3c9632-d219-4f7a-8094-36428c83afa0\") " pod="openshift-console/console-6dfd5d7545-ckkmc"
Jul 28 10:08:56.916571 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:56.916564    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b3c9632-d219-4f7a-8094-36428c83afa0-trusted-ca-bundle\") pod \"console-6dfd5d7545-ckkmc\" (UID: \"1b3c9632-d219-4f7a-8094-36428c83afa0\") " pod="openshift-console/console-6dfd5d7545-ckkmc"
Jul 28 10:08:56.917575 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:56.917513    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b3c9632-d219-4f7a-8094-36428c83afa0-service-ca\") pod \"console-6dfd5d7545-ckkmc\" (UID: \"1b3c9632-d219-4f7a-8094-36428c83afa0\") " pod="openshift-console/console-6dfd5d7545-ckkmc"
Jul 28 10:08:56.917575 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:56.917513    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b3c9632-d219-4f7a-8094-36428c83afa0-console-config\") pod \"console-6dfd5d7545-ckkmc\" (UID: \"1b3c9632-d219-4f7a-8094-36428c83afa0\") " pod="openshift-console/console-6dfd5d7545-ckkmc"
Jul 28 10:08:56.917575 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:56.917567    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b3c9632-d219-4f7a-8094-36428c83afa0-oauth-serving-cert\") pod \"console-6dfd5d7545-ckkmc\" (UID: \"1b3c9632-d219-4f7a-8094-36428c83afa0\") " pod="openshift-console/console-6dfd5d7545-ckkmc"
Jul 28 10:08:56.917806 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:56.917793    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b3c9632-d219-4f7a-8094-36428c83afa0-trusted-ca-bundle\") pod \"console-6dfd5d7545-ckkmc\" (UID: \"1b3c9632-d219-4f7a-8094-36428c83afa0\") " pod="openshift-console/console-6dfd5d7545-ckkmc"
Jul 28 10:08:56.919907 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:56.919884    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b3c9632-d219-4f7a-8094-36428c83afa0-console-oauth-config\") pod \"console-6dfd5d7545-ckkmc\" (UID: \"1b3c9632-d219-4f7a-8094-36428c83afa0\") " pod="openshift-console/console-6dfd5d7545-ckkmc"
Jul 28 10:08:56.920070 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:56.920053    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b3c9632-d219-4f7a-8094-36428c83afa0-console-serving-cert\") pod \"console-6dfd5d7545-ckkmc\" (UID: \"1b3c9632-d219-4f7a-8094-36428c83afa0\") " pod="openshift-console/console-6dfd5d7545-ckkmc"
Jul 28 10:08:56.928834 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:56.928770    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bjjv\" (UniqueName: \"kubernetes.io/projected/1b3c9632-d219-4f7a-8094-36428c83afa0-kube-api-access-4bjjv\") pod \"console-6dfd5d7545-ckkmc\" (UID: \"1b3c9632-d219-4f7a-8094-36428c83afa0\") " pod="openshift-console/console-6dfd5d7545-ckkmc"
Jul 28 10:08:56.941963 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:56.941902    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-mpcmc" podStartSLOduration=17.595066941 podStartE2EDuration="34.94188241s" totalImagesPullingTime="17.346815469s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:08:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:56.941242376 +0000 UTC m=+72.900010781" watchObservedRunningTime="2026-07-28 10:08:56.94188241 +0000 UTC m=+72.900650825"
Jul 28 10:08:56.988931 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:56.988871    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5796468f64-jtc5k" podStartSLOduration=1.64320864 podStartE2EDuration="8.988850815s" totalImagesPullingTime="7.345642175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:08:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:56.988299486 +0000 UTC m=+72.947067864" watchObservedRunningTime="2026-07-28 10:08:56.988850815 +0000 UTC m=+72.947619194"
Jul 28 10:08:56.991446 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:56.991418    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dfd5d7545-ckkmc"
Jul 28 10:08:57.160091 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:57.160058    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6dfd5d7545-ckkmc"]
Jul 28 10:08:57.166770 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:08:57.166732    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b3c9632_d219_4f7a_8094_36428c83afa0.slice/crio-0bb8e2c38aadd518b44dadb9723bd8414b88e8fe4eb155ef1325efad28e3ff23 WatchSource:0}: Error finding container 0bb8e2c38aadd518b44dadb9723bd8414b88e8fe4eb155ef1325efad28e3ff23: Status 404 returned error can't find the container with id 0bb8e2c38aadd518b44dadb9723bd8414b88e8fe4eb155ef1325efad28e3ff23
Jul 28 10:08:57.907147 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:57.907039    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dfd5d7545-ckkmc" event={"ID":"1b3c9632-d219-4f7a-8094-36428c83afa0","Type":"ContainerStarted","Data":"13a613f1d4ea000219d388d66799c6724535dd434ddedc99df97d6e3ff529b72"}
Jul 28 10:08:57.907147 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:57.907107    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dfd5d7545-ckkmc" event={"ID":"1b3c9632-d219-4f7a-8094-36428c83afa0","Type":"ContainerStarted","Data":"0bb8e2c38aadd518b44dadb9723bd8414b88e8fe4eb155ef1325efad28e3ff23"}
Jul 28 10:08:57.910920 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:57.910885    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-69b959dbcb-z2hmw"
Jul 28 10:08:57.931087 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:57.930863    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6dfd5d7545-ckkmc" podStartSLOduration=1.930840249 podStartE2EDuration="1.930840249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:08:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:57.928570373 +0000 UTC m=+73.887338767" watchObservedRunningTime="2026-07-28 10:08:57.930840249 +0000 UTC m=+73.889608629"
Jul 28 10:08:58.597646 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:58.597614    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-ngzcw"]
Jul 28 10:08:58.600094 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:58.600074    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-ngzcw"
Jul 28 10:08:58.603284 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:58.603237    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 28 10:08:58.603871 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:58.603847    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 28 10:08:58.604531 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:58.604126    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 10:08:58.604531 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:58.604342    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 10:08:58.604531 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:58.604357    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 10:08:58.605175 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:58.604959    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-8gs5r\""
Jul 28 10:08:58.618984 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:58.618957    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-ngzcw"]
Jul 28 10:08:58.730573 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:58.730539    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5796468f64-jtc5k"
Jul 28 10:08:58.730766 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:58.730634    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5796468f64-jtc5k"
Jul 28 10:08:58.736138 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:58.735046    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/949b3403-ad3f-4057-8470-4aa86a2b97c8-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-ngzcw\" (UID: \"949b3403-ad3f-4057-8470-4aa86a2b97c8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ngzcw"
Jul 28 10:08:58.736138 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:58.735132    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs45w\" (UniqueName: \"kubernetes.io/projected/949b3403-ad3f-4057-8470-4aa86a2b97c8-kube-api-access-bs45w\") pod \"prometheus-operator-84d69f794d-ngzcw\" (UID: \"949b3403-ad3f-4057-8470-4aa86a2b97c8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ngzcw"
Jul 28 10:08:58.736138 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:58.735181    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/949b3403-ad3f-4057-8470-4aa86a2b97c8-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-ngzcw\" (UID: \"949b3403-ad3f-4057-8470-4aa86a2b97c8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ngzcw"
Jul 28 10:08:58.736138 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:58.735233    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/949b3403-ad3f-4057-8470-4aa86a2b97c8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-ngzcw\" (UID: \"949b3403-ad3f-4057-8470-4aa86a2b97c8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ngzcw"
Jul 28 10:08:58.737473 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:58.737449    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5796468f64-jtc5k"
Jul 28 10:08:58.837272 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:58.836028    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs45w\" (UniqueName: \"kubernetes.io/projected/949b3403-ad3f-4057-8470-4aa86a2b97c8-kube-api-access-bs45w\") pod \"prometheus-operator-84d69f794d-ngzcw\" (UID: \"949b3403-ad3f-4057-8470-4aa86a2b97c8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ngzcw"
Jul 28 10:08:58.837272 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:58.836085    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/949b3403-ad3f-4057-8470-4aa86a2b97c8-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-ngzcw\" (UID: \"949b3403-ad3f-4057-8470-4aa86a2b97c8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ngzcw"
Jul 28 10:08:58.837272 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:58.836141    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/949b3403-ad3f-4057-8470-4aa86a2b97c8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-ngzcw\" (UID: \"949b3403-ad3f-4057-8470-4aa86a2b97c8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ngzcw"
Jul 28 10:08:58.837272 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:58.836179    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/949b3403-ad3f-4057-8470-4aa86a2b97c8-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-ngzcw\" (UID: \"949b3403-ad3f-4057-8470-4aa86a2b97c8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ngzcw"
Jul 28 10:08:58.837272 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:58.836660    2557 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 28 10:08:58.837272 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:08:58.836735    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/949b3403-ad3f-4057-8470-4aa86a2b97c8-prometheus-operator-tls podName:949b3403-ad3f-4057-8470-4aa86a2b97c8 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:59.336713585 +0000 UTC m=+75.295481956 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/949b3403-ad3f-4057-8470-4aa86a2b97c8-prometheus-operator-tls") pod "prometheus-operator-84d69f794d-ngzcw" (UID: "949b3403-ad3f-4057-8470-4aa86a2b97c8") : secret "prometheus-operator-tls" not found
Jul 28 10:08:58.837272 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:58.837007    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/949b3403-ad3f-4057-8470-4aa86a2b97c8-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-ngzcw\" (UID: \"949b3403-ad3f-4057-8470-4aa86a2b97c8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ngzcw"
Jul 28 10:08:58.845780 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:58.845712    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/949b3403-ad3f-4057-8470-4aa86a2b97c8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-ngzcw\" (UID: \"949b3403-ad3f-4057-8470-4aa86a2b97c8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ngzcw"
Jul 28 10:08:58.846634 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:58.846578    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs45w\" (UniqueName: \"kubernetes.io/projected/949b3403-ad3f-4057-8470-4aa86a2b97c8-kube-api-access-bs45w\") pod \"prometheus-operator-84d69f794d-ngzcw\" (UID: \"949b3403-ad3f-4057-8470-4aa86a2b97c8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ngzcw"
Jul 28 10:08:58.918571 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:58.918500    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-m4n6h" event={"ID":"30e72787-1717-4a7a-b2e2-43e955527f77","Type":"ContainerStarted","Data":"5df271ebcde82ba9397eef0ab0d82e784cc3092cfae790a76828e00c08f09ae2"}
Jul 28 10:08:58.923482 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:58.923074    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8fltb" event={"ID":"d23b40b0-9299-4a0b-a87c-d2f2cb921fd9","Type":"ContainerStarted","Data":"20764b592321db8289127dc7639efbbbc5ce1f2faf8c212c81452e6f78020b8b"}
Jul 28 10:08:58.928458 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:58.928393    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5796468f64-jtc5k"
Jul 28 10:08:58.967575 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:58.966148    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-m4n6h" podStartSLOduration=37.905259509 podStartE2EDuration="39.966125104s" totalImagesPullingTime="2.060865595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:08:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:58.94248245 +0000 UTC m=+74.901250829" watchObservedRunningTime="2026-07-28 10:08:58.966125104 +0000 UTC m=+74.924893490"
Jul 28 10:08:59.341528 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:59.341484    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/949b3403-ad3f-4057-8470-4aa86a2b97c8-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-ngzcw\" (UID: \"949b3403-ad3f-4057-8470-4aa86a2b97c8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ngzcw"
Jul 28 10:08:59.344994 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:59.344964    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/949b3403-ad3f-4057-8470-4aa86a2b97c8-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-ngzcw\" (UID: \"949b3403-ad3f-4057-8470-4aa86a2b97c8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ngzcw"
Jul 28 10:08:59.516770 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:59.516707    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-ngzcw"
Jul 28 10:08:59.932123 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:59.932082    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8fltb" event={"ID":"d23b40b0-9299-4a0b-a87c-d2f2cb921fd9","Type":"ContainerStarted","Data":"76ee309d9c84a3bdbf8d0e2c16b8566670da728ecf022a394887490cb56fe6f1"}
Jul 28 10:08:59.965111 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:08:59.965055    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-8fltb" podStartSLOduration=72.683826607 podStartE2EDuration="1m14.965039185s" totalImagesPullingTime="2.281212578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:07:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:59.963988711 +0000 UTC m=+75.922757091" watchObservedRunningTime="2026-07-28 10:08:59.965039185 +0000 UTC m=+75.923807564"
Jul 28 10:09:06.991910 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:06.991770    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6dfd5d7545-ckkmc"
Jul 28 10:09:06.991910 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:06.991844    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6dfd5d7545-ckkmc"
Jul 28 10:09:06.997553 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:06.997528    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6dfd5d7545-ckkmc"
Jul 28 10:09:07.972051 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:07.972023    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6dfd5d7545-ckkmc"
Jul 28 10:09:08.034823 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:08.034791    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5796468f64-jtc5k"]
Jul 28 10:09:09.860491 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:09.860466    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-64cd7c8686-n6ktg"
Jul 28 10:09:10.201581 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:10.201561    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-ngzcw"]
Jul 28 10:09:10.203392 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:09:10.203357    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod949b3403_ad3f_4057_8470_4aa86a2b97c8.slice/crio-649366c9220c40c9f159343833ab8583422033cec77e9b17b8f969a75f2b3db2 WatchSource:0}: Error finding container 649366c9220c40c9f159343833ab8583422033cec77e9b17b8f969a75f2b3db2: Status 404 returned error can't find the container with id 649366c9220c40c9f159343833ab8583422033cec77e9b17b8f969a75f2b3db2
Jul 28 10:09:10.983480 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:10.983403    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ddrhj" event={"ID":"0bc74d3d-3abf-47f5-8149-7d07a9a34e7c","Type":"ContainerStarted","Data":"496bac850b882f0efe6365ca9c0df9507e31788c95d673f9c6e78e09e3ac5ea5"}
Jul 28 10:09:10.984285 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:10.984227    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ddrhj"
Jul 28 10:09:10.986772 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:10.986701    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-xktrq" event={"ID":"a49e6876-498a-40b4-9ccf-94cf1d02bd97","Type":"ContainerStarted","Data":"6de8901279850f865baa6eb08b1f0082ca79aee9a4ed04483a3bda4225137662"}
Jul 28 10:09:10.987132 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:10.987078    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-xktrq"
Jul 28 10:09:10.989663 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:10.989635    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-ngzcw" event={"ID":"949b3403-ad3f-4057-8470-4aa86a2b97c8","Type":"ContainerStarted","Data":"649366c9220c40c9f159343833ab8583422033cec77e9b17b8f969a75f2b3db2"}
Jul 28 10:09:10.991353 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:10.991327    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jhkxw" event={"ID":"ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb","Type":"ContainerStarted","Data":"1a5686c3696b9aa098bd1525872d4299b9d77497ac8e304fad1724055c8c32f5"}
Jul 28 10:09:10.997268 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:10.997239    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-xktrq"
Jul 28 10:09:11.022452 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:11.022136    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-xktrq" podStartSLOduration=2.4835470539999998 podStartE2EDuration="24.022114095s" totalImagesPullingTime="21.538567041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:08:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:09:11.02118468 +0000 UTC m=+86.979953061" watchObservedRunningTime="2026-07-28 10:09:11.022114095 +0000 UTC m=+86.980882475"
Jul 28 10:09:11.023036 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:11.022901    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ddrhj" podStartSLOduration=72.679919116 podStartE2EDuration="1m26.022882724s" totalImagesPullingTime="13.342963608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:07:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:09:11.004307935 +0000 UTC m=+86.963076315" watchObservedRunningTime="2026-07-28 10:09:11.022882724 +0000 UTC m=+86.981651104"
Jul 28 10:09:11.040153 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:11.039203    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-jhkxw" podStartSLOduration=72.682844483 podStartE2EDuration="1m26.039180867s" totalImagesPullingTime="13.356336384s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:07:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:09:11.037873026 +0000 UTC m=+86.996641406" watchObservedRunningTime="2026-07-28 10:09:11.039180867 +0000 UTC m=+86.997949247"
Jul 28 10:09:12.923168 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:12.923123    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-69b959dbcb-z2hmw" podUID="e43afa04-2c0d-445d-a2c8-86512466e05b" containerName="registry" containerID="cri-o://97f0a69fad9f3328639d9b84ef05994c4cb180f70bb4a4a16badb3d4ed826a9f" gracePeriod=30
Jul 28 10:09:13.003485 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:13.003442    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-ngzcw" event={"ID":"949b3403-ad3f-4057-8470-4aa86a2b97c8","Type":"ContainerStarted","Data":"694861aa0c5efcf556f4bed5a39e3e3f7ae052d16f8420e3c79338beea45920a"}
Jul 28 10:09:13.003705 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:13.003512    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-ngzcw" event={"ID":"949b3403-ad3f-4057-8470-4aa86a2b97c8","Type":"ContainerStarted","Data":"81c670e9ddaab1c99796a9d0b904e9dd7fa531376d98f9d96560b90d35bb8e50"}
Jul 28 10:09:13.025290 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:13.025241    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-ngzcw" podStartSLOduration=13.104235297 podStartE2EDuration="15.025222787s" totalImagesPullingTime="1.92098749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:08:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:09:13.024020529 +0000 UTC m=+88.982788899" watchObservedRunningTime="2026-07-28 10:09:13.025222787 +0000 UTC m=+88.983991166"
Jul 28 10:09:13.212799 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:13.212769    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69b959dbcb-z2hmw"
Jul 28 10:09:13.267897 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:13.267864    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e43afa04-2c0d-445d-a2c8-86512466e05b-installation-pull-secrets\") pod \"e43afa04-2c0d-445d-a2c8-86512466e05b\" (UID: \"e43afa04-2c0d-445d-a2c8-86512466e05b\") "
Jul 28 10:09:13.267897 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:13.267902    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e43afa04-2c0d-445d-a2c8-86512466e05b-ca-trust-extracted\") pod \"e43afa04-2c0d-445d-a2c8-86512466e05b\" (UID: \"e43afa04-2c0d-445d-a2c8-86512466e05b\") "
Jul 28 10:09:13.268109 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:13.267943    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e43afa04-2c0d-445d-a2c8-86512466e05b-registry-certificates\") pod \"e43afa04-2c0d-445d-a2c8-86512466e05b\" (UID: \"e43afa04-2c0d-445d-a2c8-86512466e05b\") "
Jul 28 10:09:13.268282 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:13.268254    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e43afa04-2c0d-445d-a2c8-86512466e05b-bound-sa-token\") pod \"e43afa04-2c0d-445d-a2c8-86512466e05b\" (UID: \"e43afa04-2c0d-445d-a2c8-86512466e05b\") "
Jul 28 10:09:13.268414 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:13.268301    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e43afa04-2c0d-445d-a2c8-86512466e05b-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "e43afa04-2c0d-445d-a2c8-86512466e05b" (UID: "e43afa04-2c0d-445d-a2c8-86512466e05b"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:09:13.268414 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:13.268328    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e43afa04-2c0d-445d-a2c8-86512466e05b-trusted-ca\") pod \"e43afa04-2c0d-445d-a2c8-86512466e05b\" (UID: \"e43afa04-2c0d-445d-a2c8-86512466e05b\") "
Jul 28 10:09:13.268414 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:13.268365    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e43afa04-2c0d-445d-a2c8-86512466e05b-image-registry-private-configuration\") pod \"e43afa04-2c0d-445d-a2c8-86512466e05b\" (UID: \"e43afa04-2c0d-445d-a2c8-86512466e05b\") "
Jul 28 10:09:13.268414 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:13.268389    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e43afa04-2c0d-445d-a2c8-86512466e05b-registry-tls\") pod \"e43afa04-2c0d-445d-a2c8-86512466e05b\" (UID: \"e43afa04-2c0d-445d-a2c8-86512466e05b\") "
Jul 28 10:09:13.268641 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:13.268415    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zg72v\" (UniqueName: \"kubernetes.io/projected/e43afa04-2c0d-445d-a2c8-86512466e05b-kube-api-access-zg72v\") pod \"e43afa04-2c0d-445d-a2c8-86512466e05b\" (UID: \"e43afa04-2c0d-445d-a2c8-86512466e05b\") "
Jul 28 10:09:13.268701 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:13.268672    2557 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e43afa04-2c0d-445d-a2c8-86512466e05b-registry-certificates\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:09:13.269958 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:13.269904    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e43afa04-2c0d-445d-a2c8-86512466e05b-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "e43afa04-2c0d-445d-a2c8-86512466e05b" (UID: "e43afa04-2c0d-445d-a2c8-86512466e05b"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:09:13.271491 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:13.271464    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e43afa04-2c0d-445d-a2c8-86512466e05b-kube-api-access-zg72v" (OuterVolumeSpecName: "kube-api-access-zg72v") pod "e43afa04-2c0d-445d-a2c8-86512466e05b" (UID: "e43afa04-2c0d-445d-a2c8-86512466e05b"). InnerVolumeSpecName "kube-api-access-zg72v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:09:13.271652 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:13.271629    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e43afa04-2c0d-445d-a2c8-86512466e05b-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "e43afa04-2c0d-445d-a2c8-86512466e05b" (UID: "e43afa04-2c0d-445d-a2c8-86512466e05b"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:09:13.271812 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:13.271782    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e43afa04-2c0d-445d-a2c8-86512466e05b-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "e43afa04-2c0d-445d-a2c8-86512466e05b" (UID: "e43afa04-2c0d-445d-a2c8-86512466e05b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:09:13.271880 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:13.271825    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e43afa04-2c0d-445d-a2c8-86512466e05b-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "e43afa04-2c0d-445d-a2c8-86512466e05b" (UID: "e43afa04-2c0d-445d-a2c8-86512466e05b"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:09:13.272293 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:13.272266    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e43afa04-2c0d-445d-a2c8-86512466e05b-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "e43afa04-2c0d-445d-a2c8-86512466e05b" (UID: "e43afa04-2c0d-445d-a2c8-86512466e05b"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:09:13.281283 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:13.281250    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e43afa04-2c0d-445d-a2c8-86512466e05b-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "e43afa04-2c0d-445d-a2c8-86512466e05b" (UID: "e43afa04-2c0d-445d-a2c8-86512466e05b"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:09:13.369948 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:13.369910    2557 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e43afa04-2c0d-445d-a2c8-86512466e05b-installation-pull-secrets\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:09:13.369948 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:13.369942    2557 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e43afa04-2c0d-445d-a2c8-86512466e05b-ca-trust-extracted\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:09:13.369948 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:13.369957    2557 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e43afa04-2c0d-445d-a2c8-86512466e05b-bound-sa-token\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:09:13.370199 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:13.369970    2557 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e43afa04-2c0d-445d-a2c8-86512466e05b-trusted-ca\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:09:13.370199 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:13.369985    2557 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e43afa04-2c0d-445d-a2c8-86512466e05b-image-registry-private-configuration\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:09:13.370199 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:13.369997    2557 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e43afa04-2c0d-445d-a2c8-86512466e05b-registry-tls\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:09:13.370199 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:13.370011    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zg72v\" (UniqueName: \"kubernetes.io/projected/e43afa04-2c0d-445d-a2c8-86512466e05b-kube-api-access-zg72v\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:09:14.008544 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:14.008507    2557 generic.go:356] "Generic (PLEG): container finished" podID="e43afa04-2c0d-445d-a2c8-86512466e05b" containerID="97f0a69fad9f3328639d9b84ef05994c4cb180f70bb4a4a16badb3d4ed826a9f" exitCode=0
Jul 28 10:09:14.009137 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:14.008566    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69b959dbcb-z2hmw"
Jul 28 10:09:14.009137 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:14.008595    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69b959dbcb-z2hmw" event={"ID":"e43afa04-2c0d-445d-a2c8-86512466e05b","Type":"ContainerDied","Data":"97f0a69fad9f3328639d9b84ef05994c4cb180f70bb4a4a16badb3d4ed826a9f"}
Jul 28 10:09:14.009137 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:14.008674    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69b959dbcb-z2hmw" event={"ID":"e43afa04-2c0d-445d-a2c8-86512466e05b","Type":"ContainerDied","Data":"6d992082c109b5ed55c22eb41218984864a024ece25fa0d076d8952f2e3651d9"}
Jul 28 10:09:14.009137 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:14.008701    2557 scope.go:117] "RemoveContainer" containerID="97f0a69fad9f3328639d9b84ef05994c4cb180f70bb4a4a16badb3d4ed826a9f"
Jul 28 10:09:14.018812 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:14.018790    2557 scope.go:117] "RemoveContainer" containerID="97f0a69fad9f3328639d9b84ef05994c4cb180f70bb4a4a16badb3d4ed826a9f"
Jul 28 10:09:14.019155 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:09:14.019127    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97f0a69fad9f3328639d9b84ef05994c4cb180f70bb4a4a16badb3d4ed826a9f\": container with ID starting with 97f0a69fad9f3328639d9b84ef05994c4cb180f70bb4a4a16badb3d4ed826a9f not found: ID does not exist" containerID="97f0a69fad9f3328639d9b84ef05994c4cb180f70bb4a4a16badb3d4ed826a9f"
Jul 28 10:09:14.019233 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:14.019170    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97f0a69fad9f3328639d9b84ef05994c4cb180f70bb4a4a16badb3d4ed826a9f"} err="failed to get container status \"97f0a69fad9f3328639d9b84ef05994c4cb180f70bb4a4a16badb3d4ed826a9f\": rpc error: code = NotFound desc = could not find container \"97f0a69fad9f3328639d9b84ef05994c4cb180f70bb4a4a16badb3d4ed826a9f\": container with ID starting with 97f0a69fad9f3328639d9b84ef05994c4cb180f70bb4a4a16badb3d4ed826a9f not found: ID does not exist"
Jul 28 10:09:14.031808 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:14.031777    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-69b959dbcb-z2hmw"]
Jul 28 10:09:14.038409 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:14.035446    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-69b959dbcb-z2hmw"]
Jul 28 10:09:14.580415 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:14.580374    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e43afa04-2c0d-445d-a2c8-86512466e05b" path="/var/lib/kubelet/pods/e43afa04-2c0d-445d-a2c8-86512466e05b/volumes"
Jul 28 10:09:15.027479 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.027448    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-27zms"]
Jul 28 10:09:15.028287 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.028264    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e43afa04-2c0d-445d-a2c8-86512466e05b" containerName="registry"
Jul 28 10:09:15.028429 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.028392    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e43afa04-2c0d-445d-a2c8-86512466e05b" containerName="registry"
Jul 28 10:09:15.028516 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.028501    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="e43afa04-2c0d-445d-a2c8-86512466e05b" containerName="registry"
Jul 28 10:09:15.065001 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.064765    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-27zms"]
Jul 28 10:09:15.065001 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.064914    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-27zms"
Jul 28 10:09:15.069223 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.067638    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 28 10:09:15.070144 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.070069    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 28 10:09:15.071167 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.070376    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-754nx\""
Jul 28 10:09:15.071557 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.071528    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 28 10:09:15.078017 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.077993    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-6ps9j"]
Jul 28 10:09:15.085155 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.085120    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhxjq\" (UniqueName: \"kubernetes.io/projected/dde5d137-fa6b-4c1e-ad94-07f174179c6c-kube-api-access-rhxjq\") pod \"kube-state-metrics-56d9f944b4-27zms\" (UID: \"dde5d137-fa6b-4c1e-ad94-07f174179c6c\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-27zms"
Jul 28 10:09:15.085281 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.085193    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dde5d137-fa6b-4c1e-ad94-07f174179c6c-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-27zms\" (UID: \"dde5d137-fa6b-4c1e-ad94-07f174179c6c\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-27zms"
Jul 28 10:09:15.085281 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.085221    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/dde5d137-fa6b-4c1e-ad94-07f174179c6c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-27zms\" (UID: \"dde5d137-fa6b-4c1e-ad94-07f174179c6c\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-27zms"
Jul 28 10:09:15.085281 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.085248    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dde5d137-fa6b-4c1e-ad94-07f174179c6c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-27zms\" (UID: \"dde5d137-fa6b-4c1e-ad94-07f174179c6c\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-27zms"
Jul 28 10:09:15.085453 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.085284    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dde5d137-fa6b-4c1e-ad94-07f174179c6c-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-27zms\" (UID: \"dde5d137-fa6b-4c1e-ad94-07f174179c6c\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-27zms"
Jul 28 10:09:15.085453 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.085316    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/dde5d137-fa6b-4c1e-ad94-07f174179c6c-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-27zms\" (UID: \"dde5d137-fa6b-4c1e-ad94-07f174179c6c\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-27zms"
Jul 28 10:09:15.098987 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.098842    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6ps9j"
Jul 28 10:09:15.101630 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.101318    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xqkxz\""
Jul 28 10:09:15.104056 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.103920    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 10:09:15.105916 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.105895    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 10:09:15.106001 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.105922    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 10:09:15.186000 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.185960    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dde5d137-fa6b-4c1e-ad94-07f174179c6c-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-27zms\" (UID: \"dde5d137-fa6b-4c1e-ad94-07f174179c6c\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-27zms"
Jul 28 10:09:15.186000 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.186010    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/dde5d137-fa6b-4c1e-ad94-07f174179c6c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-27zms\" (UID: \"dde5d137-fa6b-4c1e-ad94-07f174179c6c\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-27zms"
Jul 28 10:09:15.186258 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.186038    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dde5d137-fa6b-4c1e-ad94-07f174179c6c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-27zms\" (UID: \"dde5d137-fa6b-4c1e-ad94-07f174179c6c\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-27zms"
Jul 28 10:09:15.186258 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.186068    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03488ce4-1c3e-4258-854e-913b25aac684-metrics-client-ca\") pod \"node-exporter-6ps9j\" (UID: \"03488ce4-1c3e-4258-854e-913b25aac684\") " pod="openshift-monitoring/node-exporter-6ps9j"
Jul 28 10:09:15.186258 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.186097    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/03488ce4-1c3e-4258-854e-913b25aac684-node-exporter-accelerators-collector-config\") pod \"node-exporter-6ps9j\" (UID: \"03488ce4-1c3e-4258-854e-913b25aac684\") " pod="openshift-monitoring/node-exporter-6ps9j"
Jul 28 10:09:15.186258 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.186126    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dde5d137-fa6b-4c1e-ad94-07f174179c6c-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-27zms\" (UID: \"dde5d137-fa6b-4c1e-ad94-07f174179c6c\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-27zms"
Jul 28 10:09:15.186461 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:09:15.186295    2557 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 28 10:09:15.186461 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.186359    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/03488ce4-1c3e-4258-854e-913b25aac684-node-exporter-tls\") pod \"node-exporter-6ps9j\" (UID: \"03488ce4-1c3e-4258-854e-913b25aac684\") " pod="openshift-monitoring/node-exporter-6ps9j"
Jul 28 10:09:15.186461 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:09:15.186368    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dde5d137-fa6b-4c1e-ad94-07f174179c6c-kube-state-metrics-tls podName:dde5d137-fa6b-4c1e-ad94-07f174179c6c nodeName:}" failed. No retries permitted until 2026-07-28 10:09:15.686345593 +0000 UTC m=+91.645113965 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/dde5d137-fa6b-4c1e-ad94-07f174179c6c-kube-state-metrics-tls") pod "kube-state-metrics-56d9f944b4-27zms" (UID: "dde5d137-fa6b-4c1e-ad94-07f174179c6c") : secret "kube-state-metrics-tls" not found
Jul 28 10:09:15.186461 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.186413    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/dde5d137-fa6b-4c1e-ad94-07f174179c6c-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-27zms\" (UID: \"dde5d137-fa6b-4c1e-ad94-07f174179c6c\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-27zms"
Jul 28 10:09:15.186705 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.186469    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcnbr\" (UniqueName: \"kubernetes.io/projected/03488ce4-1c3e-4258-854e-913b25aac684-kube-api-access-bcnbr\") pod \"node-exporter-6ps9j\" (UID: \"03488ce4-1c3e-4258-854e-913b25aac684\") " pod="openshift-monitoring/node-exporter-6ps9j"
Jul 28 10:09:15.186705 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.186537    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/03488ce4-1c3e-4258-854e-913b25aac684-node-exporter-textfile\") pod \"node-exporter-6ps9j\" (UID: \"03488ce4-1c3e-4258-854e-913b25aac684\") " pod="openshift-monitoring/node-exporter-6ps9j"
Jul 28 10:09:15.186705 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.186573    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/03488ce4-1c3e-4258-854e-913b25aac684-node-exporter-wtmp\") pod \"node-exporter-6ps9j\" (UID: \"03488ce4-1c3e-4258-854e-913b25aac684\") " pod="openshift-monitoring/node-exporter-6ps9j"
Jul 28 10:09:15.186705 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.186623    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhxjq\" (UniqueName: \"kubernetes.io/projected/dde5d137-fa6b-4c1e-ad94-07f174179c6c-kube-api-access-rhxjq\") pod \"kube-state-metrics-56d9f944b4-27zms\" (UID: \"dde5d137-fa6b-4c1e-ad94-07f174179c6c\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-27zms"
Jul 28 10:09:15.186705 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.186658    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/03488ce4-1c3e-4258-854e-913b25aac684-sys\") pod \"node-exporter-6ps9j\" (UID: \"03488ce4-1c3e-4258-854e-913b25aac684\") " pod="openshift-monitoring/node-exporter-6ps9j"
Jul 28 10:09:15.186705 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.186684    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/03488ce4-1c3e-4258-854e-913b25aac684-root\") pod \"node-exporter-6ps9j\" (UID: \"03488ce4-1c3e-4258-854e-913b25aac684\") " pod="openshift-monitoring/node-exporter-6ps9j"
Jul 28 10:09:15.187008 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.186712    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/03488ce4-1c3e-4258-854e-913b25aac684-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6ps9j\" (UID: \"03488ce4-1c3e-4258-854e-913b25aac684\") " pod="openshift-monitoring/node-exporter-6ps9j"
Jul 28 10:09:15.187008 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.186761    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/dde5d137-fa6b-4c1e-ad94-07f174179c6c-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-27zms\" (UID: \"dde5d137-fa6b-4c1e-ad94-07f174179c6c\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-27zms"
Jul 28 10:09:15.187122 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.187013    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dde5d137-fa6b-4c1e-ad94-07f174179c6c-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-27zms\" (UID: \"dde5d137-fa6b-4c1e-ad94-07f174179c6c\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-27zms"
Jul 28 10:09:15.187616 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.187573    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/dde5d137-fa6b-4c1e-ad94-07f174179c6c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-27zms\" (UID: \"dde5d137-fa6b-4c1e-ad94-07f174179c6c\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-27zms"
Jul 28 10:09:15.189776 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.189744    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dde5d137-fa6b-4c1e-ad94-07f174179c6c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-27zms\" (UID: \"dde5d137-fa6b-4c1e-ad94-07f174179c6c\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-27zms"
Jul 28 10:09:15.206967 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.206937    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhxjq\" (UniqueName: \"kubernetes.io/projected/dde5d137-fa6b-4c1e-ad94-07f174179c6c-kube-api-access-rhxjq\") pod \"kube-state-metrics-56d9f944b4-27zms\" (UID: \"dde5d137-fa6b-4c1e-ad94-07f174179c6c\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-27zms"
Jul 28 10:09:15.287790 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.287711    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/03488ce4-1c3e-4258-854e-913b25aac684-node-exporter-tls\") pod \"node-exporter-6ps9j\" (UID: \"03488ce4-1c3e-4258-854e-913b25aac684\") " pod="openshift-monitoring/node-exporter-6ps9j"
Jul 28 10:09:15.287790 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.287764    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcnbr\" (UniqueName: \"kubernetes.io/projected/03488ce4-1c3e-4258-854e-913b25aac684-kube-api-access-bcnbr\") pod \"node-exporter-6ps9j\" (UID: \"03488ce4-1c3e-4258-854e-913b25aac684\") " pod="openshift-monitoring/node-exporter-6ps9j"
Jul 28 10:09:15.288014 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.287814    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/03488ce4-1c3e-4258-854e-913b25aac684-node-exporter-textfile\") pod \"node-exporter-6ps9j\" (UID: \"03488ce4-1c3e-4258-854e-913b25aac684\") " pod="openshift-monitoring/node-exporter-6ps9j"
Jul 28 10:09:15.288014 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.287845    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/03488ce4-1c3e-4258-854e-913b25aac684-node-exporter-wtmp\") pod \"node-exporter-6ps9j\" (UID: \"03488ce4-1c3e-4258-854e-913b25aac684\") " pod="openshift-monitoring/node-exporter-6ps9j"
Jul 28 10:09:15.288014 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:09:15.287890    2557 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 28 10:09:15.288014 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:09:15.287965    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03488ce4-1c3e-4258-854e-913b25aac684-node-exporter-tls podName:03488ce4-1c3e-4258-854e-913b25aac684 nodeName:}" failed. No retries permitted until 2026-07-28 10:09:15.78794349 +0000 UTC m=+91.746711847 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/03488ce4-1c3e-4258-854e-913b25aac684-node-exporter-tls") pod "node-exporter-6ps9j" (UID: "03488ce4-1c3e-4258-854e-913b25aac684") : secret "node-exporter-tls" not found
Jul 28 10:09:15.288014 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.287990    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/03488ce4-1c3e-4258-854e-913b25aac684-sys\") pod \"node-exporter-6ps9j\" (UID: \"03488ce4-1c3e-4258-854e-913b25aac684\") " pod="openshift-monitoring/node-exporter-6ps9j"
Jul 28 10:09:15.288014 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.288005    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/03488ce4-1c3e-4258-854e-913b25aac684-node-exporter-wtmp\") pod \"node-exporter-6ps9j\" (UID: \"03488ce4-1c3e-4258-854e-913b25aac684\") " pod="openshift-monitoring/node-exporter-6ps9j"
Jul 28 10:09:15.288338 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.288022    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/03488ce4-1c3e-4258-854e-913b25aac684-root\") pod \"node-exporter-6ps9j\" (UID: \"03488ce4-1c3e-4258-854e-913b25aac684\") " pod="openshift-monitoring/node-exporter-6ps9j"
Jul 28 10:09:15.288338 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.288050    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/03488ce4-1c3e-4258-854e-913b25aac684-sys\") pod \"node-exporter-6ps9j\" (UID: \"03488ce4-1c3e-4258-854e-913b25aac684\") " pod="openshift-monitoring/node-exporter-6ps9j"
Jul 28 10:09:15.288338 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.288054    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/03488ce4-1c3e-4258-854e-913b25aac684-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6ps9j\" (UID: \"03488ce4-1c3e-4258-854e-913b25aac684\") " pod="openshift-monitoring/node-exporter-6ps9j"
Jul 28 10:09:15.288338 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.288068    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/03488ce4-1c3e-4258-854e-913b25aac684-root\") pod \"node-exporter-6ps9j\" (UID: \"03488ce4-1c3e-4258-854e-913b25aac684\") " pod="openshift-monitoring/node-exporter-6ps9j"
Jul 28 10:09:15.288338 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.288116    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03488ce4-1c3e-4258-854e-913b25aac684-metrics-client-ca\") pod \"node-exporter-6ps9j\" (UID: \"03488ce4-1c3e-4258-854e-913b25aac684\") " pod="openshift-monitoring/node-exporter-6ps9j"
Jul 28 10:09:15.288338 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.288149    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/03488ce4-1c3e-4258-854e-913b25aac684-node-exporter-accelerators-collector-config\") pod \"node-exporter-6ps9j\" (UID: \"03488ce4-1c3e-4258-854e-913b25aac684\") " pod="openshift-monitoring/node-exporter-6ps9j"
Jul 28 10:09:15.288842 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.288820    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03488ce4-1c3e-4258-854e-913b25aac684-metrics-client-ca\") pod \"node-exporter-6ps9j\" (UID: \"03488ce4-1c3e-4258-854e-913b25aac684\") " pod="openshift-monitoring/node-exporter-6ps9j"
Jul 28 10:09:15.289210 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.288855    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/03488ce4-1c3e-4258-854e-913b25aac684-node-exporter-accelerators-collector-config\") pod \"node-exporter-6ps9j\" (UID: \"03488ce4-1c3e-4258-854e-913b25aac684\") " pod="openshift-monitoring/node-exporter-6ps9j"
Jul 28 10:09:15.289210 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.289160    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/03488ce4-1c3e-4258-854e-913b25aac684-node-exporter-textfile\") pod \"node-exporter-6ps9j\" (UID: \"03488ce4-1c3e-4258-854e-913b25aac684\") " pod="openshift-monitoring/node-exporter-6ps9j"
Jul 28 10:09:15.290887 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.290867    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/03488ce4-1c3e-4258-854e-913b25aac684-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6ps9j\" (UID: \"03488ce4-1c3e-4258-854e-913b25aac684\") " pod="openshift-monitoring/node-exporter-6ps9j"
Jul 28 10:09:15.306700 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.306674    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcnbr\" (UniqueName: \"kubernetes.io/projected/03488ce4-1c3e-4258-854e-913b25aac684-kube-api-access-bcnbr\") pod \"node-exporter-6ps9j\" (UID: \"03488ce4-1c3e-4258-854e-913b25aac684\") " pod="openshift-monitoring/node-exporter-6ps9j"
Jul 28 10:09:15.692479 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.692395    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dde5d137-fa6b-4c1e-ad94-07f174179c6c-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-27zms\" (UID: \"dde5d137-fa6b-4c1e-ad94-07f174179c6c\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-27zms"
Jul 28 10:09:15.695416 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.695390    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dde5d137-fa6b-4c1e-ad94-07f174179c6c-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-27zms\" (UID: \"dde5d137-fa6b-4c1e-ad94-07f174179c6c\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-27zms"
Jul 28 10:09:15.793426 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.793389    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/03488ce4-1c3e-4258-854e-913b25aac684-node-exporter-tls\") pod \"node-exporter-6ps9j\" (UID: \"03488ce4-1c3e-4258-854e-913b25aac684\") " pod="openshift-monitoring/node-exporter-6ps9j"
Jul 28 10:09:15.793641 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:09:15.793617    2557 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 28 10:09:15.793771 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:09:15.793718    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03488ce4-1c3e-4258-854e-913b25aac684-node-exporter-tls podName:03488ce4-1c3e-4258-854e-913b25aac684 nodeName:}" failed. No retries permitted until 2026-07-28 10:09:16.793697043 +0000 UTC m=+92.752465416 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/03488ce4-1c3e-4258-854e-913b25aac684-node-exporter-tls") pod "node-exporter-6ps9j" (UID: "03488ce4-1c3e-4258-854e-913b25aac684") : secret "node-exporter-tls" not found
Jul 28 10:09:15.979351 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:15.979259    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-27zms"
Jul 28 10:09:16.190631 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:16.190581    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-27zms"]
Jul 28 10:09:16.195272 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:09:16.195107    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddde5d137_fa6b_4c1e_ad94_07f174179c6c.slice/crio-d1620724c1373b316d3b8f42812454c9b18fc878c903d32cff6dd220a2338bc7 WatchSource:0}: Error finding container d1620724c1373b316d3b8f42812454c9b18fc878c903d32cff6dd220a2338bc7: Status 404 returned error can't find the container with id d1620724c1373b316d3b8f42812454c9b18fc878c903d32cff6dd220a2338bc7
Jul 28 10:09:16.803213 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:16.803166    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/03488ce4-1c3e-4258-854e-913b25aac684-node-exporter-tls\") pod \"node-exporter-6ps9j\" (UID: \"03488ce4-1c3e-4258-854e-913b25aac684\") " pod="openshift-monitoring/node-exporter-6ps9j"
Jul 28 10:09:16.806128 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:16.806067    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/03488ce4-1c3e-4258-854e-913b25aac684-node-exporter-tls\") pod \"node-exporter-6ps9j\" (UID: \"03488ce4-1c3e-4258-854e-913b25aac684\") " pod="openshift-monitoring/node-exporter-6ps9j"
Jul 28 10:09:16.911866 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:16.911832    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6ps9j"
Jul 28 10:09:16.941838 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:09:16.941803    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03488ce4_1c3e_4258_854e_913b25aac684.slice/crio-19e70a950de7cac85a64245e811cd89f0b17b8d859277679ce4aa269c754d3db WatchSource:0}: Error finding container 19e70a950de7cac85a64245e811cd89f0b17b8d859277679ce4aa269c754d3db: Status 404 returned error can't find the container with id 19e70a950de7cac85a64245e811cd89f0b17b8d859277679ce4aa269c754d3db
Jul 28 10:09:17.025679 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:17.025635    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-27zms" event={"ID":"dde5d137-fa6b-4c1e-ad94-07f174179c6c","Type":"ContainerStarted","Data":"d1620724c1373b316d3b8f42812454c9b18fc878c903d32cff6dd220a2338bc7"}
Jul 28 10:09:17.027306 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:17.027272    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6ps9j" event={"ID":"03488ce4-1c3e-4258-854e-913b25aac684","Type":"ContainerStarted","Data":"19e70a950de7cac85a64245e811cd89f0b17b8d859277679ce4aa269c754d3db"}
Jul 28 10:09:19.039040 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.039001    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-27zms" event={"ID":"dde5d137-fa6b-4c1e-ad94-07f174179c6c","Type":"ContainerStarted","Data":"ae5143beeb35681e99ad3b6e36119a9974ca7d3d1e86233b42acc60a73dfaa64"}
Jul 28 10:09:19.039537 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.039065    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-27zms" event={"ID":"dde5d137-fa6b-4c1e-ad94-07f174179c6c","Type":"ContainerStarted","Data":"681cdda4d8db239617dfe4b0776f7e4caa2b61e512f175e08a28ee357348d7e7"}
Jul 28 10:09:19.039537 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.039084    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-27zms" event={"ID":"dde5d137-fa6b-4c1e-ad94-07f174179c6c","Type":"ContainerStarted","Data":"53cfa0dcb0e3865e99695125fac2adfdc7be524a68852717f2f04932de84548d"}
Jul 28 10:09:19.061580 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.061465    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-27zms" podStartSLOduration=2.151385398 podStartE2EDuration="4.061444417s" totalImagesPullingTime="1.910059019s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:09:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:09:19.059944433 +0000 UTC m=+95.018712849" watchObservedRunningTime="2026-07-28 10:09:19.061444417 +0000 UTC m=+95.020212796"
Jul 28 10:09:19.490363 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.490333    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5b7c89b76c-w2zrh"]
Jul 28 10:09:19.502557 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.502240    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5b7c89b76c-w2zrh"
Jul 28 10:09:19.504695 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.504661    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5b7c89b76c-w2zrh"]
Jul 28 10:09:19.504936 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.504910    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 28 10:09:19.504936 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.504919    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 28 10:09:19.505084 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.504948    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 28 10:09:19.505084 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.504911    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-2tq5j\""
Jul 28 10:09:19.505084 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.504911    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 28 10:09:19.505225 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.505084    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-74cu4fcpotghm\""
Jul 28 10:09:19.527862 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.527838    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8c1a3068-bb8a-4a90-97ca-f29ebebd3d87-client-ca-bundle\") pod \"metrics-server-5b7c89b76c-w2zrh\" (UID: \"8c1a3068-bb8a-4a90-97ca-f29ebebd3d87\") " pod="openshift-monitoring/metrics-server-5b7c89b76c-w2zrh"
Jul 28 10:09:19.527992 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.527870    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8c1a3068-bb8a-4a90-97ca-f29ebebd3d87-secret-metrics-server-tls\") pod \"metrics-server-5b7c89b76c-w2zrh\" (UID: \"8c1a3068-bb8a-4a90-97ca-f29ebebd3d87\") " pod="openshift-monitoring/metrics-server-5b7c89b76c-w2zrh"
Jul 28 10:09:19.527992 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.527898    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8c1a3068-bb8a-4a90-97ca-f29ebebd3d87-metrics-server-audit-profiles\") pod \"metrics-server-5b7c89b76c-w2zrh\" (UID: \"8c1a3068-bb8a-4a90-97ca-f29ebebd3d87\") " pod="openshift-monitoring/metrics-server-5b7c89b76c-w2zrh"
Jul 28 10:09:19.527992 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.527974    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp95d\" (UniqueName: \"kubernetes.io/projected/8c1a3068-bb8a-4a90-97ca-f29ebebd3d87-kube-api-access-xp95d\") pod \"metrics-server-5b7c89b76c-w2zrh\" (UID: \"8c1a3068-bb8a-4a90-97ca-f29ebebd3d87\") " pod="openshift-monitoring/metrics-server-5b7c89b76c-w2zrh"
Jul 28 10:09:19.528098 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.528012    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8c1a3068-bb8a-4a90-97ca-f29ebebd3d87-secret-metrics-server-client-certs\") pod \"metrics-server-5b7c89b76c-w2zrh\" (UID: \"8c1a3068-bb8a-4a90-97ca-f29ebebd3d87\") " pod="openshift-monitoring/metrics-server-5b7c89b76c-w2zrh"
Jul 28 10:09:19.528098 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.528039    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8c1a3068-bb8a-4a90-97ca-f29ebebd3d87-audit-log\") pod \"metrics-server-5b7c89b76c-w2zrh\" (UID: \"8c1a3068-bb8a-4a90-97ca-f29ebebd3d87\") " pod="openshift-monitoring/metrics-server-5b7c89b76c-w2zrh"
Jul 28 10:09:19.528159 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.528093    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c1a3068-bb8a-4a90-97ca-f29ebebd3d87-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5b7c89b76c-w2zrh\" (UID: \"8c1a3068-bb8a-4a90-97ca-f29ebebd3d87\") " pod="openshift-monitoring/metrics-server-5b7c89b76c-w2zrh"
Jul 28 10:09:19.628580 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.628493    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8c1a3068-bb8a-4a90-97ca-f29ebebd3d87-secret-metrics-server-tls\") pod \"metrics-server-5b7c89b76c-w2zrh\" (UID: \"8c1a3068-bb8a-4a90-97ca-f29ebebd3d87\") " pod="openshift-monitoring/metrics-server-5b7c89b76c-w2zrh"
Jul 28 10:09:19.628580 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.628537    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8c1a3068-bb8a-4a90-97ca-f29ebebd3d87-metrics-server-audit-profiles\") pod \"metrics-server-5b7c89b76c-w2zrh\" (UID: \"8c1a3068-bb8a-4a90-97ca-f29ebebd3d87\") " pod="openshift-monitoring/metrics-server-5b7c89b76c-w2zrh"
Jul 28 10:09:19.628801 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.628660    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp95d\" (UniqueName: \"kubernetes.io/projected/8c1a3068-bb8a-4a90-97ca-f29ebebd3d87-kube-api-access-xp95d\") pod \"metrics-server-5b7c89b76c-w2zrh\" (UID: \"8c1a3068-bb8a-4a90-97ca-f29ebebd3d87\") " pod="openshift-monitoring/metrics-server-5b7c89b76c-w2zrh"
Jul 28 10:09:19.628801 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.628710    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8c1a3068-bb8a-4a90-97ca-f29ebebd3d87-secret-metrics-server-client-certs\") pod \"metrics-server-5b7c89b76c-w2zrh\" (UID: \"8c1a3068-bb8a-4a90-97ca-f29ebebd3d87\") " pod="openshift-monitoring/metrics-server-5b7c89b76c-w2zrh"
Jul 28 10:09:19.628801 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.628744    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8c1a3068-bb8a-4a90-97ca-f29ebebd3d87-audit-log\") pod \"metrics-server-5b7c89b76c-w2zrh\" (UID: \"8c1a3068-bb8a-4a90-97ca-f29ebebd3d87\") " pod="openshift-monitoring/metrics-server-5b7c89b76c-w2zrh"
Jul 28 10:09:19.628801 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.628779    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c1a3068-bb8a-4a90-97ca-f29ebebd3d87-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5b7c89b76c-w2zrh\" (UID: \"8c1a3068-bb8a-4a90-97ca-f29ebebd3d87\") " pod="openshift-monitoring/metrics-server-5b7c89b76c-w2zrh"
Jul 28 10:09:19.629006 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.628824    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8c1a3068-bb8a-4a90-97ca-f29ebebd3d87-client-ca-bundle\") pod \"metrics-server-5b7c89b76c-w2zrh\" (UID: \"8c1a3068-bb8a-4a90-97ca-f29ebebd3d87\") " pod="openshift-monitoring/metrics-server-5b7c89b76c-w2zrh"
Jul 28 10:09:19.629220 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.629200    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8c1a3068-bb8a-4a90-97ca-f29ebebd3d87-audit-log\") pod \"metrics-server-5b7c89b76c-w2zrh\" (UID: \"8c1a3068-bb8a-4a90-97ca-f29ebebd3d87\") " pod="openshift-monitoring/metrics-server-5b7c89b76c-w2zrh"
Jul 28 10:09:19.629538 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.629510    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c1a3068-bb8a-4a90-97ca-f29ebebd3d87-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5b7c89b76c-w2zrh\" (UID: \"8c1a3068-bb8a-4a90-97ca-f29ebebd3d87\") " pod="openshift-monitoring/metrics-server-5b7c89b76c-w2zrh"
Jul 28 10:09:19.629653 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.629567    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8c1a3068-bb8a-4a90-97ca-f29ebebd3d87-metrics-server-audit-profiles\") pod \"metrics-server-5b7c89b76c-w2zrh\" (UID: \"8c1a3068-bb8a-4a90-97ca-f29ebebd3d87\") " pod="openshift-monitoring/metrics-server-5b7c89b76c-w2zrh"
Jul 28 10:09:19.631220 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.631190    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8c1a3068-bb8a-4a90-97ca-f29ebebd3d87-secret-metrics-server-client-certs\") pod \"metrics-server-5b7c89b76c-w2zrh\" (UID: \"8c1a3068-bb8a-4a90-97ca-f29ebebd3d87\") " pod="openshift-monitoring/metrics-server-5b7c89b76c-w2zrh"
Jul 28 10:09:19.631220 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.631215    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8c1a3068-bb8a-4a90-97ca-f29ebebd3d87-client-ca-bundle\") pod \"metrics-server-5b7c89b76c-w2zrh\" (UID: \"8c1a3068-bb8a-4a90-97ca-f29ebebd3d87\") " pod="openshift-monitoring/metrics-server-5b7c89b76c-w2zrh"
Jul 28 10:09:19.631348 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.631218    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8c1a3068-bb8a-4a90-97ca-f29ebebd3d87-secret-metrics-server-tls\") pod \"metrics-server-5b7c89b76c-w2zrh\" (UID: \"8c1a3068-bb8a-4a90-97ca-f29ebebd3d87\") " pod="openshift-monitoring/metrics-server-5b7c89b76c-w2zrh"
Jul 28 10:09:19.637286 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.637267    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp95d\" (UniqueName: \"kubernetes.io/projected/8c1a3068-bb8a-4a90-97ca-f29ebebd3d87-kube-api-access-xp95d\") pod \"metrics-server-5b7c89b76c-w2zrh\" (UID: \"8c1a3068-bb8a-4a90-97ca-f29ebebd3d87\") " pod="openshift-monitoring/metrics-server-5b7c89b76c-w2zrh"
Jul 28 10:09:19.773406 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.773367    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d967c8bb9-7w6px"]
Jul 28 10:09:19.794944 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.794899    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d967c8bb9-7w6px"
Jul 28 10:09:19.796049 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.796026    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d967c8bb9-7w6px"]
Jul 28 10:09:19.826048 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.826019    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5b7c89b76c-w2zrh"
Jul 28 10:09:19.831303 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.831277    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgst5\" (UniqueName: \"kubernetes.io/projected/23a46488-c010-4d5e-a9f2-df2576efda75-kube-api-access-kgst5\") pod \"console-5d967c8bb9-7w6px\" (UID: \"23a46488-c010-4d5e-a9f2-df2576efda75\") " pod="openshift-console/console-5d967c8bb9-7w6px"
Jul 28 10:09:19.831452 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.831312    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/23a46488-c010-4d5e-a9f2-df2576efda75-console-config\") pod \"console-5d967c8bb9-7w6px\" (UID: \"23a46488-c010-4d5e-a9f2-df2576efda75\") " pod="openshift-console/console-5d967c8bb9-7w6px"
Jul 28 10:09:19.831452 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.831339    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/23a46488-c010-4d5e-a9f2-df2576efda75-service-ca\") pod \"console-5d967c8bb9-7w6px\" (UID: \"23a46488-c010-4d5e-a9f2-df2576efda75\") " pod="openshift-console/console-5d967c8bb9-7w6px"
Jul 28 10:09:19.831452 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.831366    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/23a46488-c010-4d5e-a9f2-df2576efda75-oauth-serving-cert\") pod \"console-5d967c8bb9-7w6px\" (UID: \"23a46488-c010-4d5e-a9f2-df2576efda75\") " pod="openshift-console/console-5d967c8bb9-7w6px"
Jul 28 10:09:19.831610 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.831465    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/23a46488-c010-4d5e-a9f2-df2576efda75-console-serving-cert\") pod \"console-5d967c8bb9-7w6px\" (UID: \"23a46488-c010-4d5e-a9f2-df2576efda75\") " pod="openshift-console/console-5d967c8bb9-7w6px"
Jul 28 10:09:19.831610 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.831506    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/23a46488-c010-4d5e-a9f2-df2576efda75-console-oauth-config\") pod \"console-5d967c8bb9-7w6px\" (UID: \"23a46488-c010-4d5e-a9f2-df2576efda75\") " pod="openshift-console/console-5d967c8bb9-7w6px"
Jul 28 10:09:19.831675 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.831621    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23a46488-c010-4d5e-a9f2-df2576efda75-trusted-ca-bundle\") pod \"console-5d967c8bb9-7w6px\" (UID: \"23a46488-c010-4d5e-a9f2-df2576efda75\") " pod="openshift-console/console-5d967c8bb9-7w6px"
Jul 28 10:09:19.932790 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.932757    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23a46488-c010-4d5e-a9f2-df2576efda75-trusted-ca-bundle\") pod \"console-5d967c8bb9-7w6px\" (UID: \"23a46488-c010-4d5e-a9f2-df2576efda75\") " pod="openshift-console/console-5d967c8bb9-7w6px"
Jul 28 10:09:19.933021 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.932828    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgst5\" (UniqueName: \"kubernetes.io/projected/23a46488-c010-4d5e-a9f2-df2576efda75-kube-api-access-kgst5\") pod \"console-5d967c8bb9-7w6px\" (UID: \"23a46488-c010-4d5e-a9f2-df2576efda75\") " pod="openshift-console/console-5d967c8bb9-7w6px"
Jul 28 10:09:19.933021 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.932859    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/23a46488-c010-4d5e-a9f2-df2576efda75-console-config\") pod \"console-5d967c8bb9-7w6px\" (UID: \"23a46488-c010-4d5e-a9f2-df2576efda75\") " pod="openshift-console/console-5d967c8bb9-7w6px"
Jul 28 10:09:19.933021 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.932884    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/23a46488-c010-4d5e-a9f2-df2576efda75-service-ca\") pod \"console-5d967c8bb9-7w6px\" (UID: \"23a46488-c010-4d5e-a9f2-df2576efda75\") " pod="openshift-console/console-5d967c8bb9-7w6px"
Jul 28 10:09:19.933021 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.932911    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/23a46488-c010-4d5e-a9f2-df2576efda75-oauth-serving-cert\") pod \"console-5d967c8bb9-7w6px\" (UID: \"23a46488-c010-4d5e-a9f2-df2576efda75\") " pod="openshift-console/console-5d967c8bb9-7w6px"
Jul 28 10:09:19.934033 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.933482    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/23a46488-c010-4d5e-a9f2-df2576efda75-console-serving-cert\") pod \"console-5d967c8bb9-7w6px\" (UID: \"23a46488-c010-4d5e-a9f2-df2576efda75\") " pod="openshift-console/console-5d967c8bb9-7w6px"
Jul 28 10:09:19.934033 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.933542    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/23a46488-c010-4d5e-a9f2-df2576efda75-console-oauth-config\") pod \"console-5d967c8bb9-7w6px\" (UID: \"23a46488-c010-4d5e-a9f2-df2576efda75\") " pod="openshift-console/console-5d967c8bb9-7w6px"
Jul 28 10:09:19.934210 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.934103    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23a46488-c010-4d5e-a9f2-df2576efda75-trusted-ca-bundle\") pod \"console-5d967c8bb9-7w6px\" (UID: \"23a46488-c010-4d5e-a9f2-df2576efda75\") " pod="openshift-console/console-5d967c8bb9-7w6px"
Jul 28 10:09:19.935171 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.934989    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/23a46488-c010-4d5e-a9f2-df2576efda75-oauth-serving-cert\") pod \"console-5d967c8bb9-7w6px\" (UID: \"23a46488-c010-4d5e-a9f2-df2576efda75\") " pod="openshift-console/console-5d967c8bb9-7w6px"
Jul 28 10:09:19.936345 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.936159    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/23a46488-c010-4d5e-a9f2-df2576efda75-console-config\") pod \"console-5d967c8bb9-7w6px\" (UID: \"23a46488-c010-4d5e-a9f2-df2576efda75\") " pod="openshift-console/console-5d967c8bb9-7w6px"
Jul 28 10:09:19.939234 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.937346    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/23a46488-c010-4d5e-a9f2-df2576efda75-console-oauth-config\") pod \"console-5d967c8bb9-7w6px\" (UID: \"23a46488-c010-4d5e-a9f2-df2576efda75\") " pod="openshift-console/console-5d967c8bb9-7w6px"
Jul 28 10:09:19.942133 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.942080    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/23a46488-c010-4d5e-a9f2-df2576efda75-service-ca\") pod \"console-5d967c8bb9-7w6px\" (UID: \"23a46488-c010-4d5e-a9f2-df2576efda75\") " pod="openshift-console/console-5d967c8bb9-7w6px"
Jul 28 10:09:19.943926 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.943904    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/23a46488-c010-4d5e-a9f2-df2576efda75-console-serving-cert\") pod \"console-5d967c8bb9-7w6px\" (UID: \"23a46488-c010-4d5e-a9f2-df2576efda75\") " pod="openshift-console/console-5d967c8bb9-7w6px"
Jul 28 10:09:19.944691 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.944668    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgst5\" (UniqueName: \"kubernetes.io/projected/23a46488-c010-4d5e-a9f2-df2576efda75-kube-api-access-kgst5\") pod \"console-5d967c8bb9-7w6px\" (UID: \"23a46488-c010-4d5e-a9f2-df2576efda75\") " pod="openshift-console/console-5d967c8bb9-7w6px"
Jul 28 10:09:19.979368 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:19.979339    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5b7c89b76c-w2zrh"]
Jul 28 10:09:19.981653 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:09:19.981623    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c1a3068_bb8a_4a90_97ca_f29ebebd3d87.slice/crio-8b1d38cd928705a95a9deba234142d0a8a877664be5da4ad9cb68c595de94413 WatchSource:0}: Error finding container 8b1d38cd928705a95a9deba234142d0a8a877664be5da4ad9cb68c595de94413: Status 404 returned error can't find the container with id 8b1d38cd928705a95a9deba234142d0a8a877664be5da4ad9cb68c595de94413
Jul 28 10:09:20.046227 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:20.046190    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5b7c89b76c-w2zrh" event={"ID":"8c1a3068-bb8a-4a90-97ca-f29ebebd3d87","Type":"ContainerStarted","Data":"8b1d38cd928705a95a9deba234142d0a8a877664be5da4ad9cb68c595de94413"}
Jul 28 10:09:20.047818 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:20.047791    2557 generic.go:356] "Generic (PLEG): container finished" podID="03488ce4-1c3e-4258-854e-913b25aac684" containerID="90720f8af35915b02ffe464000d1a524204f55ed16494a9ac4fef645ac8a6746" exitCode=0
Jul 28 10:09:20.047950 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:20.047874    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6ps9j" event={"ID":"03488ce4-1c3e-4258-854e-913b25aac684","Type":"ContainerDied","Data":"90720f8af35915b02ffe464000d1a524204f55ed16494a9ac4fef645ac8a6746"}
Jul 28 10:09:20.106761 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:20.106732    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d967c8bb9-7w6px"
Jul 28 10:09:20.273080 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:20.272994    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d967c8bb9-7w6px"]
Jul 28 10:09:20.275835 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:09:20.275776    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23a46488_c010_4d5e_a9f2_df2576efda75.slice/crio-2b25770bdcc70502db22b3b1db560c1ccaad6b39fd15b1e0d4fffc69db8f8d9e WatchSource:0}: Error finding container 2b25770bdcc70502db22b3b1db560c1ccaad6b39fd15b1e0d4fffc69db8f8d9e: Status 404 returned error can't find the container with id 2b25770bdcc70502db22b3b1db560c1ccaad6b39fd15b1e0d4fffc69db8f8d9e
Jul 28 10:09:21.054965 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.054926    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6ps9j" event={"ID":"03488ce4-1c3e-4258-854e-913b25aac684","Type":"ContainerStarted","Data":"18ca5a6ef8a0eeb4dd5da3d0571937ce27b54f85361c73e17929ac0eab728c8a"}
Jul 28 10:09:21.055531 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.054990    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6ps9j" event={"ID":"03488ce4-1c3e-4258-854e-913b25aac684","Type":"ContainerStarted","Data":"ad53cfc4666774900dd1643aeff08f2e792398375db788e9849876b8f0de5e4d"}
Jul 28 10:09:21.056680 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.056646    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d967c8bb9-7w6px" event={"ID":"23a46488-c010-4d5e-a9f2-df2576efda75","Type":"ContainerStarted","Data":"52d5778dc58c1fe47d151fe7d45c0ff5dc83ddb44875f19efd19bcaf22de3549"}
Jul 28 10:09:21.056798 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.056695    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d967c8bb9-7w6px" event={"ID":"23a46488-c010-4d5e-a9f2-df2576efda75","Type":"ContainerStarted","Data":"2b25770bdcc70502db22b3b1db560c1ccaad6b39fd15b1e0d4fffc69db8f8d9e"}
Jul 28 10:09:21.084500 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.084419    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-6ps9j" podStartSLOduration=3.675209334 podStartE2EDuration="6.084399802s" totalImagesPullingTime="2.306068645s" totalInitContainerRuntime="103.121823ms" isStatefulPod=true podCreationTimestamp="2026-07-28 10:09:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:09:21.082843144 +0000 UTC m=+97.041611525" watchObservedRunningTime="2026-07-28 10:09:21.084399802 +0000 UTC m=+97.043168194"
Jul 28 10:09:21.116990 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.115807    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5d967c8bb9-7w6px" podStartSLOduration=2.115785744 podStartE2EDuration="2.115785744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:09:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:09:21.113885683 +0000 UTC m=+97.072654062" watchObservedRunningTime="2026-07-28 10:09:21.115785744 +0000 UTC m=+97.074554124"
Jul 28 10:09:21.329541 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.329446    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 10:09:21.359146 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.359104    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 10:09:21.359332 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.359265    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.363966 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.363939    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 28 10:09:21.364339 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.364314    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 28 10:09:21.364512 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.364496    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 28 10:09:21.364784 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.364761    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-5987b\""
Jul 28 10:09:21.364942 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.364927    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 28 10:09:21.365167 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.364072    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 28 10:09:21.365442 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.365422    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 28 10:09:21.365563 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.364545    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 28 10:09:21.365797 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.365011    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 28 10:09:21.366084 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.366061    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 28 10:09:21.366287 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.366270    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 28 10:09:21.368708 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.367638    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4a2kfkr7hrit\""
Jul 28 10:09:21.368708 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.364127    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 28 10:09:21.368708 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.367885    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 28 10:09:21.368708 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.368113    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 28 10:09:21.387225 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.386649    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 28 10:09:21.448381 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.448347    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bd83a23b-232a-4ed4-a5a8-562ff2704131-config\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.448381 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.448387    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bd83a23b-232a-4ed4-a5a8-562ff2704131-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.448584 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.448420    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bd83a23b-232a-4ed4-a5a8-562ff2704131-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.448584 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.448488    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bd83a23b-232a-4ed4-a5a8-562ff2704131-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.448584 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.448542    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bd83a23b-232a-4ed4-a5a8-562ff2704131-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.448584 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.448578    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd83a23b-232a-4ed4-a5a8-562ff2704131-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.448729 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.448661    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bd83a23b-232a-4ed4-a5a8-562ff2704131-config-out\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.448729 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.448693    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bd83a23b-232a-4ed4-a5a8-562ff2704131-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.448729 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.448718    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bd83a23b-232a-4ed4-a5a8-562ff2704131-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.448815 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.448733    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bd83a23b-232a-4ed4-a5a8-562ff2704131-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.448815 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.448756    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bd83a23b-232a-4ed4-a5a8-562ff2704131-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.448815 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.448784    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bd83a23b-232a-4ed4-a5a8-562ff2704131-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.448815 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.448806    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd83a23b-232a-4ed4-a5a8-562ff2704131-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.448932 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.448846    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bd83a23b-232a-4ed4-a5a8-562ff2704131-web-config\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.448932 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.448865    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bd83a23b-232a-4ed4-a5a8-562ff2704131-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.448932 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.448881    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd83a23b-232a-4ed4-a5a8-562ff2704131-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.448932 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.448910    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bd83a23b-232a-4ed4-a5a8-562ff2704131-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.449064 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.448931    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bd83a23b-232a-4ed4-a5a8-562ff2704131-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.449064 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.448948    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8bxt\" (UniqueName: \"kubernetes.io/projected/bd83a23b-232a-4ed4-a5a8-562ff2704131-kube-api-access-q8bxt\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.449064 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.448969    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd83a23b-232a-4ed4-a5a8-562ff2704131-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.550464 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.550420    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bd83a23b-232a-4ed4-a5a8-562ff2704131-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.550464 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.550458    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bd83a23b-232a-4ed4-a5a8-562ff2704131-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.550464 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.550473    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bd83a23b-232a-4ed4-a5a8-562ff2704131-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.550789 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.550491    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd83a23b-232a-4ed4-a5a8-562ff2704131-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.550789 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.550524    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bd83a23b-232a-4ed4-a5a8-562ff2704131-config-out\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.550789 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.550720    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bd83a23b-232a-4ed4-a5a8-562ff2704131-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.550949 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.550802    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bd83a23b-232a-4ed4-a5a8-562ff2704131-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.550949 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.550831    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bd83a23b-232a-4ed4-a5a8-562ff2704131-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.550949 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.550857    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bd83a23b-232a-4ed4-a5a8-562ff2704131-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.550949 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.550888    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bd83a23b-232a-4ed4-a5a8-562ff2704131-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.550949 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.550926    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd83a23b-232a-4ed4-a5a8-562ff2704131-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.551203 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.550956    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bd83a23b-232a-4ed4-a5a8-562ff2704131-web-config\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.551203 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.550966    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bd83a23b-232a-4ed4-a5a8-562ff2704131-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.551203 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.550985    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bd83a23b-232a-4ed4-a5a8-562ff2704131-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.551203 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.551012    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd83a23b-232a-4ed4-a5a8-562ff2704131-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.551203 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.551044    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bd83a23b-232a-4ed4-a5a8-562ff2704131-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.551203 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.551072    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bd83a23b-232a-4ed4-a5a8-562ff2704131-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.551203 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.551098    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8bxt\" (UniqueName: \"kubernetes.io/projected/bd83a23b-232a-4ed4-a5a8-562ff2704131-kube-api-access-q8bxt\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.551203 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.551138    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd83a23b-232a-4ed4-a5a8-562ff2704131-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.551203 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.551160    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bd83a23b-232a-4ed4-a5a8-562ff2704131-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.551203 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.551198    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bd83a23b-232a-4ed4-a5a8-562ff2704131-config\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.551696 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.551232    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bd83a23b-232a-4ed4-a5a8-562ff2704131-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.551696 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.551525    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bd83a23b-232a-4ed4-a5a8-562ff2704131-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.552200 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.552002    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd83a23b-232a-4ed4-a5a8-562ff2704131-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.553206 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.552824    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd83a23b-232a-4ed4-a5a8-562ff2704131-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.554729 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.554670    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd83a23b-232a-4ed4-a5a8-562ff2704131-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.558288 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.555101    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bd83a23b-232a-4ed4-a5a8-562ff2704131-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.558288 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.555130    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd83a23b-232a-4ed4-a5a8-562ff2704131-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.558288 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.555551    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bd83a23b-232a-4ed4-a5a8-562ff2704131-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.558288 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.556072    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bd83a23b-232a-4ed4-a5a8-562ff2704131-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.558288 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.556980    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/bd83a23b-232a-4ed4-a5a8-562ff2704131-config\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.558288 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.557586    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bd83a23b-232a-4ed4-a5a8-562ff2704131-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.558288 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.558062    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bd83a23b-232a-4ed4-a5a8-562ff2704131-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.558288 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.558147    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bd83a23b-232a-4ed4-a5a8-562ff2704131-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.558288 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.558250    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bd83a23b-232a-4ed4-a5a8-562ff2704131-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.559086 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.559040    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bd83a23b-232a-4ed4-a5a8-562ff2704131-config-out\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.559181 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.559082    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bd83a23b-232a-4ed4-a5a8-562ff2704131-web-config\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.559595 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.559571    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bd83a23b-232a-4ed4-a5a8-562ff2704131-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.560580 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.560560    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bd83a23b-232a-4ed4-a5a8-562ff2704131-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.564755 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.564713    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8bxt\" (UniqueName: \"kubernetes.io/projected/bd83a23b-232a-4ed4-a5a8-562ff2704131-kube-api-access-q8bxt\") pod \"prometheus-k8s-0\" (UID: \"bd83a23b-232a-4ed4-a5a8-562ff2704131\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.690208 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.690120    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:21.868692 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:21.868668    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 10:09:21.927973 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:09:21.927929    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd83a23b_232a_4ed4_a5a8_562ff2704131.slice/crio-d8dce7ba4ee60cd71a7e4d512005761077ac68f204834814f4299d36494008d1 WatchSource:0}: Error finding container d8dce7ba4ee60cd71a7e4d512005761077ac68f204834814f4299d36494008d1: Status 404 returned error can't find the container with id d8dce7ba4ee60cd71a7e4d512005761077ac68f204834814f4299d36494008d1
Jul 28 10:09:22.062881 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:22.062833    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bd83a23b-232a-4ed4-a5a8-562ff2704131","Type":"ContainerStarted","Data":"d8dce7ba4ee60cd71a7e4d512005761077ac68f204834814f4299d36494008d1"}
Jul 28 10:09:23.068380 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:23.068347    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5b7c89b76c-w2zrh" event={"ID":"8c1a3068-bb8a-4a90-97ca-f29ebebd3d87","Type":"ContainerStarted","Data":"156b2dc86aef383557eefd85aac10979edfb6c5c14c7ce83aaeb2b0c2036b773"}
Jul 28 10:09:23.088437 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:23.088386    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5b7c89b76c-w2zrh" podStartSLOduration=2.109655804 podStartE2EDuration="4.088367079s" totalImagesPullingTime="1.978711275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:09:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:09:23.08667467 +0000 UTC m=+99.045443051" watchObservedRunningTime="2026-07-28 10:09:23.088367079 +0000 UTC m=+99.047135461"
Jul 28 10:09:24.074152 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:24.074116    2557 generic.go:356] "Generic (PLEG): container finished" podID="bd83a23b-232a-4ed4-a5a8-562ff2704131" containerID="01ba526e472e8febde882bfbcbcd333b4baecf5f1c59a8ccf49cdf3b37c7831b" exitCode=0
Jul 28 10:09:24.074498 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:24.074199    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bd83a23b-232a-4ed4-a5a8-562ff2704131","Type":"ContainerDied","Data":"01ba526e472e8febde882bfbcbcd333b4baecf5f1c59a8ccf49cdf3b37c7831b"}
Jul 28 10:09:28.108586 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:28.108546    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bd83a23b-232a-4ed4-a5a8-562ff2704131","Type":"ContainerStarted","Data":"6e86f7b6bc7c266043189415aefb154bd15a3dd5a453f13124f8c1a9d18abea2"}
Jul 28 10:09:28.109063 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:28.108697    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bd83a23b-232a-4ed4-a5a8-562ff2704131","Type":"ContainerStarted","Data":"19287b176e53597fc14b20c955ea6dc9152150ef8480dd2906887003bf429d27"}
Jul 28 10:09:30.109237 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:30.108233    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d967c8bb9-7w6px"
Jul 28 10:09:30.109831 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:30.109396    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d967c8bb9-7w6px"
Jul 28 10:09:30.114493 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:30.114453    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d967c8bb9-7w6px"
Jul 28 10:09:30.124441 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:30.124413    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d967c8bb9-7w6px"
Jul 28 10:09:30.192151 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:30.192114    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6dfd5d7545-ckkmc"]
Jul 28 10:09:31.126828 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:31.126791    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bd83a23b-232a-4ed4-a5a8-562ff2704131","Type":"ContainerStarted","Data":"b6c5055d360b395eac03903f354e463cd9c11c83d61fd216fe53d7f64647008c"}
Jul 28 10:09:31.127247 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:31.126863    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bd83a23b-232a-4ed4-a5a8-562ff2704131","Type":"ContainerStarted","Data":"ac12c449b648fc60322c944958111552aebe64c44ea1d66b7f69226f19611902"}
Jul 28 10:09:31.127247 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:31.126889    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bd83a23b-232a-4ed4-a5a8-562ff2704131","Type":"ContainerStarted","Data":"ee8b62f5d0753b11422b54494206dc6ac8a4fdaa6d1c4b4f56c1f41d95ef1a7f"}
Jul 28 10:09:31.127247 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:31.126914    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bd83a23b-232a-4ed4-a5a8-562ff2704131","Type":"ContainerStarted","Data":"71426262219e81b598803544e1ce9e1e9b63d7458a9136826da33a93b3ad708f"}
Jul 28 10:09:31.172710 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:31.172618    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.16465955 podStartE2EDuration="10.172584909s" totalImagesPullingTime="7.994527633s" totalInitContainerRuntime="13.397726ms" isStatefulPod=false podCreationTimestamp="2026-07-28 10:09:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:09:31.171004731 +0000 UTC m=+107.129773123" watchObservedRunningTime="2026-07-28 10:09:31.172584909 +0000 UTC m=+107.131353288"
Jul 28 10:09:31.690543 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:31.690515    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:33.057017 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:33.056953    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5796468f64-jtc5k" podUID="ebd56b7c-8767-4f6b-9189-1019cdef269b" containerName="console" containerID="cri-o://5d38b6c991aa8b5f9fd16b4103e148fdf03bc0efec1bf98f3db4807daf6269ed" gracePeriod=15
Jul 28 10:09:33.350833 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:33.350808    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5796468f64-jtc5k"
Jul 28 10:09:33.468877 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:33.468844    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ebd56b7c-8767-4f6b-9189-1019cdef269b-console-oauth-config\") pod \"ebd56b7c-8767-4f6b-9189-1019cdef269b\" (UID: \"ebd56b7c-8767-4f6b-9189-1019cdef269b\") "
Jul 28 10:09:33.469042 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:33.468899    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ebd56b7c-8767-4f6b-9189-1019cdef269b-oauth-serving-cert\") pod \"ebd56b7c-8767-4f6b-9189-1019cdef269b\" (UID: \"ebd56b7c-8767-4f6b-9189-1019cdef269b\") "
Jul 28 10:09:33.469042 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:33.468926    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45c94\" (UniqueName: \"kubernetes.io/projected/ebd56b7c-8767-4f6b-9189-1019cdef269b-kube-api-access-45c94\") pod \"ebd56b7c-8767-4f6b-9189-1019cdef269b\" (UID: \"ebd56b7c-8767-4f6b-9189-1019cdef269b\") "
Jul 28 10:09:33.469042 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:33.468964    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ebd56b7c-8767-4f6b-9189-1019cdef269b-service-ca\") pod \"ebd56b7c-8767-4f6b-9189-1019cdef269b\" (UID: \"ebd56b7c-8767-4f6b-9189-1019cdef269b\") "
Jul 28 10:09:33.469042 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:33.468996    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ebd56b7c-8767-4f6b-9189-1019cdef269b-console-serving-cert\") pod \"ebd56b7c-8767-4f6b-9189-1019cdef269b\" (UID: \"ebd56b7c-8767-4f6b-9189-1019cdef269b\") "
Jul 28 10:09:33.469042 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:33.469021    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ebd56b7c-8767-4f6b-9189-1019cdef269b-console-config\") pod \"ebd56b7c-8767-4f6b-9189-1019cdef269b\" (UID: \"ebd56b7c-8767-4f6b-9189-1019cdef269b\") "
Jul 28 10:09:33.469382 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:33.469349    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebd56b7c-8767-4f6b-9189-1019cdef269b-service-ca" (OuterVolumeSpecName: "service-ca") pod "ebd56b7c-8767-4f6b-9189-1019cdef269b" (UID: "ebd56b7c-8767-4f6b-9189-1019cdef269b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:09:33.469505 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:33.469386    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebd56b7c-8767-4f6b-9189-1019cdef269b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ebd56b7c-8767-4f6b-9189-1019cdef269b" (UID: "ebd56b7c-8767-4f6b-9189-1019cdef269b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:09:33.469505 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:33.469465    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebd56b7c-8767-4f6b-9189-1019cdef269b-console-config" (OuterVolumeSpecName: "console-config") pod "ebd56b7c-8767-4f6b-9189-1019cdef269b" (UID: "ebd56b7c-8767-4f6b-9189-1019cdef269b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:09:33.469505 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:33.469491    2557 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ebd56b7c-8767-4f6b-9189-1019cdef269b-oauth-serving-cert\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:09:33.469652 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:33.469509    2557 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ebd56b7c-8767-4f6b-9189-1019cdef269b-service-ca\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:09:33.471355 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:33.471330    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebd56b7c-8767-4f6b-9189-1019cdef269b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ebd56b7c-8767-4f6b-9189-1019cdef269b" (UID: "ebd56b7c-8767-4f6b-9189-1019cdef269b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:09:33.471455 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:33.471363    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebd56b7c-8767-4f6b-9189-1019cdef269b-kube-api-access-45c94" (OuterVolumeSpecName: "kube-api-access-45c94") pod "ebd56b7c-8767-4f6b-9189-1019cdef269b" (UID: "ebd56b7c-8767-4f6b-9189-1019cdef269b"). InnerVolumeSpecName "kube-api-access-45c94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:09:33.471455 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:33.471392    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebd56b7c-8767-4f6b-9189-1019cdef269b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ebd56b7c-8767-4f6b-9189-1019cdef269b" (UID: "ebd56b7c-8767-4f6b-9189-1019cdef269b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:09:33.570918 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:33.570834    2557 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ebd56b7c-8767-4f6b-9189-1019cdef269b-console-oauth-config\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:09:33.570918 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:33.570863    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45c94\" (UniqueName: \"kubernetes.io/projected/ebd56b7c-8767-4f6b-9189-1019cdef269b-kube-api-access-45c94\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:09:33.570918 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:33.570875    2557 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ebd56b7c-8767-4f6b-9189-1019cdef269b-console-serving-cert\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:09:33.570918 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:33.570885    2557 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ebd56b7c-8767-4f6b-9189-1019cdef269b-console-config\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:09:34.145017 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:34.144979    2557 generic.go:356] "Generic (PLEG): container finished" podID="ebd56b7c-8767-4f6b-9189-1019cdef269b" containerID="5d38b6c991aa8b5f9fd16b4103e148fdf03bc0efec1bf98f3db4807daf6269ed" exitCode=0
Jul 28 10:09:34.145406 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:34.145037    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5796468f64-jtc5k"
Jul 28 10:09:34.145406 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:34.145054    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5796468f64-jtc5k" event={"ID":"ebd56b7c-8767-4f6b-9189-1019cdef269b","Type":"ContainerDied","Data":"5d38b6c991aa8b5f9fd16b4103e148fdf03bc0efec1bf98f3db4807daf6269ed"}
Jul 28 10:09:34.145406 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:34.145132    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5796468f64-jtc5k" event={"ID":"ebd56b7c-8767-4f6b-9189-1019cdef269b","Type":"ContainerDied","Data":"4ce7a1d5160912af821251dd14b9550b34539374336201e9e7ce158730353451"}
Jul 28 10:09:34.145406 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:34.145154    2557 scope.go:117] "RemoveContainer" containerID="5d38b6c991aa8b5f9fd16b4103e148fdf03bc0efec1bf98f3db4807daf6269ed"
Jul 28 10:09:34.153961 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:34.153941    2557 scope.go:117] "RemoveContainer" containerID="5d38b6c991aa8b5f9fd16b4103e148fdf03bc0efec1bf98f3db4807daf6269ed"
Jul 28 10:09:34.154234 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:09:34.154214    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d38b6c991aa8b5f9fd16b4103e148fdf03bc0efec1bf98f3db4807daf6269ed\": container with ID starting with 5d38b6c991aa8b5f9fd16b4103e148fdf03bc0efec1bf98f3db4807daf6269ed not found: ID does not exist" containerID="5d38b6c991aa8b5f9fd16b4103e148fdf03bc0efec1bf98f3db4807daf6269ed"
Jul 28 10:09:34.154278 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:34.154243    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d38b6c991aa8b5f9fd16b4103e148fdf03bc0efec1bf98f3db4807daf6269ed"} err="failed to get container status \"5d38b6c991aa8b5f9fd16b4103e148fdf03bc0efec1bf98f3db4807daf6269ed\": rpc error: code = NotFound desc = could not find container \"5d38b6c991aa8b5f9fd16b4103e148fdf03bc0efec1bf98f3db4807daf6269ed\": container with ID starting with 5d38b6c991aa8b5f9fd16b4103e148fdf03bc0efec1bf98f3db4807daf6269ed not found: ID does not exist"
Jul 28 10:09:34.169450 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:34.169427    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5796468f64-jtc5k"]
Jul 28 10:09:34.172997 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:34.172975    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5796468f64-jtc5k"]
Jul 28 10:09:34.578905 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:34.578872    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebd56b7c-8767-4f6b-9189-1019cdef269b" path="/var/lib/kubelet/pods/ebd56b7c-8767-4f6b-9189-1019cdef269b/volumes"
Jul 28 10:09:39.826778 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:39.826740    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5b7c89b76c-w2zrh"
Jul 28 10:09:39.827287 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:39.826830    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5b7c89b76c-w2zrh"
Jul 28 10:09:43.006458 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:43.006426    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ddrhj"
Jul 28 10:09:55.217312 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:55.217245    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6dfd5d7545-ckkmc" podUID="1b3c9632-d219-4f7a-8094-36428c83afa0" containerName="console" containerID="cri-o://13a613f1d4ea000219d388d66799c6724535dd434ddedc99df97d6e3ff529b72" gracePeriod=15
Jul 28 10:09:55.544012 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:55.543984    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dfd5d7545-ckkmc"
Jul 28 10:09:55.577051 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:55.575323    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b3c9632-d219-4f7a-8094-36428c83afa0-console-serving-cert\") pod \"1b3c9632-d219-4f7a-8094-36428c83afa0\" (UID: \"1b3c9632-d219-4f7a-8094-36428c83afa0\") "
Jul 28 10:09:55.577051 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:55.575396    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b3c9632-d219-4f7a-8094-36428c83afa0-console-config\") pod \"1b3c9632-d219-4f7a-8094-36428c83afa0\" (UID: \"1b3c9632-d219-4f7a-8094-36428c83afa0\") "
Jul 28 10:09:55.577051 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:55.575434    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b3c9632-d219-4f7a-8094-36428c83afa0-trusted-ca-bundle\") pod \"1b3c9632-d219-4f7a-8094-36428c83afa0\" (UID: \"1b3c9632-d219-4f7a-8094-36428c83afa0\") "
Jul 28 10:09:55.577051 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:55.575457    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b3c9632-d219-4f7a-8094-36428c83afa0-service-ca\") pod \"1b3c9632-d219-4f7a-8094-36428c83afa0\" (UID: \"1b3c9632-d219-4f7a-8094-36428c83afa0\") "
Jul 28 10:09:55.577051 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:55.575507    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4bjjv\" (UniqueName: \"kubernetes.io/projected/1b3c9632-d219-4f7a-8094-36428c83afa0-kube-api-access-4bjjv\") pod \"1b3c9632-d219-4f7a-8094-36428c83afa0\" (UID: \"1b3c9632-d219-4f7a-8094-36428c83afa0\") "
Jul 28 10:09:55.577051 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:55.575539    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b3c9632-d219-4f7a-8094-36428c83afa0-oauth-serving-cert\") pod \"1b3c9632-d219-4f7a-8094-36428c83afa0\" (UID: \"1b3c9632-d219-4f7a-8094-36428c83afa0\") "
Jul 28 10:09:55.577051 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:55.575580    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b3c9632-d219-4f7a-8094-36428c83afa0-console-oauth-config\") pod \"1b3c9632-d219-4f7a-8094-36428c83afa0\" (UID: \"1b3c9632-d219-4f7a-8094-36428c83afa0\") "
Jul 28 10:09:55.577051 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:55.576501    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b3c9632-d219-4f7a-8094-36428c83afa0-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "1b3c9632-d219-4f7a-8094-36428c83afa0" (UID: "1b3c9632-d219-4f7a-8094-36428c83afa0"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:09:55.577051 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:55.576782    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b3c9632-d219-4f7a-8094-36428c83afa0-console-config" (OuterVolumeSpecName: "console-config") pod "1b3c9632-d219-4f7a-8094-36428c83afa0" (UID: "1b3c9632-d219-4f7a-8094-36428c83afa0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:09:55.577051 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:55.576906    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b3c9632-d219-4f7a-8094-36428c83afa0-service-ca" (OuterVolumeSpecName: "service-ca") pod "1b3c9632-d219-4f7a-8094-36428c83afa0" (UID: "1b3c9632-d219-4f7a-8094-36428c83afa0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:09:55.577051 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:55.577014    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b3c9632-d219-4f7a-8094-36428c83afa0-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1b3c9632-d219-4f7a-8094-36428c83afa0" (UID: "1b3c9632-d219-4f7a-8094-36428c83afa0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:09:55.583016 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:55.582981    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b3c9632-d219-4f7a-8094-36428c83afa0-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1b3c9632-d219-4f7a-8094-36428c83afa0" (UID: "1b3c9632-d219-4f7a-8094-36428c83afa0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:09:55.583339 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:55.583314    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b3c9632-d219-4f7a-8094-36428c83afa0-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1b3c9632-d219-4f7a-8094-36428c83afa0" (UID: "1b3c9632-d219-4f7a-8094-36428c83afa0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:09:55.583432 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:55.583367    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b3c9632-d219-4f7a-8094-36428c83afa0-kube-api-access-4bjjv" (OuterVolumeSpecName: "kube-api-access-4bjjv") pod "1b3c9632-d219-4f7a-8094-36428c83afa0" (UID: "1b3c9632-d219-4f7a-8094-36428c83afa0"). InnerVolumeSpecName "kube-api-access-4bjjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:09:55.676757 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:55.676712    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bjjv\" (UniqueName: \"kubernetes.io/projected/1b3c9632-d219-4f7a-8094-36428c83afa0-kube-api-access-4bjjv\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:09:55.676757 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:55.676758    2557 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b3c9632-d219-4f7a-8094-36428c83afa0-oauth-serving-cert\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:09:55.676992 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:55.676775    2557 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b3c9632-d219-4f7a-8094-36428c83afa0-console-oauth-config\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:09:55.676992 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:55.676790    2557 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b3c9632-d219-4f7a-8094-36428c83afa0-console-serving-cert\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:09:55.676992 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:55.676806    2557 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b3c9632-d219-4f7a-8094-36428c83afa0-console-config\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:09:55.676992 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:55.676823    2557 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b3c9632-d219-4f7a-8094-36428c83afa0-trusted-ca-bundle\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:09:55.676992 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:55.676837    2557 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b3c9632-d219-4f7a-8094-36428c83afa0-service-ca\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:09:56.236539 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:56.236494    2557 generic.go:356] "Generic (PLEG): container finished" podID="1b3c9632-d219-4f7a-8094-36428c83afa0" containerID="13a613f1d4ea000219d388d66799c6724535dd434ddedc99df97d6e3ff529b72" exitCode=0
Jul 28 10:09:56.237045 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:56.236622    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dfd5d7545-ckkmc" event={"ID":"1b3c9632-d219-4f7a-8094-36428c83afa0","Type":"ContainerDied","Data":"13a613f1d4ea000219d388d66799c6724535dd434ddedc99df97d6e3ff529b72"}
Jul 28 10:09:56.237045 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:56.236679    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dfd5d7545-ckkmc" event={"ID":"1b3c9632-d219-4f7a-8094-36428c83afa0","Type":"ContainerDied","Data":"0bb8e2c38aadd518b44dadb9723bd8414b88e8fe4eb155ef1325efad28e3ff23"}
Jul 28 10:09:56.237045 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:56.236711    2557 scope.go:117] "RemoveContainer" containerID="13a613f1d4ea000219d388d66799c6724535dd434ddedc99df97d6e3ff529b72"
Jul 28 10:09:56.237045 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:56.236887    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dfd5d7545-ckkmc"
Jul 28 10:09:56.252225 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:56.252201    2557 scope.go:117] "RemoveContainer" containerID="13a613f1d4ea000219d388d66799c6724535dd434ddedc99df97d6e3ff529b72"
Jul 28 10:09:56.252511 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:09:56.252490    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13a613f1d4ea000219d388d66799c6724535dd434ddedc99df97d6e3ff529b72\": container with ID starting with 13a613f1d4ea000219d388d66799c6724535dd434ddedc99df97d6e3ff529b72 not found: ID does not exist" containerID="13a613f1d4ea000219d388d66799c6724535dd434ddedc99df97d6e3ff529b72"
Jul 28 10:09:56.252583 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:56.252519    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13a613f1d4ea000219d388d66799c6724535dd434ddedc99df97d6e3ff529b72"} err="failed to get container status \"13a613f1d4ea000219d388d66799c6724535dd434ddedc99df97d6e3ff529b72\": rpc error: code = NotFound desc = could not find container \"13a613f1d4ea000219d388d66799c6724535dd434ddedc99df97d6e3ff529b72\": container with ID starting with 13a613f1d4ea000219d388d66799c6724535dd434ddedc99df97d6e3ff529b72 not found: ID does not exist"
Jul 28 10:09:56.263539 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:56.263507    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6dfd5d7545-ckkmc"]
Jul 28 10:09:56.265739 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:56.265711    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6dfd5d7545-ckkmc"]
Jul 28 10:09:56.579459 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:56.579423    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b3c9632-d219-4f7a-8094-36428c83afa0" path="/var/lib/kubelet/pods/1b3c9632-d219-4f7a-8094-36428c83afa0/volumes"
Jul 28 10:09:59.832462 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:59.832433    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5b7c89b76c-w2zrh"
Jul 28 10:09:59.836424 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:09:59.836403    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5b7c89b76c-w2zrh"
Jul 28 10:10:21.690938 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:21.690834    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:21.710845 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:21.710814    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:22.350904 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:22.350878    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:43.019422 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:43.019344    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-54975fd499-78vrl"]
Jul 28 10:10:43.019885 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:43.019644    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebd56b7c-8767-4f6b-9189-1019cdef269b" containerName="console"
Jul 28 10:10:43.019885 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:43.019656    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebd56b7c-8767-4f6b-9189-1019cdef269b" containerName="console"
Jul 28 10:10:43.019885 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:43.019666    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b3c9632-d219-4f7a-8094-36428c83afa0" containerName="console"
Jul 28 10:10:43.019885 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:43.019672    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b3c9632-d219-4f7a-8094-36428c83afa0" containerName="console"
Jul 28 10:10:43.019885 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:43.019731    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebd56b7c-8767-4f6b-9189-1019cdef269b" containerName="console"
Jul 28 10:10:43.019885 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:43.019738    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b3c9632-d219-4f7a-8094-36428c83afa0" containerName="console"
Jul 28 10:10:43.021882 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:43.021861    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54975fd499-78vrl"
Jul 28 10:10:43.072184 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:43.072151    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54975fd499-78vrl"]
Jul 28 10:10:43.085016 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:43.084978    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f869e1ec-7459-4227-9496-a81836bf59fa-console-config\") pod \"console-54975fd499-78vrl\" (UID: \"f869e1ec-7459-4227-9496-a81836bf59fa\") " pod="openshift-console/console-54975fd499-78vrl"
Jul 28 10:10:43.085186 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:43.085025    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f869e1ec-7459-4227-9496-a81836bf59fa-oauth-serving-cert\") pod \"console-54975fd499-78vrl\" (UID: \"f869e1ec-7459-4227-9496-a81836bf59fa\") " pod="openshift-console/console-54975fd499-78vrl"
Jul 28 10:10:43.085186 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:43.085100    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f869e1ec-7459-4227-9496-a81836bf59fa-service-ca\") pod \"console-54975fd499-78vrl\" (UID: \"f869e1ec-7459-4227-9496-a81836bf59fa\") " pod="openshift-console/console-54975fd499-78vrl"
Jul 28 10:10:43.085186 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:43.085170    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f869e1ec-7459-4227-9496-a81836bf59fa-console-serving-cert\") pod \"console-54975fd499-78vrl\" (UID: \"f869e1ec-7459-4227-9496-a81836bf59fa\") " pod="openshift-console/console-54975fd499-78vrl"
Jul 28 10:10:43.085288 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:43.085199    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f869e1ec-7459-4227-9496-a81836bf59fa-console-oauth-config\") pod \"console-54975fd499-78vrl\" (UID: \"f869e1ec-7459-4227-9496-a81836bf59fa\") " pod="openshift-console/console-54975fd499-78vrl"
Jul 28 10:10:43.085288 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:43.085214    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dz7m\" (UniqueName: \"kubernetes.io/projected/f869e1ec-7459-4227-9496-a81836bf59fa-kube-api-access-5dz7m\") pod \"console-54975fd499-78vrl\" (UID: \"f869e1ec-7459-4227-9496-a81836bf59fa\") " pod="openshift-console/console-54975fd499-78vrl"
Jul 28 10:10:43.085288 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:43.085260    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f869e1ec-7459-4227-9496-a81836bf59fa-trusted-ca-bundle\") pod \"console-54975fd499-78vrl\" (UID: \"f869e1ec-7459-4227-9496-a81836bf59fa\") " pod="openshift-console/console-54975fd499-78vrl"
Jul 28 10:10:43.186220 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:43.186167    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f869e1ec-7459-4227-9496-a81836bf59fa-console-config\") pod \"console-54975fd499-78vrl\" (UID: \"f869e1ec-7459-4227-9496-a81836bf59fa\") " pod="openshift-console/console-54975fd499-78vrl"
Jul 28 10:10:43.186220 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:43.186215    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f869e1ec-7459-4227-9496-a81836bf59fa-oauth-serving-cert\") pod \"console-54975fd499-78vrl\" (UID: \"f869e1ec-7459-4227-9496-a81836bf59fa\") " pod="openshift-console/console-54975fd499-78vrl"
Jul 28 10:10:43.186444 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:43.186257    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f869e1ec-7459-4227-9496-a81836bf59fa-service-ca\") pod \"console-54975fd499-78vrl\" (UID: \"f869e1ec-7459-4227-9496-a81836bf59fa\") " pod="openshift-console/console-54975fd499-78vrl"
Jul 28 10:10:43.186444 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:43.186282    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f869e1ec-7459-4227-9496-a81836bf59fa-console-serving-cert\") pod \"console-54975fd499-78vrl\" (UID: \"f869e1ec-7459-4227-9496-a81836bf59fa\") " pod="openshift-console/console-54975fd499-78vrl"
Jul 28 10:10:43.186444 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:43.186302    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f869e1ec-7459-4227-9496-a81836bf59fa-console-oauth-config\") pod \"console-54975fd499-78vrl\" (UID: \"f869e1ec-7459-4227-9496-a81836bf59fa\") " pod="openshift-console/console-54975fd499-78vrl"
Jul 28 10:10:43.186593 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:43.186429    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dz7m\" (UniqueName: \"kubernetes.io/projected/f869e1ec-7459-4227-9496-a81836bf59fa-kube-api-access-5dz7m\") pod \"console-54975fd499-78vrl\" (UID: \"f869e1ec-7459-4227-9496-a81836bf59fa\") " pod="openshift-console/console-54975fd499-78vrl"
Jul 28 10:10:43.186593 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:43.186535    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f869e1ec-7459-4227-9496-a81836bf59fa-trusted-ca-bundle\") pod \"console-54975fd499-78vrl\" (UID: \"f869e1ec-7459-4227-9496-a81836bf59fa\") " pod="openshift-console/console-54975fd499-78vrl"
Jul 28 10:10:43.187102 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:43.187074    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f869e1ec-7459-4227-9496-a81836bf59fa-console-config\") pod \"console-54975fd499-78vrl\" (UID: \"f869e1ec-7459-4227-9496-a81836bf59fa\") " pod="openshift-console/console-54975fd499-78vrl"
Jul 28 10:10:43.187284 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:43.187176    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f869e1ec-7459-4227-9496-a81836bf59fa-service-ca\") pod \"console-54975fd499-78vrl\" (UID: \"f869e1ec-7459-4227-9496-a81836bf59fa\") " pod="openshift-console/console-54975fd499-78vrl"
Jul 28 10:10:43.187284 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:43.187176    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f869e1ec-7459-4227-9496-a81836bf59fa-oauth-serving-cert\") pod \"console-54975fd499-78vrl\" (UID: \"f869e1ec-7459-4227-9496-a81836bf59fa\") " pod="openshift-console/console-54975fd499-78vrl"
Jul 28 10:10:43.187414 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:43.187343    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f869e1ec-7459-4227-9496-a81836bf59fa-trusted-ca-bundle\") pod \"console-54975fd499-78vrl\" (UID: \"f869e1ec-7459-4227-9496-a81836bf59fa\") " pod="openshift-console/console-54975fd499-78vrl"
Jul 28 10:10:43.188801 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:43.188785    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f869e1ec-7459-4227-9496-a81836bf59fa-console-serving-cert\") pod \"console-54975fd499-78vrl\" (UID: \"f869e1ec-7459-4227-9496-a81836bf59fa\") " pod="openshift-console/console-54975fd499-78vrl"
Jul 28 10:10:43.188856 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:43.188822    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f869e1ec-7459-4227-9496-a81836bf59fa-console-oauth-config\") pod \"console-54975fd499-78vrl\" (UID: \"f869e1ec-7459-4227-9496-a81836bf59fa\") " pod="openshift-console/console-54975fd499-78vrl"
Jul 28 10:10:43.194357 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:43.194337    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dz7m\" (UniqueName: \"kubernetes.io/projected/f869e1ec-7459-4227-9496-a81836bf59fa-kube-api-access-5dz7m\") pod \"console-54975fd499-78vrl\" (UID: \"f869e1ec-7459-4227-9496-a81836bf59fa\") " pod="openshift-console/console-54975fd499-78vrl"
Jul 28 10:10:43.332411 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:43.332314    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54975fd499-78vrl"
Jul 28 10:10:43.461637 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:43.461542    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54975fd499-78vrl"]
Jul 28 10:10:43.466983 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:10:43.466948    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf869e1ec_7459_4227_9496_a81836bf59fa.slice/crio-3fd7ee6a47c7877b54a215f1779fdb2df04afb40c2035b3cd49230bda79ea3f9 WatchSource:0}: Error finding container 3fd7ee6a47c7877b54a215f1779fdb2df04afb40c2035b3cd49230bda79ea3f9: Status 404 returned error can't find the container with id 3fd7ee6a47c7877b54a215f1779fdb2df04afb40c2035b3cd49230bda79ea3f9
Jul 28 10:10:44.423247 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:44.423213    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54975fd499-78vrl" event={"ID":"f869e1ec-7459-4227-9496-a81836bf59fa","Type":"ContainerStarted","Data":"122fd6f8ecaddd8faf67b508bfeb8352baf9f2f60c21c7c68ce2d9fe0d89af62"}
Jul 28 10:10:44.423574 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:44.423263    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54975fd499-78vrl" event={"ID":"f869e1ec-7459-4227-9496-a81836bf59fa","Type":"ContainerStarted","Data":"3fd7ee6a47c7877b54a215f1779fdb2df04afb40c2035b3cd49230bda79ea3f9"}
Jul 28 10:10:44.442229 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:44.442182    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-54975fd499-78vrl" podStartSLOduration=1.442165619 podStartE2EDuration="1.442165619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:10:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:10:44.440875236 +0000 UTC m=+180.399643617" watchObservedRunningTime="2026-07-28 10:10:44.442165619 +0000 UTC m=+180.400933997"
Jul 28 10:10:51.274343 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:51.274303    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54975fd499-78vrl"]
Jul 28 10:10:51.302885 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:51.302855    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5bc7759579-5w9mr"]
Jul 28 10:10:51.306006 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:51.305985    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bc7759579-5w9mr"
Jul 28 10:10:51.320833 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:51.320806    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bc7759579-5w9mr"]
Jul 28 10:10:51.361702 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:51.361669    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b91f9084-96cf-4b3a-b44f-7d0707c20cfc-console-oauth-config\") pod \"console-5bc7759579-5w9mr\" (UID: \"b91f9084-96cf-4b3a-b44f-7d0707c20cfc\") " pod="openshift-console/console-5bc7759579-5w9mr"
Jul 28 10:10:51.361876 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:51.361719    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b91f9084-96cf-4b3a-b44f-7d0707c20cfc-trusted-ca-bundle\") pod \"console-5bc7759579-5w9mr\" (UID: \"b91f9084-96cf-4b3a-b44f-7d0707c20cfc\") " pod="openshift-console/console-5bc7759579-5w9mr"
Jul 28 10:10:51.361876 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:51.361741    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b91f9084-96cf-4b3a-b44f-7d0707c20cfc-service-ca\") pod \"console-5bc7759579-5w9mr\" (UID: \"b91f9084-96cf-4b3a-b44f-7d0707c20cfc\") " pod="openshift-console/console-5bc7759579-5w9mr"
Jul 28 10:10:51.361876 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:51.361778    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b91f9084-96cf-4b3a-b44f-7d0707c20cfc-console-config\") pod \"console-5bc7759579-5w9mr\" (UID: \"b91f9084-96cf-4b3a-b44f-7d0707c20cfc\") " pod="openshift-console/console-5bc7759579-5w9mr"
Jul 28 10:10:51.361876 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:51.361804    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b91f9084-96cf-4b3a-b44f-7d0707c20cfc-oauth-serving-cert\") pod \"console-5bc7759579-5w9mr\" (UID: \"b91f9084-96cf-4b3a-b44f-7d0707c20cfc\") " pod="openshift-console/console-5bc7759579-5w9mr"
Jul 28 10:10:51.361876 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:51.361842    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ckfh\" (UniqueName: \"kubernetes.io/projected/b91f9084-96cf-4b3a-b44f-7d0707c20cfc-kube-api-access-9ckfh\") pod \"console-5bc7759579-5w9mr\" (UID: \"b91f9084-96cf-4b3a-b44f-7d0707c20cfc\") " pod="openshift-console/console-5bc7759579-5w9mr"
Jul 28 10:10:51.362054 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:51.361903    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b91f9084-96cf-4b3a-b44f-7d0707c20cfc-console-serving-cert\") pod \"console-5bc7759579-5w9mr\" (UID: \"b91f9084-96cf-4b3a-b44f-7d0707c20cfc\") " pod="openshift-console/console-5bc7759579-5w9mr"
Jul 28 10:10:51.463280 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:51.463243    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b91f9084-96cf-4b3a-b44f-7d0707c20cfc-console-serving-cert\") pod \"console-5bc7759579-5w9mr\" (UID: \"b91f9084-96cf-4b3a-b44f-7d0707c20cfc\") " pod="openshift-console/console-5bc7759579-5w9mr"
Jul 28 10:10:51.463516 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:51.463290    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b91f9084-96cf-4b3a-b44f-7d0707c20cfc-console-oauth-config\") pod \"console-5bc7759579-5w9mr\" (UID: \"b91f9084-96cf-4b3a-b44f-7d0707c20cfc\") " pod="openshift-console/console-5bc7759579-5w9mr"
Jul 28 10:10:51.463516 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:51.463331    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b91f9084-96cf-4b3a-b44f-7d0707c20cfc-trusted-ca-bundle\") pod \"console-5bc7759579-5w9mr\" (UID: \"b91f9084-96cf-4b3a-b44f-7d0707c20cfc\") " pod="openshift-console/console-5bc7759579-5w9mr"
Jul 28 10:10:51.463516 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:51.463354    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b91f9084-96cf-4b3a-b44f-7d0707c20cfc-service-ca\") pod \"console-5bc7759579-5w9mr\" (UID: \"b91f9084-96cf-4b3a-b44f-7d0707c20cfc\") " pod="openshift-console/console-5bc7759579-5w9mr"
Jul 28 10:10:51.463516 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:51.463377    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b91f9084-96cf-4b3a-b44f-7d0707c20cfc-console-config\") pod \"console-5bc7759579-5w9mr\" (UID: \"b91f9084-96cf-4b3a-b44f-7d0707c20cfc\") " pod="openshift-console/console-5bc7759579-5w9mr"
Jul 28 10:10:51.463516 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:51.463397    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b91f9084-96cf-4b3a-b44f-7d0707c20cfc-oauth-serving-cert\") pod \"console-5bc7759579-5w9mr\" (UID: \"b91f9084-96cf-4b3a-b44f-7d0707c20cfc\") " pod="openshift-console/console-5bc7759579-5w9mr"
Jul 28 10:10:51.463516 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:51.463427    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ckfh\" (UniqueName: \"kubernetes.io/projected/b91f9084-96cf-4b3a-b44f-7d0707c20cfc-kube-api-access-9ckfh\") pod \"console-5bc7759579-5w9mr\" (UID: \"b91f9084-96cf-4b3a-b44f-7d0707c20cfc\") " pod="openshift-console/console-5bc7759579-5w9mr"
Jul 28 10:10:51.464313 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:51.464278    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b91f9084-96cf-4b3a-b44f-7d0707c20cfc-console-config\") pod \"console-5bc7759579-5w9mr\" (UID: \"b91f9084-96cf-4b3a-b44f-7d0707c20cfc\") " pod="openshift-console/console-5bc7759579-5w9mr"
Jul 28 10:10:51.464425 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:51.464278    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b91f9084-96cf-4b3a-b44f-7d0707c20cfc-oauth-serving-cert\") pod \"console-5bc7759579-5w9mr\" (UID: \"b91f9084-96cf-4b3a-b44f-7d0707c20cfc\") " pod="openshift-console/console-5bc7759579-5w9mr"
Jul 28 10:10:51.464425 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:51.464383    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b91f9084-96cf-4b3a-b44f-7d0707c20cfc-service-ca\") pod \"console-5bc7759579-5w9mr\" (UID: \"b91f9084-96cf-4b3a-b44f-7d0707c20cfc\") " pod="openshift-console/console-5bc7759579-5w9mr"
Jul 28 10:10:51.464513 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:51.464465    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b91f9084-96cf-4b3a-b44f-7d0707c20cfc-trusted-ca-bundle\") pod \"console-5bc7759579-5w9mr\" (UID: \"b91f9084-96cf-4b3a-b44f-7d0707c20cfc\") " pod="openshift-console/console-5bc7759579-5w9mr"
Jul 28 10:10:51.465897 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:51.465861    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b91f9084-96cf-4b3a-b44f-7d0707c20cfc-console-serving-cert\") pod \"console-5bc7759579-5w9mr\" (UID: \"b91f9084-96cf-4b3a-b44f-7d0707c20cfc\") " pod="openshift-console/console-5bc7759579-5w9mr"
Jul 28 10:10:51.466203 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:51.466178    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b91f9084-96cf-4b3a-b44f-7d0707c20cfc-console-oauth-config\") pod \"console-5bc7759579-5w9mr\" (UID: \"b91f9084-96cf-4b3a-b44f-7d0707c20cfc\") " pod="openshift-console/console-5bc7759579-5w9mr"
Jul 28 10:10:51.472454 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:51.472430    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ckfh\" (UniqueName: \"kubernetes.io/projected/b91f9084-96cf-4b3a-b44f-7d0707c20cfc-kube-api-access-9ckfh\") pod \"console-5bc7759579-5w9mr\" (UID: \"b91f9084-96cf-4b3a-b44f-7d0707c20cfc\") " pod="openshift-console/console-5bc7759579-5w9mr"
Jul 28 10:10:51.614679 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:51.614580    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bc7759579-5w9mr"
Jul 28 10:10:51.745015 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:51.744987    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bc7759579-5w9mr"]
Jul 28 10:10:51.747374 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:10:51.747343    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb91f9084_96cf_4b3a_b44f_7d0707c20cfc.slice/crio-e4804b6602501e273512d8b7ad2eb0381da99d465538e8bef1ad83e8d9e949f6 WatchSource:0}: Error finding container e4804b6602501e273512d8b7ad2eb0381da99d465538e8bef1ad83e8d9e949f6: Status 404 returned error can't find the container with id e4804b6602501e273512d8b7ad2eb0381da99d465538e8bef1ad83e8d9e949f6
Jul 28 10:10:52.454951 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:52.454900    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bc7759579-5w9mr" event={"ID":"b91f9084-96cf-4b3a-b44f-7d0707c20cfc","Type":"ContainerStarted","Data":"da744b474e1943b6de7fcbec4f8f3f295f3b674eddc7d5373481fffed3a67b61"}
Jul 28 10:10:52.455316 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:52.455016    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bc7759579-5w9mr" event={"ID":"b91f9084-96cf-4b3a-b44f-7d0707c20cfc","Type":"ContainerStarted","Data":"e4804b6602501e273512d8b7ad2eb0381da99d465538e8bef1ad83e8d9e949f6"}
Jul 28 10:10:52.476619 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:52.476555    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5bc7759579-5w9mr" podStartSLOduration=1.476537081 podStartE2EDuration="1.476537081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:10:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:10:52.474813855 +0000 UTC m=+188.433582233" watchObservedRunningTime="2026-07-28 10:10:52.476537081 +0000 UTC m=+188.435305460"
Jul 28 10:10:53.333396 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:10:53.333361    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-54975fd499-78vrl"
Jul 28 10:11:01.615231 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:01.615188    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5bc7759579-5w9mr"
Jul 28 10:11:01.615841 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:01.615266    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5bc7759579-5w9mr"
Jul 28 10:11:01.620006 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:01.619984    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5bc7759579-5w9mr"
Jul 28 10:11:02.502548 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:02.502519    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5bc7759579-5w9mr"
Jul 28 10:11:02.559582 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:02.559547    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d967c8bb9-7w6px"]
Jul 28 10:11:16.293258 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:16.293201    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-54975fd499-78vrl" podUID="f869e1ec-7459-4227-9496-a81836bf59fa" containerName="console" containerID="cri-o://122fd6f8ecaddd8faf67b508bfeb8352baf9f2f60c21c7c68ce2d9fe0d89af62" gracePeriod=15
Jul 28 10:11:16.526365 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:16.526343    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54975fd499-78vrl"
Jul 28 10:11:16.549860 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:16.549774    2557 generic.go:356] "Generic (PLEG): container finished" podID="f869e1ec-7459-4227-9496-a81836bf59fa" containerID="122fd6f8ecaddd8faf67b508bfeb8352baf9f2f60c21c7c68ce2d9fe0d89af62" exitCode=0
Jul 28 10:11:16.549860 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:16.549824    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54975fd499-78vrl"
Jul 28 10:11:16.550091 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:16.549856    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54975fd499-78vrl" event={"ID":"f869e1ec-7459-4227-9496-a81836bf59fa","Type":"ContainerDied","Data":"122fd6f8ecaddd8faf67b508bfeb8352baf9f2f60c21c7c68ce2d9fe0d89af62"}
Jul 28 10:11:16.550091 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:16.549956    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54975fd499-78vrl" event={"ID":"f869e1ec-7459-4227-9496-a81836bf59fa","Type":"ContainerDied","Data":"3fd7ee6a47c7877b54a215f1779fdb2df04afb40c2035b3cd49230bda79ea3f9"}
Jul 28 10:11:16.550091 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:16.549995    2557 scope.go:117] "RemoveContainer" containerID="122fd6f8ecaddd8faf67b508bfeb8352baf9f2f60c21c7c68ce2d9fe0d89af62"
Jul 28 10:11:16.558206 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:16.558178    2557 scope.go:117] "RemoveContainer" containerID="122fd6f8ecaddd8faf67b508bfeb8352baf9f2f60c21c7c68ce2d9fe0d89af62"
Jul 28 10:11:16.558526 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:11:16.558499    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"122fd6f8ecaddd8faf67b508bfeb8352baf9f2f60c21c7c68ce2d9fe0d89af62\": container with ID starting with 122fd6f8ecaddd8faf67b508bfeb8352baf9f2f60c21c7c68ce2d9fe0d89af62 not found: ID does not exist" containerID="122fd6f8ecaddd8faf67b508bfeb8352baf9f2f60c21c7c68ce2d9fe0d89af62"
Jul 28 10:11:16.558615 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:16.558539    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"122fd6f8ecaddd8faf67b508bfeb8352baf9f2f60c21c7c68ce2d9fe0d89af62"} err="failed to get container status \"122fd6f8ecaddd8faf67b508bfeb8352baf9f2f60c21c7c68ce2d9fe0d89af62\": rpc error: code = NotFound desc = could not find container \"122fd6f8ecaddd8faf67b508bfeb8352baf9f2f60c21c7c68ce2d9fe0d89af62\": container with ID starting with 122fd6f8ecaddd8faf67b508bfeb8352baf9f2f60c21c7c68ce2d9fe0d89af62 not found: ID does not exist"
Jul 28 10:11:16.692658 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:16.692622    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f869e1ec-7459-4227-9496-a81836bf59fa-console-config\") pod \"f869e1ec-7459-4227-9496-a81836bf59fa\" (UID: \"f869e1ec-7459-4227-9496-a81836bf59fa\") "
Jul 28 10:11:16.692840 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:16.692671    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f869e1ec-7459-4227-9496-a81836bf59fa-oauth-serving-cert\") pod \"f869e1ec-7459-4227-9496-a81836bf59fa\" (UID: \"f869e1ec-7459-4227-9496-a81836bf59fa\") "
Jul 28 10:11:16.692840 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:16.692704    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f869e1ec-7459-4227-9496-a81836bf59fa-console-serving-cert\") pod \"f869e1ec-7459-4227-9496-a81836bf59fa\" (UID: \"f869e1ec-7459-4227-9496-a81836bf59fa\") "
Jul 28 10:11:16.692840 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:16.692768    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f869e1ec-7459-4227-9496-a81836bf59fa-service-ca\") pod \"f869e1ec-7459-4227-9496-a81836bf59fa\" (UID: \"f869e1ec-7459-4227-9496-a81836bf59fa\") "
Jul 28 10:11:16.692840 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:16.692801    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5dz7m\" (UniqueName: \"kubernetes.io/projected/f869e1ec-7459-4227-9496-a81836bf59fa-kube-api-access-5dz7m\") pod \"f869e1ec-7459-4227-9496-a81836bf59fa\" (UID: \"f869e1ec-7459-4227-9496-a81836bf59fa\") "
Jul 28 10:11:16.692840 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:16.692833    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f869e1ec-7459-4227-9496-a81836bf59fa-console-oauth-config\") pod \"f869e1ec-7459-4227-9496-a81836bf59fa\" (UID: \"f869e1ec-7459-4227-9496-a81836bf59fa\") "
Jul 28 10:11:16.693099 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:16.692863    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f869e1ec-7459-4227-9496-a81836bf59fa-trusted-ca-bundle\") pod \"f869e1ec-7459-4227-9496-a81836bf59fa\" (UID: \"f869e1ec-7459-4227-9496-a81836bf59fa\") "
Jul 28 10:11:16.693158 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:16.693024    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f869e1ec-7459-4227-9496-a81836bf59fa-console-config" (OuterVolumeSpecName: "console-config") pod "f869e1ec-7459-4227-9496-a81836bf59fa" (UID: "f869e1ec-7459-4227-9496-a81836bf59fa"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:11:16.693158 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:16.693137    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f869e1ec-7459-4227-9496-a81836bf59fa-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f869e1ec-7459-4227-9496-a81836bf59fa" (UID: "f869e1ec-7459-4227-9496-a81836bf59fa"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:11:16.693232 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:16.693178    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f869e1ec-7459-4227-9496-a81836bf59fa-service-ca" (OuterVolumeSpecName: "service-ca") pod "f869e1ec-7459-4227-9496-a81836bf59fa" (UID: "f869e1ec-7459-4227-9496-a81836bf59fa"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:11:16.693474 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:16.693450    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f869e1ec-7459-4227-9496-a81836bf59fa-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f869e1ec-7459-4227-9496-a81836bf59fa" (UID: "f869e1ec-7459-4227-9496-a81836bf59fa"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:11:16.695147 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:16.695105    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f869e1ec-7459-4227-9496-a81836bf59fa-kube-api-access-5dz7m" (OuterVolumeSpecName: "kube-api-access-5dz7m") pod "f869e1ec-7459-4227-9496-a81836bf59fa" (UID: "f869e1ec-7459-4227-9496-a81836bf59fa"). InnerVolumeSpecName "kube-api-access-5dz7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:11:16.695147 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:16.695134    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f869e1ec-7459-4227-9496-a81836bf59fa-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f869e1ec-7459-4227-9496-a81836bf59fa" (UID: "f869e1ec-7459-4227-9496-a81836bf59fa"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:11:16.695311 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:16.695153    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f869e1ec-7459-4227-9496-a81836bf59fa-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f869e1ec-7459-4227-9496-a81836bf59fa" (UID: "f869e1ec-7459-4227-9496-a81836bf59fa"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:11:16.794074 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:16.794028    2557 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f869e1ec-7459-4227-9496-a81836bf59fa-service-ca\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:11:16.794074 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:16.794068    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dz7m\" (UniqueName: \"kubernetes.io/projected/f869e1ec-7459-4227-9496-a81836bf59fa-kube-api-access-5dz7m\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:11:16.794074 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:16.794079    2557 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f869e1ec-7459-4227-9496-a81836bf59fa-console-oauth-config\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:11:16.794074 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:16.794089    2557 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f869e1ec-7459-4227-9496-a81836bf59fa-trusted-ca-bundle\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:11:16.794338 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:16.794097    2557 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f869e1ec-7459-4227-9496-a81836bf59fa-console-config\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:11:16.794338 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:16.794105    2557 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f869e1ec-7459-4227-9496-a81836bf59fa-oauth-serving-cert\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:11:16.794338 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:16.794113    2557 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f869e1ec-7459-4227-9496-a81836bf59fa-console-serving-cert\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:11:16.870906 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:16.870875    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54975fd499-78vrl"]
Jul 28 10:11:16.874253 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:16.874229    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-54975fd499-78vrl"]
Jul 28 10:11:18.579493 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:18.579462    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f869e1ec-7459-4227-9496-a81836bf59fa" path="/var/lib/kubelet/pods/f869e1ec-7459-4227-9496-a81836bf59fa/volumes"
Jul 28 10:11:27.579139 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:27.579086    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5d967c8bb9-7w6px" podUID="23a46488-c010-4d5e-a9f2-df2576efda75" containerName="console" containerID="cri-o://52d5778dc58c1fe47d151fe7d45c0ff5dc83ddb44875f19efd19bcaf22de3549" gracePeriod=15
Jul 28 10:11:27.816976 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:27.816953    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d967c8bb9-7w6px"
Jul 28 10:11:27.888418 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:27.888318    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/23a46488-c010-4d5e-a9f2-df2576efda75-console-oauth-config\") pod \"23a46488-c010-4d5e-a9f2-df2576efda75\" (UID: \"23a46488-c010-4d5e-a9f2-df2576efda75\") "
Jul 28 10:11:27.888418 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:27.888378    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/23a46488-c010-4d5e-a9f2-df2576efda75-service-ca\") pod \"23a46488-c010-4d5e-a9f2-df2576efda75\" (UID: \"23a46488-c010-4d5e-a9f2-df2576efda75\") "
Jul 28 10:11:27.888418 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:27.888413    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kgst5\" (UniqueName: \"kubernetes.io/projected/23a46488-c010-4d5e-a9f2-df2576efda75-kube-api-access-kgst5\") pod \"23a46488-c010-4d5e-a9f2-df2576efda75\" (UID: \"23a46488-c010-4d5e-a9f2-df2576efda75\") "
Jul 28 10:11:27.888724 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:27.888476    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/23a46488-c010-4d5e-a9f2-df2576efda75-console-serving-cert\") pod \"23a46488-c010-4d5e-a9f2-df2576efda75\" (UID: \"23a46488-c010-4d5e-a9f2-df2576efda75\") "
Jul 28 10:11:27.888724 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:27.888516    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23a46488-c010-4d5e-a9f2-df2576efda75-trusted-ca-bundle\") pod \"23a46488-c010-4d5e-a9f2-df2576efda75\" (UID: \"23a46488-c010-4d5e-a9f2-df2576efda75\") "
Jul 28 10:11:27.888724 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:27.888551    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/23a46488-c010-4d5e-a9f2-df2576efda75-oauth-serving-cert\") pod \"23a46488-c010-4d5e-a9f2-df2576efda75\" (UID: \"23a46488-c010-4d5e-a9f2-df2576efda75\") "
Jul 28 10:11:27.888724 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:27.888576    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/23a46488-c010-4d5e-a9f2-df2576efda75-console-config\") pod \"23a46488-c010-4d5e-a9f2-df2576efda75\" (UID: \"23a46488-c010-4d5e-a9f2-df2576efda75\") "
Jul 28 10:11:27.888917 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:27.888857    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23a46488-c010-4d5e-a9f2-df2576efda75-service-ca" (OuterVolumeSpecName: "service-ca") pod "23a46488-c010-4d5e-a9f2-df2576efda75" (UID: "23a46488-c010-4d5e-a9f2-df2576efda75"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:11:27.889023 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:27.888991    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23a46488-c010-4d5e-a9f2-df2576efda75-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "23a46488-c010-4d5e-a9f2-df2576efda75" (UID: "23a46488-c010-4d5e-a9f2-df2576efda75"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:11:27.889085 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:27.889028    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23a46488-c010-4d5e-a9f2-df2576efda75-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "23a46488-c010-4d5e-a9f2-df2576efda75" (UID: "23a46488-c010-4d5e-a9f2-df2576efda75"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:11:27.889170 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:27.889132    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23a46488-c010-4d5e-a9f2-df2576efda75-console-config" (OuterVolumeSpecName: "console-config") pod "23a46488-c010-4d5e-a9f2-df2576efda75" (UID: "23a46488-c010-4d5e-a9f2-df2576efda75"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:11:27.890640 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:27.890589    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23a46488-c010-4d5e-a9f2-df2576efda75-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "23a46488-c010-4d5e-a9f2-df2576efda75" (UID: "23a46488-c010-4d5e-a9f2-df2576efda75"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:11:27.890717 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:27.890656    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23a46488-c010-4d5e-a9f2-df2576efda75-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "23a46488-c010-4d5e-a9f2-df2576efda75" (UID: "23a46488-c010-4d5e-a9f2-df2576efda75"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:11:27.890717 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:27.890684    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23a46488-c010-4d5e-a9f2-df2576efda75-kube-api-access-kgst5" (OuterVolumeSpecName: "kube-api-access-kgst5") pod "23a46488-c010-4d5e-a9f2-df2576efda75" (UID: "23a46488-c010-4d5e-a9f2-df2576efda75"). InnerVolumeSpecName "kube-api-access-kgst5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:11:27.989898 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:27.989858    2557 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23a46488-c010-4d5e-a9f2-df2576efda75-trusted-ca-bundle\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:11:27.989898 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:27.989888    2557 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/23a46488-c010-4d5e-a9f2-df2576efda75-oauth-serving-cert\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:11:27.989898 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:27.989898    2557 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/23a46488-c010-4d5e-a9f2-df2576efda75-console-config\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:11:27.989898 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:27.989907    2557 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/23a46488-c010-4d5e-a9f2-df2576efda75-console-oauth-config\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:11:27.990156 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:27.989916    2557 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/23a46488-c010-4d5e-a9f2-df2576efda75-service-ca\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:11:27.990156 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:27.989924    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgst5\" (UniqueName: \"kubernetes.io/projected/23a46488-c010-4d5e-a9f2-df2576efda75-kube-api-access-kgst5\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:11:27.990156 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:27.989932    2557 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/23a46488-c010-4d5e-a9f2-df2576efda75-console-serving-cert\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:11:28.598566 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:28.598538    2557 generic.go:356] "Generic (PLEG): container finished" podID="23a46488-c010-4d5e-a9f2-df2576efda75" containerID="52d5778dc58c1fe47d151fe7d45c0ff5dc83ddb44875f19efd19bcaf22de3549" exitCode=0
Jul 28 10:11:28.598942 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:28.598578    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d967c8bb9-7w6px"
Jul 28 10:11:28.598942 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:28.598625    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d967c8bb9-7w6px" event={"ID":"23a46488-c010-4d5e-a9f2-df2576efda75","Type":"ContainerDied","Data":"52d5778dc58c1fe47d151fe7d45c0ff5dc83ddb44875f19efd19bcaf22de3549"}
Jul 28 10:11:28.598942 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:28.598674    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d967c8bb9-7w6px" event={"ID":"23a46488-c010-4d5e-a9f2-df2576efda75","Type":"ContainerDied","Data":"2b25770bdcc70502db22b3b1db560c1ccaad6b39fd15b1e0d4fffc69db8f8d9e"}
Jul 28 10:11:28.598942 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:28.598694    2557 scope.go:117] "RemoveContainer" containerID="52d5778dc58c1fe47d151fe7d45c0ff5dc83ddb44875f19efd19bcaf22de3549"
Jul 28 10:11:28.606758 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:28.606735    2557 scope.go:117] "RemoveContainer" containerID="52d5778dc58c1fe47d151fe7d45c0ff5dc83ddb44875f19efd19bcaf22de3549"
Jul 28 10:11:28.607052 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:11:28.607025    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52d5778dc58c1fe47d151fe7d45c0ff5dc83ddb44875f19efd19bcaf22de3549\": container with ID starting with 52d5778dc58c1fe47d151fe7d45c0ff5dc83ddb44875f19efd19bcaf22de3549 not found: ID does not exist" containerID="52d5778dc58c1fe47d151fe7d45c0ff5dc83ddb44875f19efd19bcaf22de3549"
Jul 28 10:11:28.607136 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:28.607057    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52d5778dc58c1fe47d151fe7d45c0ff5dc83ddb44875f19efd19bcaf22de3549"} err="failed to get container status \"52d5778dc58c1fe47d151fe7d45c0ff5dc83ddb44875f19efd19bcaf22de3549\": rpc error: code = NotFound desc = could not find container \"52d5778dc58c1fe47d151fe7d45c0ff5dc83ddb44875f19efd19bcaf22de3549\": container with ID starting with 52d5778dc58c1fe47d151fe7d45c0ff5dc83ddb44875f19efd19bcaf22de3549 not found: ID does not exist"
Jul 28 10:11:28.616306 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:28.616279    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d967c8bb9-7w6px"]
Jul 28 10:11:28.618028 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:28.618006    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5d967c8bb9-7w6px"]
Jul 28 10:11:30.579196 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:11:30.579163    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23a46488-c010-4d5e-a9f2-df2576efda75" path="/var/lib/kubelet/pods/23a46488-c010-4d5e-a9f2-df2576efda75/volumes"
Jul 28 10:12:44.417278 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:12:44.417246    2557 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 10:15:48.184256 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:15:48.184218    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-598bbfdd7c-tb6nc"]
Jul 28 10:15:48.184730 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:15:48.184503    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f869e1ec-7459-4227-9496-a81836bf59fa" containerName="console"
Jul 28 10:15:48.184730 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:15:48.184514    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f869e1ec-7459-4227-9496-a81836bf59fa" containerName="console"
Jul 28 10:15:48.184730 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:15:48.184534    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23a46488-c010-4d5e-a9f2-df2576efda75" containerName="console"
Jul 28 10:15:48.184730 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:15:48.184540    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="23a46488-c010-4d5e-a9f2-df2576efda75" containerName="console"
Jul 28 10:15:48.184730 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:15:48.184589    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="f869e1ec-7459-4227-9496-a81836bf59fa" containerName="console"
Jul 28 10:15:48.184730 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:15:48.184617    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="23a46488-c010-4d5e-a9f2-df2576efda75" containerName="console"
Jul 28 10:15:48.186457 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:15:48.186436    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-598bbfdd7c-tb6nc"
Jul 28 10:15:48.204178 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:15:48.204147    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-598bbfdd7c-tb6nc"]
Jul 28 10:15:48.316237 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:15:48.316194    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a9aa4162-c8f4-410b-b596-cbeb8f904aa6-service-ca\") pod \"console-598bbfdd7c-tb6nc\" (UID: \"a9aa4162-c8f4-410b-b596-cbeb8f904aa6\") " pod="openshift-console/console-598bbfdd7c-tb6nc"
Jul 28 10:15:48.316419 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:15:48.316269    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a9aa4162-c8f4-410b-b596-cbeb8f904aa6-console-serving-cert\") pod \"console-598bbfdd7c-tb6nc\" (UID: \"a9aa4162-c8f4-410b-b596-cbeb8f904aa6\") " pod="openshift-console/console-598bbfdd7c-tb6nc"
Jul 28 10:15:48.316419 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:15:48.316295    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a9aa4162-c8f4-410b-b596-cbeb8f904aa6-console-oauth-config\") pod \"console-598bbfdd7c-tb6nc\" (UID: \"a9aa4162-c8f4-410b-b596-cbeb8f904aa6\") " pod="openshift-console/console-598bbfdd7c-tb6nc"
Jul 28 10:15:48.316419 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:15:48.316323    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c7xf\" (UniqueName: \"kubernetes.io/projected/a9aa4162-c8f4-410b-b596-cbeb8f904aa6-kube-api-access-2c7xf\") pod \"console-598bbfdd7c-tb6nc\" (UID: \"a9aa4162-c8f4-410b-b596-cbeb8f904aa6\") " pod="openshift-console/console-598bbfdd7c-tb6nc"
Jul 28 10:15:48.316419 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:15:48.316393    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9aa4162-c8f4-410b-b596-cbeb8f904aa6-trusted-ca-bundle\") pod \"console-598bbfdd7c-tb6nc\" (UID: \"a9aa4162-c8f4-410b-b596-cbeb8f904aa6\") " pod="openshift-console/console-598bbfdd7c-tb6nc"
Jul 28 10:15:48.316595 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:15:48.316428    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a9aa4162-c8f4-410b-b596-cbeb8f904aa6-oauth-serving-cert\") pod \"console-598bbfdd7c-tb6nc\" (UID: \"a9aa4162-c8f4-410b-b596-cbeb8f904aa6\") " pod="openshift-console/console-598bbfdd7c-tb6nc"
Jul 28 10:15:48.316595 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:15:48.316527    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a9aa4162-c8f4-410b-b596-cbeb8f904aa6-console-config\") pod \"console-598bbfdd7c-tb6nc\" (UID: \"a9aa4162-c8f4-410b-b596-cbeb8f904aa6\") " pod="openshift-console/console-598bbfdd7c-tb6nc"
Jul 28 10:15:48.417943 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:15:48.417906    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a9aa4162-c8f4-410b-b596-cbeb8f904aa6-console-config\") pod \"console-598bbfdd7c-tb6nc\" (UID: \"a9aa4162-c8f4-410b-b596-cbeb8f904aa6\") " pod="openshift-console/console-598bbfdd7c-tb6nc"
Jul 28 10:15:48.418139 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:15:48.417953    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a9aa4162-c8f4-410b-b596-cbeb8f904aa6-service-ca\") pod \"console-598bbfdd7c-tb6nc\" (UID: \"a9aa4162-c8f4-410b-b596-cbeb8f904aa6\") " pod="openshift-console/console-598bbfdd7c-tb6nc"
Jul 28 10:15:48.418139 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:15:48.417984    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a9aa4162-c8f4-410b-b596-cbeb8f904aa6-console-serving-cert\") pod \"console-598bbfdd7c-tb6nc\" (UID: \"a9aa4162-c8f4-410b-b596-cbeb8f904aa6\") " pod="openshift-console/console-598bbfdd7c-tb6nc"
Jul 28 10:15:48.418139 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:15:48.417998    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a9aa4162-c8f4-410b-b596-cbeb8f904aa6-console-oauth-config\") pod \"console-598bbfdd7c-tb6nc\" (UID: \"a9aa4162-c8f4-410b-b596-cbeb8f904aa6\") " pod="openshift-console/console-598bbfdd7c-tb6nc"
Jul 28 10:15:48.418139 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:15:48.418015    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c7xf\" (UniqueName: \"kubernetes.io/projected/a9aa4162-c8f4-410b-b596-cbeb8f904aa6-kube-api-access-2c7xf\") pod \"console-598bbfdd7c-tb6nc\" (UID: \"a9aa4162-c8f4-410b-b596-cbeb8f904aa6\") " pod="openshift-console/console-598bbfdd7c-tb6nc"
Jul 28 10:15:48.418139 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:15:48.418045    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9aa4162-c8f4-410b-b596-cbeb8f904aa6-trusted-ca-bundle\") pod \"console-598bbfdd7c-tb6nc\" (UID: \"a9aa4162-c8f4-410b-b596-cbeb8f904aa6\") " pod="openshift-console/console-598bbfdd7c-tb6nc"
Jul 28 10:15:48.418139 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:15:48.418077    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a9aa4162-c8f4-410b-b596-cbeb8f904aa6-oauth-serving-cert\") pod \"console-598bbfdd7c-tb6nc\" (UID: \"a9aa4162-c8f4-410b-b596-cbeb8f904aa6\") " pod="openshift-console/console-598bbfdd7c-tb6nc"
Jul 28 10:15:48.418857 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:15:48.418806    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a9aa4162-c8f4-410b-b596-cbeb8f904aa6-console-config\") pod \"console-598bbfdd7c-tb6nc\" (UID: \"a9aa4162-c8f4-410b-b596-cbeb8f904aa6\") " pod="openshift-console/console-598bbfdd7c-tb6nc"
Jul 28 10:15:48.418857 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:15:48.418816    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a9aa4162-c8f4-410b-b596-cbeb8f904aa6-oauth-serving-cert\") pod \"console-598bbfdd7c-tb6nc\" (UID: \"a9aa4162-c8f4-410b-b596-cbeb8f904aa6\") " pod="openshift-console/console-598bbfdd7c-tb6nc"
Jul 28 10:15:48.419051 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:15:48.419031    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a9aa4162-c8f4-410b-b596-cbeb8f904aa6-service-ca\") pod \"console-598bbfdd7c-tb6nc\" (UID: \"a9aa4162-c8f4-410b-b596-cbeb8f904aa6\") " pod="openshift-console/console-598bbfdd7c-tb6nc"
Jul 28 10:15:48.419276 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:15:48.419249    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9aa4162-c8f4-410b-b596-cbeb8f904aa6-trusted-ca-bundle\") pod \"console-598bbfdd7c-tb6nc\" (UID: \"a9aa4162-c8f4-410b-b596-cbeb8f904aa6\") " pod="openshift-console/console-598bbfdd7c-tb6nc"
Jul 28 10:15:48.420827 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:15:48.420802    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a9aa4162-c8f4-410b-b596-cbeb8f904aa6-console-oauth-config\") pod \"console-598bbfdd7c-tb6nc\" (UID: \"a9aa4162-c8f4-410b-b596-cbeb8f904aa6\") " pod="openshift-console/console-598bbfdd7c-tb6nc"
Jul 28 10:15:48.421153 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:15:48.421138    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a9aa4162-c8f4-410b-b596-cbeb8f904aa6-console-serving-cert\") pod \"console-598bbfdd7c-tb6nc\" (UID: \"a9aa4162-c8f4-410b-b596-cbeb8f904aa6\") " pod="openshift-console/console-598bbfdd7c-tb6nc"
Jul 28 10:15:48.461012 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:15:48.460943    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c7xf\" (UniqueName: \"kubernetes.io/projected/a9aa4162-c8f4-410b-b596-cbeb8f904aa6-kube-api-access-2c7xf\") pod \"console-598bbfdd7c-tb6nc\" (UID: \"a9aa4162-c8f4-410b-b596-cbeb8f904aa6\") " pod="openshift-console/console-598bbfdd7c-tb6nc"
Jul 28 10:15:48.495625 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:15:48.495561    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-598bbfdd7c-tb6nc"
Jul 28 10:15:48.655081 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:15:48.654907    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-598bbfdd7c-tb6nc"]
Jul 28 10:15:48.657938 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:15:48.657915    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9aa4162_c8f4_410b_b596_cbeb8f904aa6.slice/crio-22f60683afc8dddfeb894fab4ffd9b29cf1dfea2ccd754ab415c120830ac941e WatchSource:0}: Error finding container 22f60683afc8dddfeb894fab4ffd9b29cf1dfea2ccd754ab415c120830ac941e: Status 404 returned error can't find the container with id 22f60683afc8dddfeb894fab4ffd9b29cf1dfea2ccd754ab415c120830ac941e
Jul 28 10:15:49.581107 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:15:49.581065    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-598bbfdd7c-tb6nc" event={"ID":"a9aa4162-c8f4-410b-b596-cbeb8f904aa6","Type":"ContainerStarted","Data":"91fa216cdd894d4572d0b753cc50585dce89078be2cd611dec02c9ab7c2680fd"}
Jul 28 10:15:49.581505 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:15:49.581126    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-598bbfdd7c-tb6nc" event={"ID":"a9aa4162-c8f4-410b-b596-cbeb8f904aa6","Type":"ContainerStarted","Data":"22f60683afc8dddfeb894fab4ffd9b29cf1dfea2ccd754ab415c120830ac941e"}
Jul 28 10:15:49.600169 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:15:49.600113    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-598bbfdd7c-tb6nc" podStartSLOduration=1.600095358 podStartE2EDuration="1.600095358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:15:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:15:49.598447553 +0000 UTC m=+485.557215943" watchObservedRunningTime="2026-07-28 10:15:49.600095358 +0000 UTC m=+485.558863737"
Jul 28 10:15:58.496193 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:15:58.496149    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-598bbfdd7c-tb6nc"
Jul 28 10:15:58.496713 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:15:58.496219    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-598bbfdd7c-tb6nc"
Jul 28 10:15:58.501112 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:15:58.501085    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-598bbfdd7c-tb6nc"
Jul 28 10:15:58.618941 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:15:58.618915    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-598bbfdd7c-tb6nc"
Jul 28 10:15:58.670705 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:15:58.670670    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bc7759579-5w9mr"]
Jul 28 10:16:23.690924 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:16:23.690865    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5bc7759579-5w9mr" podUID="b91f9084-96cf-4b3a-b44f-7d0707c20cfc" containerName="console" containerID="cri-o://da744b474e1943b6de7fcbec4f8f3f295f3b674eddc7d5373481fffed3a67b61" gracePeriod=15
Jul 28 10:16:23.933848 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:16:23.933819    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bc7759579-5w9mr"
Jul 28 10:16:24.034717 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:16:24.034683    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b91f9084-96cf-4b3a-b44f-7d0707c20cfc-console-config\") pod \"b91f9084-96cf-4b3a-b44f-7d0707c20cfc\" (UID: \"b91f9084-96cf-4b3a-b44f-7d0707c20cfc\") "
Jul 28 10:16:24.034900 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:16:24.034744    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b91f9084-96cf-4b3a-b44f-7d0707c20cfc-oauth-serving-cert\") pod \"b91f9084-96cf-4b3a-b44f-7d0707c20cfc\" (UID: \"b91f9084-96cf-4b3a-b44f-7d0707c20cfc\") "
Jul 28 10:16:24.034900 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:16:24.034775    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b91f9084-96cf-4b3a-b44f-7d0707c20cfc-console-serving-cert\") pod \"b91f9084-96cf-4b3a-b44f-7d0707c20cfc\" (UID: \"b91f9084-96cf-4b3a-b44f-7d0707c20cfc\") "
Jul 28 10:16:24.034900 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:16:24.034799    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9ckfh\" (UniqueName: \"kubernetes.io/projected/b91f9084-96cf-4b3a-b44f-7d0707c20cfc-kube-api-access-9ckfh\") pod \"b91f9084-96cf-4b3a-b44f-7d0707c20cfc\" (UID: \"b91f9084-96cf-4b3a-b44f-7d0707c20cfc\") "
Jul 28 10:16:24.034900 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:16:24.034855    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b91f9084-96cf-4b3a-b44f-7d0707c20cfc-console-oauth-config\") pod \"b91f9084-96cf-4b3a-b44f-7d0707c20cfc\" (UID: \"b91f9084-96cf-4b3a-b44f-7d0707c20cfc\") "
Jul 28 10:16:24.034900 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:16:24.034885    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b91f9084-96cf-4b3a-b44f-7d0707c20cfc-trusted-ca-bundle\") pod \"b91f9084-96cf-4b3a-b44f-7d0707c20cfc\" (UID: \"b91f9084-96cf-4b3a-b44f-7d0707c20cfc\") "
Jul 28 10:16:24.035155 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:16:24.034908    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b91f9084-96cf-4b3a-b44f-7d0707c20cfc-service-ca\") pod \"b91f9084-96cf-4b3a-b44f-7d0707c20cfc\" (UID: \"b91f9084-96cf-4b3a-b44f-7d0707c20cfc\") "
Jul 28 10:16:24.035155 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:16:24.035127    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b91f9084-96cf-4b3a-b44f-7d0707c20cfc-console-config" (OuterVolumeSpecName: "console-config") pod "b91f9084-96cf-4b3a-b44f-7d0707c20cfc" (UID: "b91f9084-96cf-4b3a-b44f-7d0707c20cfc"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:16:24.035259 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:16:24.035242    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b91f9084-96cf-4b3a-b44f-7d0707c20cfc-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b91f9084-96cf-4b3a-b44f-7d0707c20cfc" (UID: "b91f9084-96cf-4b3a-b44f-7d0707c20cfc"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:16:24.035416 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:16:24.035392    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b91f9084-96cf-4b3a-b44f-7d0707c20cfc-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b91f9084-96cf-4b3a-b44f-7d0707c20cfc" (UID: "b91f9084-96cf-4b3a-b44f-7d0707c20cfc"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:16:24.035472 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:16:24.035436    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b91f9084-96cf-4b3a-b44f-7d0707c20cfc-service-ca" (OuterVolumeSpecName: "service-ca") pod "b91f9084-96cf-4b3a-b44f-7d0707c20cfc" (UID: "b91f9084-96cf-4b3a-b44f-7d0707c20cfc"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:16:24.037035 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:16:24.037001    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b91f9084-96cf-4b3a-b44f-7d0707c20cfc-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b91f9084-96cf-4b3a-b44f-7d0707c20cfc" (UID: "b91f9084-96cf-4b3a-b44f-7d0707c20cfc"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:16:24.037035 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:16:24.037018    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b91f9084-96cf-4b3a-b44f-7d0707c20cfc-kube-api-access-9ckfh" (OuterVolumeSpecName: "kube-api-access-9ckfh") pod "b91f9084-96cf-4b3a-b44f-7d0707c20cfc" (UID: "b91f9084-96cf-4b3a-b44f-7d0707c20cfc"). InnerVolumeSpecName "kube-api-access-9ckfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:16:24.037179 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:16:24.037014    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b91f9084-96cf-4b3a-b44f-7d0707c20cfc-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b91f9084-96cf-4b3a-b44f-7d0707c20cfc" (UID: "b91f9084-96cf-4b3a-b44f-7d0707c20cfc"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:16:24.136674 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:16:24.136638    2557 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b91f9084-96cf-4b3a-b44f-7d0707c20cfc-console-oauth-config\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:16:24.136674 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:16:24.136666    2557 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b91f9084-96cf-4b3a-b44f-7d0707c20cfc-trusted-ca-bundle\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:16:24.136674 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:16:24.136675    2557 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b91f9084-96cf-4b3a-b44f-7d0707c20cfc-service-ca\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:16:24.136674 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:16:24.136683    2557 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b91f9084-96cf-4b3a-b44f-7d0707c20cfc-console-config\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:16:24.136931 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:16:24.136691    2557 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b91f9084-96cf-4b3a-b44f-7d0707c20cfc-oauth-serving-cert\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:16:24.136931 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:16:24.136699    2557 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b91f9084-96cf-4b3a-b44f-7d0707c20cfc-console-serving-cert\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:16:24.136931 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:16:24.136708    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ckfh\" (UniqueName: \"kubernetes.io/projected/b91f9084-96cf-4b3a-b44f-7d0707c20cfc-kube-api-access-9ckfh\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:16:24.718240 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:16:24.718197    2557 generic.go:356] "Generic (PLEG): container finished" podID="b91f9084-96cf-4b3a-b44f-7d0707c20cfc" containerID="da744b474e1943b6de7fcbec4f8f3f295f3b674eddc7d5373481fffed3a67b61" exitCode=0
Jul 28 10:16:24.718647 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:16:24.718261    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bc7759579-5w9mr"
Jul 28 10:16:24.718647 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:16:24.718285    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bc7759579-5w9mr" event={"ID":"b91f9084-96cf-4b3a-b44f-7d0707c20cfc","Type":"ContainerDied","Data":"da744b474e1943b6de7fcbec4f8f3f295f3b674eddc7d5373481fffed3a67b61"}
Jul 28 10:16:24.718647 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:16:24.718343    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bc7759579-5w9mr" event={"ID":"b91f9084-96cf-4b3a-b44f-7d0707c20cfc","Type":"ContainerDied","Data":"e4804b6602501e273512d8b7ad2eb0381da99d465538e8bef1ad83e8d9e949f6"}
Jul 28 10:16:24.718647 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:16:24.718365    2557 scope.go:117] "RemoveContainer" containerID="da744b474e1943b6de7fcbec4f8f3f295f3b674eddc7d5373481fffed3a67b61"
Jul 28 10:16:24.726748 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:16:24.726729    2557 scope.go:117] "RemoveContainer" containerID="da744b474e1943b6de7fcbec4f8f3f295f3b674eddc7d5373481fffed3a67b61"
Jul 28 10:16:24.727007 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:16:24.726988    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da744b474e1943b6de7fcbec4f8f3f295f3b674eddc7d5373481fffed3a67b61\": container with ID starting with da744b474e1943b6de7fcbec4f8f3f295f3b674eddc7d5373481fffed3a67b61 not found: ID does not exist" containerID="da744b474e1943b6de7fcbec4f8f3f295f3b674eddc7d5373481fffed3a67b61"
Jul 28 10:16:24.727071 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:16:24.727015    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da744b474e1943b6de7fcbec4f8f3f295f3b674eddc7d5373481fffed3a67b61"} err="failed to get container status \"da744b474e1943b6de7fcbec4f8f3f295f3b674eddc7d5373481fffed3a67b61\": rpc error: code = NotFound desc = could not find container \"da744b474e1943b6de7fcbec4f8f3f295f3b674eddc7d5373481fffed3a67b61\": container with ID starting with da744b474e1943b6de7fcbec4f8f3f295f3b674eddc7d5373481fffed3a67b61 not found: ID does not exist"
Jul 28 10:16:24.734274 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:16:24.734250    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bc7759579-5w9mr"]
Jul 28 10:16:24.737627 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:16:24.737588    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5bc7759579-5w9mr"]
Jul 28 10:16:26.579148 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:16:26.579118    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b91f9084-96cf-4b3a-b44f-7d0707c20cfc" path="/var/lib/kubelet/pods/b91f9084-96cf-4b3a-b44f-7d0707c20cfc/volumes"
Jul 28 10:17:07.019511 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:07.019479    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-2xbqs"]
Jul 28 10:17:07.019939 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:07.019804    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b91f9084-96cf-4b3a-b44f-7d0707c20cfc" containerName="console"
Jul 28 10:17:07.019939 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:07.019815    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b91f9084-96cf-4b3a-b44f-7d0707c20cfc" containerName="console"
Jul 28 10:17:07.019939 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:07.019868    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b91f9084-96cf-4b3a-b44f-7d0707c20cfc" containerName="console"
Jul 28 10:17:07.024030 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:07.024013    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-2xbqs"
Jul 28 10:17:07.026906 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:07.026885    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 28 10:17:07.026906 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:07.026900    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 28 10:17:07.027424 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:07.027409    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-fkfg5\""
Jul 28 10:17:07.032773 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:07.032750    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-2xbqs"]
Jul 28 10:17:07.100457 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:07.100422    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qpsv\" (UniqueName: \"kubernetes.io/projected/dbbca43c-746c-4f4e-9306-21b52b7fdd17-kube-api-access-2qpsv\") pod \"cert-manager-webhook-6486ffd997-2xbqs\" (UID: \"dbbca43c-746c-4f4e-9306-21b52b7fdd17\") " pod="cert-manager/cert-manager-webhook-6486ffd997-2xbqs"
Jul 28 10:17:07.100648 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:07.100466    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dbbca43c-746c-4f4e-9306-21b52b7fdd17-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-2xbqs\" (UID: \"dbbca43c-746c-4f4e-9306-21b52b7fdd17\") " pod="cert-manager/cert-manager-webhook-6486ffd997-2xbqs"
Jul 28 10:17:07.201383 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:07.201341    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qpsv\" (UniqueName: \"kubernetes.io/projected/dbbca43c-746c-4f4e-9306-21b52b7fdd17-kube-api-access-2qpsv\") pod \"cert-manager-webhook-6486ffd997-2xbqs\" (UID: \"dbbca43c-746c-4f4e-9306-21b52b7fdd17\") " pod="cert-manager/cert-manager-webhook-6486ffd997-2xbqs"
Jul 28 10:17:07.201542 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:07.201392    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dbbca43c-746c-4f4e-9306-21b52b7fdd17-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-2xbqs\" (UID: \"dbbca43c-746c-4f4e-9306-21b52b7fdd17\") " pod="cert-manager/cert-manager-webhook-6486ffd997-2xbqs"
Jul 28 10:17:07.216669 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:07.216637    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qpsv\" (UniqueName: \"kubernetes.io/projected/dbbca43c-746c-4f4e-9306-21b52b7fdd17-kube-api-access-2qpsv\") pod \"cert-manager-webhook-6486ffd997-2xbqs\" (UID: \"dbbca43c-746c-4f4e-9306-21b52b7fdd17\") " pod="cert-manager/cert-manager-webhook-6486ffd997-2xbqs"
Jul 28 10:17:07.217667 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:07.217644    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dbbca43c-746c-4f4e-9306-21b52b7fdd17-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-2xbqs\" (UID: \"dbbca43c-746c-4f4e-9306-21b52b7fdd17\") " pod="cert-manager/cert-manager-webhook-6486ffd997-2xbqs"
Jul 28 10:17:07.355063 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:07.354984    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-2xbqs"
Jul 28 10:17:07.515069 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:07.515045    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-2xbqs"]
Jul 28 10:17:07.517182 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:17:07.517154    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddbbca43c_746c_4f4e_9306_21b52b7fdd17.slice/crio-2acd44cfec4b9765918e47c89381239c1a3f3b2ab3abf55f19ab9e82e0b0c055 WatchSource:0}: Error finding container 2acd44cfec4b9765918e47c89381239c1a3f3b2ab3abf55f19ab9e82e0b0c055: Status 404 returned error can't find the container with id 2acd44cfec4b9765918e47c89381239c1a3f3b2ab3abf55f19ab9e82e0b0c055
Jul 28 10:17:07.519237 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:07.519221    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 10:17:07.884288 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:07.884251    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-2xbqs" event={"ID":"dbbca43c-746c-4f4e-9306-21b52b7fdd17","Type":"ContainerStarted","Data":"2acd44cfec4b9765918e47c89381239c1a3f3b2ab3abf55f19ab9e82e0b0c055"}
Jul 28 10:17:10.899201 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:10.899109    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-2xbqs" event={"ID":"dbbca43c-746c-4f4e-9306-21b52b7fdd17","Type":"ContainerStarted","Data":"2b1e3c27cdc8ccf34da986774fd238fc29134ea73bb9fb50c24b5274632872c9"}
Jul 28 10:17:10.899566 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:10.899242    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-2xbqs"
Jul 28 10:17:10.920284 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:10.920235    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-2xbqs" podStartSLOduration=1.814928726 podStartE2EDuration="4.920217894s" totalImagesPullingTime="3.105289168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:17:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:17:10.919154437 +0000 UTC m=+566.877922820" watchObservedRunningTime="2026-07-28 10:17:10.920217894 +0000 UTC m=+566.878986272"
Jul 28 10:17:13.810939 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:13.810903    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-chwv8"]
Jul 28 10:17:13.814191 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:13.814175    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-chwv8"
Jul 28 10:17:13.816230 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:13.816207    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-dpw4q\""
Jul 28 10:17:13.821654 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:13.821629    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-chwv8"]
Jul 28 10:17:13.964592 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:13.964550    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a8097167-50b8-4d05-9a2f-11da1ad8c384-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-chwv8\" (UID: \"a8097167-50b8-4d05-9a2f-11da1ad8c384\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-chwv8"
Jul 28 10:17:13.964807 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:13.964675    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdp2s\" (UniqueName: \"kubernetes.io/projected/a8097167-50b8-4d05-9a2f-11da1ad8c384-kube-api-access-fdp2s\") pod \"cert-manager-cainjector-559cdd8c87-chwv8\" (UID: \"a8097167-50b8-4d05-9a2f-11da1ad8c384\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-chwv8"
Jul 28 10:17:14.066009 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:14.065912    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdp2s\" (UniqueName: \"kubernetes.io/projected/a8097167-50b8-4d05-9a2f-11da1ad8c384-kube-api-access-fdp2s\") pod \"cert-manager-cainjector-559cdd8c87-chwv8\" (UID: \"a8097167-50b8-4d05-9a2f-11da1ad8c384\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-chwv8"
Jul 28 10:17:14.066009 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:14.065977    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a8097167-50b8-4d05-9a2f-11da1ad8c384-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-chwv8\" (UID: \"a8097167-50b8-4d05-9a2f-11da1ad8c384\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-chwv8"
Jul 28 10:17:14.073934 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:14.073900    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a8097167-50b8-4d05-9a2f-11da1ad8c384-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-chwv8\" (UID: \"a8097167-50b8-4d05-9a2f-11da1ad8c384\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-chwv8"
Jul 28 10:17:14.074177 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:14.074155    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdp2s\" (UniqueName: \"kubernetes.io/projected/a8097167-50b8-4d05-9a2f-11da1ad8c384-kube-api-access-fdp2s\") pod \"cert-manager-cainjector-559cdd8c87-chwv8\" (UID: \"a8097167-50b8-4d05-9a2f-11da1ad8c384\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-chwv8"
Jul 28 10:17:14.124118 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:14.124070    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-chwv8"
Jul 28 10:17:14.250649 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:14.250624    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-chwv8"]
Jul 28 10:17:14.253096 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:17:14.253068    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda8097167_50b8_4d05_9a2f_11da1ad8c384.slice/crio-c8e7d01cb0cdb4561a009be1ee5a7fe095e95bc2f82e072bec0515efae9653cf WatchSource:0}: Error finding container c8e7d01cb0cdb4561a009be1ee5a7fe095e95bc2f82e072bec0515efae9653cf: Status 404 returned error can't find the container with id c8e7d01cb0cdb4561a009be1ee5a7fe095e95bc2f82e072bec0515efae9653cf
Jul 28 10:17:14.917018 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:14.916977    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-chwv8" event={"ID":"a8097167-50b8-4d05-9a2f-11da1ad8c384","Type":"ContainerStarted","Data":"cdecb86dace9fed2cee69826c127f8a114a690746cbdeca6379f39cced3780be"}
Jul 28 10:17:14.917018 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:14.917021    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-chwv8" event={"ID":"a8097167-50b8-4d05-9a2f-11da1ad8c384","Type":"ContainerStarted","Data":"c8e7d01cb0cdb4561a009be1ee5a7fe095e95bc2f82e072bec0515efae9653cf"}
Jul 28 10:17:14.933392 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:14.933336    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-chwv8" podStartSLOduration=1.933320727 podStartE2EDuration="1.933320727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:17:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:17:14.932320987 +0000 UTC m=+570.891089366" watchObservedRunningTime="2026-07-28 10:17:14.933320727 +0000 UTC m=+570.892089106"
Jul 28 10:17:16.906728 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:16.906693    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-2xbqs"
Jul 28 10:17:19.693439 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:19.693396    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-wxn4j"]
Jul 28 10:17:19.696836 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:19.696813    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-wxn4j"
Jul 28 10:17:19.698840 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:19.698822    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-jbdjw\""
Jul 28 10:17:19.704221 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:19.704148    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-wxn4j"]
Jul 28 10:17:19.816961 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:19.816920    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b6abb7fd-2032-4108-8e6a-2c5332245c39-bound-sa-token\") pod \"cert-manager-55b4d954d5-wxn4j\" (UID: \"b6abb7fd-2032-4108-8e6a-2c5332245c39\") " pod="cert-manager/cert-manager-55b4d954d5-wxn4j"
Jul 28 10:17:19.817141 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:19.817029    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kqzz\" (UniqueName: \"kubernetes.io/projected/b6abb7fd-2032-4108-8e6a-2c5332245c39-kube-api-access-8kqzz\") pod \"cert-manager-55b4d954d5-wxn4j\" (UID: \"b6abb7fd-2032-4108-8e6a-2c5332245c39\") " pod="cert-manager/cert-manager-55b4d954d5-wxn4j"
Jul 28 10:17:19.917863 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:19.917820    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b6abb7fd-2032-4108-8e6a-2c5332245c39-bound-sa-token\") pod \"cert-manager-55b4d954d5-wxn4j\" (UID: \"b6abb7fd-2032-4108-8e6a-2c5332245c39\") " pod="cert-manager/cert-manager-55b4d954d5-wxn4j"
Jul 28 10:17:19.918044 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:19.917883    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kqzz\" (UniqueName: \"kubernetes.io/projected/b6abb7fd-2032-4108-8e6a-2c5332245c39-kube-api-access-8kqzz\") pod \"cert-manager-55b4d954d5-wxn4j\" (UID: \"b6abb7fd-2032-4108-8e6a-2c5332245c39\") " pod="cert-manager/cert-manager-55b4d954d5-wxn4j"
Jul 28 10:17:19.926131 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:19.926106    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b6abb7fd-2032-4108-8e6a-2c5332245c39-bound-sa-token\") pod \"cert-manager-55b4d954d5-wxn4j\" (UID: \"b6abb7fd-2032-4108-8e6a-2c5332245c39\") " pod="cert-manager/cert-manager-55b4d954d5-wxn4j"
Jul 28 10:17:19.926255 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:19.926148    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kqzz\" (UniqueName: \"kubernetes.io/projected/b6abb7fd-2032-4108-8e6a-2c5332245c39-kube-api-access-8kqzz\") pod \"cert-manager-55b4d954d5-wxn4j\" (UID: \"b6abb7fd-2032-4108-8e6a-2c5332245c39\") " pod="cert-manager/cert-manager-55b4d954d5-wxn4j"
Jul 28 10:17:20.007726 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:20.007697    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-wxn4j"
Jul 28 10:17:20.167075 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:17:20.167043    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb6abb7fd_2032_4108_8e6a_2c5332245c39.slice/crio-cf8cbcdb398a95121e27c8e7513d36663d2e4b12ad5e26467fb0ae42ec4def0f WatchSource:0}: Error finding container cf8cbcdb398a95121e27c8e7513d36663d2e4b12ad5e26467fb0ae42ec4def0f: Status 404 returned error can't find the container with id cf8cbcdb398a95121e27c8e7513d36663d2e4b12ad5e26467fb0ae42ec4def0f
Jul 28 10:17:20.168176 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:20.168085    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-wxn4j"]
Jul 28 10:17:20.946069 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:20.946032    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-wxn4j" event={"ID":"b6abb7fd-2032-4108-8e6a-2c5332245c39","Type":"ContainerStarted","Data":"0686d9afa15e916d28765e21cffb6a776f0131e095decb8fca4d46c3df6b49e9"}
Jul 28 10:17:20.946069 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:20.946077    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-wxn4j" event={"ID":"b6abb7fd-2032-4108-8e6a-2c5332245c39","Type":"ContainerStarted","Data":"cf8cbcdb398a95121e27c8e7513d36663d2e4b12ad5e26467fb0ae42ec4def0f"}
Jul 28 10:17:20.963450 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:20.963395    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-wxn4j" podStartSLOduration=1.963378363 podStartE2EDuration="1.963378363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:17:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:17:20.962036895 +0000 UTC m=+576.920805287" watchObservedRunningTime="2026-07-28 10:17:20.963378363 +0000 UTC m=+576.922146742"
Jul 28 10:17:46.697114 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:46.697082    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-68cc75d5fc-8886b"]
Jul 28 10:17:46.699414 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:46.699392    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-68cc75d5fc-8886b"
Jul 28 10:17:46.703501 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:46.703470    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 28 10:17:46.703692 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:46.703674    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 28 10:17:46.704013 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:46.703995    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-n2wjv\""
Jul 28 10:17:46.704013 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:46.704011    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 28 10:17:46.717519 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:46.717495    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 28 10:17:46.717878 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:46.717855    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 28 10:17:46.719738 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:46.719717    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-68cc75d5fc-8886b"]
Jul 28 10:17:46.748991 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:46.748958    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67rd4\" (UniqueName: \"kubernetes.io/projected/96f79162-8c39-4e0e-8c22-0513f8b708c7-kube-api-access-67rd4\") pod \"lws-controller-manager-68cc75d5fc-8886b\" (UID: \"96f79162-8c39-4e0e-8c22-0513f8b708c7\") " pod="openshift-lws-operator/lws-controller-manager-68cc75d5fc-8886b"
Jul 28 10:17:46.749143 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:46.748997    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/96f79162-8c39-4e0e-8c22-0513f8b708c7-manager-config\") pod \"lws-controller-manager-68cc75d5fc-8886b\" (UID: \"96f79162-8c39-4e0e-8c22-0513f8b708c7\") " pod="openshift-lws-operator/lws-controller-manager-68cc75d5fc-8886b"
Jul 28 10:17:46.749143 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:46.749026    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/96f79162-8c39-4e0e-8c22-0513f8b708c7-metrics-cert\") pod \"lws-controller-manager-68cc75d5fc-8886b\" (UID: \"96f79162-8c39-4e0e-8c22-0513f8b708c7\") " pod="openshift-lws-operator/lws-controller-manager-68cc75d5fc-8886b"
Jul 28 10:17:46.749143 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:46.749112    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/96f79162-8c39-4e0e-8c22-0513f8b708c7-cert\") pod \"lws-controller-manager-68cc75d5fc-8886b\" (UID: \"96f79162-8c39-4e0e-8c22-0513f8b708c7\") " pod="openshift-lws-operator/lws-controller-manager-68cc75d5fc-8886b"
Jul 28 10:17:46.850460 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:46.850406    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/96f79162-8c39-4e0e-8c22-0513f8b708c7-cert\") pod \"lws-controller-manager-68cc75d5fc-8886b\" (UID: \"96f79162-8c39-4e0e-8c22-0513f8b708c7\") " pod="openshift-lws-operator/lws-controller-manager-68cc75d5fc-8886b"
Jul 28 10:17:46.850680 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:46.850502    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67rd4\" (UniqueName: \"kubernetes.io/projected/96f79162-8c39-4e0e-8c22-0513f8b708c7-kube-api-access-67rd4\") pod \"lws-controller-manager-68cc75d5fc-8886b\" (UID: \"96f79162-8c39-4e0e-8c22-0513f8b708c7\") " pod="openshift-lws-operator/lws-controller-manager-68cc75d5fc-8886b"
Jul 28 10:17:46.850680 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:46.850643    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/96f79162-8c39-4e0e-8c22-0513f8b708c7-manager-config\") pod \"lws-controller-manager-68cc75d5fc-8886b\" (UID: \"96f79162-8c39-4e0e-8c22-0513f8b708c7\") " pod="openshift-lws-operator/lws-controller-manager-68cc75d5fc-8886b"
Jul 28 10:17:46.850848 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:46.850685    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/96f79162-8c39-4e0e-8c22-0513f8b708c7-metrics-cert\") pod \"lws-controller-manager-68cc75d5fc-8886b\" (UID: \"96f79162-8c39-4e0e-8c22-0513f8b708c7\") " pod="openshift-lws-operator/lws-controller-manager-68cc75d5fc-8886b"
Jul 28 10:17:46.851269 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:46.851251    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/96f79162-8c39-4e0e-8c22-0513f8b708c7-manager-config\") pod \"lws-controller-manager-68cc75d5fc-8886b\" (UID: \"96f79162-8c39-4e0e-8c22-0513f8b708c7\") " pod="openshift-lws-operator/lws-controller-manager-68cc75d5fc-8886b"
Jul 28 10:17:46.853021 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:46.852994    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/96f79162-8c39-4e0e-8c22-0513f8b708c7-cert\") pod \"lws-controller-manager-68cc75d5fc-8886b\" (UID: \"96f79162-8c39-4e0e-8c22-0513f8b708c7\") " pod="openshift-lws-operator/lws-controller-manager-68cc75d5fc-8886b"
Jul 28 10:17:46.853132 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:46.853115    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/96f79162-8c39-4e0e-8c22-0513f8b708c7-metrics-cert\") pod \"lws-controller-manager-68cc75d5fc-8886b\" (UID: \"96f79162-8c39-4e0e-8c22-0513f8b708c7\") " pod="openshift-lws-operator/lws-controller-manager-68cc75d5fc-8886b"
Jul 28 10:17:46.859592 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:46.859566    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67rd4\" (UniqueName: \"kubernetes.io/projected/96f79162-8c39-4e0e-8c22-0513f8b708c7-kube-api-access-67rd4\") pod \"lws-controller-manager-68cc75d5fc-8886b\" (UID: \"96f79162-8c39-4e0e-8c22-0513f8b708c7\") " pod="openshift-lws-operator/lws-controller-manager-68cc75d5fc-8886b"
Jul 28 10:17:47.008360 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:47.008317    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-68cc75d5fc-8886b"
Jul 28 10:17:47.175196 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:47.175117    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-68cc75d5fc-8886b"]
Jul 28 10:17:47.179337 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:17:47.179301    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96f79162_8c39_4e0e_8c22_0513f8b708c7.slice/crio-429e6b8babff8c935550e0113c7e6cf8435bcb9873d2a1b6d5b2ec228691e03d WatchSource:0}: Error finding container 429e6b8babff8c935550e0113c7e6cf8435bcb9873d2a1b6d5b2ec228691e03d: Status 404 returned error can't find the container with id 429e6b8babff8c935550e0113c7e6cf8435bcb9873d2a1b6d5b2ec228691e03d
Jul 28 10:17:48.054005 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:48.053962    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-68cc75d5fc-8886b" event={"ID":"96f79162-8c39-4e0e-8c22-0513f8b708c7","Type":"ContainerStarted","Data":"429e6b8babff8c935550e0113c7e6cf8435bcb9873d2a1b6d5b2ec228691e03d"}
Jul 28 10:17:50.065323 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:50.065285    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-68cc75d5fc-8886b" event={"ID":"96f79162-8c39-4e0e-8c22-0513f8b708c7","Type":"ContainerStarted","Data":"9b10bf653abf3d5407a510ab58d668ca5a5b39802f22925d0875176fca9da26f"}
Jul 28 10:17:50.065798 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:50.065433    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-68cc75d5fc-8886b"
Jul 28 10:17:50.082430 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:17:50.082372    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-68cc75d5fc-8886b" podStartSLOduration=1.680449473 podStartE2EDuration="4.082356074s" totalImagesPullingTime="2.401906601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:17:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:17:50.081362732 +0000 UTC m=+606.040131111" watchObservedRunningTime="2026-07-28 10:17:50.082356074 +0000 UTC m=+606.041124516"
Jul 28 10:18:01.072160 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:18:01.072130    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-68cc75d5fc-8886b"
Jul 28 10:18:17.157867 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:18:17.157781    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-774d94476f-5grlb"]
Jul 28 10:18:17.161333 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:18:17.161310    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-5grlb"
Jul 28 10:18:17.167251 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:18:17.167227    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 28 10:18:17.167796 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:18:17.167772    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-bl7h5\""
Jul 28 10:18:17.167906 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:18:17.167799    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 28 10:18:17.176132 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:18:17.176108    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-774d94476f-5grlb"]
Jul 28 10:18:17.220075 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:18:17.220029    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdtlq\" (UniqueName: \"kubernetes.io/projected/59142a1a-3fd6-4148-bad7-92e0b43f979f-kube-api-access-cdtlq\") pod \"limitador-operator-controller-manager-774d94476f-5grlb\" (UID: \"59142a1a-3fd6-4148-bad7-92e0b43f979f\") " pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-5grlb"
Jul 28 10:18:17.321406 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:18:17.321361    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdtlq\" (UniqueName: \"kubernetes.io/projected/59142a1a-3fd6-4148-bad7-92e0b43f979f-kube-api-access-cdtlq\") pod \"limitador-operator-controller-manager-774d94476f-5grlb\" (UID: \"59142a1a-3fd6-4148-bad7-92e0b43f979f\") " pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-5grlb"
Jul 28 10:18:17.335232 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:18:17.335194    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdtlq\" (UniqueName: \"kubernetes.io/projected/59142a1a-3fd6-4148-bad7-92e0b43f979f-kube-api-access-cdtlq\") pod \"limitador-operator-controller-manager-774d94476f-5grlb\" (UID: \"59142a1a-3fd6-4148-bad7-92e0b43f979f\") " pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-5grlb"
Jul 28 10:18:17.471530 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:18:17.471439    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-5grlb"
Jul 28 10:18:17.610590 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:18:17.610565    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-774d94476f-5grlb"]
Jul 28 10:18:17.613069 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:18:17.613044    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59142a1a_3fd6_4148_bad7_92e0b43f979f.slice/crio-cb0cbb19f6294c5cb2f81bc3e726c41bd437ee32132ec58a18af5c801de034a9 WatchSource:0}: Error finding container cb0cbb19f6294c5cb2f81bc3e726c41bd437ee32132ec58a18af5c801de034a9: Status 404 returned error can't find the container with id cb0cbb19f6294c5cb2f81bc3e726c41bd437ee32132ec58a18af5c801de034a9
Jul 28 10:18:18.172642 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:18:18.172588    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-5grlb" event={"ID":"59142a1a-3fd6-4148-bad7-92e0b43f979f","Type":"ContainerStarted","Data":"cb0cbb19f6294c5cb2f81bc3e726c41bd437ee32132ec58a18af5c801de034a9"}
Jul 28 10:18:20.183535 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:18:20.183496    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-5grlb" event={"ID":"59142a1a-3fd6-4148-bad7-92e0b43f979f","Type":"ContainerStarted","Data":"942c02002cb1fbe378807cf5275007c52758aefa4a63dc9c87df945288d6b17b"}
Jul 28 10:18:20.184052 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:18:20.183643    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-5grlb"
Jul 28 10:18:20.200623 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:18:20.200554    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-5grlb" podStartSLOduration=0.843674124 podStartE2EDuration="3.200538517s" totalImagesPullingTime="2.356864393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:18:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:18:20.199512532 +0000 UTC m=+636.158280912" watchObservedRunningTime="2026-07-28 10:18:20.200538517 +0000 UTC m=+636.159306896"
Jul 28 10:18:31.190871 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:18:31.190840    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-5grlb"
Jul 28 10:19:06.275680 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:19:06.275645    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c548b5d7d-5s2rs"]
Jul 28 10:19:06.279875 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:19:06.279854    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-5s2rs"
Jul 28 10:19:06.281896 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:19:06.281876    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-xc8zx\""
Jul 28 10:19:06.286273 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:19:06.286248    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-5s2rs"]
Jul 28 10:19:06.355342 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:19:06.355299    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z246\" (UniqueName: \"kubernetes.io/projected/2463b01e-bd74-4813-a7ed-bfcae404dbdc-kube-api-access-7z246\") pod \"authorino-c548b5d7d-5s2rs\" (UID: \"2463b01e-bd74-4813-a7ed-bfcae404dbdc\") " pod="kuadrant-system/authorino-c548b5d7d-5s2rs"
Jul 28 10:19:06.456681 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:19:06.456624    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z246\" (UniqueName: \"kubernetes.io/projected/2463b01e-bd74-4813-a7ed-bfcae404dbdc-kube-api-access-7z246\") pod \"authorino-c548b5d7d-5s2rs\" (UID: \"2463b01e-bd74-4813-a7ed-bfcae404dbdc\") " pod="kuadrant-system/authorino-c548b5d7d-5s2rs"
Jul 28 10:19:06.468455 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:19:06.468424    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z246\" (UniqueName: \"kubernetes.io/projected/2463b01e-bd74-4813-a7ed-bfcae404dbdc-kube-api-access-7z246\") pod \"authorino-c548b5d7d-5s2rs\" (UID: \"2463b01e-bd74-4813-a7ed-bfcae404dbdc\") " pod="kuadrant-system/authorino-c548b5d7d-5s2rs"
Jul 28 10:19:06.589717 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:19:06.589643    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-5s2rs"
Jul 28 10:19:06.717973 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:19:06.717806    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-5s2rs"]
Jul 28 10:19:06.720681 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:19:06.720648    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2463b01e_bd74_4813_a7ed_bfcae404dbdc.slice/crio-1d0f984e286e92e824f1dc6b68c5be7e75a4303fa7a3aca68cab5a187f121d14 WatchSource:0}: Error finding container 1d0f984e286e92e824f1dc6b68c5be7e75a4303fa7a3aca68cab5a187f121d14: Status 404 returned error can't find the container with id 1d0f984e286e92e824f1dc6b68c5be7e75a4303fa7a3aca68cab5a187f121d14
Jul 28 10:19:07.385541 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:19:07.385505    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-5s2rs" event={"ID":"2463b01e-bd74-4813-a7ed-bfcae404dbdc","Type":"ContainerStarted","Data":"1d0f984e286e92e824f1dc6b68c5be7e75a4303fa7a3aca68cab5a187f121d14"}
Jul 28 10:19:09.397612 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:19:09.397566    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-5s2rs" event={"ID":"2463b01e-bd74-4813-a7ed-bfcae404dbdc","Type":"ContainerStarted","Data":"d1621ce6992a1bfbe3acc58f99b850fec007814c2d46df8d690288eddc21e939"}
Jul 28 10:19:09.414141 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:19:09.414087    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c548b5d7d-5s2rs" podStartSLOduration=1.238472171 podStartE2EDuration="3.414069206s" totalImagesPullingTime="2.175597035s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:19:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:19:09.412297556 +0000 UTC m=+685.371065934" watchObservedRunningTime="2026-07-28 10:19:09.414069206 +0000 UTC m=+685.372837584"
Jul 28 10:19:31.719797 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:19:31.719765    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7cf4765b85-5xmkm"]
Jul 28 10:19:31.723237 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:19:31.723219    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cf4765b85-5xmkm"
Jul 28 10:19:31.725701 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:19:31.725677    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 28 10:19:31.730711 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:19:31.730691    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cf4765b85-5xmkm"]
Jul 28 10:19:31.885702 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:19:31.885657    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/676a3b2e-543d-47a4-8805-fea3a7ae14a7-tls-cert\") pod \"authorino-7cf4765b85-5xmkm\" (UID: \"676a3b2e-543d-47a4-8805-fea3a7ae14a7\") " pod="kuadrant-system/authorino-7cf4765b85-5xmkm"
Jul 28 10:19:31.885898 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:19:31.885746    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcbzn\" (UniqueName: \"kubernetes.io/projected/676a3b2e-543d-47a4-8805-fea3a7ae14a7-kube-api-access-pcbzn\") pod \"authorino-7cf4765b85-5xmkm\" (UID: \"676a3b2e-543d-47a4-8805-fea3a7ae14a7\") " pod="kuadrant-system/authorino-7cf4765b85-5xmkm"
Jul 28 10:19:31.986459 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:19:31.986371    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcbzn\" (UniqueName: \"kubernetes.io/projected/676a3b2e-543d-47a4-8805-fea3a7ae14a7-kube-api-access-pcbzn\") pod \"authorino-7cf4765b85-5xmkm\" (UID: \"676a3b2e-543d-47a4-8805-fea3a7ae14a7\") " pod="kuadrant-system/authorino-7cf4765b85-5xmkm"
Jul 28 10:19:31.986649 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:19:31.986541    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/676a3b2e-543d-47a4-8805-fea3a7ae14a7-tls-cert\") pod \"authorino-7cf4765b85-5xmkm\" (UID: \"676a3b2e-543d-47a4-8805-fea3a7ae14a7\") " pod="kuadrant-system/authorino-7cf4765b85-5xmkm"
Jul 28 10:19:31.988920 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:19:31.988900    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/676a3b2e-543d-47a4-8805-fea3a7ae14a7-tls-cert\") pod \"authorino-7cf4765b85-5xmkm\" (UID: \"676a3b2e-543d-47a4-8805-fea3a7ae14a7\") " pod="kuadrant-system/authorino-7cf4765b85-5xmkm"
Jul 28 10:19:31.994388 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:19:31.994368    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcbzn\" (UniqueName: \"kubernetes.io/projected/676a3b2e-543d-47a4-8805-fea3a7ae14a7-kube-api-access-pcbzn\") pod \"authorino-7cf4765b85-5xmkm\" (UID: \"676a3b2e-543d-47a4-8805-fea3a7ae14a7\") " pod="kuadrant-system/authorino-7cf4765b85-5xmkm"
Jul 28 10:19:32.033431 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:19:32.033378    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cf4765b85-5xmkm"
Jul 28 10:19:32.166546 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:19:32.166520    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cf4765b85-5xmkm"]
Jul 28 10:19:32.169262 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:19:32.169233    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod676a3b2e_543d_47a4_8805_fea3a7ae14a7.slice/crio-f50724cbbf671cf517b9beb85cdea4f94e027561c06f4e7a582c45dd3bde438e WatchSource:0}: Error finding container f50724cbbf671cf517b9beb85cdea4f94e027561c06f4e7a582c45dd3bde438e: Status 404 returned error can't find the container with id f50724cbbf671cf517b9beb85cdea4f94e027561c06f4e7a582c45dd3bde438e
Jul 28 10:19:32.504662 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:19:32.504590    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cf4765b85-5xmkm" event={"ID":"676a3b2e-543d-47a4-8805-fea3a7ae14a7","Type":"ContainerStarted","Data":"f50724cbbf671cf517b9beb85cdea4f94e027561c06f4e7a582c45dd3bde438e"}
Jul 28 10:19:33.516511 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:19:33.516471    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cf4765b85-5xmkm" event={"ID":"676a3b2e-543d-47a4-8805-fea3a7ae14a7","Type":"ContainerStarted","Data":"cb05a207ba32ccb928c538ac3c00f5c1f9804c670a5a60e8ca81782699c4428a"}
Jul 28 10:19:33.531128 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:19:33.531071    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7cf4765b85-5xmkm" podStartSLOduration=2.058196005 podStartE2EDuration="2.531055198s" totalImagesPullingTime="472.859193ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:19:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:19:33.530576793 +0000 UTC m=+709.489345173" watchObservedRunningTime="2026-07-28 10:19:33.531055198 +0000 UTC m=+709.489823578"
Jul 28 10:19:33.554214 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:19:33.554175    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-5s2rs"]
Jul 28 10:19:33.554399 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:19:33.554375    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c548b5d7d-5s2rs" podUID="2463b01e-bd74-4813-a7ed-bfcae404dbdc" containerName="authorino" containerID="cri-o://d1621ce6992a1bfbe3acc58f99b850fec007814c2d46df8d690288eddc21e939" gracePeriod=30
Jul 28 10:19:33.808560 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:19:33.808524    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-5s2rs"
Jul 28 10:19:33.814104 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:19:33.814079    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7z246\" (UniqueName: \"kubernetes.io/projected/2463b01e-bd74-4813-a7ed-bfcae404dbdc-kube-api-access-7z246\") pod \"2463b01e-bd74-4813-a7ed-bfcae404dbdc\" (UID: \"2463b01e-bd74-4813-a7ed-bfcae404dbdc\") "
Jul 28 10:19:33.816225 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:19:33.816196    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2463b01e-bd74-4813-a7ed-bfcae404dbdc-kube-api-access-7z246" (OuterVolumeSpecName: "kube-api-access-7z246") pod "2463b01e-bd74-4813-a7ed-bfcae404dbdc" (UID: "2463b01e-bd74-4813-a7ed-bfcae404dbdc"). InnerVolumeSpecName "kube-api-access-7z246". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:19:33.914947 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:19:33.914911    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7z246\" (UniqueName: \"kubernetes.io/projected/2463b01e-bd74-4813-a7ed-bfcae404dbdc-kube-api-access-7z246\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:19:34.522454 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:19:34.522423    2557 generic.go:356] "Generic (PLEG): container finished" podID="2463b01e-bd74-4813-a7ed-bfcae404dbdc" containerID="d1621ce6992a1bfbe3acc58f99b850fec007814c2d46df8d690288eddc21e939" exitCode=0
Jul 28 10:19:34.522913 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:19:34.522462    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-5s2rs"
Jul 28 10:19:34.522913 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:19:34.522516    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-5s2rs" event={"ID":"2463b01e-bd74-4813-a7ed-bfcae404dbdc","Type":"ContainerDied","Data":"d1621ce6992a1bfbe3acc58f99b850fec007814c2d46df8d690288eddc21e939"}
Jul 28 10:19:34.522913 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:19:34.522567    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-5s2rs" event={"ID":"2463b01e-bd74-4813-a7ed-bfcae404dbdc","Type":"ContainerDied","Data":"1d0f984e286e92e824f1dc6b68c5be7e75a4303fa7a3aca68cab5a187f121d14"}
Jul 28 10:19:34.522913 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:19:34.522588    2557 scope.go:117] "RemoveContainer" containerID="d1621ce6992a1bfbe3acc58f99b850fec007814c2d46df8d690288eddc21e939"
Jul 28 10:19:34.531504 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:19:34.531476    2557 scope.go:117] "RemoveContainer" containerID="d1621ce6992a1bfbe3acc58f99b850fec007814c2d46df8d690288eddc21e939"
Jul 28 10:19:34.531809 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:19:34.531785    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1621ce6992a1bfbe3acc58f99b850fec007814c2d46df8d690288eddc21e939\": container with ID starting with d1621ce6992a1bfbe3acc58f99b850fec007814c2d46df8d690288eddc21e939 not found: ID does not exist" containerID="d1621ce6992a1bfbe3acc58f99b850fec007814c2d46df8d690288eddc21e939"
Jul 28 10:19:34.531886 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:19:34.531819    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1621ce6992a1bfbe3acc58f99b850fec007814c2d46df8d690288eddc21e939"} err="failed to get container status \"d1621ce6992a1bfbe3acc58f99b850fec007814c2d46df8d690288eddc21e939\": rpc error: code = NotFound desc = could not find container \"d1621ce6992a1bfbe3acc58f99b850fec007814c2d46df8d690288eddc21e939\": container with ID starting with d1621ce6992a1bfbe3acc58f99b850fec007814c2d46df8d690288eddc21e939 not found: ID does not exist"
Jul 28 10:19:34.543295 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:19:34.543234    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-5s2rs"]
Jul 28 10:19:34.544762 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:19:34.544743    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-5s2rs"]
Jul 28 10:19:34.579870 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:19:34.579841    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2463b01e-bd74-4813-a7ed-bfcae404dbdc" path="/var/lib/kubelet/pods/2463b01e-bd74-4813-a7ed-bfcae404dbdc/volumes"
Jul 28 10:22:17.285230 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:22:17.285195    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v5snb"]
Jul 28 10:22:17.285709 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:22:17.285514    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2463b01e-bd74-4813-a7ed-bfcae404dbdc" containerName="authorino"
Jul 28 10:22:17.285709 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:22:17.285525    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="2463b01e-bd74-4813-a7ed-bfcae404dbdc" containerName="authorino"
Jul 28 10:22:17.285709 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:22:17.285570    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="2463b01e-bd74-4813-a7ed-bfcae404dbdc" containerName="authorino"
Jul 28 10:22:17.288514 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:22:17.288489    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v5snb"
Jul 28 10:22:17.290809 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:22:17.290788    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Jul 28 10:22:17.290920 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:22:17.290863    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-m68p6\""
Jul 28 10:22:17.291498 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:22:17.291481    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Jul 28 10:22:17.291619 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:22:17.291557    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Jul 28 10:22:17.298239 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:22:17.298216    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v5snb"]
Jul 28 10:22:17.347092 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:22:17.347053    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bcc75e5-167c-4ee2-878e-2ded3b70035f-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-v5snb\" (UID: \"2bcc75e5-167c-4ee2-878e-2ded3b70035f\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v5snb"
Jul 28 10:22:17.347092 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:22:17.347090    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bcc75e5-167c-4ee2-878e-2ded3b70035f-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-v5snb\" (UID: \"2bcc75e5-167c-4ee2-878e-2ded3b70035f\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v5snb"
Jul 28 10:22:17.347322 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:22:17.347109    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x2fl\" (UniqueName: \"kubernetes.io/projected/2bcc75e5-167c-4ee2-878e-2ded3b70035f-kube-api-access-2x2fl\") pod \"fc-smoke-test-kserve-7f4687897-v5snb\" (UID: \"2bcc75e5-167c-4ee2-878e-2ded3b70035f\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v5snb"
Jul 28 10:22:17.347322 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:22:17.347137    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bcc75e5-167c-4ee2-878e-2ded3b70035f-dshm\") pod \"fc-smoke-test-kserve-7f4687897-v5snb\" (UID: \"2bcc75e5-167c-4ee2-878e-2ded3b70035f\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v5snb"
Jul 28 10:22:17.347322 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:22:17.347203    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bcc75e5-167c-4ee2-878e-2ded3b70035f-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-v5snb\" (UID: \"2bcc75e5-167c-4ee2-878e-2ded3b70035f\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v5snb"
Jul 28 10:22:17.347322 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:22:17.347289    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bcc75e5-167c-4ee2-878e-2ded3b70035f-home\") pod \"fc-smoke-test-kserve-7f4687897-v5snb\" (UID: \"2bcc75e5-167c-4ee2-878e-2ded3b70035f\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v5snb"
Jul 28 10:22:17.448006 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:22:17.447966    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bcc75e5-167c-4ee2-878e-2ded3b70035f-home\") pod \"fc-smoke-test-kserve-7f4687897-v5snb\" (UID: \"2bcc75e5-167c-4ee2-878e-2ded3b70035f\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v5snb"
Jul 28 10:22:17.448194 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:22:17.448031    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bcc75e5-167c-4ee2-878e-2ded3b70035f-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-v5snb\" (UID: \"2bcc75e5-167c-4ee2-878e-2ded3b70035f\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v5snb"
Jul 28 10:22:17.448194 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:22:17.448054    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bcc75e5-167c-4ee2-878e-2ded3b70035f-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-v5snb\" (UID: \"2bcc75e5-167c-4ee2-878e-2ded3b70035f\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v5snb"
Jul 28 10:22:17.448194 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:22:17.448076    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x2fl\" (UniqueName: \"kubernetes.io/projected/2bcc75e5-167c-4ee2-878e-2ded3b70035f-kube-api-access-2x2fl\") pod \"fc-smoke-test-kserve-7f4687897-v5snb\" (UID: \"2bcc75e5-167c-4ee2-878e-2ded3b70035f\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v5snb"
Jul 28 10:22:17.448194 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:22:17.448111    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bcc75e5-167c-4ee2-878e-2ded3b70035f-dshm\") pod \"fc-smoke-test-kserve-7f4687897-v5snb\" (UID: \"2bcc75e5-167c-4ee2-878e-2ded3b70035f\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v5snb"
Jul 28 10:22:17.448194 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:22:17.448144    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bcc75e5-167c-4ee2-878e-2ded3b70035f-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-v5snb\" (UID: \"2bcc75e5-167c-4ee2-878e-2ded3b70035f\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v5snb"
Jul 28 10:22:17.448452 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:22:17.448432    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bcc75e5-167c-4ee2-878e-2ded3b70035f-home\") pod \"fc-smoke-test-kserve-7f4687897-v5snb\" (UID: \"2bcc75e5-167c-4ee2-878e-2ded3b70035f\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v5snb"
Jul 28 10:22:17.448510 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:22:17.448488    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bcc75e5-167c-4ee2-878e-2ded3b70035f-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-v5snb\" (UID: \"2bcc75e5-167c-4ee2-878e-2ded3b70035f\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v5snb"
Jul 28 10:22:17.448580 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:22:17.448559    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bcc75e5-167c-4ee2-878e-2ded3b70035f-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-v5snb\" (UID: \"2bcc75e5-167c-4ee2-878e-2ded3b70035f\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v5snb"
Jul 28 10:22:17.450352 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:22:17.450331    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bcc75e5-167c-4ee2-878e-2ded3b70035f-dshm\") pod \"fc-smoke-test-kserve-7f4687897-v5snb\" (UID: \"2bcc75e5-167c-4ee2-878e-2ded3b70035f\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v5snb"
Jul 28 10:22:17.450595 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:22:17.450580    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bcc75e5-167c-4ee2-878e-2ded3b70035f-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-v5snb\" (UID: \"2bcc75e5-167c-4ee2-878e-2ded3b70035f\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v5snb"
Jul 28 10:22:17.459136 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:22:17.459106    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x2fl\" (UniqueName: \"kubernetes.io/projected/2bcc75e5-167c-4ee2-878e-2ded3b70035f-kube-api-access-2x2fl\") pod \"fc-smoke-test-kserve-7f4687897-v5snb\" (UID: \"2bcc75e5-167c-4ee2-878e-2ded3b70035f\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v5snb"
Jul 28 10:22:17.600867 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:22:17.600759    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v5snb"
Jul 28 10:22:17.735480 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:22:17.735455    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v5snb"]
Jul 28 10:22:17.737345 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:22:17.737318    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bcc75e5_167c_4ee2_878e_2ded3b70035f.slice/crio-c1aecbbc5b9cbd77ab6928bb2e35f6d4e1a446e1ca588d16aee6ade8d256ae99 WatchSource:0}: Error finding container c1aecbbc5b9cbd77ab6928bb2e35f6d4e1a446e1ca588d16aee6ade8d256ae99: Status 404 returned error can't find the container with id c1aecbbc5b9cbd77ab6928bb2e35f6d4e1a446e1ca588d16aee6ade8d256ae99
Jul 28 10:22:17.739138 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:22:17.739122    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 10:22:18.239016 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:22:18.238972    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v5snb" event={"ID":"2bcc75e5-167c-4ee2-878e-2ded3b70035f","Type":"ContainerStarted","Data":"c1aecbbc5b9cbd77ab6928bb2e35f6d4e1a446e1ca588d16aee6ade8d256ae99"}
Jul 28 10:22:20.252663 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:22:20.252626    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v5snb" event={"ID":"2bcc75e5-167c-4ee2-878e-2ded3b70035f","Type":"ContainerStarted","Data":"d24851a115b449a2f2f3fc2d8df32494fd279acd46ed747736b35f10e1a40131"}
Jul 28 10:22:20.270213 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:22:20.270159    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v5snb" podStartSLOduration=1.5055273439999999 podStartE2EDuration="3.270143702s" totalImagesPullingTime="1.764616358s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:22:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:22:20.26836389 +0000 UTC m=+876.227132270" watchObservedRunningTime="2026-07-28 10:22:20.270143702 +0000 UTC m=+876.228912080"
Jul 28 10:22:20.601308 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:22:20.601222    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v5snb"
Jul 28 10:22:27.601441 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:22:27.601400    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v5snb"
Jul 28 10:22:27.614248 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:22:27.614223    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v5snb"
Jul 28 10:22:28.298665 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:22:28.298636    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v5snb"
Jul 28 10:23:06.243176 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:06.243082    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v5snb"]
Jul 28 10:23:06.253266 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:23:06.253234    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 10:23:06.253266 ip-10-0-131-88 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:23:06.253266 ip-10-0-131-88 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 28 10:23:06.253266 ip-10-0-131-88 kubenswrapper[2557]:  > containerID="d24851a115b449a2f2f3fc2d8df32494fd279acd46ed747736b35f10e1a40131" cmd=["/bin/sleep","15"]
Jul 28 10:23:06.253509 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:06.253285    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v5snb" message="" err=<
Jul 28 10:23:06.253509 ip-10-0-131-88 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:23:06.253509 ip-10-0-131-88 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 28 10:23:06.253509 ip-10-0-131-88 kubenswrapper[2557]:  >
Jul 28 10:23:06.253509 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:23:06.253297    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 10:23:06.253509 ip-10-0-131-88 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:23:06.253509 ip-10-0-131-88 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 28 10:23:06.253509 ip-10-0-131-88 kubenswrapper[2557]:  > pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v5snb" podUID="2bcc75e5-167c-4ee2-878e-2ded3b70035f" containerName="main" containerID="cri-o://d24851a115b449a2f2f3fc2d8df32494fd279acd46ed747736b35f10e1a40131"
Jul 28 10:23:06.253509 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:06.253330    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v5snb" podUID="2bcc75e5-167c-4ee2-878e-2ded3b70035f" containerName="main" containerID="cri-o://d24851a115b449a2f2f3fc2d8df32494fd279acd46ed747736b35f10e1a40131" gracePeriod=60
Jul 28 10:23:06.288453 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:06.288396    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v5snb" podUID="2bcc75e5-167c-4ee2-878e-2ded3b70035f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": dial tcp 10.133.0.32:8000: connect: connection refused"
Jul 28 10:23:06.606585 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:06.606556    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v5snb"
Jul 28 10:23:06.679550 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:06.679514    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bcc75e5-167c-4ee2-878e-2ded3b70035f-home\") pod \"2bcc75e5-167c-4ee2-878e-2ded3b70035f\" (UID: \"2bcc75e5-167c-4ee2-878e-2ded3b70035f\") "
Jul 28 10:23:06.679755 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:06.679572    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bcc75e5-167c-4ee2-878e-2ded3b70035f-tmp-dir\") pod \"2bcc75e5-167c-4ee2-878e-2ded3b70035f\" (UID: \"2bcc75e5-167c-4ee2-878e-2ded3b70035f\") "
Jul 28 10:23:06.679755 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:06.679620    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bcc75e5-167c-4ee2-878e-2ded3b70035f-tls-certs\") pod \"2bcc75e5-167c-4ee2-878e-2ded3b70035f\" (UID: \"2bcc75e5-167c-4ee2-878e-2ded3b70035f\") "
Jul 28 10:23:06.679755 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:06.679659    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2x2fl\" (UniqueName: \"kubernetes.io/projected/2bcc75e5-167c-4ee2-878e-2ded3b70035f-kube-api-access-2x2fl\") pod \"2bcc75e5-167c-4ee2-878e-2ded3b70035f\" (UID: \"2bcc75e5-167c-4ee2-878e-2ded3b70035f\") "
Jul 28 10:23:06.679755 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:06.679689    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bcc75e5-167c-4ee2-878e-2ded3b70035f-model-cache\") pod \"2bcc75e5-167c-4ee2-878e-2ded3b70035f\" (UID: \"2bcc75e5-167c-4ee2-878e-2ded3b70035f\") "
Jul 28 10:23:06.679755 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:06.679733    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bcc75e5-167c-4ee2-878e-2ded3b70035f-dshm\") pod \"2bcc75e5-167c-4ee2-878e-2ded3b70035f\" (UID: \"2bcc75e5-167c-4ee2-878e-2ded3b70035f\") "
Jul 28 10:23:06.680018 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:06.679858    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bcc75e5-167c-4ee2-878e-2ded3b70035f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2bcc75e5-167c-4ee2-878e-2ded3b70035f" (UID: "2bcc75e5-167c-4ee2-878e-2ded3b70035f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:23:06.680018 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:06.679886    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bcc75e5-167c-4ee2-878e-2ded3b70035f-home" (OuterVolumeSpecName: "home") pod "2bcc75e5-167c-4ee2-878e-2ded3b70035f" (UID: "2bcc75e5-167c-4ee2-878e-2ded3b70035f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:23:06.680018 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:06.679999    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bcc75e5-167c-4ee2-878e-2ded3b70035f-model-cache" (OuterVolumeSpecName: "model-cache") pod "2bcc75e5-167c-4ee2-878e-2ded3b70035f" (UID: "2bcc75e5-167c-4ee2-878e-2ded3b70035f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:23:06.680150 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:06.680082    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bcc75e5-167c-4ee2-878e-2ded3b70035f-tmp-dir\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:23:06.680150 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:06.680093    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bcc75e5-167c-4ee2-878e-2ded3b70035f-model-cache\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:23:06.680150 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:06.680103    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bcc75e5-167c-4ee2-878e-2ded3b70035f-home\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:23:06.681933 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:06.681904    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bcc75e5-167c-4ee2-878e-2ded3b70035f-kube-api-access-2x2fl" (OuterVolumeSpecName: "kube-api-access-2x2fl") pod "2bcc75e5-167c-4ee2-878e-2ded3b70035f" (UID: "2bcc75e5-167c-4ee2-878e-2ded3b70035f"). InnerVolumeSpecName "kube-api-access-2x2fl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:23:06.682037 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:06.681932    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bcc75e5-167c-4ee2-878e-2ded3b70035f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2bcc75e5-167c-4ee2-878e-2ded3b70035f" (UID: "2bcc75e5-167c-4ee2-878e-2ded3b70035f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:23:06.682037 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:06.681970    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bcc75e5-167c-4ee2-878e-2ded3b70035f-dshm" (OuterVolumeSpecName: "dshm") pod "2bcc75e5-167c-4ee2-878e-2ded3b70035f" (UID: "2bcc75e5-167c-4ee2-878e-2ded3b70035f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:23:06.781094 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:06.781006    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bcc75e5-167c-4ee2-878e-2ded3b70035f-tls-certs\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:23:06.781094 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:06.781040    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x2fl\" (UniqueName: \"kubernetes.io/projected/2bcc75e5-167c-4ee2-878e-2ded3b70035f-kube-api-access-2x2fl\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:23:06.781094 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:06.781049    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bcc75e5-167c-4ee2-878e-2ded3b70035f-dshm\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:23:07.465577 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:07.465544    2557 generic.go:356] "Generic (PLEG): container finished" podID="2bcc75e5-167c-4ee2-878e-2ded3b70035f" containerID="d24851a115b449a2f2f3fc2d8df32494fd279acd46ed747736b35f10e1a40131" exitCode=0
Jul 28 10:23:07.466010 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:07.465618    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v5snb"
Jul 28 10:23:07.466010 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:07.465627    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v5snb" event={"ID":"2bcc75e5-167c-4ee2-878e-2ded3b70035f","Type":"ContainerDied","Data":"d24851a115b449a2f2f3fc2d8df32494fd279acd46ed747736b35f10e1a40131"}
Jul 28 10:23:07.466010 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:07.465710    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v5snb" event={"ID":"2bcc75e5-167c-4ee2-878e-2ded3b70035f","Type":"ContainerDied","Data":"c1aecbbc5b9cbd77ab6928bb2e35f6d4e1a446e1ca588d16aee6ade8d256ae99"}
Jul 28 10:23:07.466010 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:07.465733    2557 scope.go:117] "RemoveContainer" containerID="d24851a115b449a2f2f3fc2d8df32494fd279acd46ed747736b35f10e1a40131"
Jul 28 10:23:07.474414 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:07.474396    2557 scope.go:117] "RemoveContainer" containerID="d24851a115b449a2f2f3fc2d8df32494fd279acd46ed747736b35f10e1a40131"
Jul 28 10:23:07.474698 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:23:07.474681    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d24851a115b449a2f2f3fc2d8df32494fd279acd46ed747736b35f10e1a40131\": container with ID starting with d24851a115b449a2f2f3fc2d8df32494fd279acd46ed747736b35f10e1a40131 not found: ID does not exist" containerID="d24851a115b449a2f2f3fc2d8df32494fd279acd46ed747736b35f10e1a40131"
Jul 28 10:23:07.474774 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:07.474709    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d24851a115b449a2f2f3fc2d8df32494fd279acd46ed747736b35f10e1a40131"} err="failed to get container status \"d24851a115b449a2f2f3fc2d8df32494fd279acd46ed747736b35f10e1a40131\": rpc error: code = NotFound desc = could not find container \"d24851a115b449a2f2f3fc2d8df32494fd279acd46ed747736b35f10e1a40131\": container with ID starting with d24851a115b449a2f2f3fc2d8df32494fd279acd46ed747736b35f10e1a40131 not found: ID does not exist"
Jul 28 10:23:07.484633 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:07.484591    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v5snb"]
Jul 28 10:23:07.488415 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:07.488394    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v5snb"]
Jul 28 10:23:08.579799 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:08.579764    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bcc75e5-167c-4ee2-878e-2ded3b70035f" path="/var/lib/kubelet/pods/2bcc75e5-167c-4ee2-878e-2ded3b70035f/volumes"
Jul 28 10:23:39.281760 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:39.281725    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rkkdb"]
Jul 28 10:23:39.284043 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:39.282037    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bcc75e5-167c-4ee2-878e-2ded3b70035f" containerName="main"
Jul 28 10:23:39.284043 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:39.282047    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bcc75e5-167c-4ee2-878e-2ded3b70035f" containerName="main"
Jul 28 10:23:39.284043 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:39.282104    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bcc75e5-167c-4ee2-878e-2ded3b70035f" containerName="main"
Jul 28 10:23:39.284930 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:39.284914    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rkkdb"
Jul 28 10:23:39.287712 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:39.287685    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-9rknv\""
Jul 28 10:23:39.287712 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:39.287707    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Jul 28 10:23:39.287881 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:39.287724    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Jul 28 10:23:39.287881 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:39.287791    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Jul 28 10:23:39.295498 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:39.295478    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rkkdb"]
Jul 28 10:23:39.462122 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:39.462088    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2da20806-7d2a-4c29-9b20-1fbc83d124bf-home\") pod \"fc-concurrency-test-kserve-b5d44467c-rkkdb\" (UID: \"2da20806-7d2a-4c29-9b20-1fbc83d124bf\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rkkdb"
Jul 28 10:23:39.462294 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:39.462142    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2da20806-7d2a-4c29-9b20-1fbc83d124bf-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-rkkdb\" (UID: \"2da20806-7d2a-4c29-9b20-1fbc83d124bf\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rkkdb"
Jul 28 10:23:39.462294 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:39.462176    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2da20806-7d2a-4c29-9b20-1fbc83d124bf-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-rkkdb\" (UID: \"2da20806-7d2a-4c29-9b20-1fbc83d124bf\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rkkdb"
Jul 28 10:23:39.462294 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:39.462215    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc964\" (UniqueName: \"kubernetes.io/projected/2da20806-7d2a-4c29-9b20-1fbc83d124bf-kube-api-access-wc964\") pod \"fc-concurrency-test-kserve-b5d44467c-rkkdb\" (UID: \"2da20806-7d2a-4c29-9b20-1fbc83d124bf\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rkkdb"
Jul 28 10:23:39.462294 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:39.462273    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2da20806-7d2a-4c29-9b20-1fbc83d124bf-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-rkkdb\" (UID: \"2da20806-7d2a-4c29-9b20-1fbc83d124bf\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rkkdb"
Jul 28 10:23:39.462425 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:39.462302    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2da20806-7d2a-4c29-9b20-1fbc83d124bf-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-rkkdb\" (UID: \"2da20806-7d2a-4c29-9b20-1fbc83d124bf\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rkkdb"
Jul 28 10:23:39.563444 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:39.563359    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2da20806-7d2a-4c29-9b20-1fbc83d124bf-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-rkkdb\" (UID: \"2da20806-7d2a-4c29-9b20-1fbc83d124bf\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rkkdb"
Jul 28 10:23:39.563444 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:39.563400    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2da20806-7d2a-4c29-9b20-1fbc83d124bf-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-rkkdb\" (UID: \"2da20806-7d2a-4c29-9b20-1fbc83d124bf\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rkkdb"
Jul 28 10:23:39.563671 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:39.563527    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc964\" (UniqueName: \"kubernetes.io/projected/2da20806-7d2a-4c29-9b20-1fbc83d124bf-kube-api-access-wc964\") pod \"fc-concurrency-test-kserve-b5d44467c-rkkdb\" (UID: \"2da20806-7d2a-4c29-9b20-1fbc83d124bf\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rkkdb"
Jul 28 10:23:39.563671 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:39.563571    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2da20806-7d2a-4c29-9b20-1fbc83d124bf-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-rkkdb\" (UID: \"2da20806-7d2a-4c29-9b20-1fbc83d124bf\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rkkdb"
Jul 28 10:23:39.563751 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:39.563682    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2da20806-7d2a-4c29-9b20-1fbc83d124bf-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-rkkdb\" (UID: \"2da20806-7d2a-4c29-9b20-1fbc83d124bf\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rkkdb"
Jul 28 10:23:39.563751 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:39.563719    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2da20806-7d2a-4c29-9b20-1fbc83d124bf-home\") pod \"fc-concurrency-test-kserve-b5d44467c-rkkdb\" (UID: \"2da20806-7d2a-4c29-9b20-1fbc83d124bf\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rkkdb"
Jul 28 10:23:39.563919 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:39.563891    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2da20806-7d2a-4c29-9b20-1fbc83d124bf-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-rkkdb\" (UID: \"2da20806-7d2a-4c29-9b20-1fbc83d124bf\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rkkdb"
Jul 28 10:23:39.563987 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:39.563915    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2da20806-7d2a-4c29-9b20-1fbc83d124bf-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-rkkdb\" (UID: \"2da20806-7d2a-4c29-9b20-1fbc83d124bf\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rkkdb"
Jul 28 10:23:39.564103 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:39.564082    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2da20806-7d2a-4c29-9b20-1fbc83d124bf-home\") pod \"fc-concurrency-test-kserve-b5d44467c-rkkdb\" (UID: \"2da20806-7d2a-4c29-9b20-1fbc83d124bf\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rkkdb"
Jul 28 10:23:39.565665 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:39.565646    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2da20806-7d2a-4c29-9b20-1fbc83d124bf-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-rkkdb\" (UID: \"2da20806-7d2a-4c29-9b20-1fbc83d124bf\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rkkdb"
Jul 28 10:23:39.566118 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:39.566099    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2da20806-7d2a-4c29-9b20-1fbc83d124bf-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-rkkdb\" (UID: \"2da20806-7d2a-4c29-9b20-1fbc83d124bf\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rkkdb"
Jul 28 10:23:39.570965 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:39.570943    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc964\" (UniqueName: \"kubernetes.io/projected/2da20806-7d2a-4c29-9b20-1fbc83d124bf-kube-api-access-wc964\") pod \"fc-concurrency-test-kserve-b5d44467c-rkkdb\" (UID: \"2da20806-7d2a-4c29-9b20-1fbc83d124bf\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rkkdb"
Jul 28 10:23:39.594814 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:39.594776    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rkkdb"
Jul 28 10:23:39.741894 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:39.741860    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rkkdb"]
Jul 28 10:23:39.745062 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:23:39.745033    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2da20806_7d2a_4c29_9b20_1fbc83d124bf.slice/crio-4a79a420d4c75e5c8273824db5fc3f7c67419392f491bc9bc8cc821798742cc3 WatchSource:0}: Error finding container 4a79a420d4c75e5c8273824db5fc3f7c67419392f491bc9bc8cc821798742cc3: Status 404 returned error can't find the container with id 4a79a420d4c75e5c8273824db5fc3f7c67419392f491bc9bc8cc821798742cc3
Jul 28 10:23:40.622917 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:40.622874    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rkkdb" event={"ID":"2da20806-7d2a-4c29-9b20-1fbc83d124bf","Type":"ContainerStarted","Data":"d2ac7d59ab3ca8e31ac2f0f922229290b9630dccc459f6d5248f60cf04ac9c20"}
Jul 28 10:23:40.623389 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:40.622930    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rkkdb" event={"ID":"2da20806-7d2a-4c29-9b20-1fbc83d124bf","Type":"ContainerStarted","Data":"4a79a420d4c75e5c8273824db5fc3f7c67419392f491bc9bc8cc821798742cc3"}
Jul 28 10:23:40.657802 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:40.657742    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rkkdb" podStartSLOduration=1.657726297 podStartE2EDuration="1.657726297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:23:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:23:40.656243782 +0000 UTC m=+956.615012161" watchObservedRunningTime="2026-07-28 10:23:40.657726297 +0000 UTC m=+956.616494676"
Jul 28 10:23:41.595486 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:41.595451    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rkkdb"
Jul 28 10:23:49.595476 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:49.595429    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rkkdb"
Jul 28 10:23:49.608139 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:49.608114    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rkkdb"
Jul 28 10:23:49.674215 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:23:49.674182    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rkkdb"
Jul 28 10:24:17.414063 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:17.413965    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rkkdb"]
Jul 28 10:24:17.423891 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:24:17.423851    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 10:24:17.423891 ip-10-0-131-88 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:24:17.423891 ip-10-0-131-88 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 28 10:24:17.423891 ip-10-0-131-88 kubenswrapper[2557]:  > containerID="d2ac7d59ab3ca8e31ac2f0f922229290b9630dccc459f6d5248f60cf04ac9c20" cmd=["/bin/sleep","15"]
Jul 28 10:24:17.424159 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:17.423910    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rkkdb" message="" err=<
Jul 28 10:24:17.424159 ip-10-0-131-88 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:24:17.424159 ip-10-0-131-88 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 28 10:24:17.424159 ip-10-0-131-88 kubenswrapper[2557]:  >
Jul 28 10:24:17.424159 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:24:17.423922    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 10:24:17.424159 ip-10-0-131-88 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:24:17.424159 ip-10-0-131-88 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 28 10:24:17.424159 ip-10-0-131-88 kubenswrapper[2557]:  > pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rkkdb" podUID="2da20806-7d2a-4c29-9b20-1fbc83d124bf" containerName="main" containerID="cri-o://d2ac7d59ab3ca8e31ac2f0f922229290b9630dccc459f6d5248f60cf04ac9c20"
Jul 28 10:24:17.424159 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:17.423984    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rkkdb" podUID="2da20806-7d2a-4c29-9b20-1fbc83d124bf" containerName="main" containerID="cri-o://d2ac7d59ab3ca8e31ac2f0f922229290b9630dccc459f6d5248f60cf04ac9c20" gracePeriod=60
Jul 28 10:24:17.663040 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:17.662991    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rkkdb" podUID="2da20806-7d2a-4c29-9b20-1fbc83d124bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused"
Jul 28 10:24:17.784187 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:17.784161    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rkkdb"
Jul 28 10:24:17.786045 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:17.786015    2557 generic.go:356] "Generic (PLEG): container finished" podID="2da20806-7d2a-4c29-9b20-1fbc83d124bf" containerID="d2ac7d59ab3ca8e31ac2f0f922229290b9630dccc459f6d5248f60cf04ac9c20" exitCode=0
Jul 28 10:24:17.786190 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:17.786054    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rkkdb" event={"ID":"2da20806-7d2a-4c29-9b20-1fbc83d124bf","Type":"ContainerDied","Data":"d2ac7d59ab3ca8e31ac2f0f922229290b9630dccc459f6d5248f60cf04ac9c20"}
Jul 28 10:24:17.786190 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:17.786130    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rkkdb" event={"ID":"2da20806-7d2a-4c29-9b20-1fbc83d124bf","Type":"ContainerDied","Data":"4a79a420d4c75e5c8273824db5fc3f7c67419392f491bc9bc8cc821798742cc3"}
Jul 28 10:24:17.786190 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:17.786062    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rkkdb"
Jul 28 10:24:17.786190 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:17.786165    2557 scope.go:117] "RemoveContainer" containerID="d2ac7d59ab3ca8e31ac2f0f922229290b9630dccc459f6d5248f60cf04ac9c20"
Jul 28 10:24:17.795851 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:17.795827    2557 scope.go:117] "RemoveContainer" containerID="d2ac7d59ab3ca8e31ac2f0f922229290b9630dccc459f6d5248f60cf04ac9c20"
Jul 28 10:24:17.796169 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:24:17.796148    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2ac7d59ab3ca8e31ac2f0f922229290b9630dccc459f6d5248f60cf04ac9c20\": container with ID starting with d2ac7d59ab3ca8e31ac2f0f922229290b9630dccc459f6d5248f60cf04ac9c20 not found: ID does not exist" containerID="d2ac7d59ab3ca8e31ac2f0f922229290b9630dccc459f6d5248f60cf04ac9c20"
Jul 28 10:24:17.796265 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:17.796173    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2ac7d59ab3ca8e31ac2f0f922229290b9630dccc459f6d5248f60cf04ac9c20"} err="failed to get container status \"d2ac7d59ab3ca8e31ac2f0f922229290b9630dccc459f6d5248f60cf04ac9c20\": rpc error: code = NotFound desc = could not find container \"d2ac7d59ab3ca8e31ac2f0f922229290b9630dccc459f6d5248f60cf04ac9c20\": container with ID starting with d2ac7d59ab3ca8e31ac2f0f922229290b9630dccc459f6d5248f60cf04ac9c20 not found: ID does not exist"
Jul 28 10:24:17.925441 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:17.925408    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2da20806-7d2a-4c29-9b20-1fbc83d124bf-tmp-dir\") pod \"2da20806-7d2a-4c29-9b20-1fbc83d124bf\" (UID: \"2da20806-7d2a-4c29-9b20-1fbc83d124bf\") "
Jul 28 10:24:17.925641 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:17.925453    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wc964\" (UniqueName: \"kubernetes.io/projected/2da20806-7d2a-4c29-9b20-1fbc83d124bf-kube-api-access-wc964\") pod \"2da20806-7d2a-4c29-9b20-1fbc83d124bf\" (UID: \"2da20806-7d2a-4c29-9b20-1fbc83d124bf\") "
Jul 28 10:24:17.925641 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:17.925503    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2da20806-7d2a-4c29-9b20-1fbc83d124bf-dshm\") pod \"2da20806-7d2a-4c29-9b20-1fbc83d124bf\" (UID: \"2da20806-7d2a-4c29-9b20-1fbc83d124bf\") "
Jul 28 10:24:17.925641 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:17.925530    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2da20806-7d2a-4c29-9b20-1fbc83d124bf-home\") pod \"2da20806-7d2a-4c29-9b20-1fbc83d124bf\" (UID: \"2da20806-7d2a-4c29-9b20-1fbc83d124bf\") "
Jul 28 10:24:17.925641 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:17.925562    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2da20806-7d2a-4c29-9b20-1fbc83d124bf-tls-certs\") pod \"2da20806-7d2a-4c29-9b20-1fbc83d124bf\" (UID: \"2da20806-7d2a-4c29-9b20-1fbc83d124bf\") "
Jul 28 10:24:17.925876 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:17.925705    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2da20806-7d2a-4c29-9b20-1fbc83d124bf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2da20806-7d2a-4c29-9b20-1fbc83d124bf" (UID: "2da20806-7d2a-4c29-9b20-1fbc83d124bf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:24:17.925876 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:17.925772    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2da20806-7d2a-4c29-9b20-1fbc83d124bf-model-cache\") pod \"2da20806-7d2a-4c29-9b20-1fbc83d124bf\" (UID: \"2da20806-7d2a-4c29-9b20-1fbc83d124bf\") "
Jul 28 10:24:17.925876 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:17.925808    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2da20806-7d2a-4c29-9b20-1fbc83d124bf-home" (OuterVolumeSpecName: "home") pod "2da20806-7d2a-4c29-9b20-1fbc83d124bf" (UID: "2da20806-7d2a-4c29-9b20-1fbc83d124bf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:24:17.926079 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:17.926052    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2da20806-7d2a-4c29-9b20-1fbc83d124bf-tmp-dir\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:24:17.926079 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:17.926078    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2da20806-7d2a-4c29-9b20-1fbc83d124bf-home\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:24:17.926255 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:17.926047    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2da20806-7d2a-4c29-9b20-1fbc83d124bf-model-cache" (OuterVolumeSpecName: "model-cache") pod "2da20806-7d2a-4c29-9b20-1fbc83d124bf" (UID: "2da20806-7d2a-4c29-9b20-1fbc83d124bf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:24:17.927910 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:17.927884    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2da20806-7d2a-4c29-9b20-1fbc83d124bf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2da20806-7d2a-4c29-9b20-1fbc83d124bf" (UID: "2da20806-7d2a-4c29-9b20-1fbc83d124bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:24:17.927991 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:17.927947    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2da20806-7d2a-4c29-9b20-1fbc83d124bf-kube-api-access-wc964" (OuterVolumeSpecName: "kube-api-access-wc964") pod "2da20806-7d2a-4c29-9b20-1fbc83d124bf" (UID: "2da20806-7d2a-4c29-9b20-1fbc83d124bf"). InnerVolumeSpecName "kube-api-access-wc964". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:24:17.928108 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:17.928093    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2da20806-7d2a-4c29-9b20-1fbc83d124bf-dshm" (OuterVolumeSpecName: "dshm") pod "2da20806-7d2a-4c29-9b20-1fbc83d124bf" (UID: "2da20806-7d2a-4c29-9b20-1fbc83d124bf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:24:18.026864 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:18.026817    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2da20806-7d2a-4c29-9b20-1fbc83d124bf-model-cache\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:24:18.026864 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:18.026860    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc964\" (UniqueName: \"kubernetes.io/projected/2da20806-7d2a-4c29-9b20-1fbc83d124bf-kube-api-access-wc964\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:24:18.026864 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:18.026871    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2da20806-7d2a-4c29-9b20-1fbc83d124bf-dshm\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:24:18.026864 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:18.026880    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2da20806-7d2a-4c29-9b20-1fbc83d124bf-tls-certs\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:24:18.107938 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:18.107901    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rkkdb"]
Jul 28 10:24:18.111726 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:18.111698    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rkkdb"]
Jul 28 10:24:18.581138 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:18.581103    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2da20806-7d2a-4c29-9b20-1fbc83d124bf" path="/var/lib/kubelet/pods/2da20806-7d2a-4c29-9b20-1fbc83d124bf/volumes"
Jul 28 10:24:49.591168 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:49.591134    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-r5ts7"]
Jul 28 10:24:49.591616 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:49.591514    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2da20806-7d2a-4c29-9b20-1fbc83d124bf" containerName="main"
Jul 28 10:24:49.591616 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:49.591526    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="2da20806-7d2a-4c29-9b20-1fbc83d124bf" containerName="main"
Jul 28 10:24:49.591616 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:49.591590    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="2da20806-7d2a-4c29-9b20-1fbc83d124bf" containerName="main"
Jul 28 10:24:49.596101 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:49.596080    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-r5ts7"
Jul 28 10:24:49.599239 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:49.599216    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Jul 28 10:24:49.599361 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:49.599242    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Jul 28 10:24:49.599361 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:49.599328    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
Jul 28 10:24:49.599361 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:49.599347    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"default-dockercfg-z8s7s\""
Jul 28 10:24:49.606488 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:49.606464    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-r5ts7"]
Jul 28 10:24:49.702125 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:49.702088    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20e82100-4861-4511-ab7a-a123a8fcffb8-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-r5ts7\" (UID: \"20e82100-4861-4511-ab7a-a123a8fcffb8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-r5ts7"
Jul 28 10:24:49.702306 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:49.702132    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20e82100-4861-4511-ab7a-a123a8fcffb8-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-r5ts7\" (UID: \"20e82100-4861-4511-ab7a-a123a8fcffb8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-r5ts7"
Jul 28 10:24:49.702306 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:49.702153    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t8nm\" (UniqueName: \"kubernetes.io/projected/20e82100-4861-4511-ab7a-a123a8fcffb8-kube-api-access-9t8nm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-r5ts7\" (UID: \"20e82100-4861-4511-ab7a-a123a8fcffb8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-r5ts7"
Jul 28 10:24:49.702306 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:49.702244    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20e82100-4861-4511-ab7a-a123a8fcffb8-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-r5ts7\" (UID: \"20e82100-4861-4511-ab7a-a123a8fcffb8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-r5ts7"
Jul 28 10:24:49.702306 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:49.702281    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20e82100-4861-4511-ab7a-a123a8fcffb8-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-r5ts7\" (UID: \"20e82100-4861-4511-ab7a-a123a8fcffb8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-r5ts7"
Jul 28 10:24:49.702468 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:49.702352    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20e82100-4861-4511-ab7a-a123a8fcffb8-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-r5ts7\" (UID: \"20e82100-4861-4511-ab7a-a123a8fcffb8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-r5ts7"
Jul 28 10:24:49.802918 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:49.802878    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20e82100-4861-4511-ab7a-a123a8fcffb8-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-r5ts7\" (UID: \"20e82100-4861-4511-ab7a-a123a8fcffb8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-r5ts7"
Jul 28 10:24:49.802918 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:49.802922    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20e82100-4861-4511-ab7a-a123a8fcffb8-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-r5ts7\" (UID: \"20e82100-4861-4511-ab7a-a123a8fcffb8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-r5ts7"
Jul 28 10:24:49.803163 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:49.802939    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t8nm\" (UniqueName: \"kubernetes.io/projected/20e82100-4861-4511-ab7a-a123a8fcffb8-kube-api-access-9t8nm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-r5ts7\" (UID: \"20e82100-4861-4511-ab7a-a123a8fcffb8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-r5ts7"
Jul 28 10:24:49.803163 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:49.802973    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20e82100-4861-4511-ab7a-a123a8fcffb8-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-r5ts7\" (UID: \"20e82100-4861-4511-ab7a-a123a8fcffb8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-r5ts7"
Jul 28 10:24:49.803163 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:49.802992    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20e82100-4861-4511-ab7a-a123a8fcffb8-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-r5ts7\" (UID: \"20e82100-4861-4511-ab7a-a123a8fcffb8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-r5ts7"
Jul 28 10:24:49.803163 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:49.803034    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20e82100-4861-4511-ab7a-a123a8fcffb8-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-r5ts7\" (UID: \"20e82100-4861-4511-ab7a-a123a8fcffb8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-r5ts7"
Jul 28 10:24:49.803369 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:49.803348    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20e82100-4861-4511-ab7a-a123a8fcffb8-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-r5ts7\" (UID: \"20e82100-4861-4511-ab7a-a123a8fcffb8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-r5ts7"
Jul 28 10:24:49.803427 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:49.803388    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20e82100-4861-4511-ab7a-a123a8fcffb8-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-r5ts7\" (UID: \"20e82100-4861-4511-ab7a-a123a8fcffb8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-r5ts7"
Jul 28 10:24:49.803488 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:49.803444    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20e82100-4861-4511-ab7a-a123a8fcffb8-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-r5ts7\" (UID: \"20e82100-4861-4511-ab7a-a123a8fcffb8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-r5ts7"
Jul 28 10:24:49.805315 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:49.805297    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20e82100-4861-4511-ab7a-a123a8fcffb8-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-r5ts7\" (UID: \"20e82100-4861-4511-ab7a-a123a8fcffb8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-r5ts7"
Jul 28 10:24:49.805610 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:49.805577    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20e82100-4861-4511-ab7a-a123a8fcffb8-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-r5ts7\" (UID: \"20e82100-4861-4511-ab7a-a123a8fcffb8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-r5ts7"
Jul 28 10:24:49.810821 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:49.810801    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t8nm\" (UniqueName: \"kubernetes.io/projected/20e82100-4861-4511-ab7a-a123a8fcffb8-kube-api-access-9t8nm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-r5ts7\" (UID: \"20e82100-4861-4511-ab7a-a123a8fcffb8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-r5ts7"
Jul 28 10:24:49.906972 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:49.906888    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-r5ts7"
Jul 28 10:24:50.043888 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:50.043859    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-r5ts7"]
Jul 28 10:24:50.046000 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:24:50.045964    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20e82100_4861_4511_ab7a_a123a8fcffb8.slice/crio-4c92e31d54bb163c57d6792cb06374315c7dc8fbd701f61cb0fcdaf337f98154 WatchSource:0}: Error finding container 4c92e31d54bb163c57d6792cb06374315c7dc8fbd701f61cb0fcdaf337f98154: Status 404 returned error can't find the container with id 4c92e31d54bb163c57d6792cb06374315c7dc8fbd701f61cb0fcdaf337f98154
Jul 28 10:24:50.933167 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:50.933130    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-r5ts7" event={"ID":"20e82100-4861-4511-ab7a-a123a8fcffb8","Type":"ContainerStarted","Data":"a30bf3f054dd32bb8cdccb4d8cde2c05c05f619f186f386281cc14306db267d2"}
Jul 28 10:24:50.933657 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:50.933184    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-r5ts7" event={"ID":"20e82100-4861-4511-ab7a-a123a8fcffb8","Type":"ContainerStarted","Data":"4c92e31d54bb163c57d6792cb06374315c7dc8fbd701f61cb0fcdaf337f98154"}
Jul 28 10:24:50.949695 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:50.949641    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-r5ts7" podStartSLOduration=1.949623886 podStartE2EDuration="1.949623886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:24:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:24:50.948536974 +0000 UTC m=+1026.907305346" watchObservedRunningTime="2026-07-28 10:24:50.949623886 +0000 UTC m=+1026.908392267"
Jul 28 10:24:51.735311 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:51.735275    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-r5ts7"]
Jul 28 10:24:51.907520 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:51.907491    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-r5ts7"
Jul 28 10:24:52.025159 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:24:52.025127    2557 secret.go:189] Couldn't get secret e2e-test-gateway-section-name-propagation-49b5edd9/gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs: secret "gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs" not found
Jul 28 10:24:52.025536 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:24:52.025207    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20e82100-4861-4511-ab7a-a123a8fcffb8-tls-certs podName:20e82100-4861-4511-ab7a-a123a8fcffb8 nodeName:}" failed. No retries permitted until 2026-07-28 10:24:52.525191251 +0000 UTC m=+1028.483959611 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/20e82100-4861-4511-ab7a-a123a8fcffb8-tls-certs") pod "gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-r5ts7" (UID: "20e82100-4861-4511-ab7a-a123a8fcffb8") : secret "gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs" not found
Jul 28 10:24:52.530801 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:24:52.530764    2557 secret.go:189] Couldn't get secret e2e-test-gateway-section-name-propagation-49b5edd9/gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs: secret "gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs" not found
Jul 28 10:24:52.531018 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:24:52.530875    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20e82100-4861-4511-ab7a-a123a8fcffb8-tls-certs podName:20e82100-4861-4511-ab7a-a123a8fcffb8 nodeName:}" failed. No retries permitted until 2026-07-28 10:24:53.530852562 +0000 UTC m=+1029.489620926 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/20e82100-4861-4511-ab7a-a123a8fcffb8-tls-certs") pod "gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-r5ts7" (UID: "20e82100-4861-4511-ab7a-a123a8fcffb8") : secret "gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs" not found
Jul 28 10:24:52.951047 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:24:52.950970    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 10:24:52.951047 ip-10-0-131-88 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:24:52.951047 ip-10-0-131-88 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 28 10:24:52.951047 ip-10-0-131-88 kubenswrapper[2557]:  > containerID="a30bf3f054dd32bb8cdccb4d8cde2c05c05f619f186f386281cc14306db267d2" cmd=["/bin/sleep","15"]
Jul 28 10:24:52.951047 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:52.951017    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-r5ts7" message="" err=<
Jul 28 10:24:52.951047 ip-10-0-131-88 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:24:52.951047 ip-10-0-131-88 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 28 10:24:52.951047 ip-10-0-131-88 kubenswrapper[2557]:  >
Jul 28 10:24:52.951047 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:24:52.951030    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 10:24:52.951047 ip-10-0-131-88 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:24:52.951047 ip-10-0-131-88 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 28 10:24:52.951047 ip-10-0-131-88 kubenswrapper[2557]:  > pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-r5ts7" podUID="20e82100-4861-4511-ab7a-a123a8fcffb8" containerName="main" containerID="cri-o://a30bf3f054dd32bb8cdccb4d8cde2c05c05f619f186f386281cc14306db267d2"
Jul 28 10:24:52.951541 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:52.951063    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-r5ts7" podUID="20e82100-4861-4511-ab7a-a123a8fcffb8" containerName="main" containerID="cri-o://a30bf3f054dd32bb8cdccb4d8cde2c05c05f619f186f386281cc14306db267d2" gracePeriod=60
Jul 28 10:24:53.213522 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:53.213456    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-r5ts7"
Jul 28 10:24:53.338760 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:53.338727    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20e82100-4861-4511-ab7a-a123a8fcffb8-model-cache\") pod \"20e82100-4861-4511-ab7a-a123a8fcffb8\" (UID: \"20e82100-4861-4511-ab7a-a123a8fcffb8\") "
Jul 28 10:24:53.338760 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:53.338768    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20e82100-4861-4511-ab7a-a123a8fcffb8-dshm\") pod \"20e82100-4861-4511-ab7a-a123a8fcffb8\" (UID: \"20e82100-4861-4511-ab7a-a123a8fcffb8\") "
Jul 28 10:24:53.339010 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:53.338788    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9t8nm\" (UniqueName: \"kubernetes.io/projected/20e82100-4861-4511-ab7a-a123a8fcffb8-kube-api-access-9t8nm\") pod \"20e82100-4861-4511-ab7a-a123a8fcffb8\" (UID: \"20e82100-4861-4511-ab7a-a123a8fcffb8\") "
Jul 28 10:24:53.339010 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:53.338875    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20e82100-4861-4511-ab7a-a123a8fcffb8-tmp-dir\") pod \"20e82100-4861-4511-ab7a-a123a8fcffb8\" (UID: \"20e82100-4861-4511-ab7a-a123a8fcffb8\") "
Jul 28 10:24:53.339122 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:53.339017    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20e82100-4861-4511-ab7a-a123a8fcffb8-model-cache" (OuterVolumeSpecName: "model-cache") pod "20e82100-4861-4511-ab7a-a123a8fcffb8" (UID: "20e82100-4861-4511-ab7a-a123a8fcffb8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:24:53.339122 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:53.339057    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20e82100-4861-4511-ab7a-a123a8fcffb8-home\") pod \"20e82100-4861-4511-ab7a-a123a8fcffb8\" (UID: \"20e82100-4861-4511-ab7a-a123a8fcffb8\") "
Jul 28 10:24:53.339122 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:53.339101    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20e82100-4861-4511-ab7a-a123a8fcffb8-tls-certs\") pod \"20e82100-4861-4511-ab7a-a123a8fcffb8\" (UID: \"20e82100-4861-4511-ab7a-a123a8fcffb8\") "
Jul 28 10:24:53.339122 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:53.339111    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20e82100-4861-4511-ab7a-a123a8fcffb8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "20e82100-4861-4511-ab7a-a123a8fcffb8" (UID: "20e82100-4861-4511-ab7a-a123a8fcffb8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:24:53.339279 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:53.339206    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20e82100-4861-4511-ab7a-a123a8fcffb8-home" (OuterVolumeSpecName: "home") pod "20e82100-4861-4511-ab7a-a123a8fcffb8" (UID: "20e82100-4861-4511-ab7a-a123a8fcffb8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:24:53.339371 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:53.339356    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20e82100-4861-4511-ab7a-a123a8fcffb8-tmp-dir\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:24:53.339371 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:53.339371    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20e82100-4861-4511-ab7a-a123a8fcffb8-home\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:24:53.339464 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:53.339382    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20e82100-4861-4511-ab7a-a123a8fcffb8-model-cache\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:24:53.341112 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:53.341090    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20e82100-4861-4511-ab7a-a123a8fcffb8-dshm" (OuterVolumeSpecName: "dshm") pod "20e82100-4861-4511-ab7a-a123a8fcffb8" (UID: "20e82100-4861-4511-ab7a-a123a8fcffb8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:24:53.341200 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:53.341120    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20e82100-4861-4511-ab7a-a123a8fcffb8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "20e82100-4861-4511-ab7a-a123a8fcffb8" (UID: "20e82100-4861-4511-ab7a-a123a8fcffb8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:24:53.341248 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:53.341196    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20e82100-4861-4511-ab7a-a123a8fcffb8-kube-api-access-9t8nm" (OuterVolumeSpecName: "kube-api-access-9t8nm") pod "20e82100-4861-4511-ab7a-a123a8fcffb8" (UID: "20e82100-4861-4511-ab7a-a123a8fcffb8"). InnerVolumeSpecName "kube-api-access-9t8nm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:24:53.440860 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:53.440819    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20e82100-4861-4511-ab7a-a123a8fcffb8-tls-certs\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:24:53.440860 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:53.440851    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20e82100-4861-4511-ab7a-a123a8fcffb8-dshm\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:24:53.440860 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:53.440860    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t8nm\" (UniqueName: \"kubernetes.io/projected/20e82100-4861-4511-ab7a-a123a8fcffb8-kube-api-access-9t8nm\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:24:53.947464 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:53.947427    2557 generic.go:356] "Generic (PLEG): container finished" podID="20e82100-4861-4511-ab7a-a123a8fcffb8" containerID="a30bf3f054dd32bb8cdccb4d8cde2c05c05f619f186f386281cc14306db267d2" exitCode=0
Jul 28 10:24:53.947690 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:53.947472    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-r5ts7"
Jul 28 10:24:53.947690 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:53.947490    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-r5ts7" event={"ID":"20e82100-4861-4511-ab7a-a123a8fcffb8","Type":"ContainerDied","Data":"a30bf3f054dd32bb8cdccb4d8cde2c05c05f619f186f386281cc14306db267d2"}
Jul 28 10:24:53.947690 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:53.947544    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-r5ts7" event={"ID":"20e82100-4861-4511-ab7a-a123a8fcffb8","Type":"ContainerDied","Data":"4c92e31d54bb163c57d6792cb06374315c7dc8fbd701f61cb0fcdaf337f98154"}
Jul 28 10:24:53.947690 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:53.947572    2557 scope.go:117] "RemoveContainer" containerID="a30bf3f054dd32bb8cdccb4d8cde2c05c05f619f186f386281cc14306db267d2"
Jul 28 10:24:53.956489 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:53.956359    2557 scope.go:117] "RemoveContainer" containerID="a30bf3f054dd32bb8cdccb4d8cde2c05c05f619f186f386281cc14306db267d2"
Jul 28 10:24:53.956659 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:24:53.956639    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a30bf3f054dd32bb8cdccb4d8cde2c05c05f619f186f386281cc14306db267d2\": container with ID starting with a30bf3f054dd32bb8cdccb4d8cde2c05c05f619f186f386281cc14306db267d2 not found: ID does not exist" containerID="a30bf3f054dd32bb8cdccb4d8cde2c05c05f619f186f386281cc14306db267d2"
Jul 28 10:24:53.956704 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:53.956667    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a30bf3f054dd32bb8cdccb4d8cde2c05c05f619f186f386281cc14306db267d2"} err="failed to get container status \"a30bf3f054dd32bb8cdccb4d8cde2c05c05f619f186f386281cc14306db267d2\": rpc error: code = NotFound desc = could not find container \"a30bf3f054dd32bb8cdccb4d8cde2c05c05f619f186f386281cc14306db267d2\": container with ID starting with a30bf3f054dd32bb8cdccb4d8cde2c05c05f619f186f386281cc14306db267d2 not found: ID does not exist"
Jul 28 10:24:53.969447 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:53.969423    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-r5ts7"]
Jul 28 10:24:53.973686 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:53.973664    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-r5ts7"]
Jul 28 10:24:54.580242 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:54.580201    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20e82100-4861-4511-ab7a-a123a8fcffb8" path="/var/lib/kubelet/pods/20e82100-4861-4511-ab7a-a123a8fcffb8/volumes"
Jul 28 10:24:58.968388 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:58.968351    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf9ljfv"]
Jul 28 10:24:58.968916 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:58.968869    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20e82100-4861-4511-ab7a-a123a8fcffb8" containerName="main"
Jul 28 10:24:58.968916 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:58.968887    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="20e82100-4861-4511-ab7a-a123a8fcffb8" containerName="main"
Jul 28 10:24:58.969039 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:58.968957    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="20e82100-4861-4511-ab7a-a123a8fcffb8" containerName="main"
Jul 28 10:24:58.973322 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:58.973305    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf9ljfv"
Jul 28 10:24:58.976714 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:58.976689    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\""
Jul 28 10:24:58.976714 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:58.976707    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Jul 28 10:24:58.976874 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:58.976757    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"default-dockercfg-pcztj\""
Jul 28 10:24:58.976874 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:58.976786    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Jul 28 10:24:58.984278 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:58.983940    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf9ljfv"]
Jul 28 10:24:59.089392 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:59.089353    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35f63890-0ede-4baf-bf15-f7b009a29624-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf9ljfv\" (UID: \"35f63890-0ede-4baf-bf15-f7b009a29624\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf9ljfv"
Jul 28 10:24:59.089564 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:59.089413    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29pr7\" (UniqueName: \"kubernetes.io/projected/35f63890-0ede-4baf-bf15-f7b009a29624-kube-api-access-29pr7\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf9ljfv\" (UID: \"35f63890-0ede-4baf-bf15-f7b009a29624\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf9ljfv"
Jul 28 10:24:59.089564 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:59.089468    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35f63890-0ede-4baf-bf15-f7b009a29624-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf9ljfv\" (UID: \"35f63890-0ede-4baf-bf15-f7b009a29624\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf9ljfv"
Jul 28 10:24:59.089564 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:59.089540    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35f63890-0ede-4baf-bf15-f7b009a29624-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf9ljfv\" (UID: \"35f63890-0ede-4baf-bf15-f7b009a29624\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf9ljfv"
Jul 28 10:24:59.089719 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:59.089582    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35f63890-0ede-4baf-bf15-f7b009a29624-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf9ljfv\" (UID: \"35f63890-0ede-4baf-bf15-f7b009a29624\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf9ljfv"
Jul 28 10:24:59.089719 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:59.089631    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35f63890-0ede-4baf-bf15-f7b009a29624-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf9ljfv\" (UID: \"35f63890-0ede-4baf-bf15-f7b009a29624\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf9ljfv"
Jul 28 10:24:59.190908 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:59.190869    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35f63890-0ede-4baf-bf15-f7b009a29624-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf9ljfv\" (UID: \"35f63890-0ede-4baf-bf15-f7b009a29624\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf9ljfv"
Jul 28 10:24:59.191099 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:59.190934    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35f63890-0ede-4baf-bf15-f7b009a29624-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf9ljfv\" (UID: \"35f63890-0ede-4baf-bf15-f7b009a29624\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf9ljfv"
Jul 28 10:24:59.191099 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:59.190956    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29pr7\" (UniqueName: \"kubernetes.io/projected/35f63890-0ede-4baf-bf15-f7b009a29624-kube-api-access-29pr7\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf9ljfv\" (UID: \"35f63890-0ede-4baf-bf15-f7b009a29624\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf9ljfv"
Jul 28 10:24:59.191099 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:59.190984    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35f63890-0ede-4baf-bf15-f7b009a29624-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf9ljfv\" (UID: \"35f63890-0ede-4baf-bf15-f7b009a29624\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf9ljfv"
Jul 28 10:24:59.191099 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:59.191008    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35f63890-0ede-4baf-bf15-f7b009a29624-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf9ljfv\" (UID: \"35f63890-0ede-4baf-bf15-f7b009a29624\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf9ljfv"
Jul 28 10:24:59.191099 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:59.191036    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35f63890-0ede-4baf-bf15-f7b009a29624-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf9ljfv\" (UID: \"35f63890-0ede-4baf-bf15-f7b009a29624\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf9ljfv"
Jul 28 10:24:59.191390 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:59.191369    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35f63890-0ede-4baf-bf15-f7b009a29624-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf9ljfv\" (UID: \"35f63890-0ede-4baf-bf15-f7b009a29624\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf9ljfv"
Jul 28 10:24:59.191461 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:59.191421    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35f63890-0ede-4baf-bf15-f7b009a29624-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf9ljfv\" (UID: \"35f63890-0ede-4baf-bf15-f7b009a29624\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf9ljfv"
Jul 28 10:24:59.191461 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:59.191442    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35f63890-0ede-4baf-bf15-f7b009a29624-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf9ljfv\" (UID: \"35f63890-0ede-4baf-bf15-f7b009a29624\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf9ljfv"
Jul 28 10:24:59.193166 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:59.193147    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35f63890-0ede-4baf-bf15-f7b009a29624-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf9ljfv\" (UID: \"35f63890-0ede-4baf-bf15-f7b009a29624\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf9ljfv"
Jul 28 10:24:59.193540 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:59.193517    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35f63890-0ede-4baf-bf15-f7b009a29624-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf9ljfv\" (UID: \"35f63890-0ede-4baf-bf15-f7b009a29624\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf9ljfv"
Jul 28 10:24:59.206594 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:59.206564    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29pr7\" (UniqueName: \"kubernetes.io/projected/35f63890-0ede-4baf-bf15-f7b009a29624-kube-api-access-29pr7\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf9ljfv\" (UID: \"35f63890-0ede-4baf-bf15-f7b009a29624\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf9ljfv"
Jul 28 10:24:59.283518 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:59.283460    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf9ljfv"
Jul 28 10:24:59.427951 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:59.427923    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf9ljfv"]
Jul 28 10:24:59.430588 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:24:59.430560    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35f63890_0ede_4baf_bf15_f7b009a29624.slice/crio-e0aadd41c5b9f473342a9c6dbae7087b01d6c95e16594ca8547996a1a883cbaf WatchSource:0}: Error finding container e0aadd41c5b9f473342a9c6dbae7087b01d6c95e16594ca8547996a1a883cbaf: Status 404 returned error can't find the container with id e0aadd41c5b9f473342a9c6dbae7087b01d6c95e16594ca8547996a1a883cbaf
Jul 28 10:24:59.976261 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:59.976226    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf9ljfv" event={"ID":"35f63890-0ede-4baf-bf15-f7b009a29624","Type":"ContainerStarted","Data":"231c84b0345e5b7815b1febfc9a88761b5f1ae240a03d70b11116b427a05be7b"}
Jul 28 10:24:59.976709 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:59.976276    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf9ljfv" event={"ID":"35f63890-0ede-4baf-bf15-f7b009a29624","Type":"ContainerStarted","Data":"e0aadd41c5b9f473342a9c6dbae7087b01d6c95e16594ca8547996a1a883cbaf"}
Jul 28 10:24:59.993206 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:24:59.993125    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf9ljfv" podStartSLOduration=1.9931069689999998 podStartE2EDuration="1.993106969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:24:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:24:59.99277289 +0000 UTC m=+1035.951541270" watchObservedRunningTime="2026-07-28 10:24:59.993106969 +0000 UTC m=+1035.951875352"
Jul 28 10:25:00.284038 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:00.283988    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf9ljfv"
Jul 28 10:25:01.971175 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:01.971130    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf9ljfv"]
Jul 28 10:25:02.122550 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:25:02.122511    2557 secret.go:189] Couldn't get secret e2e-test-gateway-section-name-propagation-94799d44/gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs: secret "gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs" not found
Jul 28 10:25:02.122746 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:25:02.122588    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35f63890-0ede-4baf-bf15-f7b009a29624-tls-certs podName:35f63890-0ede-4baf-bf15-f7b009a29624 nodeName:}" failed. No retries permitted until 2026-07-28 10:25:02.62257165 +0000 UTC m=+1038.581340007 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/35f63890-0ede-4baf-bf15-f7b009a29624-tls-certs") pod "gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf9ljfv" (UID: "35f63890-0ede-4baf-bf15-f7b009a29624") : secret "gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs" not found
Jul 28 10:25:02.629097 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:25:02.629062    2557 secret.go:189] Couldn't get secret e2e-test-gateway-section-name-propagation-94799d44/gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs: secret "gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs" not found
Jul 28 10:25:02.629248 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:25:02.629144    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35f63890-0ede-4baf-bf15-f7b009a29624-tls-certs podName:35f63890-0ede-4baf-bf15-f7b009a29624 nodeName:}" failed. No retries permitted until 2026-07-28 10:25:03.629127994 +0000 UTC m=+1039.587896350 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/35f63890-0ede-4baf-bf15-f7b009a29624-tls-certs") pod "gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf9ljfv" (UID: "35f63890-0ede-4baf-bf15-f7b009a29624") : secret "gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs" not found
Jul 28 10:25:02.998708 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:25:02.998624    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 10:25:02.998708 ip-10-0-131-88 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:25:02.998708 ip-10-0-131-88 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 28 10:25:02.998708 ip-10-0-131-88 kubenswrapper[2557]:  > containerID="231c84b0345e5b7815b1febfc9a88761b5f1ae240a03d70b11116b427a05be7b" cmd=["/bin/sleep","15"]
Jul 28 10:25:02.998708 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:02.998674    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf9ljfv" message="" err=<
Jul 28 10:25:02.998708 ip-10-0-131-88 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:25:02.998708 ip-10-0-131-88 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 28 10:25:02.998708 ip-10-0-131-88 kubenswrapper[2557]:  >
Jul 28 10:25:02.998708 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:25:02.998688    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 10:25:02.998708 ip-10-0-131-88 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:25:02.998708 ip-10-0-131-88 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 28 10:25:02.998708 ip-10-0-131-88 kubenswrapper[2557]:  > pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf9ljfv" podUID="35f63890-0ede-4baf-bf15-f7b009a29624" containerName="main" containerID="cri-o://231c84b0345e5b7815b1febfc9a88761b5f1ae240a03d70b11116b427a05be7b"
Jul 28 10:25:02.999380 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:02.998725    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf9ljfv" podUID="35f63890-0ede-4baf-bf15-f7b009a29624" containerName="main" containerID="cri-o://231c84b0345e5b7815b1febfc9a88761b5f1ae240a03d70b11116b427a05be7b" gracePeriod=60
Jul 28 10:25:03.242319 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:03.242296    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf9ljfv"
Jul 28 10:25:03.335404 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:03.335369    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35f63890-0ede-4baf-bf15-f7b009a29624-model-cache\") pod \"35f63890-0ede-4baf-bf15-f7b009a29624\" (UID: \"35f63890-0ede-4baf-bf15-f7b009a29624\") "
Jul 28 10:25:03.335646 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:03.335425    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35f63890-0ede-4baf-bf15-f7b009a29624-tls-certs\") pod \"35f63890-0ede-4baf-bf15-f7b009a29624\" (UID: \"35f63890-0ede-4baf-bf15-f7b009a29624\") "
Jul 28 10:25:03.335646 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:03.335458    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35f63890-0ede-4baf-bf15-f7b009a29624-dshm\") pod \"35f63890-0ede-4baf-bf15-f7b009a29624\" (UID: \"35f63890-0ede-4baf-bf15-f7b009a29624\") "
Jul 28 10:25:03.335646 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:03.335474    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35f63890-0ede-4baf-bf15-f7b009a29624-tmp-dir\") pod \"35f63890-0ede-4baf-bf15-f7b009a29624\" (UID: \"35f63890-0ede-4baf-bf15-f7b009a29624\") "
Jul 28 10:25:03.335646 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:03.335562    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35f63890-0ede-4baf-bf15-f7b009a29624-home\") pod \"35f63890-0ede-4baf-bf15-f7b009a29624\" (UID: \"35f63890-0ede-4baf-bf15-f7b009a29624\") "
Jul 28 10:25:03.335646 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:03.335593    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29pr7\" (UniqueName: \"kubernetes.io/projected/35f63890-0ede-4baf-bf15-f7b009a29624-kube-api-access-29pr7\") pod \"35f63890-0ede-4baf-bf15-f7b009a29624\" (UID: \"35f63890-0ede-4baf-bf15-f7b009a29624\") "
Jul 28 10:25:03.335946 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:03.335703    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f63890-0ede-4baf-bf15-f7b009a29624-model-cache" (OuterVolumeSpecName: "model-cache") pod "35f63890-0ede-4baf-bf15-f7b009a29624" (UID: "35f63890-0ede-4baf-bf15-f7b009a29624"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:25:03.335946 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:03.335900    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35f63890-0ede-4baf-bf15-f7b009a29624-model-cache\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:25:03.336045 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:03.335918    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f63890-0ede-4baf-bf15-f7b009a29624-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "35f63890-0ede-4baf-bf15-f7b009a29624" (UID: "35f63890-0ede-4baf-bf15-f7b009a29624"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:25:03.336045 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:03.335980    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f63890-0ede-4baf-bf15-f7b009a29624-home" (OuterVolumeSpecName: "home") pod "35f63890-0ede-4baf-bf15-f7b009a29624" (UID: "35f63890-0ede-4baf-bf15-f7b009a29624"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:25:03.337841 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:03.337813    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f63890-0ede-4baf-bf15-f7b009a29624-dshm" (OuterVolumeSpecName: "dshm") pod "35f63890-0ede-4baf-bf15-f7b009a29624" (UID: "35f63890-0ede-4baf-bf15-f7b009a29624"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:25:03.337956 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:03.337839    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35f63890-0ede-4baf-bf15-f7b009a29624-kube-api-access-29pr7" (OuterVolumeSpecName: "kube-api-access-29pr7") pod "35f63890-0ede-4baf-bf15-f7b009a29624" (UID: "35f63890-0ede-4baf-bf15-f7b009a29624"). InnerVolumeSpecName "kube-api-access-29pr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:25:03.337956 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:03.337921    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35f63890-0ede-4baf-bf15-f7b009a29624-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "35f63890-0ede-4baf-bf15-f7b009a29624" (UID: "35f63890-0ede-4baf-bf15-f7b009a29624"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:25:03.436911 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:03.436879    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35f63890-0ede-4baf-bf15-f7b009a29624-home\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:25:03.436911 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:03.436908    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29pr7\" (UniqueName: \"kubernetes.io/projected/35f63890-0ede-4baf-bf15-f7b009a29624-kube-api-access-29pr7\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:25:03.437112 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:03.436921    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35f63890-0ede-4baf-bf15-f7b009a29624-tls-certs\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:25:03.437112 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:03.436932    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35f63890-0ede-4baf-bf15-f7b009a29624-dshm\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:25:03.437112 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:03.436942    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35f63890-0ede-4baf-bf15-f7b009a29624-tmp-dir\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:25:03.996021 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:03.995984    2557 generic.go:356] "Generic (PLEG): container finished" podID="35f63890-0ede-4baf-bf15-f7b009a29624" containerID="231c84b0345e5b7815b1febfc9a88761b5f1ae240a03d70b11116b427a05be7b" exitCode=0
Jul 28 10:25:03.996021 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:03.996026    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf9ljfv"
Jul 28 10:25:03.996224 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:03.996064    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf9ljfv" event={"ID":"35f63890-0ede-4baf-bf15-f7b009a29624","Type":"ContainerDied","Data":"231c84b0345e5b7815b1febfc9a88761b5f1ae240a03d70b11116b427a05be7b"}
Jul 28 10:25:03.996224 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:03.996108    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf9ljfv" event={"ID":"35f63890-0ede-4baf-bf15-f7b009a29624","Type":"ContainerDied","Data":"e0aadd41c5b9f473342a9c6dbae7087b01d6c95e16594ca8547996a1a883cbaf"}
Jul 28 10:25:03.996224 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:03.996133    2557 scope.go:117] "RemoveContainer" containerID="231c84b0345e5b7815b1febfc9a88761b5f1ae240a03d70b11116b427a05be7b"
Jul 28 10:25:04.005143 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:04.004956    2557 scope.go:117] "RemoveContainer" containerID="231c84b0345e5b7815b1febfc9a88761b5f1ae240a03d70b11116b427a05be7b"
Jul 28 10:25:04.005382 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:25:04.005253    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"231c84b0345e5b7815b1febfc9a88761b5f1ae240a03d70b11116b427a05be7b\": container with ID starting with 231c84b0345e5b7815b1febfc9a88761b5f1ae240a03d70b11116b427a05be7b not found: ID does not exist" containerID="231c84b0345e5b7815b1febfc9a88761b5f1ae240a03d70b11116b427a05be7b"
Jul 28 10:25:04.005382 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:04.005290    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"231c84b0345e5b7815b1febfc9a88761b5f1ae240a03d70b11116b427a05be7b"} err="failed to get container status \"231c84b0345e5b7815b1febfc9a88761b5f1ae240a03d70b11116b427a05be7b\": rpc error: code = NotFound desc = could not find container \"231c84b0345e5b7815b1febfc9a88761b5f1ae240a03d70b11116b427a05be7b\": container with ID starting with 231c84b0345e5b7815b1febfc9a88761b5f1ae240a03d70b11116b427a05be7b not found: ID does not exist"
Jul 28 10:25:04.015875 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:04.015849    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf9ljfv"]
Jul 28 10:25:04.020256 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:04.020235    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf9ljfv"]
Jul 28 10:25:04.580715 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:04.580675    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35f63890-0ede-4baf-bf15-f7b009a29624" path="/var/lib/kubelet/pods/35f63890-0ede-4baf-bf15-f7b009a29624/volumes"
Jul 28 10:25:14.494818 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:14.494689    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr"]
Jul 28 10:25:14.495452 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:14.495429    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35f63890-0ede-4baf-bf15-f7b009a29624" containerName="main"
Jul 28 10:25:14.495452 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:14.495454    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="35f63890-0ede-4baf-bf15-f7b009a29624" containerName="main"
Jul 28 10:25:14.495645 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:14.495631    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="35f63890-0ede-4baf-bf15-f7b009a29624" containerName="main"
Jul 28 10:25:14.499581 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:14.499547    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr"
Jul 28 10:25:14.502046 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:14.502025    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-977dm\""
Jul 28 10:25:14.502170 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:14.502058    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"seaweedfs-s3-creds\""
Jul 28 10:25:14.502433 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:14.502417    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Jul 28 10:25:14.502506 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:14.502488    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Jul 28 10:25:14.502629 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:14.502579    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Jul 28 10:25:14.509769 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:14.509748    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr"]
Jul 28 10:25:14.534440 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:14.534410    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb11e741-9113-4639-a324-6f63a2ab3f30-model-cache\") pod \"auth-enabled-test-kserve-789975db9c-fhgpr\" (UID: \"fb11e741-9113-4639-a324-6f63a2ab3f30\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr"
Jul 28 10:25:14.534585 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:14.534448    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb11e741-9113-4639-a324-6f63a2ab3f30-kserve-provision-location\") pod \"auth-enabled-test-kserve-789975db9c-fhgpr\" (UID: \"fb11e741-9113-4639-a324-6f63a2ab3f30\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr"
Jul 28 10:25:14.534585 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:14.534468    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb11e741-9113-4639-a324-6f63a2ab3f30-tls-certs\") pod \"auth-enabled-test-kserve-789975db9c-fhgpr\" (UID: \"fb11e741-9113-4639-a324-6f63a2ab3f30\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr"
Jul 28 10:25:14.534585 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:14.534502    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb11e741-9113-4639-a324-6f63a2ab3f30-dshm\") pod \"auth-enabled-test-kserve-789975db9c-fhgpr\" (UID: \"fb11e741-9113-4639-a324-6f63a2ab3f30\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr"
Jul 28 10:25:14.534585 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:14.534529    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb11e741-9113-4639-a324-6f63a2ab3f30-home\") pod \"auth-enabled-test-kserve-789975db9c-fhgpr\" (UID: \"fb11e741-9113-4639-a324-6f63a2ab3f30\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr"
Jul 28 10:25:14.534585 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:14.534545    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb11e741-9113-4639-a324-6f63a2ab3f30-tmp-dir\") pod \"auth-enabled-test-kserve-789975db9c-fhgpr\" (UID: \"fb11e741-9113-4639-a324-6f63a2ab3f30\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr"
Jul 28 10:25:14.534585 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:14.534559    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgdzv\" (UniqueName: \"kubernetes.io/projected/fb11e741-9113-4639-a324-6f63a2ab3f30-kube-api-access-lgdzv\") pod \"auth-enabled-test-kserve-789975db9c-fhgpr\" (UID: \"fb11e741-9113-4639-a324-6f63a2ab3f30\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr"
Jul 28 10:25:14.635326 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:14.635286    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb11e741-9113-4639-a324-6f63a2ab3f30-model-cache\") pod \"auth-enabled-test-kserve-789975db9c-fhgpr\" (UID: \"fb11e741-9113-4639-a324-6f63a2ab3f30\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr"
Jul 28 10:25:14.635326 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:14.635329    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb11e741-9113-4639-a324-6f63a2ab3f30-kserve-provision-location\") pod \"auth-enabled-test-kserve-789975db9c-fhgpr\" (UID: \"fb11e741-9113-4639-a324-6f63a2ab3f30\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr"
Jul 28 10:25:14.635571 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:14.635352    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb11e741-9113-4639-a324-6f63a2ab3f30-tls-certs\") pod \"auth-enabled-test-kserve-789975db9c-fhgpr\" (UID: \"fb11e741-9113-4639-a324-6f63a2ab3f30\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr"
Jul 28 10:25:14.635571 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:14.635374    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb11e741-9113-4639-a324-6f63a2ab3f30-dshm\") pod \"auth-enabled-test-kserve-789975db9c-fhgpr\" (UID: \"fb11e741-9113-4639-a324-6f63a2ab3f30\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr"
Jul 28 10:25:14.635571 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:14.635391    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb11e741-9113-4639-a324-6f63a2ab3f30-home\") pod \"auth-enabled-test-kserve-789975db9c-fhgpr\" (UID: \"fb11e741-9113-4639-a324-6f63a2ab3f30\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr"
Jul 28 10:25:14.635571 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:14.635407    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb11e741-9113-4639-a324-6f63a2ab3f30-tmp-dir\") pod \"auth-enabled-test-kserve-789975db9c-fhgpr\" (UID: \"fb11e741-9113-4639-a324-6f63a2ab3f30\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr"
Jul 28 10:25:14.635571 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:14.635423    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgdzv\" (UniqueName: \"kubernetes.io/projected/fb11e741-9113-4639-a324-6f63a2ab3f30-kube-api-access-lgdzv\") pod \"auth-enabled-test-kserve-789975db9c-fhgpr\" (UID: \"fb11e741-9113-4639-a324-6f63a2ab3f30\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr"
Jul 28 10:25:14.635854 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:14.635800    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb11e741-9113-4639-a324-6f63a2ab3f30-kserve-provision-location\") pod \"auth-enabled-test-kserve-789975db9c-fhgpr\" (UID: \"fb11e741-9113-4639-a324-6f63a2ab3f30\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr"
Jul 28 10:25:14.635907 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:14.635882    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb11e741-9113-4639-a324-6f63a2ab3f30-model-cache\") pod \"auth-enabled-test-kserve-789975db9c-fhgpr\" (UID: \"fb11e741-9113-4639-a324-6f63a2ab3f30\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr"
Jul 28 10:25:14.635944 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:14.635905    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb11e741-9113-4639-a324-6f63a2ab3f30-home\") pod \"auth-enabled-test-kserve-789975db9c-fhgpr\" (UID: \"fb11e741-9113-4639-a324-6f63a2ab3f30\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr"
Jul 28 10:25:14.635984 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:14.635947    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb11e741-9113-4639-a324-6f63a2ab3f30-tmp-dir\") pod \"auth-enabled-test-kserve-789975db9c-fhgpr\" (UID: \"fb11e741-9113-4639-a324-6f63a2ab3f30\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr"
Jul 28 10:25:14.637786 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:14.637764    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb11e741-9113-4639-a324-6f63a2ab3f30-dshm\") pod \"auth-enabled-test-kserve-789975db9c-fhgpr\" (UID: \"fb11e741-9113-4639-a324-6f63a2ab3f30\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr"
Jul 28 10:25:14.637929 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:14.637911    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb11e741-9113-4639-a324-6f63a2ab3f30-tls-certs\") pod \"auth-enabled-test-kserve-789975db9c-fhgpr\" (UID: \"fb11e741-9113-4639-a324-6f63a2ab3f30\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr"
Jul 28 10:25:14.643322 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:14.643300    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgdzv\" (UniqueName: \"kubernetes.io/projected/fb11e741-9113-4639-a324-6f63a2ab3f30-kube-api-access-lgdzv\") pod \"auth-enabled-test-kserve-789975db9c-fhgpr\" (UID: \"fb11e741-9113-4639-a324-6f63a2ab3f30\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr"
Jul 28 10:25:14.811793 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:14.811762    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr"
Jul 28 10:25:14.956225 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:14.956196    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr"]
Jul 28 10:25:14.959592 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:25:14.959567    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb11e741_9113_4639_a324_6f63a2ab3f30.slice/crio-7f157f59886b6669e9a9a3d6b2347544fb00c8d03279b8556551cf04f55ad84c WatchSource:0}: Error finding container 7f157f59886b6669e9a9a3d6b2347544fb00c8d03279b8556551cf04f55ad84c: Status 404 returned error can't find the container with id 7f157f59886b6669e9a9a3d6b2347544fb00c8d03279b8556551cf04f55ad84c
Jul 28 10:25:15.046742 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:15.046708    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr" event={"ID":"fb11e741-9113-4639-a324-6f63a2ab3f30","Type":"ContainerStarted","Data":"7f157f59886b6669e9a9a3d6b2347544fb00c8d03279b8556551cf04f55ad84c"}
Jul 28 10:25:19.070765 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:19.070726    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr" event={"ID":"fb11e741-9113-4639-a324-6f63a2ab3f30","Type":"ContainerStarted","Data":"35175a9c33a85a8e4171e42fe59092dff469dcb2175b7a8fd8727f11410373ef"}
Jul 28 10:25:22.088368 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:22.088335    2557 generic.go:356] "Generic (PLEG): container finished" podID="fb11e741-9113-4639-a324-6f63a2ab3f30" containerID="35175a9c33a85a8e4171e42fe59092dff469dcb2175b7a8fd8727f11410373ef" exitCode=0
Jul 28 10:25:22.088763 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:22.088397    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr" event={"ID":"fb11e741-9113-4639-a324-6f63a2ab3f30","Type":"ContainerDied","Data":"35175a9c33a85a8e4171e42fe59092dff469dcb2175b7a8fd8727f11410373ef"}
Jul 28 10:25:51.260812 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:51.260764    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr" event={"ID":"fb11e741-9113-4639-a324-6f63a2ab3f30","Type":"ContainerStarted","Data":"6238d17c1bfc8e07dcfc5cde30b8f0e788541df688b49ba80942faf570a24806"}
Jul 28 10:25:51.283715 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:51.283645    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr" podStartSLOduration=1.574487259 podStartE2EDuration="37.283624788s" totalImagesPullingTime="32.325262322s" totalInitContainerRuntime="3.383875207s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:25:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:25:51.280657885 +0000 UTC m=+1087.239426266" watchObservedRunningTime="2026-07-28 10:25:51.283624788 +0000 UTC m=+1087.242393168"
Jul 28 10:25:51.812862 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:51.812825    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr"
Jul 28 10:25:54.812820 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:54.812780    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr"
Jul 28 10:25:54.814466 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:25:54.814438    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr" podUID="fb11e741-9113-4639-a324-6f63a2ab3f30" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
Jul 28 10:26:00.954463 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:26:00.954422    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 28 10:26:01.167772 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:26:01.167735    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 28 10:26:01.167936 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:26:01.167858    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 10:26:01.170038 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:26:01.170014    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-l5vsp\""
Jul 28 10:26:01.170038 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:26:01.170033    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Jul 28 10:26:01.170591 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:26:01.170572    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Jul 28 10:26:01.170591 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:26:01.170582    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-tb698\""
Jul 28 10:26:01.170792 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:26:01.170778    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Jul 28 10:26:01.170920 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:26:01.170899    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Jul 28 10:26:01.285832 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:26:01.285797    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5ca62e0-c282-4f68-a580-f5504ca253c3-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b5ca62e0-c282-4f68-a580-f5504ca253c3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 10:26:01.285832 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:26:01.285834    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5ca62e0-c282-4f68-a580-f5504ca253c3-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b5ca62e0-c282-4f68-a580-f5504ca253c3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 10:26:01.286084 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:26:01.285965    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5ca62e0-c282-4f68-a580-f5504ca253c3-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b5ca62e0-c282-4f68-a580-f5504ca253c3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 10:26:01.286084 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:26:01.286014    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5ca62e0-c282-4f68-a580-f5504ca253c3-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b5ca62e0-c282-4f68-a580-f5504ca253c3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 10:26:01.286084 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:26:01.286042    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzbr7\" (UniqueName: \"kubernetes.io/projected/b5ca62e0-c282-4f68-a580-f5504ca253c3-kube-api-access-kzbr7\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b5ca62e0-c282-4f68-a580-f5504ca253c3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 10:26:01.286243 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:26:01.286106    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5ca62e0-c282-4f68-a580-f5504ca253c3-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b5ca62e0-c282-4f68-a580-f5504ca253c3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 10:26:01.286243 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:26:01.286134    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5ca62e0-c282-4f68-a580-f5504ca253c3-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b5ca62e0-c282-4f68-a580-f5504ca253c3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 10:26:01.387053 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:26:01.387011    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5ca62e0-c282-4f68-a580-f5504ca253c3-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b5ca62e0-c282-4f68-a580-f5504ca253c3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 10:26:01.387053 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:26:01.387054    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzbr7\" (UniqueName: \"kubernetes.io/projected/b5ca62e0-c282-4f68-a580-f5504ca253c3-kube-api-access-kzbr7\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b5ca62e0-c282-4f68-a580-f5504ca253c3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 10:26:01.387303 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:26:01.387093    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5ca62e0-c282-4f68-a580-f5504ca253c3-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b5ca62e0-c282-4f68-a580-f5504ca253c3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 10:26:01.387303 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:26:01.387235    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5ca62e0-c282-4f68-a580-f5504ca253c3-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b5ca62e0-c282-4f68-a580-f5504ca253c3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 10:26:01.387408 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:26:01.387325    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5ca62e0-c282-4f68-a580-f5504ca253c3-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b5ca62e0-c282-4f68-a580-f5504ca253c3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 10:26:01.387408 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:26:01.387364    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5ca62e0-c282-4f68-a580-f5504ca253c3-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b5ca62e0-c282-4f68-a580-f5504ca253c3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 10:26:01.387480 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:26:01.387422    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5ca62e0-c282-4f68-a580-f5504ca253c3-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b5ca62e0-c282-4f68-a580-f5504ca253c3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 10:26:01.387480 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:26:01.387434    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5ca62e0-c282-4f68-a580-f5504ca253c3-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b5ca62e0-c282-4f68-a580-f5504ca253c3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 10:26:01.387657 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:26:01.387491    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5ca62e0-c282-4f68-a580-f5504ca253c3-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b5ca62e0-c282-4f68-a580-f5504ca253c3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 10:26:01.387657 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:26:01.387579    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5ca62e0-c282-4f68-a580-f5504ca253c3-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b5ca62e0-c282-4f68-a580-f5504ca253c3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 10:26:01.387810 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:26:01.387788    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5ca62e0-c282-4f68-a580-f5504ca253c3-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b5ca62e0-c282-4f68-a580-f5504ca253c3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 10:26:01.389490 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:26:01.389456    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5ca62e0-c282-4f68-a580-f5504ca253c3-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b5ca62e0-c282-4f68-a580-f5504ca253c3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 10:26:01.389689 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:26:01.389674    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5ca62e0-c282-4f68-a580-f5504ca253c3-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b5ca62e0-c282-4f68-a580-f5504ca253c3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 10:26:01.394835 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:26:01.394813    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzbr7\" (UniqueName: \"kubernetes.io/projected/b5ca62e0-c282-4f68-a580-f5504ca253c3-kube-api-access-kzbr7\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b5ca62e0-c282-4f68-a580-f5504ca253c3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 10:26:01.478839 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:26:01.478802    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 10:26:01.675708 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:26:01.675405    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 28 10:26:01.678571 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:26:01.678540    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5ca62e0_c282_4f68_a580_f5504ca253c3.slice/crio-091bc7e610b01e55ff461547dc1ffbc00729dea20ec461305639fe07493183d7 WatchSource:0}: Error finding container 091bc7e610b01e55ff461547dc1ffbc00729dea20ec461305639fe07493183d7: Status 404 returned error can't find the container with id 091bc7e610b01e55ff461547dc1ffbc00729dea20ec461305639fe07493183d7
Jul 28 10:26:02.314685 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:26:02.314640    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"b5ca62e0-c282-4f68-a580-f5504ca253c3","Type":"ContainerStarted","Data":"d58dcc06f6d1a12b6eb8ce76ac79524f1fba5aedbc2b797bd02207ed5bbdc84d"}
Jul 28 10:26:02.315116 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:26:02.314708    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"b5ca62e0-c282-4f68-a580-f5504ca253c3","Type":"ContainerStarted","Data":"091bc7e610b01e55ff461547dc1ffbc00729dea20ec461305639fe07493183d7"}
Jul 28 10:26:04.812782 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:26:04.812739    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr" podUID="fb11e741-9113-4639-a324-6f63a2ab3f30" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
Jul 28 10:26:07.341920 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:26:07.341877    2557 generic.go:356] "Generic (PLEG): container finished" podID="b5ca62e0-c282-4f68-a580-f5504ca253c3" containerID="d58dcc06f6d1a12b6eb8ce76ac79524f1fba5aedbc2b797bd02207ed5bbdc84d" exitCode=0
Jul 28 10:26:07.342286 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:26:07.341949    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"b5ca62e0-c282-4f68-a580-f5504ca253c3","Type":"ContainerDied","Data":"d58dcc06f6d1a12b6eb8ce76ac79524f1fba5aedbc2b797bd02207ed5bbdc84d"}
Jul 28 10:26:08.350034 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:26:08.350000    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"b5ca62e0-c282-4f68-a580-f5504ca253c3","Type":"ContainerStarted","Data":"631b5ae4efb1615c840179adaa6cc6180c54bd156f6d35a8f2380be6b4a53aa3"}
Jul 28 10:26:08.374222 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:26:08.374160    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=3.795153902 podStartE2EDuration="8.374138818s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.578984916s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:26:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:26:08.371953699 +0000 UTC m=+1104.330722114" watchObservedRunningTime="2026-07-28 10:26:08.374138818 +0000 UTC m=+1104.332907201"
Jul 28 10:26:14.812295 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:26:14.812252    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr" podUID="fb11e741-9113-4639-a324-6f63a2ab3f30" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
Jul 28 10:26:24.812956 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:26:24.812905    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr" podUID="fb11e741-9113-4639-a324-6f63a2ab3f30" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
Jul 28 10:26:34.812749 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:26:34.812695    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr" podUID="fb11e741-9113-4639-a324-6f63a2ab3f30" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
Jul 28 10:26:44.812423 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:26:44.812377    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr" podUID="fb11e741-9113-4639-a324-6f63a2ab3f30" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
Jul 28 10:26:54.813079 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:26:54.813036    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr" podUID="fb11e741-9113-4639-a324-6f63a2ab3f30" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
Jul 28 10:27:04.812181 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:27:04.812142    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr" podUID="fb11e741-9113-4639-a324-6f63a2ab3f30" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
Jul 28 10:27:14.812268 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:27:14.812177    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr" podUID="fb11e741-9113-4639-a324-6f63a2ab3f30" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
Jul 28 10:27:24.812697 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:27:24.812650    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr" podUID="fb11e741-9113-4639-a324-6f63a2ab3f30" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
Jul 28 10:27:34.812251 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:27:34.812198    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr" podUID="fb11e741-9113-4639-a324-6f63a2ab3f30" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
Jul 28 10:27:44.812848 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:27:44.812808    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr" podUID="fb11e741-9113-4639-a324-6f63a2ab3f30" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
Jul 28 10:27:54.812524 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:27:54.812481    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr" podUID="fb11e741-9113-4639-a324-6f63a2ab3f30" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
Jul 28 10:28:04.813236 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:28:04.813193    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr" podUID="fb11e741-9113-4639-a324-6f63a2ab3f30" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
Jul 28 10:28:14.812298 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:28:14.812253    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr" podUID="fb11e741-9113-4639-a324-6f63a2ab3f30" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
Jul 28 10:28:24.813081 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:28:24.813037    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr" podUID="fb11e741-9113-4639-a324-6f63a2ab3f30" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
Jul 28 10:28:34.822370 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:28:34.822339    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr"
Jul 28 10:28:34.830221 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:28:34.830197    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr"
Jul 28 10:28:40.487689 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:28:40.487646    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr"]
Jul 28 10:28:55.510825 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:28:55.510784    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr" podUID="fb11e741-9113-4639-a324-6f63a2ab3f30" containerName="main" containerID="cri-o://6238d17c1bfc8e07dcfc5cde30b8f0e788541df688b49ba80942faf570a24806" gracePeriod=45
Jul 28 10:28:55.823962 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:28:55.823858    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr" podUID="fb11e741-9113-4639-a324-6f63a2ab3f30" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
Jul 28 10:28:56.172752 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:28:56.172722    2557 generic.go:356] "Generic (PLEG): container finished" podID="fb11e741-9113-4639-a324-6f63a2ab3f30" containerID="6238d17c1bfc8e07dcfc5cde30b8f0e788541df688b49ba80942faf570a24806" exitCode=0
Jul 28 10:28:56.172906 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:28:56.172776    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr" event={"ID":"fb11e741-9113-4639-a324-6f63a2ab3f30","Type":"ContainerDied","Data":"6238d17c1bfc8e07dcfc5cde30b8f0e788541df688b49ba80942faf570a24806"}
Jul 28 10:28:56.262009 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:28:56.261984    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr"
Jul 28 10:28:56.356872 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:28:56.356834    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb11e741-9113-4639-a324-6f63a2ab3f30-kserve-provision-location\") pod \"fb11e741-9113-4639-a324-6f63a2ab3f30\" (UID: \"fb11e741-9113-4639-a324-6f63a2ab3f30\") "
Jul 28 10:28:56.357035 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:28:56.356885    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb11e741-9113-4639-a324-6f63a2ab3f30-dshm\") pod \"fb11e741-9113-4639-a324-6f63a2ab3f30\" (UID: \"fb11e741-9113-4639-a324-6f63a2ab3f30\") "
Jul 28 10:28:56.357035 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:28:56.356903    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb11e741-9113-4639-a324-6f63a2ab3f30-home\") pod \"fb11e741-9113-4639-a324-6f63a2ab3f30\" (UID: \"fb11e741-9113-4639-a324-6f63a2ab3f30\") "
Jul 28 10:28:56.357035 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:28:56.356936    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb11e741-9113-4639-a324-6f63a2ab3f30-model-cache\") pod \"fb11e741-9113-4639-a324-6f63a2ab3f30\" (UID: \"fb11e741-9113-4639-a324-6f63a2ab3f30\") "
Jul 28 10:28:56.357035 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:28:56.356964    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb11e741-9113-4639-a324-6f63a2ab3f30-tmp-dir\") pod \"fb11e741-9113-4639-a324-6f63a2ab3f30\" (UID: \"fb11e741-9113-4639-a324-6f63a2ab3f30\") "
Jul 28 10:28:56.357035 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:28:56.356998    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb11e741-9113-4639-a324-6f63a2ab3f30-tls-certs\") pod \"fb11e741-9113-4639-a324-6f63a2ab3f30\" (UID: \"fb11e741-9113-4639-a324-6f63a2ab3f30\") "
Jul 28 10:28:56.357035 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:28:56.357036    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgdzv\" (UniqueName: \"kubernetes.io/projected/fb11e741-9113-4639-a324-6f63a2ab3f30-kube-api-access-lgdzv\") pod \"fb11e741-9113-4639-a324-6f63a2ab3f30\" (UID: \"fb11e741-9113-4639-a324-6f63a2ab3f30\") "
Jul 28 10:28:56.357316 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:28:56.357255    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb11e741-9113-4639-a324-6f63a2ab3f30-model-cache" (OuterVolumeSpecName: "model-cache") pod "fb11e741-9113-4639-a324-6f63a2ab3f30" (UID: "fb11e741-9113-4639-a324-6f63a2ab3f30"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:28:56.357699 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:28:56.357668    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb11e741-9113-4639-a324-6f63a2ab3f30-home" (OuterVolumeSpecName: "home") pod "fb11e741-9113-4639-a324-6f63a2ab3f30" (UID: "fb11e741-9113-4639-a324-6f63a2ab3f30"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:28:56.359290 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:28:56.359264    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb11e741-9113-4639-a324-6f63a2ab3f30-dshm" (OuterVolumeSpecName: "dshm") pod "fb11e741-9113-4639-a324-6f63a2ab3f30" (UID: "fb11e741-9113-4639-a324-6f63a2ab3f30"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:28:56.359393 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:28:56.359305    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb11e741-9113-4639-a324-6f63a2ab3f30-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fb11e741-9113-4639-a324-6f63a2ab3f30" (UID: "fb11e741-9113-4639-a324-6f63a2ab3f30"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:28:56.359572 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:28:56.359554    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb11e741-9113-4639-a324-6f63a2ab3f30-kube-api-access-lgdzv" (OuterVolumeSpecName: "kube-api-access-lgdzv") pod "fb11e741-9113-4639-a324-6f63a2ab3f30" (UID: "fb11e741-9113-4639-a324-6f63a2ab3f30"). InnerVolumeSpecName "kube-api-access-lgdzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:28:56.371778 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:28:56.371745    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb11e741-9113-4639-a324-6f63a2ab3f30-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fb11e741-9113-4639-a324-6f63a2ab3f30" (UID: "fb11e741-9113-4639-a324-6f63a2ab3f30"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:28:56.420102 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:28:56.420019    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb11e741-9113-4639-a324-6f63a2ab3f30-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fb11e741-9113-4639-a324-6f63a2ab3f30" (UID: "fb11e741-9113-4639-a324-6f63a2ab3f30"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:28:56.458634 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:28:56.458580    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb11e741-9113-4639-a324-6f63a2ab3f30-tmp-dir\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:28:56.458634 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:28:56.458631    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb11e741-9113-4639-a324-6f63a2ab3f30-tls-certs\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:28:56.458904 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:28:56.458647    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgdzv\" (UniqueName: \"kubernetes.io/projected/fb11e741-9113-4639-a324-6f63a2ab3f30-kube-api-access-lgdzv\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:28:56.458904 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:28:56.458661    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb11e741-9113-4639-a324-6f63a2ab3f30-kserve-provision-location\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:28:56.458904 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:28:56.458674    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb11e741-9113-4639-a324-6f63a2ab3f30-dshm\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:28:56.458904 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:28:56.458684    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb11e741-9113-4639-a324-6f63a2ab3f30-home\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:28:56.458904 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:28:56.458694    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb11e741-9113-4639-a324-6f63a2ab3f30-model-cache\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:28:57.179037 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:28:57.179005    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr"
Jul 28 10:28:57.179037 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:28:57.179022    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr" event={"ID":"fb11e741-9113-4639-a324-6f63a2ab3f30","Type":"ContainerDied","Data":"7f157f59886b6669e9a9a3d6b2347544fb00c8d03279b8556551cf04f55ad84c"}
Jul 28 10:28:57.179575 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:28:57.179079    2557 scope.go:117] "RemoveContainer" containerID="6238d17c1bfc8e07dcfc5cde30b8f0e788541df688b49ba80942faf570a24806"
Jul 28 10:28:57.187544 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:28:57.187526    2557 scope.go:117] "RemoveContainer" containerID="35175a9c33a85a8e4171e42fe59092dff469dcb2175b7a8fd8727f11410373ef"
Jul 28 10:28:57.197229 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:28:57.197207    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr"]
Jul 28 10:28:57.200893 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:28:57.200868    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-789975db9c-fhgpr"]
Jul 28 10:28:58.580149 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:28:58.580114    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb11e741-9113-4639-a324-6f63a2ab3f30" path="/var/lib/kubelet/pods/fb11e741-9113-4639-a324-6f63a2ab3f30/volumes"
Jul 28 10:29:02.297255 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:02.297219    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 28 10:29:17.319990 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:17.319928    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="b5ca62e0-c282-4f68-a580-f5504ca253c3" containerName="main" containerID="cri-o://631b5ae4efb1615c840179adaa6cc6180c54bd156f6d35a8f2380be6b4a53aa3" gracePeriod=45
Jul 28 10:29:17.991441 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:17.991417    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 10:29:18.036305 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:18.036277    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5ca62e0-c282-4f68-a580-f5504ca253c3-home\") pod \"b5ca62e0-c282-4f68-a580-f5504ca253c3\" (UID: \"b5ca62e0-c282-4f68-a580-f5504ca253c3\") "
Jul 28 10:29:18.036503 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:18.036325    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kzbr7\" (UniqueName: \"kubernetes.io/projected/b5ca62e0-c282-4f68-a580-f5504ca253c3-kube-api-access-kzbr7\") pod \"b5ca62e0-c282-4f68-a580-f5504ca253c3\" (UID: \"b5ca62e0-c282-4f68-a580-f5504ca253c3\") "
Jul 28 10:29:18.036503 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:18.036369    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5ca62e0-c282-4f68-a580-f5504ca253c3-dshm\") pod \"b5ca62e0-c282-4f68-a580-f5504ca253c3\" (UID: \"b5ca62e0-c282-4f68-a580-f5504ca253c3\") "
Jul 28 10:29:18.036503 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:18.036401    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5ca62e0-c282-4f68-a580-f5504ca253c3-tls-certs\") pod \"b5ca62e0-c282-4f68-a580-f5504ca253c3\" (UID: \"b5ca62e0-c282-4f68-a580-f5504ca253c3\") "
Jul 28 10:29:18.036712 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:18.036544    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5ca62e0-c282-4f68-a580-f5504ca253c3-model-cache\") pod \"b5ca62e0-c282-4f68-a580-f5504ca253c3\" (UID: \"b5ca62e0-c282-4f68-a580-f5504ca253c3\") "
Jul 28 10:29:18.036712 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:18.036619    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5ca62e0-c282-4f68-a580-f5504ca253c3-tmp-dir\") pod \"b5ca62e0-c282-4f68-a580-f5504ca253c3\" (UID: \"b5ca62e0-c282-4f68-a580-f5504ca253c3\") "
Jul 28 10:29:18.036712 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:18.036653    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5ca62e0-c282-4f68-a580-f5504ca253c3-kserve-provision-location\") pod \"b5ca62e0-c282-4f68-a580-f5504ca253c3\" (UID: \"b5ca62e0-c282-4f68-a580-f5504ca253c3\") "
Jul 28 10:29:18.036872 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:18.036811    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5ca62e0-c282-4f68-a580-f5504ca253c3-model-cache" (OuterVolumeSpecName: "model-cache") pod "b5ca62e0-c282-4f68-a580-f5504ca253c3" (UID: "b5ca62e0-c282-4f68-a580-f5504ca253c3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:29:18.036959 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:18.036930    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5ca62e0-c282-4f68-a580-f5504ca253c3-home" (OuterVolumeSpecName: "home") pod "b5ca62e0-c282-4f68-a580-f5504ca253c3" (UID: "b5ca62e0-c282-4f68-a580-f5504ca253c3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:29:18.037032 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:18.037016    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5ca62e0-c282-4f68-a580-f5504ca253c3-model-cache\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:29:18.037091 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:18.037037    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5ca62e0-c282-4f68-a580-f5504ca253c3-home\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:29:18.039161 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:18.039132    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5ca62e0-c282-4f68-a580-f5504ca253c3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b5ca62e0-c282-4f68-a580-f5504ca253c3" (UID: "b5ca62e0-c282-4f68-a580-f5504ca253c3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:29:18.039161 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:18.039141    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5ca62e0-c282-4f68-a580-f5504ca253c3-kube-api-access-kzbr7" (OuterVolumeSpecName: "kube-api-access-kzbr7") pod "b5ca62e0-c282-4f68-a580-f5504ca253c3" (UID: "b5ca62e0-c282-4f68-a580-f5504ca253c3"). InnerVolumeSpecName "kube-api-access-kzbr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:29:18.039294 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:18.039149    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5ca62e0-c282-4f68-a580-f5504ca253c3-dshm" (OuterVolumeSpecName: "dshm") pod "b5ca62e0-c282-4f68-a580-f5504ca253c3" (UID: "b5ca62e0-c282-4f68-a580-f5504ca253c3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:29:18.054779 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:18.054746    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5ca62e0-c282-4f68-a580-f5504ca253c3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b5ca62e0-c282-4f68-a580-f5504ca253c3" (UID: "b5ca62e0-c282-4f68-a580-f5504ca253c3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:29:18.093875 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:18.093809    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5ca62e0-c282-4f68-a580-f5504ca253c3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b5ca62e0-c282-4f68-a580-f5504ca253c3" (UID: "b5ca62e0-c282-4f68-a580-f5504ca253c3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:29:18.137703 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:18.137672    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5ca62e0-c282-4f68-a580-f5504ca253c3-tls-certs\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:29:18.137703 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:18.137700    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5ca62e0-c282-4f68-a580-f5504ca253c3-tmp-dir\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:29:18.137859 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:18.137710    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5ca62e0-c282-4f68-a580-f5504ca253c3-kserve-provision-location\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:29:18.137859 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:18.137720    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzbr7\" (UniqueName: \"kubernetes.io/projected/b5ca62e0-c282-4f68-a580-f5504ca253c3-kube-api-access-kzbr7\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:29:18.137859 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:18.137729    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5ca62e0-c282-4f68-a580-f5504ca253c3-dshm\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:29:18.276443 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:18.276413    2557 generic.go:356] "Generic (PLEG): container finished" podID="b5ca62e0-c282-4f68-a580-f5504ca253c3" containerID="631b5ae4efb1615c840179adaa6cc6180c54bd156f6d35a8f2380be6b4a53aa3" exitCode=0
Jul 28 10:29:18.276633 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:18.276466    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 10:29:18.276633 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:18.276495    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"b5ca62e0-c282-4f68-a580-f5504ca253c3","Type":"ContainerDied","Data":"631b5ae4efb1615c840179adaa6cc6180c54bd156f6d35a8f2380be6b4a53aa3"}
Jul 28 10:29:18.276633 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:18.276546    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"b5ca62e0-c282-4f68-a580-f5504ca253c3","Type":"ContainerDied","Data":"091bc7e610b01e55ff461547dc1ffbc00729dea20ec461305639fe07493183d7"}
Jul 28 10:29:18.276633 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:18.276569    2557 scope.go:117] "RemoveContainer" containerID="631b5ae4efb1615c840179adaa6cc6180c54bd156f6d35a8f2380be6b4a53aa3"
Jul 28 10:29:18.285884 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:18.285860    2557 scope.go:117] "RemoveContainer" containerID="d58dcc06f6d1a12b6eb8ce76ac79524f1fba5aedbc2b797bd02207ed5bbdc84d"
Jul 28 10:29:18.293330 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:18.293313    2557 scope.go:117] "RemoveContainer" containerID="631b5ae4efb1615c840179adaa6cc6180c54bd156f6d35a8f2380be6b4a53aa3"
Jul 28 10:29:18.293573 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:29:18.293555    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"631b5ae4efb1615c840179adaa6cc6180c54bd156f6d35a8f2380be6b4a53aa3\": container with ID starting with 631b5ae4efb1615c840179adaa6cc6180c54bd156f6d35a8f2380be6b4a53aa3 not found: ID does not exist" containerID="631b5ae4efb1615c840179adaa6cc6180c54bd156f6d35a8f2380be6b4a53aa3"
Jul 28 10:29:18.293636 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:18.293583    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"631b5ae4efb1615c840179adaa6cc6180c54bd156f6d35a8f2380be6b4a53aa3"} err="failed to get container status \"631b5ae4efb1615c840179adaa6cc6180c54bd156f6d35a8f2380be6b4a53aa3\": rpc error: code = NotFound desc = could not find container \"631b5ae4efb1615c840179adaa6cc6180c54bd156f6d35a8f2380be6b4a53aa3\": container with ID starting with 631b5ae4efb1615c840179adaa6cc6180c54bd156f6d35a8f2380be6b4a53aa3 not found: ID does not exist"
Jul 28 10:29:18.293636 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:18.293613    2557 scope.go:117] "RemoveContainer" containerID="d58dcc06f6d1a12b6eb8ce76ac79524f1fba5aedbc2b797bd02207ed5bbdc84d"
Jul 28 10:29:18.293851 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:29:18.293836    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d58dcc06f6d1a12b6eb8ce76ac79524f1fba5aedbc2b797bd02207ed5bbdc84d\": container with ID starting with d58dcc06f6d1a12b6eb8ce76ac79524f1fba5aedbc2b797bd02207ed5bbdc84d not found: ID does not exist" containerID="d58dcc06f6d1a12b6eb8ce76ac79524f1fba5aedbc2b797bd02207ed5bbdc84d"
Jul 28 10:29:18.293901 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:18.293855    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d58dcc06f6d1a12b6eb8ce76ac79524f1fba5aedbc2b797bd02207ed5bbdc84d"} err="failed to get container status \"d58dcc06f6d1a12b6eb8ce76ac79524f1fba5aedbc2b797bd02207ed5bbdc84d\": rpc error: code = NotFound desc = could not find container \"d58dcc06f6d1a12b6eb8ce76ac79524f1fba5aedbc2b797bd02207ed5bbdc84d\": container with ID starting with d58dcc06f6d1a12b6eb8ce76ac79524f1fba5aedbc2b797bd02207ed5bbdc84d not found: ID does not exist"
Jul 28 10:29:18.298565 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:18.298541    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 28 10:29:18.304625 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:18.304587    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 28 10:29:18.580217 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:18.580184    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5ca62e0-c282-4f68-a580-f5504ca253c3" path="/var/lib/kubelet/pods/b5ca62e0-c282-4f68-a580-f5504ca253c3/volumes"
Jul 28 10:29:47.083721 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:47.083689    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-76br8"]
Jul 28 10:29:47.084151 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:47.083989    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5ca62e0-c282-4f68-a580-f5504ca253c3" containerName="storage-initializer"
Jul 28 10:29:47.084151 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:47.083999    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5ca62e0-c282-4f68-a580-f5504ca253c3" containerName="storage-initializer"
Jul 28 10:29:47.084151 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:47.084019    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5ca62e0-c282-4f68-a580-f5504ca253c3" containerName="main"
Jul 28 10:29:47.084151 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:47.084024    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5ca62e0-c282-4f68-a580-f5504ca253c3" containerName="main"
Jul 28 10:29:47.084151 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:47.084031    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb11e741-9113-4639-a324-6f63a2ab3f30" containerName="storage-initializer"
Jul 28 10:29:47.084151 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:47.084037    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb11e741-9113-4639-a324-6f63a2ab3f30" containerName="storage-initializer"
Jul 28 10:29:47.084151 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:47.084047    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb11e741-9113-4639-a324-6f63a2ab3f30" containerName="main"
Jul 28 10:29:47.084151 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:47.084054    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb11e741-9113-4639-a324-6f63a2ab3f30" containerName="main"
Jul 28 10:29:47.084635 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:47.084594    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb11e741-9113-4639-a324-6f63a2ab3f30" containerName="main"
Jul 28 10:29:47.090067 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:47.090038    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5ca62e0-c282-4f68-a580-f5504ca253c3" containerName="main"
Jul 28 10:29:47.093369 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:47.093344    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-76br8"
Jul 28 10:29:47.096793 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:47.096770    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-52jn5\""
Jul 28 10:29:47.097258 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:47.097241    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Jul 28 10:29:47.097381 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:47.097243    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Jul 28 10:29:47.098567 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:47.098551    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Jul 28 10:29:47.103983 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:47.103955    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-76br8"]
Jul 28 10:29:47.208136 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:47.208092    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bc8aa1c-5edc-441e-9972-b00d4caa2fe2-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-76br8\" (UID: \"0bc8aa1c-5edc-441e-9972-b00d4caa2fe2\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-76br8"
Jul 28 10:29:47.208313 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:47.208179    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bc8aa1c-5edc-441e-9972-b00d4caa2fe2-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-76br8\" (UID: \"0bc8aa1c-5edc-441e-9972-b00d4caa2fe2\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-76br8"
Jul 28 10:29:47.208313 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:47.208209    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9pjq\" (UniqueName: \"kubernetes.io/projected/0bc8aa1c-5edc-441e-9972-b00d4caa2fe2-kube-api-access-j9pjq\") pod \"scheduler-inline-config-test-kserve-8655bc757-76br8\" (UID: \"0bc8aa1c-5edc-441e-9972-b00d4caa2fe2\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-76br8"
Jul 28 10:29:47.208313 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:47.208243    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bc8aa1c-5edc-441e-9972-b00d4caa2fe2-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-76br8\" (UID: \"0bc8aa1c-5edc-441e-9972-b00d4caa2fe2\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-76br8"
Jul 28 10:29:47.208423 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:47.208312    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bc8aa1c-5edc-441e-9972-b00d4caa2fe2-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-76br8\" (UID: \"0bc8aa1c-5edc-441e-9972-b00d4caa2fe2\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-76br8"
Jul 28 10:29:47.208423 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:47.208377    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bc8aa1c-5edc-441e-9972-b00d4caa2fe2-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-76br8\" (UID: \"0bc8aa1c-5edc-441e-9972-b00d4caa2fe2\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-76br8"
Jul 28 10:29:47.309387 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:47.309352    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bc8aa1c-5edc-441e-9972-b00d4caa2fe2-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-76br8\" (UID: \"0bc8aa1c-5edc-441e-9972-b00d4caa2fe2\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-76br8"
Jul 28 10:29:47.309653 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:47.309414    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bc8aa1c-5edc-441e-9972-b00d4caa2fe2-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-76br8\" (UID: \"0bc8aa1c-5edc-441e-9972-b00d4caa2fe2\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-76br8"
Jul 28 10:29:47.309653 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:47.309439    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9pjq\" (UniqueName: \"kubernetes.io/projected/0bc8aa1c-5edc-441e-9972-b00d4caa2fe2-kube-api-access-j9pjq\") pod \"scheduler-inline-config-test-kserve-8655bc757-76br8\" (UID: \"0bc8aa1c-5edc-441e-9972-b00d4caa2fe2\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-76br8"
Jul 28 10:29:47.309653 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:47.309465    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bc8aa1c-5edc-441e-9972-b00d4caa2fe2-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-76br8\" (UID: \"0bc8aa1c-5edc-441e-9972-b00d4caa2fe2\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-76br8"
Jul 28 10:29:47.309653 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:47.309492    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bc8aa1c-5edc-441e-9972-b00d4caa2fe2-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-76br8\" (UID: \"0bc8aa1c-5edc-441e-9972-b00d4caa2fe2\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-76br8"
Jul 28 10:29:47.309653 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:47.309523    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bc8aa1c-5edc-441e-9972-b00d4caa2fe2-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-76br8\" (UID: \"0bc8aa1c-5edc-441e-9972-b00d4caa2fe2\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-76br8"
Jul 28 10:29:47.310020 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:47.309929    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bc8aa1c-5edc-441e-9972-b00d4caa2fe2-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-76br8\" (UID: \"0bc8aa1c-5edc-441e-9972-b00d4caa2fe2\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-76br8"
Jul 28 10:29:47.310127 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:47.310088    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bc8aa1c-5edc-441e-9972-b00d4caa2fe2-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-76br8\" (UID: \"0bc8aa1c-5edc-441e-9972-b00d4caa2fe2\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-76br8"
Jul 28 10:29:47.310189 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:47.310152    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bc8aa1c-5edc-441e-9972-b00d4caa2fe2-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-76br8\" (UID: \"0bc8aa1c-5edc-441e-9972-b00d4caa2fe2\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-76br8"
Jul 28 10:29:47.312499 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:47.312471    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bc8aa1c-5edc-441e-9972-b00d4caa2fe2-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-76br8\" (UID: \"0bc8aa1c-5edc-441e-9972-b00d4caa2fe2\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-76br8"
Jul 28 10:29:47.312702 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:47.312682    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bc8aa1c-5edc-441e-9972-b00d4caa2fe2-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-76br8\" (UID: \"0bc8aa1c-5edc-441e-9972-b00d4caa2fe2\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-76br8"
Jul 28 10:29:47.323761 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:47.323733    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9pjq\" (UniqueName: \"kubernetes.io/projected/0bc8aa1c-5edc-441e-9972-b00d4caa2fe2-kube-api-access-j9pjq\") pod \"scheduler-inline-config-test-kserve-8655bc757-76br8\" (UID: \"0bc8aa1c-5edc-441e-9972-b00d4caa2fe2\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-76br8"
Jul 28 10:29:47.405239 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:47.405155    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-76br8"
Jul 28 10:29:47.549070 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:47.549045    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-76br8"]
Jul 28 10:29:47.551434 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:29:47.551409    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bc8aa1c_5edc_441e_9972_b00d4caa2fe2.slice/crio-23011577caed4acf708ddc11e1cf421bc6e6d8ed431cdf124e1c017e00e1ad1a WatchSource:0}: Error finding container 23011577caed4acf708ddc11e1cf421bc6e6d8ed431cdf124e1c017e00e1ad1a: Status 404 returned error can't find the container with id 23011577caed4acf708ddc11e1cf421bc6e6d8ed431cdf124e1c017e00e1ad1a
Jul 28 10:29:48.412345 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:48.412298    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-76br8" event={"ID":"0bc8aa1c-5edc-441e-9972-b00d4caa2fe2","Type":"ContainerStarted","Data":"0e635bbd1d9834e453a4975a545b058c55111bf2b52712f3f5944d35099b7119"}
Jul 28 10:29:48.412834 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:48.412372    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-76br8" event={"ID":"0bc8aa1c-5edc-441e-9972-b00d4caa2fe2","Type":"ContainerStarted","Data":"23011577caed4acf708ddc11e1cf421bc6e6d8ed431cdf124e1c017e00e1ad1a"}
Jul 28 10:29:48.434917 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:48.434853    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-76br8" podStartSLOduration=1.434830131 podStartE2EDuration="1.434830131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:29:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:29:48.432129822 +0000 UTC m=+1324.390898200" watchObservedRunningTime="2026-07-28 10:29:48.434830131 +0000 UTC m=+1324.393598511"
Jul 28 10:29:49.405444 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:49.405412    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-76br8"
Jul 28 10:29:57.405665 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:57.405631    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-76br8"
Jul 28 10:29:57.417984 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:57.417960    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-76br8"
Jul 28 10:29:57.461956 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:29:57.461928    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-76br8"
Jul 28 10:30:25.319710 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:25.319613    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-76br8"]
Jul 28 10:30:25.332445 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:30:25.332394    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 10:30:25.332445 ip-10-0-131-88 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:30:25.332445 ip-10-0-131-88 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 28 10:30:25.332445 ip-10-0-131-88 kubenswrapper[2557]:  > containerID="0e635bbd1d9834e453a4975a545b058c55111bf2b52712f3f5944d35099b7119" cmd=["/bin/sleep","15"]
Jul 28 10:30:25.332715 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:25.332475    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-76br8" message="" err=<
Jul 28 10:30:25.332715 ip-10-0-131-88 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:30:25.332715 ip-10-0-131-88 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 28 10:30:25.332715 ip-10-0-131-88 kubenswrapper[2557]:  >
Jul 28 10:30:25.332715 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:30:25.332493    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 10:30:25.332715 ip-10-0-131-88 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:30:25.332715 ip-10-0-131-88 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 28 10:30:25.332715 ip-10-0-131-88 kubenswrapper[2557]:  > pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-76br8" podUID="0bc8aa1c-5edc-441e-9972-b00d4caa2fe2" containerName="main" containerID="cri-o://0e635bbd1d9834e453a4975a545b058c55111bf2b52712f3f5944d35099b7119"
Jul 28 10:30:25.332715 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:25.332557    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-76br8" podUID="0bc8aa1c-5edc-441e-9972-b00d4caa2fe2" containerName="main" containerID="cri-o://0e635bbd1d9834e453a4975a545b058c55111bf2b52712f3f5944d35099b7119" gracePeriod=60
Jul 28 10:30:25.451161 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:25.451129    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-76br8" podUID="0bc8aa1c-5edc-441e-9972-b00d4caa2fe2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 28 10:30:25.576273 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:25.576198    2557 generic.go:356] "Generic (PLEG): container finished" podID="0bc8aa1c-5edc-441e-9972-b00d4caa2fe2" containerID="0e635bbd1d9834e453a4975a545b058c55111bf2b52712f3f5944d35099b7119" exitCode=0
Jul 28 10:30:25.576273 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:25.576247    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-76br8" event={"ID":"0bc8aa1c-5edc-441e-9972-b00d4caa2fe2","Type":"ContainerDied","Data":"0e635bbd1d9834e453a4975a545b058c55111bf2b52712f3f5944d35099b7119"}
Jul 28 10:30:25.689897 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:25.689867    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-76br8"
Jul 28 10:30:25.844942 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:25.844842    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bc8aa1c-5edc-441e-9972-b00d4caa2fe2-model-cache\") pod \"0bc8aa1c-5edc-441e-9972-b00d4caa2fe2\" (UID: \"0bc8aa1c-5edc-441e-9972-b00d4caa2fe2\") "
Jul 28 10:30:25.844942 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:25.844928    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bc8aa1c-5edc-441e-9972-b00d4caa2fe2-home\") pod \"0bc8aa1c-5edc-441e-9972-b00d4caa2fe2\" (UID: \"0bc8aa1c-5edc-441e-9972-b00d4caa2fe2\") "
Jul 28 10:30:25.845189 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:25.844977    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bc8aa1c-5edc-441e-9972-b00d4caa2fe2-tmp-dir\") pod \"0bc8aa1c-5edc-441e-9972-b00d4caa2fe2\" (UID: \"0bc8aa1c-5edc-441e-9972-b00d4caa2fe2\") "
Jul 28 10:30:25.845189 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:25.844998    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bc8aa1c-5edc-441e-9972-b00d4caa2fe2-tls-certs\") pod \"0bc8aa1c-5edc-441e-9972-b00d4caa2fe2\" (UID: \"0bc8aa1c-5edc-441e-9972-b00d4caa2fe2\") "
Jul 28 10:30:25.845189 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:25.845039    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bc8aa1c-5edc-441e-9972-b00d4caa2fe2-dshm\") pod \"0bc8aa1c-5edc-441e-9972-b00d4caa2fe2\" (UID: \"0bc8aa1c-5edc-441e-9972-b00d4caa2fe2\") "
Jul 28 10:30:25.845189 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:25.845063    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j9pjq\" (UniqueName: \"kubernetes.io/projected/0bc8aa1c-5edc-441e-9972-b00d4caa2fe2-kube-api-access-j9pjq\") pod \"0bc8aa1c-5edc-441e-9972-b00d4caa2fe2\" (UID: \"0bc8aa1c-5edc-441e-9972-b00d4caa2fe2\") "
Jul 28 10:30:25.845189 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:25.845139    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bc8aa1c-5edc-441e-9972-b00d4caa2fe2-model-cache" (OuterVolumeSpecName: "model-cache") pod "0bc8aa1c-5edc-441e-9972-b00d4caa2fe2" (UID: "0bc8aa1c-5edc-441e-9972-b00d4caa2fe2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:30:25.845424 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:25.845239    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bc8aa1c-5edc-441e-9972-b00d4caa2fe2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0bc8aa1c-5edc-441e-9972-b00d4caa2fe2" (UID: "0bc8aa1c-5edc-441e-9972-b00d4caa2fe2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:30:25.845424 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:25.845251    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bc8aa1c-5edc-441e-9972-b00d4caa2fe2-home" (OuterVolumeSpecName: "home") pod "0bc8aa1c-5edc-441e-9972-b00d4caa2fe2" (UID: "0bc8aa1c-5edc-441e-9972-b00d4caa2fe2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:30:25.845424 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:25.845351    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bc8aa1c-5edc-441e-9972-b00d4caa2fe2-home\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:30:25.845424 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:25.845370    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bc8aa1c-5edc-441e-9972-b00d4caa2fe2-tmp-dir\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:30:25.845424 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:25.845382    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bc8aa1c-5edc-441e-9972-b00d4caa2fe2-model-cache\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:30:25.847241 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:25.847219    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bc8aa1c-5edc-441e-9972-b00d4caa2fe2-dshm" (OuterVolumeSpecName: "dshm") pod "0bc8aa1c-5edc-441e-9972-b00d4caa2fe2" (UID: "0bc8aa1c-5edc-441e-9972-b00d4caa2fe2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:30:25.847325 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:25.847269    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bc8aa1c-5edc-441e-9972-b00d4caa2fe2-kube-api-access-j9pjq" (OuterVolumeSpecName: "kube-api-access-j9pjq") pod "0bc8aa1c-5edc-441e-9972-b00d4caa2fe2" (UID: "0bc8aa1c-5edc-441e-9972-b00d4caa2fe2"). InnerVolumeSpecName "kube-api-access-j9pjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:30:25.847325 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:25.847299    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bc8aa1c-5edc-441e-9972-b00d4caa2fe2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0bc8aa1c-5edc-441e-9972-b00d4caa2fe2" (UID: "0bc8aa1c-5edc-441e-9972-b00d4caa2fe2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:30:25.946716 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:25.946685    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bc8aa1c-5edc-441e-9972-b00d4caa2fe2-tls-certs\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:30:25.946716 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:25.946714    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bc8aa1c-5edc-441e-9972-b00d4caa2fe2-dshm\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:30:25.946845 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:25.946724    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9pjq\" (UniqueName: \"kubernetes.io/projected/0bc8aa1c-5edc-441e-9972-b00d4caa2fe2-kube-api-access-j9pjq\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:30:26.583353 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:26.583313    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-76br8"
Jul 28 10:30:26.583353 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:26.583333    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-76br8" event={"ID":"0bc8aa1c-5edc-441e-9972-b00d4caa2fe2","Type":"ContainerDied","Data":"23011577caed4acf708ddc11e1cf421bc6e6d8ed431cdf124e1c017e00e1ad1a"}
Jul 28 10:30:26.583905 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:26.583391    2557 scope.go:117] "RemoveContainer" containerID="0e635bbd1d9834e453a4975a545b058c55111bf2b52712f3f5944d35099b7119"
Jul 28 10:30:26.603560 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:26.603526    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-76br8"]
Jul 28 10:30:26.611634 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:26.608379    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-76br8"]
Jul 28 10:30:28.579662 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:28.579627    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bc8aa1c-5edc-441e-9972-b00d4caa2fe2" path="/var/lib/kubelet/pods/0bc8aa1c-5edc-441e-9972-b00d4caa2fe2/volumes"
Jul 28 10:30:51.290078 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:51.290040    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgtmlm"]
Jul 28 10:30:51.290483 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:51.290376    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bc8aa1c-5edc-441e-9972-b00d4caa2fe2" containerName="main"
Jul 28 10:30:51.290483 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:51.290387    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bc8aa1c-5edc-441e-9972-b00d4caa2fe2" containerName="main"
Jul 28 10:30:51.290483 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:51.290444    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bc8aa1c-5edc-441e-9972-b00d4caa2fe2" containerName="main"
Jul 28 10:30:51.294376 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:51.294358    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgtmlm"
Jul 28 10:30:51.296876 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:51.296854    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Jul 28 10:30:51.297354 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:51.297337    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Jul 28 10:30:51.297456 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:51.297403    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-klrxv\""
Jul 28 10:30:51.297519 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:51.297459    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Jul 28 10:30:51.301949 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:51.301913    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgtmlm"]
Jul 28 10:30:51.373072 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:51.373034    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqmz8\" (UniqueName: \"kubernetes.io/projected/e1c16e66-0cd8-4d98-95b3-a243ec4a211d-kube-api-access-zqmz8\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgtmlm\" (UID: \"e1c16e66-0cd8-4d98-95b3-a243ec4a211d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgtmlm"
Jul 28 10:30:51.373072 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:51.373073    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1c16e66-0cd8-4d98-95b3-a243ec4a211d-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgtmlm\" (UID: \"e1c16e66-0cd8-4d98-95b3-a243ec4a211d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgtmlm"
Jul 28 10:30:51.373277 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:51.373098    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1c16e66-0cd8-4d98-95b3-a243ec4a211d-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgtmlm\" (UID: \"e1c16e66-0cd8-4d98-95b3-a243ec4a211d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgtmlm"
Jul 28 10:30:51.373277 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:51.373178    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1c16e66-0cd8-4d98-95b3-a243ec4a211d-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgtmlm\" (UID: \"e1c16e66-0cd8-4d98-95b3-a243ec4a211d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgtmlm"
Jul 28 10:30:51.373277 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:51.373222    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1c16e66-0cd8-4d98-95b3-a243ec4a211d-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgtmlm\" (UID: \"e1c16e66-0cd8-4d98-95b3-a243ec4a211d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgtmlm"
Jul 28 10:30:51.373277 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:51.373252    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1c16e66-0cd8-4d98-95b3-a243ec4a211d-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgtmlm\" (UID: \"e1c16e66-0cd8-4d98-95b3-a243ec4a211d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgtmlm"
Jul 28 10:30:51.474732 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:51.474688    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqmz8\" (UniqueName: \"kubernetes.io/projected/e1c16e66-0cd8-4d98-95b3-a243ec4a211d-kube-api-access-zqmz8\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgtmlm\" (UID: \"e1c16e66-0cd8-4d98-95b3-a243ec4a211d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgtmlm"
Jul 28 10:30:51.474934 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:51.474742    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1c16e66-0cd8-4d98-95b3-a243ec4a211d-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgtmlm\" (UID: \"e1c16e66-0cd8-4d98-95b3-a243ec4a211d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgtmlm"
Jul 28 10:30:51.474934 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:51.474767    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1c16e66-0cd8-4d98-95b3-a243ec4a211d-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgtmlm\" (UID: \"e1c16e66-0cd8-4d98-95b3-a243ec4a211d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgtmlm"
Jul 28 10:30:51.474934 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:51.474809    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1c16e66-0cd8-4d98-95b3-a243ec4a211d-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgtmlm\" (UID: \"e1c16e66-0cd8-4d98-95b3-a243ec4a211d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgtmlm"
Jul 28 10:30:51.474934 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:51.474833    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1c16e66-0cd8-4d98-95b3-a243ec4a211d-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgtmlm\" (UID: \"e1c16e66-0cd8-4d98-95b3-a243ec4a211d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgtmlm"
Jul 28 10:30:51.474934 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:51.474864    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1c16e66-0cd8-4d98-95b3-a243ec4a211d-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgtmlm\" (UID: \"e1c16e66-0cd8-4d98-95b3-a243ec4a211d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgtmlm"
Jul 28 10:30:51.475218 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:51.475196    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1c16e66-0cd8-4d98-95b3-a243ec4a211d-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgtmlm\" (UID: \"e1c16e66-0cd8-4d98-95b3-a243ec4a211d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgtmlm"
Jul 28 10:30:51.475282 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:51.475204    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1c16e66-0cd8-4d98-95b3-a243ec4a211d-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgtmlm\" (UID: \"e1c16e66-0cd8-4d98-95b3-a243ec4a211d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgtmlm"
Jul 28 10:30:51.475335 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:51.475290    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1c16e66-0cd8-4d98-95b3-a243ec4a211d-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgtmlm\" (UID: \"e1c16e66-0cd8-4d98-95b3-a243ec4a211d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgtmlm"
Jul 28 10:30:51.477084 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:51.477058    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1c16e66-0cd8-4d98-95b3-a243ec4a211d-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgtmlm\" (UID: \"e1c16e66-0cd8-4d98-95b3-a243ec4a211d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgtmlm"
Jul 28 10:30:51.477259 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:51.477241    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1c16e66-0cd8-4d98-95b3-a243ec4a211d-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgtmlm\" (UID: \"e1c16e66-0cd8-4d98-95b3-a243ec4a211d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgtmlm"
Jul 28 10:30:51.481743 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:51.481721    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqmz8\" (UniqueName: \"kubernetes.io/projected/e1c16e66-0cd8-4d98-95b3-a243ec4a211d-kube-api-access-zqmz8\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgtmlm\" (UID: \"e1c16e66-0cd8-4d98-95b3-a243ec4a211d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgtmlm"
Jul 28 10:30:51.605795 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:51.605712    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgtmlm"
Jul 28 10:30:51.742491 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:51.742432    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgtmlm"]
Jul 28 10:30:52.698819 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:52.698779    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgtmlm" event={"ID":"e1c16e66-0cd8-4d98-95b3-a243ec4a211d","Type":"ContainerStarted","Data":"5ea7305b9b97b2aac48e5ced052df8b62e205c9487c52977997d08abdcd6d2b1"}
Jul 28 10:30:52.699278 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:52.698842    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgtmlm" event={"ID":"e1c16e66-0cd8-4d98-95b3-a243ec4a211d","Type":"ContainerStarted","Data":"11e6891174d04b384ac13ca631e32073bb570a20533047f043ec9db817dde5e6"}
Jul 28 10:30:52.719644 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:52.719562    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgtmlm" podStartSLOduration=1.719543101 podStartE2EDuration="1.719543101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:30:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:30:52.717805607 +0000 UTC m=+1388.676574031" watchObservedRunningTime="2026-07-28 10:30:52.719543101 +0000 UTC m=+1388.678311535"
Jul 28 10:30:53.606057 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:30:53.606017    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgtmlm"
Jul 28 10:31:01.606023 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:01.605979    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgtmlm"
Jul 28 10:31:01.618506 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:01.618474    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgtmlm"
Jul 28 10:31:01.748774 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:01.748739    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgtmlm"
Jul 28 10:31:24.946834 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:24.946799    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgtmlm"]
Jul 28 10:31:24.956072 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:31:24.956025    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 10:31:24.956072 ip-10-0-131-88 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:31:24.956072 ip-10-0-131-88 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 28 10:31:24.956072 ip-10-0-131-88 kubenswrapper[2557]:  > containerID="5ea7305b9b97b2aac48e5ced052df8b62e205c9487c52977997d08abdcd6d2b1" cmd=["/bin/sleep","15"]
Jul 28 10:31:24.956385 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:24.956105    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgtmlm" message="" err=<
Jul 28 10:31:24.956385 ip-10-0-131-88 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:31:24.956385 ip-10-0-131-88 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 28 10:31:24.956385 ip-10-0-131-88 kubenswrapper[2557]:  >
Jul 28 10:31:24.956385 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:31:24.956127    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 10:31:24.956385 ip-10-0-131-88 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:31:24.956385 ip-10-0-131-88 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 28 10:31:24.956385 ip-10-0-131-88 kubenswrapper[2557]:  > pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgtmlm" podUID="e1c16e66-0cd8-4d98-95b3-a243ec4a211d" containerName="main" containerID="cri-o://5ea7305b9b97b2aac48e5ced052df8b62e205c9487c52977997d08abdcd6d2b1"
Jul 28 10:31:24.956385 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:24.956181    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgtmlm" podUID="e1c16e66-0cd8-4d98-95b3-a243ec4a211d" containerName="main" containerID="cri-o://5ea7305b9b97b2aac48e5ced052df8b62e205c9487c52977997d08abdcd6d2b1" gracePeriod=60
Jul 28 10:31:25.313564 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:25.313538    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgtmlm"
Jul 28 10:31:25.383522 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:25.383490    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zqmz8\" (UniqueName: \"kubernetes.io/projected/e1c16e66-0cd8-4d98-95b3-a243ec4a211d-kube-api-access-zqmz8\") pod \"e1c16e66-0cd8-4d98-95b3-a243ec4a211d\" (UID: \"e1c16e66-0cd8-4d98-95b3-a243ec4a211d\") "
Jul 28 10:31:25.383697 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:25.383538    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1c16e66-0cd8-4d98-95b3-a243ec4a211d-dshm\") pod \"e1c16e66-0cd8-4d98-95b3-a243ec4a211d\" (UID: \"e1c16e66-0cd8-4d98-95b3-a243ec4a211d\") "
Jul 28 10:31:25.383697 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:25.383576    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1c16e66-0cd8-4d98-95b3-a243ec4a211d-model-cache\") pod \"e1c16e66-0cd8-4d98-95b3-a243ec4a211d\" (UID: \"e1c16e66-0cd8-4d98-95b3-a243ec4a211d\") "
Jul 28 10:31:25.383697 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:25.383594    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1c16e66-0cd8-4d98-95b3-a243ec4a211d-home\") pod \"e1c16e66-0cd8-4d98-95b3-a243ec4a211d\" (UID: \"e1c16e66-0cd8-4d98-95b3-a243ec4a211d\") "
Jul 28 10:31:25.383697 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:25.383632    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1c16e66-0cd8-4d98-95b3-a243ec4a211d-tls-certs\") pod \"e1c16e66-0cd8-4d98-95b3-a243ec4a211d\" (UID: \"e1c16e66-0cd8-4d98-95b3-a243ec4a211d\") "
Jul 28 10:31:25.383697 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:25.383674    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1c16e66-0cd8-4d98-95b3-a243ec4a211d-tmp-dir\") pod \"e1c16e66-0cd8-4d98-95b3-a243ec4a211d\" (UID: \"e1c16e66-0cd8-4d98-95b3-a243ec4a211d\") "
Jul 28 10:31:25.383961 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:25.383893    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1c16e66-0cd8-4d98-95b3-a243ec4a211d-model-cache" (OuterVolumeSpecName: "model-cache") pod "e1c16e66-0cd8-4d98-95b3-a243ec4a211d" (UID: "e1c16e66-0cd8-4d98-95b3-a243ec4a211d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:31:25.383961 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:25.383914    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1c16e66-0cd8-4d98-95b3-a243ec4a211d-home" (OuterVolumeSpecName: "home") pod "e1c16e66-0cd8-4d98-95b3-a243ec4a211d" (UID: "e1c16e66-0cd8-4d98-95b3-a243ec4a211d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:31:25.384080 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:25.384062    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1c16e66-0cd8-4d98-95b3-a243ec4a211d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e1c16e66-0cd8-4d98-95b3-a243ec4a211d" (UID: "e1c16e66-0cd8-4d98-95b3-a243ec4a211d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:31:25.385871 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:25.385841    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1c16e66-0cd8-4d98-95b3-a243ec4a211d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e1c16e66-0cd8-4d98-95b3-a243ec4a211d" (UID: "e1c16e66-0cd8-4d98-95b3-a243ec4a211d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:31:25.385954 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:25.385882    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1c16e66-0cd8-4d98-95b3-a243ec4a211d-dshm" (OuterVolumeSpecName: "dshm") pod "e1c16e66-0cd8-4d98-95b3-a243ec4a211d" (UID: "e1c16e66-0cd8-4d98-95b3-a243ec4a211d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:31:25.385954 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:25.385901    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1c16e66-0cd8-4d98-95b3-a243ec4a211d-kube-api-access-zqmz8" (OuterVolumeSpecName: "kube-api-access-zqmz8") pod "e1c16e66-0cd8-4d98-95b3-a243ec4a211d" (UID: "e1c16e66-0cd8-4d98-95b3-a243ec4a211d"). InnerVolumeSpecName "kube-api-access-zqmz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:31:25.484559 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:25.484460    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqmz8\" (UniqueName: \"kubernetes.io/projected/e1c16e66-0cd8-4d98-95b3-a243ec4a211d-kube-api-access-zqmz8\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:31:25.484559 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:25.484493    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1c16e66-0cd8-4d98-95b3-a243ec4a211d-dshm\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:31:25.484559 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:25.484508    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1c16e66-0cd8-4d98-95b3-a243ec4a211d-model-cache\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:31:25.484559 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:25.484520    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1c16e66-0cd8-4d98-95b3-a243ec4a211d-home\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:31:25.484559 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:25.484531    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1c16e66-0cd8-4d98-95b3-a243ec4a211d-tls-certs\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:31:25.484559 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:25.484542    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1c16e66-0cd8-4d98-95b3-a243ec4a211d-tmp-dir\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:31:25.840254 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:25.840222    2557 generic.go:356] "Generic (PLEG): container finished" podID="e1c16e66-0cd8-4d98-95b3-a243ec4a211d" containerID="5ea7305b9b97b2aac48e5ced052df8b62e205c9487c52977997d08abdcd6d2b1" exitCode=0
Jul 28 10:31:25.840432 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:25.840269    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgtmlm"
Jul 28 10:31:25.840432 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:25.840270    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgtmlm" event={"ID":"e1c16e66-0cd8-4d98-95b3-a243ec4a211d","Type":"ContainerDied","Data":"5ea7305b9b97b2aac48e5ced052df8b62e205c9487c52977997d08abdcd6d2b1"}
Jul 28 10:31:25.840432 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:25.840385    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgtmlm" event={"ID":"e1c16e66-0cd8-4d98-95b3-a243ec4a211d","Type":"ContainerDied","Data":"11e6891174d04b384ac13ca631e32073bb570a20533047f043ec9db817dde5e6"}
Jul 28 10:31:25.840432 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:25.840414    2557 scope.go:117] "RemoveContainer" containerID="5ea7305b9b97b2aac48e5ced052df8b62e205c9487c52977997d08abdcd6d2b1"
Jul 28 10:31:25.849394 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:25.849371    2557 scope.go:117] "RemoveContainer" containerID="5ea7305b9b97b2aac48e5ced052df8b62e205c9487c52977997d08abdcd6d2b1"
Jul 28 10:31:25.849666 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:31:25.849645    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ea7305b9b97b2aac48e5ced052df8b62e205c9487c52977997d08abdcd6d2b1\": container with ID starting with 5ea7305b9b97b2aac48e5ced052df8b62e205c9487c52977997d08abdcd6d2b1 not found: ID does not exist" containerID="5ea7305b9b97b2aac48e5ced052df8b62e205c9487c52977997d08abdcd6d2b1"
Jul 28 10:31:25.849726 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:25.849675    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ea7305b9b97b2aac48e5ced052df8b62e205c9487c52977997d08abdcd6d2b1"} err="failed to get container status \"5ea7305b9b97b2aac48e5ced052df8b62e205c9487c52977997d08abdcd6d2b1\": rpc error: code = NotFound desc = could not find container \"5ea7305b9b97b2aac48e5ced052df8b62e205c9487c52977997d08abdcd6d2b1\": container with ID starting with 5ea7305b9b97b2aac48e5ced052df8b62e205c9487c52977997d08abdcd6d2b1 not found: ID does not exist"
Jul 28 10:31:25.861435 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:25.861404    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgtmlm"]
Jul 28 10:31:25.865326 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:25.865302    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgtmlm"]
Jul 28 10:31:26.580325 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:26.580293    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1c16e66-0cd8-4d98-95b3-a243ec4a211d" path="/var/lib/kubelet/pods/e1c16e66-0cd8-4d98-95b3-a243ec4a211d/volumes"
Jul 28 10:31:58.385013 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:58.384979    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f64786547-58j6j"]
Jul 28 10:31:58.385460 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:58.385321    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1c16e66-0cd8-4d98-95b3-a243ec4a211d" containerName="main"
Jul 28 10:31:58.385460 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:58.385335    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1c16e66-0cd8-4d98-95b3-a243ec4a211d" containerName="main"
Jul 28 10:31:58.385460 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:58.385403    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1c16e66-0cd8-4d98-95b3-a243ec4a211d" containerName="main"
Jul 28 10:31:58.389741 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:58.389719    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f64786547-58j6j"
Jul 28 10:31:58.392388 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:58.392364    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"seaweedfs-s3-creds\""
Jul 28 10:31:58.392521 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:58.392406    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Jul 28 10:31:58.392521 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:58.392362    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Jul 28 10:31:58.392521 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:58.392362    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-rmxfc\""
Jul 28 10:31:58.392689 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:58.392521    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Jul 28 10:31:58.402430 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:58.402404    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f64786547-58j6j"]
Jul 28 10:31:58.484571 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:58.484535    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxd47\" (UniqueName: \"kubernetes.io/projected/98493a25-9ee5-40ac-830b-190d0576f418-kube-api-access-rxd47\") pod \"auth-disabled-test-kserve-f64786547-58j6j\" (UID: \"98493a25-9ee5-40ac-830b-190d0576f418\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f64786547-58j6j"
Jul 28 10:31:58.484806 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:58.484584    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98493a25-9ee5-40ac-830b-190d0576f418-tmp-dir\") pod \"auth-disabled-test-kserve-f64786547-58j6j\" (UID: \"98493a25-9ee5-40ac-830b-190d0576f418\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f64786547-58j6j"
Jul 28 10:31:58.484806 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:58.484707    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98493a25-9ee5-40ac-830b-190d0576f418-kserve-provision-location\") pod \"auth-disabled-test-kserve-f64786547-58j6j\" (UID: \"98493a25-9ee5-40ac-830b-190d0576f418\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f64786547-58j6j"
Jul 28 10:31:58.484806 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:58.484748    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98493a25-9ee5-40ac-830b-190d0576f418-dshm\") pod \"auth-disabled-test-kserve-f64786547-58j6j\" (UID: \"98493a25-9ee5-40ac-830b-190d0576f418\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f64786547-58j6j"
Jul 28 10:31:58.484971 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:58.484810    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98493a25-9ee5-40ac-830b-190d0576f418-tls-certs\") pod \"auth-disabled-test-kserve-f64786547-58j6j\" (UID: \"98493a25-9ee5-40ac-830b-190d0576f418\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f64786547-58j6j"
Jul 28 10:31:58.484971 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:58.484841    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98493a25-9ee5-40ac-830b-190d0576f418-home\") pod \"auth-disabled-test-kserve-f64786547-58j6j\" (UID: \"98493a25-9ee5-40ac-830b-190d0576f418\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f64786547-58j6j"
Jul 28 10:31:58.484971 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:58.484869    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98493a25-9ee5-40ac-830b-190d0576f418-model-cache\") pod \"auth-disabled-test-kserve-f64786547-58j6j\" (UID: \"98493a25-9ee5-40ac-830b-190d0576f418\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f64786547-58j6j"
Jul 28 10:31:58.585699 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:58.585663    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98493a25-9ee5-40ac-830b-190d0576f418-tmp-dir\") pod \"auth-disabled-test-kserve-f64786547-58j6j\" (UID: \"98493a25-9ee5-40ac-830b-190d0576f418\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f64786547-58j6j"
Jul 28 10:31:58.585877 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:58.585711    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98493a25-9ee5-40ac-830b-190d0576f418-kserve-provision-location\") pod \"auth-disabled-test-kserve-f64786547-58j6j\" (UID: \"98493a25-9ee5-40ac-830b-190d0576f418\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f64786547-58j6j"
Jul 28 10:31:58.585877 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:58.585728    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98493a25-9ee5-40ac-830b-190d0576f418-dshm\") pod \"auth-disabled-test-kserve-f64786547-58j6j\" (UID: \"98493a25-9ee5-40ac-830b-190d0576f418\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f64786547-58j6j"
Jul 28 10:31:58.585877 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:58.585765    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98493a25-9ee5-40ac-830b-190d0576f418-tls-certs\") pod \"auth-disabled-test-kserve-f64786547-58j6j\" (UID: \"98493a25-9ee5-40ac-830b-190d0576f418\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f64786547-58j6j"
Jul 28 10:31:58.585877 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:58.585782    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98493a25-9ee5-40ac-830b-190d0576f418-home\") pod \"auth-disabled-test-kserve-f64786547-58j6j\" (UID: \"98493a25-9ee5-40ac-830b-190d0576f418\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f64786547-58j6j"
Jul 28 10:31:58.585877 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:58.585799    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98493a25-9ee5-40ac-830b-190d0576f418-model-cache\") pod \"auth-disabled-test-kserve-f64786547-58j6j\" (UID: \"98493a25-9ee5-40ac-830b-190d0576f418\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f64786547-58j6j"
Jul 28 10:31:58.585877 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:58.585852    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxd47\" (UniqueName: \"kubernetes.io/projected/98493a25-9ee5-40ac-830b-190d0576f418-kube-api-access-rxd47\") pod \"auth-disabled-test-kserve-f64786547-58j6j\" (UID: \"98493a25-9ee5-40ac-830b-190d0576f418\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f64786547-58j6j"
Jul 28 10:31:58.586217 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:58.586067    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98493a25-9ee5-40ac-830b-190d0576f418-tmp-dir\") pod \"auth-disabled-test-kserve-f64786547-58j6j\" (UID: \"98493a25-9ee5-40ac-830b-190d0576f418\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f64786547-58j6j"
Jul 28 10:31:58.586217 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:58.586114    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98493a25-9ee5-40ac-830b-190d0576f418-kserve-provision-location\") pod \"auth-disabled-test-kserve-f64786547-58j6j\" (UID: \"98493a25-9ee5-40ac-830b-190d0576f418\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f64786547-58j6j"
Jul 28 10:31:58.586325 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:58.586260    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98493a25-9ee5-40ac-830b-190d0576f418-home\") pod \"auth-disabled-test-kserve-f64786547-58j6j\" (UID: \"98493a25-9ee5-40ac-830b-190d0576f418\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f64786547-58j6j"
Jul 28 10:31:58.586325 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:58.586268    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98493a25-9ee5-40ac-830b-190d0576f418-model-cache\") pod \"auth-disabled-test-kserve-f64786547-58j6j\" (UID: \"98493a25-9ee5-40ac-830b-190d0576f418\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f64786547-58j6j"
Jul 28 10:31:58.588030 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:58.588007    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98493a25-9ee5-40ac-830b-190d0576f418-dshm\") pod \"auth-disabled-test-kserve-f64786547-58j6j\" (UID: \"98493a25-9ee5-40ac-830b-190d0576f418\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f64786547-58j6j"
Jul 28 10:31:58.588273 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:58.588257    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98493a25-9ee5-40ac-830b-190d0576f418-tls-certs\") pod \"auth-disabled-test-kserve-f64786547-58j6j\" (UID: \"98493a25-9ee5-40ac-830b-190d0576f418\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f64786547-58j6j"
Jul 28 10:31:58.595638 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:58.595592    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxd47\" (UniqueName: \"kubernetes.io/projected/98493a25-9ee5-40ac-830b-190d0576f418-kube-api-access-rxd47\") pod \"auth-disabled-test-kserve-f64786547-58j6j\" (UID: \"98493a25-9ee5-40ac-830b-190d0576f418\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f64786547-58j6j"
Jul 28 10:31:58.700648 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:58.700542    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f64786547-58j6j"
Jul 28 10:31:58.844995 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:58.844966    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f64786547-58j6j"]
Jul 28 10:31:58.847394 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:31:58.847366    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98493a25_9ee5_40ac_830b_190d0576f418.slice/crio-c6e4de40d623c6a4317f738d88a64fe9b2d73773b6f7548f2cb2bf32312e0c04 WatchSource:0}: Error finding container c6e4de40d623c6a4317f738d88a64fe9b2d73773b6f7548f2cb2bf32312e0c04: Status 404 returned error can't find the container with id c6e4de40d623c6a4317f738d88a64fe9b2d73773b6f7548f2cb2bf32312e0c04
Jul 28 10:31:58.988900 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:58.988812    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f64786547-58j6j" event={"ID":"98493a25-9ee5-40ac-830b-190d0576f418","Type":"ContainerStarted","Data":"86b4b9df392e00b449e419f6a3b890d6aa9d2b2370dc198ca788151fb138c73f"}
Jul 28 10:31:58.988900 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:31:58.988870    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f64786547-58j6j" event={"ID":"98493a25-9ee5-40ac-830b-190d0576f418","Type":"ContainerStarted","Data":"c6e4de40d623c6a4317f738d88a64fe9b2d73773b6f7548f2cb2bf32312e0c04"}
Jul 28 10:32:03.009771 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:32:03.009739    2557 generic.go:356] "Generic (PLEG): container finished" podID="98493a25-9ee5-40ac-830b-190d0576f418" containerID="86b4b9df392e00b449e419f6a3b890d6aa9d2b2370dc198ca788151fb138c73f" exitCode=0
Jul 28 10:32:03.010128 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:32:03.009813    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f64786547-58j6j" event={"ID":"98493a25-9ee5-40ac-830b-190d0576f418","Type":"ContainerDied","Data":"86b4b9df392e00b449e419f6a3b890d6aa9d2b2370dc198ca788151fb138c73f"}
Jul 28 10:32:04.017163 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:32:04.017129    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f64786547-58j6j" event={"ID":"98493a25-9ee5-40ac-830b-190d0576f418","Type":"ContainerStarted","Data":"17be3c35f935e14a286cc30db5aeaa85fe01c67d5fdf2fa83916a74c3cdc35a9"}
Jul 28 10:32:04.039073 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:32:04.038999    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f64786547-58j6j" podStartSLOduration=2.323940131 podStartE2EDuration="6.038978476s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.715038345s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:31:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:32:04.036498168 +0000 UTC m=+1459.995266569" watchObservedRunningTime="2026-07-28 10:32:04.038978476 +0000 UTC m=+1459.997746856"
Jul 28 10:32:04.701347 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:32:04.701310    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f64786547-58j6j"
Jul 28 10:32:08.700934 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:32:08.700898    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f64786547-58j6j"
Jul 28 10:32:08.702267 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:32:08.702238    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f64786547-58j6j" podUID="98493a25-9ee5-40ac-830b-190d0576f418" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 28 10:32:18.701959 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:32:18.701915    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f64786547-58j6j" podUID="98493a25-9ee5-40ac-830b-190d0576f418" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 28 10:32:20.450962 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:32:20.450927    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565875lg6f"]
Jul 28 10:32:20.454281 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:32:20.454255    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565875lg6f"
Jul 28 10:32:20.458341 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:32:20.458302    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Jul 28 10:32:20.458618 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:32:20.458582    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Jul 28 10:32:20.458895 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:32:20.458765    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-fxjnj\""
Jul 28 10:32:20.458895 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:32:20.458662    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Jul 28 10:32:20.458895 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:32:20.458694    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-gcrjb\""
Jul 28 10:32:20.466790 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:32:20.466764    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565875lg6f"]
Jul 28 10:32:20.572179 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:32:20.572144    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xv54\" (UniqueName: \"kubernetes.io/projected/02f9c13f-d31d-4c91-a5c3-1a08026427a6-kube-api-access-9xv54\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-565875lg6f\" (UID: \"02f9c13f-d31d-4c91-a5c3-1a08026427a6\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565875lg6f"
Jul 28 10:32:20.572362 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:32:20.572213    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02f9c13f-d31d-4c91-a5c3-1a08026427a6-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-565875lg6f\" (UID: \"02f9c13f-d31d-4c91-a5c3-1a08026427a6\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565875lg6f"
Jul 28 10:32:20.673234 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:32:20.673195    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02f9c13f-d31d-4c91-a5c3-1a08026427a6-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-565875lg6f\" (UID: \"02f9c13f-d31d-4c91-a5c3-1a08026427a6\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565875lg6f"
Jul 28 10:32:20.673416 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:32:20.673286    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xv54\" (UniqueName: \"kubernetes.io/projected/02f9c13f-d31d-4c91-a5c3-1a08026427a6-kube-api-access-9xv54\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-565875lg6f\" (UID: \"02f9c13f-d31d-4c91-a5c3-1a08026427a6\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565875lg6f"
Jul 28 10:32:20.676152 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:32:20.676124    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02f9c13f-d31d-4c91-a5c3-1a08026427a6-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-565875lg6f\" (UID: \"02f9c13f-d31d-4c91-a5c3-1a08026427a6\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565875lg6f"
Jul 28 10:32:20.687426 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:32:20.687398    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xv54\" (UniqueName: \"kubernetes.io/projected/02f9c13f-d31d-4c91-a5c3-1a08026427a6-kube-api-access-9xv54\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-565875lg6f\" (UID: \"02f9c13f-d31d-4c91-a5c3-1a08026427a6\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565875lg6f"
Jul 28 10:32:20.769053 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:32:20.769022    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565875lg6f"
Jul 28 10:32:20.906234 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:32:20.906202    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565875lg6f"]
Jul 28 10:32:20.908952 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:32:20.908929    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02f9c13f_d31d_4c91_a5c3_1a08026427a6.slice/crio-9cf8e56305e31d9e42a150164a7721c5dc4dfd82fd494ee5d9d7dc0dddf37e6c WatchSource:0}: Error finding container 9cf8e56305e31d9e42a150164a7721c5dc4dfd82fd494ee5d9d7dc0dddf37e6c: Status 404 returned error can't find the container with id 9cf8e56305e31d9e42a150164a7721c5dc4dfd82fd494ee5d9d7dc0dddf37e6c
Jul 28 10:32:20.911014 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:32:20.910992    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 10:32:21.099070 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:32:21.098980    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565875lg6f" event={"ID":"02f9c13f-d31d-4c91-a5c3-1a08026427a6","Type":"ContainerStarted","Data":"9cf8e56305e31d9e42a150164a7721c5dc4dfd82fd494ee5d9d7dc0dddf37e6c"}
Jul 28 10:32:24.118467 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:32:24.118430    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565875lg6f" event={"ID":"02f9c13f-d31d-4c91-a5c3-1a08026427a6","Type":"ContainerStarted","Data":"5ee65e5a1d1fff211a529ddc12479093b74e33c5bf35d5dd417d777b3127ca1e"}
Jul 28 10:32:24.118901 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:32:24.118547    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565875lg6f"
Jul 28 10:32:24.135866 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:32:24.135819    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565875lg6f" podStartSLOduration=1.7202493909999999 podStartE2EDuration="4.135801621s" totalImagesPullingTime="2.41555223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:32:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:32:24.133961565 +0000 UTC m=+1480.092729944" watchObservedRunningTime="2026-07-28 10:32:24.135801621 +0000 UTC m=+1480.094569999"
Jul 28 10:32:28.701583 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:32:28.701540    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f64786547-58j6j" podUID="98493a25-9ee5-40ac-830b-190d0576f418" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 28 10:32:38.701897 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:32:38.701853    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f64786547-58j6j" podUID="98493a25-9ee5-40ac-830b-190d0576f418" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 28 10:32:48.701905 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:32:48.701857    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f64786547-58j6j" podUID="98493a25-9ee5-40ac-830b-190d0576f418" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 28 10:32:55.124553 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:32:55.124525    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565875lg6f"
Jul 28 10:32:56.693570 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:32:56.693522    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565875lg6f"]
Jul 28 10:32:58.701876 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:32:58.701839    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f64786547-58j6j" podUID="98493a25-9ee5-40ac-830b-190d0576f418" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 28 10:33:05.124963 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:33:05.124922    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565875lg6f" podUID="02f9c13f-d31d-4c91-a5c3-1a08026427a6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 10:33:08.701164 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:33:08.701057    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f64786547-58j6j" podUID="98493a25-9ee5-40ac-830b-190d0576f418" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 28 10:33:11.721865 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:33:11.721829    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565875lg6f" podUID="02f9c13f-d31d-4c91-a5c3-1a08026427a6" containerName="main" containerID="cri-o://5ee65e5a1d1fff211a529ddc12479093b74e33c5bf35d5dd417d777b3127ca1e" gracePeriod=45
Jul 28 10:33:11.982997 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:33:11.982931    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565875lg6f"
Jul 28 10:33:12.050714 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:33:12.050677    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02f9c13f-d31d-4c91-a5c3-1a08026427a6-tls-certs\") pod \"02f9c13f-d31d-4c91-a5c3-1a08026427a6\" (UID: \"02f9c13f-d31d-4c91-a5c3-1a08026427a6\") "
Jul 28 10:33:12.050884 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:33:12.050795    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9xv54\" (UniqueName: \"kubernetes.io/projected/02f9c13f-d31d-4c91-a5c3-1a08026427a6-kube-api-access-9xv54\") pod \"02f9c13f-d31d-4c91-a5c3-1a08026427a6\" (UID: \"02f9c13f-d31d-4c91-a5c3-1a08026427a6\") "
Jul 28 10:33:12.052979 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:33:12.052947    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02f9c13f-d31d-4c91-a5c3-1a08026427a6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "02f9c13f-d31d-4c91-a5c3-1a08026427a6" (UID: "02f9c13f-d31d-4c91-a5c3-1a08026427a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:33:12.053099 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:33:12.053015    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02f9c13f-d31d-4c91-a5c3-1a08026427a6-kube-api-access-9xv54" (OuterVolumeSpecName: "kube-api-access-9xv54") pod "02f9c13f-d31d-4c91-a5c3-1a08026427a6" (UID: "02f9c13f-d31d-4c91-a5c3-1a08026427a6"). InnerVolumeSpecName "kube-api-access-9xv54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:33:12.151748 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:33:12.151709    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02f9c13f-d31d-4c91-a5c3-1a08026427a6-tls-certs\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:33:12.151748 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:33:12.151748    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xv54\" (UniqueName: \"kubernetes.io/projected/02f9c13f-d31d-4c91-a5c3-1a08026427a6-kube-api-access-9xv54\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:33:12.351088 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:33:12.351035    2557 generic.go:356] "Generic (PLEG): container finished" podID="02f9c13f-d31d-4c91-a5c3-1a08026427a6" containerID="5ee65e5a1d1fff211a529ddc12479093b74e33c5bf35d5dd417d777b3127ca1e" exitCode=0
Jul 28 10:33:12.351295 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:33:12.351099    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565875lg6f"
Jul 28 10:33:12.351295 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:33:12.351139    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565875lg6f" event={"ID":"02f9c13f-d31d-4c91-a5c3-1a08026427a6","Type":"ContainerDied","Data":"5ee65e5a1d1fff211a529ddc12479093b74e33c5bf35d5dd417d777b3127ca1e"}
Jul 28 10:33:12.351295 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:33:12.351189    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565875lg6f" event={"ID":"02f9c13f-d31d-4c91-a5c3-1a08026427a6","Type":"ContainerDied","Data":"9cf8e56305e31d9e42a150164a7721c5dc4dfd82fd494ee5d9d7dc0dddf37e6c"}
Jul 28 10:33:12.351295 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:33:12.351224    2557 scope.go:117] "RemoveContainer" containerID="5ee65e5a1d1fff211a529ddc12479093b74e33c5bf35d5dd417d777b3127ca1e"
Jul 28 10:33:12.360328 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:33:12.360307    2557 scope.go:117] "RemoveContainer" containerID="5ee65e5a1d1fff211a529ddc12479093b74e33c5bf35d5dd417d777b3127ca1e"
Jul 28 10:33:12.360642 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:33:12.360618    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ee65e5a1d1fff211a529ddc12479093b74e33c5bf35d5dd417d777b3127ca1e\": container with ID starting with 5ee65e5a1d1fff211a529ddc12479093b74e33c5bf35d5dd417d777b3127ca1e not found: ID does not exist" containerID="5ee65e5a1d1fff211a529ddc12479093b74e33c5bf35d5dd417d777b3127ca1e"
Jul 28 10:33:12.360700 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:33:12.360654    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ee65e5a1d1fff211a529ddc12479093b74e33c5bf35d5dd417d777b3127ca1e"} err="failed to get container status \"5ee65e5a1d1fff211a529ddc12479093b74e33c5bf35d5dd417d777b3127ca1e\": rpc error: code = NotFound desc = could not find container \"5ee65e5a1d1fff211a529ddc12479093b74e33c5bf35d5dd417d777b3127ca1e\": container with ID starting with 5ee65e5a1d1fff211a529ddc12479093b74e33c5bf35d5dd417d777b3127ca1e not found: ID does not exist"
Jul 28 10:33:12.371897 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:33:12.371867    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565875lg6f"]
Jul 28 10:33:12.375777 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:33:12.375751    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565875lg6f"]
Jul 28 10:33:12.579749 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:33:12.579713    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02f9c13f-d31d-4c91-a5c3-1a08026427a6" path="/var/lib/kubelet/pods/02f9c13f-d31d-4c91-a5c3-1a08026427a6/volumes"
Jul 28 10:33:18.701310 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:33:18.701272    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f64786547-58j6j" podUID="98493a25-9ee5-40ac-830b-190d0576f418" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 28 10:33:28.702010 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:33:28.701971    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f64786547-58j6j" podUID="98493a25-9ee5-40ac-830b-190d0576f418" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 28 10:33:29.165026 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:33:29.164984    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfdmdcj"]
Jul 28 10:33:29.165489 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:33:29.165468    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02f9c13f-d31d-4c91-a5c3-1a08026427a6" containerName="main"
Jul 28 10:33:29.165489 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:33:29.165491    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="02f9c13f-d31d-4c91-a5c3-1a08026427a6" containerName="main"
Jul 28 10:33:29.165653 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:33:29.165620    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="02f9c13f-d31d-4c91-a5c3-1a08026427a6" containerName="main"
Jul 28 10:33:29.168240 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:33:29.168218    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfdmdcj"
Jul 28 10:33:29.171382 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:33:29.171358    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-k84ps\""
Jul 28 10:33:29.171545 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:33:29.171367    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Jul 28 10:33:29.171713 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:33:29.171399    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Jul 28 10:33:29.171794 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:33:29.171447    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Jul 28 10:33:29.171794 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:33:29.171453    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-4sjg9\""
Jul 28 10:33:29.181685 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:33:29.181401    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfdmdcj"]
Jul 28 10:33:29.310914 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:33:29.310872    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqkd9\" (UniqueName: \"kubernetes.io/projected/3f0888ab-be84-4e53-b583-f08a15066c4c-kube-api-access-hqkd9\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfdmdcj\" (UID: \"3f0888ab-be84-4e53-b583-f08a15066c4c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfdmdcj"
Jul 28 10:33:29.311094 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:33:29.310926    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f0888ab-be84-4e53-b583-f08a15066c4c-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfdmdcj\" (UID: \"3f0888ab-be84-4e53-b583-f08a15066c4c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfdmdcj"
Jul 28 10:33:29.412313 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:33:29.412270    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f0888ab-be84-4e53-b583-f08a15066c4c-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfdmdcj\" (UID: \"3f0888ab-be84-4e53-b583-f08a15066c4c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfdmdcj"
Jul 28 10:33:29.412489 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:33:29.412449    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqkd9\" (UniqueName: \"kubernetes.io/projected/3f0888ab-be84-4e53-b583-f08a15066c4c-kube-api-access-hqkd9\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfdmdcj\" (UID: \"3f0888ab-be84-4e53-b583-f08a15066c4c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfdmdcj"
Jul 28 10:33:29.414723 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:33:29.414698    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f0888ab-be84-4e53-b583-f08a15066c4c-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfdmdcj\" (UID: \"3f0888ab-be84-4e53-b583-f08a15066c4c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfdmdcj"
Jul 28 10:33:29.421132 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:33:29.421066    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqkd9\" (UniqueName: \"kubernetes.io/projected/3f0888ab-be84-4e53-b583-f08a15066c4c-kube-api-access-hqkd9\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfdmdcj\" (UID: \"3f0888ab-be84-4e53-b583-f08a15066c4c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfdmdcj"
Jul 28 10:33:29.482622 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:33:29.482556    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfdmdcj"
Jul 28 10:33:29.639925 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:33:29.639894    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfdmdcj"]
Jul 28 10:33:29.642280 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:33:29.642250    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f0888ab_be84_4e53_b583_f08a15066c4c.slice/crio-35796107041f3e9c3c88d2fb368b451aeb4038e0107363161a0787d264ec22a5 WatchSource:0}: Error finding container 35796107041f3e9c3c88d2fb368b451aeb4038e0107363161a0787d264ec22a5: Status 404 returned error can't find the container with id 35796107041f3e9c3c88d2fb368b451aeb4038e0107363161a0787d264ec22a5
Jul 28 10:33:30.454626 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:33:30.454574    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfdmdcj" event={"ID":"3f0888ab-be84-4e53-b583-f08a15066c4c","Type":"ContainerStarted","Data":"cd3259faec50ab7e318cd38b1932cc8650b836401b70a379441e04451844a218"}
Jul 28 10:33:30.455050 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:33:30.454648    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfdmdcj" event={"ID":"3f0888ab-be84-4e53-b583-f08a15066c4c","Type":"ContainerStarted","Data":"35796107041f3e9c3c88d2fb368b451aeb4038e0107363161a0787d264ec22a5"}
Jul 28 10:33:30.455050 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:33:30.454775    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfdmdcj"
Jul 28 10:33:30.475340 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:33:30.475280    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfdmdcj" podStartSLOduration=1.475257038 podStartE2EDuration="1.475257038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:33:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:33:30.472405359 +0000 UTC m=+1546.431173740" watchObservedRunningTime="2026-07-28 10:33:30.475257038 +0000 UTC m=+1546.434025418"
Jul 28 10:33:38.701365 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:33:38.701320    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f64786547-58j6j" podUID="98493a25-9ee5-40ac-830b-190d0576f418" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 28 10:33:48.701329 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:33:48.701283    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f64786547-58j6j" podUID="98493a25-9ee5-40ac-830b-190d0576f418" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 28 10:33:58.711385 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:33:58.711355    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f64786547-58j6j"
Jul 28 10:33:58.719358 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:33:58.719333    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f64786547-58j6j"
Jul 28 10:34:01.461413 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:01.461373    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfdmdcj" podUID="3f0888ab-be84-4e53-b583-f08a15066c4c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 10:34:02.726135 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:02.726089    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfdmdcj"]
Jul 28 10:34:02.727985 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:02.727951    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfdmdcj" podUID="3f0888ab-be84-4e53-b583-f08a15066c4c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 10:34:04.263764 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:04.263723    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f64786547-58j6j"]
Jul 28 10:34:12.728720 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:12.728679    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfdmdcj" podUID="3f0888ab-be84-4e53-b583-f08a15066c4c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 10:34:17.746966 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:17.746909    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfdmdcj" podUID="3f0888ab-be84-4e53-b583-f08a15066c4c" containerName="main" containerID="cri-o://cd3259faec50ab7e318cd38b1932cc8650b836401b70a379441e04451844a218" gracePeriod=45
Jul 28 10:34:18.002761 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:18.002734    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfdmdcj"
Jul 28 10:34:18.146982 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:18.146941    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hqkd9\" (UniqueName: \"kubernetes.io/projected/3f0888ab-be84-4e53-b583-f08a15066c4c-kube-api-access-hqkd9\") pod \"3f0888ab-be84-4e53-b583-f08a15066c4c\" (UID: \"3f0888ab-be84-4e53-b583-f08a15066c4c\") "
Jul 28 10:34:18.147169 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:18.147060    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f0888ab-be84-4e53-b583-f08a15066c4c-tls-certs\") pod \"3f0888ab-be84-4e53-b583-f08a15066c4c\" (UID: \"3f0888ab-be84-4e53-b583-f08a15066c4c\") "
Jul 28 10:34:18.149146 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:18.149114    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f0888ab-be84-4e53-b583-f08a15066c4c-kube-api-access-hqkd9" (OuterVolumeSpecName: "kube-api-access-hqkd9") pod "3f0888ab-be84-4e53-b583-f08a15066c4c" (UID: "3f0888ab-be84-4e53-b583-f08a15066c4c"). InnerVolumeSpecName "kube-api-access-hqkd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:34:18.149259 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:18.149163    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f0888ab-be84-4e53-b583-f08a15066c4c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3f0888ab-be84-4e53-b583-f08a15066c4c" (UID: "3f0888ab-be84-4e53-b583-f08a15066c4c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:34:18.247889 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:18.247846    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f0888ab-be84-4e53-b583-f08a15066c4c-tls-certs\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:34:18.247889 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:18.247879    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqkd9\" (UniqueName: \"kubernetes.io/projected/3f0888ab-be84-4e53-b583-f08a15066c4c-kube-api-access-hqkd9\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:34:18.691206 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:18.691172    2557 generic.go:356] "Generic (PLEG): container finished" podID="3f0888ab-be84-4e53-b583-f08a15066c4c" containerID="cd3259faec50ab7e318cd38b1932cc8650b836401b70a379441e04451844a218" exitCode=0
Jul 28 10:34:18.691404 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:18.691236    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfdmdcj"
Jul 28 10:34:18.691404 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:18.691250    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfdmdcj" event={"ID":"3f0888ab-be84-4e53-b583-f08a15066c4c","Type":"ContainerDied","Data":"cd3259faec50ab7e318cd38b1932cc8650b836401b70a379441e04451844a218"}
Jul 28 10:34:18.691404 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:18.691307    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfdmdcj" event={"ID":"3f0888ab-be84-4e53-b583-f08a15066c4c","Type":"ContainerDied","Data":"35796107041f3e9c3c88d2fb368b451aeb4038e0107363161a0787d264ec22a5"}
Jul 28 10:34:18.691404 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:18.691334    2557 scope.go:117] "RemoveContainer" containerID="cd3259faec50ab7e318cd38b1932cc8650b836401b70a379441e04451844a218"
Jul 28 10:34:18.700079 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:18.700061    2557 scope.go:117] "RemoveContainer" containerID="cd3259faec50ab7e318cd38b1932cc8650b836401b70a379441e04451844a218"
Jul 28 10:34:18.700391 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:34:18.700370    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd3259faec50ab7e318cd38b1932cc8650b836401b70a379441e04451844a218\": container with ID starting with cd3259faec50ab7e318cd38b1932cc8650b836401b70a379441e04451844a218 not found: ID does not exist" containerID="cd3259faec50ab7e318cd38b1932cc8650b836401b70a379441e04451844a218"
Jul 28 10:34:18.700459 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:18.700403    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd3259faec50ab7e318cd38b1932cc8650b836401b70a379441e04451844a218"} err="failed to get container status \"cd3259faec50ab7e318cd38b1932cc8650b836401b70a379441e04451844a218\": rpc error: code = NotFound desc = could not find container \"cd3259faec50ab7e318cd38b1932cc8650b836401b70a379441e04451844a218\": container with ID starting with cd3259faec50ab7e318cd38b1932cc8650b836401b70a379441e04451844a218 not found: ID does not exist"
Jul 28 10:34:18.707851 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:18.707817    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfdmdcj"]
Jul 28 10:34:18.714407 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:18.714382    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfdmdcj"]
Jul 28 10:34:19.289960 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:19.289894    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f64786547-58j6j" podUID="98493a25-9ee5-40ac-830b-190d0576f418" containerName="main" containerID="cri-o://17be3c35f935e14a286cc30db5aeaa85fe01c67d5fdf2fa83916a74c3cdc35a9" gracePeriod=45
Jul 28 10:34:19.712177 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:19.712083    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f64786547-58j6j" podUID="98493a25-9ee5-40ac-830b-190d0576f418" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 28 10:34:20.053721 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:20.053692    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f64786547-58j6j"
Jul 28 10:34:20.166946 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:20.166910    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98493a25-9ee5-40ac-830b-190d0576f418-tmp-dir\") pod \"98493a25-9ee5-40ac-830b-190d0576f418\" (UID: \"98493a25-9ee5-40ac-830b-190d0576f418\") "
Jul 28 10:34:20.166946 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:20.166948    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98493a25-9ee5-40ac-830b-190d0576f418-home\") pod \"98493a25-9ee5-40ac-830b-190d0576f418\" (UID: \"98493a25-9ee5-40ac-830b-190d0576f418\") "
Jul 28 10:34:20.167174 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:20.166995    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98493a25-9ee5-40ac-830b-190d0576f418-model-cache\") pod \"98493a25-9ee5-40ac-830b-190d0576f418\" (UID: \"98493a25-9ee5-40ac-830b-190d0576f418\") "
Jul 28 10:34:20.167174 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:20.167043    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98493a25-9ee5-40ac-830b-190d0576f418-dshm\") pod \"98493a25-9ee5-40ac-830b-190d0576f418\" (UID: \"98493a25-9ee5-40ac-830b-190d0576f418\") "
Jul 28 10:34:20.167174 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:20.167070    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98493a25-9ee5-40ac-830b-190d0576f418-kserve-provision-location\") pod \"98493a25-9ee5-40ac-830b-190d0576f418\" (UID: \"98493a25-9ee5-40ac-830b-190d0576f418\") "
Jul 28 10:34:20.167174 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:20.167109    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98493a25-9ee5-40ac-830b-190d0576f418-tls-certs\") pod \"98493a25-9ee5-40ac-830b-190d0576f418\" (UID: \"98493a25-9ee5-40ac-830b-190d0576f418\") "
Jul 28 10:34:20.167174 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:20.167140    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rxd47\" (UniqueName: \"kubernetes.io/projected/98493a25-9ee5-40ac-830b-190d0576f418-kube-api-access-rxd47\") pod \"98493a25-9ee5-40ac-830b-190d0576f418\" (UID: \"98493a25-9ee5-40ac-830b-190d0576f418\") "
Jul 28 10:34:20.167429 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:20.167260    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98493a25-9ee5-40ac-830b-190d0576f418-model-cache" (OuterVolumeSpecName: "model-cache") pod "98493a25-9ee5-40ac-830b-190d0576f418" (UID: "98493a25-9ee5-40ac-830b-190d0576f418"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:34:20.167429 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:20.167422    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98493a25-9ee5-40ac-830b-190d0576f418-model-cache\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:34:20.167902 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:20.167875    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98493a25-9ee5-40ac-830b-190d0576f418-home" (OuterVolumeSpecName: "home") pod "98493a25-9ee5-40ac-830b-190d0576f418" (UID: "98493a25-9ee5-40ac-830b-190d0576f418"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:34:20.169396 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:20.169363    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98493a25-9ee5-40ac-830b-190d0576f418-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "98493a25-9ee5-40ac-830b-190d0576f418" (UID: "98493a25-9ee5-40ac-830b-190d0576f418"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:34:20.169396 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:20.169373    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98493a25-9ee5-40ac-830b-190d0576f418-dshm" (OuterVolumeSpecName: "dshm") pod "98493a25-9ee5-40ac-830b-190d0576f418" (UID: "98493a25-9ee5-40ac-830b-190d0576f418"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:34:20.170180 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:20.170156    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98493a25-9ee5-40ac-830b-190d0576f418-kube-api-access-rxd47" (OuterVolumeSpecName: "kube-api-access-rxd47") pod "98493a25-9ee5-40ac-830b-190d0576f418" (UID: "98493a25-9ee5-40ac-830b-190d0576f418"). InnerVolumeSpecName "kube-api-access-rxd47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:34:20.179408 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:20.179384    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98493a25-9ee5-40ac-830b-190d0576f418-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "98493a25-9ee5-40ac-830b-190d0576f418" (UID: "98493a25-9ee5-40ac-830b-190d0576f418"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:34:20.234891 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:20.234836    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98493a25-9ee5-40ac-830b-190d0576f418-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "98493a25-9ee5-40ac-830b-190d0576f418" (UID: "98493a25-9ee5-40ac-830b-190d0576f418"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:34:20.268160 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:20.268126    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98493a25-9ee5-40ac-830b-190d0576f418-dshm\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:34:20.268160 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:20.268155    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98493a25-9ee5-40ac-830b-190d0576f418-kserve-provision-location\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:34:20.268160 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:20.268166    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98493a25-9ee5-40ac-830b-190d0576f418-tls-certs\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:34:20.268380 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:20.268176    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxd47\" (UniqueName: \"kubernetes.io/projected/98493a25-9ee5-40ac-830b-190d0576f418-kube-api-access-rxd47\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:34:20.268380 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:20.268185    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98493a25-9ee5-40ac-830b-190d0576f418-tmp-dir\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:34:20.268380 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:20.268195    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98493a25-9ee5-40ac-830b-190d0576f418-home\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:34:20.579932 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:20.579901    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f0888ab-be84-4e53-b583-f08a15066c4c" path="/var/lib/kubelet/pods/3f0888ab-be84-4e53-b583-f08a15066c4c/volumes"
Jul 28 10:34:20.702956 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:20.702913    2557 generic.go:356] "Generic (PLEG): container finished" podID="98493a25-9ee5-40ac-830b-190d0576f418" containerID="17be3c35f935e14a286cc30db5aeaa85fe01c67d5fdf2fa83916a74c3cdc35a9" exitCode=0
Jul 28 10:34:20.703129 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:20.702967    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f64786547-58j6j" event={"ID":"98493a25-9ee5-40ac-830b-190d0576f418","Type":"ContainerDied","Data":"17be3c35f935e14a286cc30db5aeaa85fe01c67d5fdf2fa83916a74c3cdc35a9"}
Jul 28 10:34:20.703129 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:20.702991    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f64786547-58j6j"
Jul 28 10:34:20.703129 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:20.703053    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f64786547-58j6j" event={"ID":"98493a25-9ee5-40ac-830b-190d0576f418","Type":"ContainerDied","Data":"c6e4de40d623c6a4317f738d88a64fe9b2d73773b6f7548f2cb2bf32312e0c04"}
Jul 28 10:34:20.703129 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:20.703091    2557 scope.go:117] "RemoveContainer" containerID="17be3c35f935e14a286cc30db5aeaa85fe01c67d5fdf2fa83916a74c3cdc35a9"
Jul 28 10:34:20.711692 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:20.711629    2557 scope.go:117] "RemoveContainer" containerID="86b4b9df392e00b449e419f6a3b890d6aa9d2b2370dc198ca788151fb138c73f"
Jul 28 10:34:20.719764 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:20.719740    2557 scope.go:117] "RemoveContainer" containerID="17be3c35f935e14a286cc30db5aeaa85fe01c67d5fdf2fa83916a74c3cdc35a9"
Jul 28 10:34:20.720455 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:34:20.720431    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17be3c35f935e14a286cc30db5aeaa85fe01c67d5fdf2fa83916a74c3cdc35a9\": container with ID starting with 17be3c35f935e14a286cc30db5aeaa85fe01c67d5fdf2fa83916a74c3cdc35a9 not found: ID does not exist" containerID="17be3c35f935e14a286cc30db5aeaa85fe01c67d5fdf2fa83916a74c3cdc35a9"
Jul 28 10:34:20.720564 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:20.720466    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17be3c35f935e14a286cc30db5aeaa85fe01c67d5fdf2fa83916a74c3cdc35a9"} err="failed to get container status \"17be3c35f935e14a286cc30db5aeaa85fe01c67d5fdf2fa83916a74c3cdc35a9\": rpc error: code = NotFound desc = could not find container \"17be3c35f935e14a286cc30db5aeaa85fe01c67d5fdf2fa83916a74c3cdc35a9\": container with ID starting with 17be3c35f935e14a286cc30db5aeaa85fe01c67d5fdf2fa83916a74c3cdc35a9 not found: ID does not exist"
Jul 28 10:34:20.720564 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:20.720493    2557 scope.go:117] "RemoveContainer" containerID="86b4b9df392e00b449e419f6a3b890d6aa9d2b2370dc198ca788151fb138c73f"
Jul 28 10:34:20.720863 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:34:20.720833    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86b4b9df392e00b449e419f6a3b890d6aa9d2b2370dc198ca788151fb138c73f\": container with ID starting with 86b4b9df392e00b449e419f6a3b890d6aa9d2b2370dc198ca788151fb138c73f not found: ID does not exist" containerID="86b4b9df392e00b449e419f6a3b890d6aa9d2b2370dc198ca788151fb138c73f"
Jul 28 10:34:20.720935 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:20.720888    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86b4b9df392e00b449e419f6a3b890d6aa9d2b2370dc198ca788151fb138c73f"} err="failed to get container status \"86b4b9df392e00b449e419f6a3b890d6aa9d2b2370dc198ca788151fb138c73f\": rpc error: code = NotFound desc = could not find container \"86b4b9df392e00b449e419f6a3b890d6aa9d2b2370dc198ca788151fb138c73f\": container with ID starting with 86b4b9df392e00b449e419f6a3b890d6aa9d2b2370dc198ca788151fb138c73f not found: ID does not exist"
Jul 28 10:34:20.722124 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:20.722097    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f64786547-58j6j"]
Jul 28 10:34:20.725718 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:20.725690    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f64786547-58j6j"]
Jul 28 10:34:22.579878 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:22.579840    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98493a25-9ee5-40ac-830b-190d0576f418" path="/var/lib/kubelet/pods/98493a25-9ee5-40ac-830b-190d0576f418/volumes"
Jul 28 10:34:31.698149 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:31.698119    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-v62hx"]
Jul 28 10:34:31.698753 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:31.698584    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98493a25-9ee5-40ac-830b-190d0576f418" containerName="storage-initializer"
Jul 28 10:34:31.698753 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:31.698622    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="98493a25-9ee5-40ac-830b-190d0576f418" containerName="storage-initializer"
Jul 28 10:34:31.698753 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:31.698662    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98493a25-9ee5-40ac-830b-190d0576f418" containerName="main"
Jul 28 10:34:31.698753 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:31.698671    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="98493a25-9ee5-40ac-830b-190d0576f418" containerName="main"
Jul 28 10:34:31.698753 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:31.698684    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f0888ab-be84-4e53-b583-f08a15066c4c" containerName="main"
Jul 28 10:34:31.698753 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:31.698692    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f0888ab-be84-4e53-b583-f08a15066c4c" containerName="main"
Jul 28 10:34:31.699065 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:31.698763    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f0888ab-be84-4e53-b583-f08a15066c4c" containerName="main"
Jul 28 10:34:31.699065 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:31.698776    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="98493a25-9ee5-40ac-830b-190d0576f418" containerName="main"
Jul 28 10:34:31.701896 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:31.701876    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-v62hx"
Jul 28 10:34:31.704055 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:31.704036    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Jul 28 10:34:31.704680 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:31.704652    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-c6wgl\""
Jul 28 10:34:31.704795 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:31.704722    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Jul 28 10:34:31.704865 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:31.704800    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Jul 28 10:34:31.713761 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:31.713736    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-v62hx"]
Jul 28 10:34:31.762200 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:31.762171    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44d000da-910c-4455-8e5f-207e721228aa-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-v62hx\" (UID: \"44d000da-910c-4455-8e5f-207e721228aa\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-v62hx"
Jul 28 10:34:31.762200 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:31.762200    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44d000da-910c-4455-8e5f-207e721228aa-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-v62hx\" (UID: \"44d000da-910c-4455-8e5f-207e721228aa\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-v62hx"
Jul 28 10:34:31.762409 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:31.762226    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44d000da-910c-4455-8e5f-207e721228aa-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-v62hx\" (UID: \"44d000da-910c-4455-8e5f-207e721228aa\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-v62hx"
Jul 28 10:34:31.762409 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:31.762302    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44d000da-910c-4455-8e5f-207e721228aa-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-v62hx\" (UID: \"44d000da-910c-4455-8e5f-207e721228aa\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-v62hx"
Jul 28 10:34:31.762409 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:31.762319    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-649z7\" (UniqueName: \"kubernetes.io/projected/44d000da-910c-4455-8e5f-207e721228aa-kube-api-access-649z7\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-v62hx\" (UID: \"44d000da-910c-4455-8e5f-207e721228aa\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-v62hx"
Jul 28 10:34:31.762409 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:31.762378    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44d000da-910c-4455-8e5f-207e721228aa-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-v62hx\" (UID: \"44d000da-910c-4455-8e5f-207e721228aa\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-v62hx"
Jul 28 10:34:31.863650 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:31.863596    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44d000da-910c-4455-8e5f-207e721228aa-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-v62hx\" (UID: \"44d000da-910c-4455-8e5f-207e721228aa\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-v62hx"
Jul 28 10:34:31.863823 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:31.863662    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44d000da-910c-4455-8e5f-207e721228aa-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-v62hx\" (UID: \"44d000da-910c-4455-8e5f-207e721228aa\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-v62hx"
Jul 28 10:34:31.863823 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:31.863705    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44d000da-910c-4455-8e5f-207e721228aa-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-v62hx\" (UID: \"44d000da-910c-4455-8e5f-207e721228aa\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-v62hx"
Jul 28 10:34:31.863823 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:31.863773    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44d000da-910c-4455-8e5f-207e721228aa-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-v62hx\" (UID: \"44d000da-910c-4455-8e5f-207e721228aa\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-v62hx"
Jul 28 10:34:31.863823 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:31.863798    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-649z7\" (UniqueName: \"kubernetes.io/projected/44d000da-910c-4455-8e5f-207e721228aa-kube-api-access-649z7\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-v62hx\" (UID: \"44d000da-910c-4455-8e5f-207e721228aa\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-v62hx"
Jul 28 10:34:31.864026 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:31.863841    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44d000da-910c-4455-8e5f-207e721228aa-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-v62hx\" (UID: \"44d000da-910c-4455-8e5f-207e721228aa\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-v62hx"
Jul 28 10:34:31.864081 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:31.864064    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44d000da-910c-4455-8e5f-207e721228aa-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-v62hx\" (UID: \"44d000da-910c-4455-8e5f-207e721228aa\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-v62hx"
Jul 28 10:34:31.864151 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:31.864128    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44d000da-910c-4455-8e5f-207e721228aa-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-v62hx\" (UID: \"44d000da-910c-4455-8e5f-207e721228aa\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-v62hx"
Jul 28 10:34:31.864228 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:31.864150    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44d000da-910c-4455-8e5f-207e721228aa-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-v62hx\" (UID: \"44d000da-910c-4455-8e5f-207e721228aa\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-v62hx"
Jul 28 10:34:31.865851 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:31.865815    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44d000da-910c-4455-8e5f-207e721228aa-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-v62hx\" (UID: \"44d000da-910c-4455-8e5f-207e721228aa\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-v62hx"
Jul 28 10:34:31.866141 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:31.866126    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44d000da-910c-4455-8e5f-207e721228aa-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-v62hx\" (UID: \"44d000da-910c-4455-8e5f-207e721228aa\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-v62hx"
Jul 28 10:34:31.873257 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:31.873235    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-649z7\" (UniqueName: \"kubernetes.io/projected/44d000da-910c-4455-8e5f-207e721228aa-kube-api-access-649z7\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-v62hx\" (UID: \"44d000da-910c-4455-8e5f-207e721228aa\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-v62hx"
Jul 28 10:34:32.013517 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:32.013473    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-v62hx"
Jul 28 10:34:32.152576 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:32.152546    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-v62hx"]
Jul 28 10:34:32.155294 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:34:32.155263    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44d000da_910c_4455_8e5f_207e721228aa.slice/crio-15dd410b8850d8d90656128070120f05303c3a8d38ca919eb40d0d33f01d4bea WatchSource:0}: Error finding container 15dd410b8850d8d90656128070120f05303c3a8d38ca919eb40d0d33f01d4bea: Status 404 returned error can't find the container with id 15dd410b8850d8d90656128070120f05303c3a8d38ca919eb40d0d33f01d4bea
Jul 28 10:34:32.760283 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:32.760245    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-v62hx" event={"ID":"44d000da-910c-4455-8e5f-207e721228aa","Type":"ContainerStarted","Data":"f81af067d5476367d6533c63c56dd190a3345930fb6d80946c06d4c75782be3d"}
Jul 28 10:34:32.760777 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:32.760312    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-v62hx" event={"ID":"44d000da-910c-4455-8e5f-207e721228aa","Type":"ContainerStarted","Data":"15dd410b8850d8d90656128070120f05303c3a8d38ca919eb40d0d33f01d4bea"}
Jul 28 10:34:32.777795 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:32.777749    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-v62hx" podStartSLOduration=1.777736352 podStartE2EDuration="1.777736352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:34:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:34:32.775968968 +0000 UTC m=+1608.734737348" watchObservedRunningTime="2026-07-28 10:34:32.777736352 +0000 UTC m=+1608.736504732"
Jul 28 10:34:33.014401 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:33.014314    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-v62hx"
Jul 28 10:34:42.014097 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:42.014000    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-v62hx"
Jul 28 10:34:42.026268 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:42.026240    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-v62hx"
Jul 28 10:34:42.814097 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:34:42.814069    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-v62hx"
Jul 28 10:35:04.804051 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:04.804019    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-v62hx"]
Jul 28 10:35:04.813043 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:35:04.813009    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 10:35:04.813043 ip-10-0-131-88 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:35:04.813043 ip-10-0-131-88 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 28 10:35:04.813043 ip-10-0-131-88 kubenswrapper[2557]:  > containerID="f81af067d5476367d6533c63c56dd190a3345930fb6d80946c06d4c75782be3d" cmd=["/bin/sleep","15"]
Jul 28 10:35:04.813294 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:04.813060    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-v62hx" message="" err=<
Jul 28 10:35:04.813294 ip-10-0-131-88 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:35:04.813294 ip-10-0-131-88 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 28 10:35:04.813294 ip-10-0-131-88 kubenswrapper[2557]:  >
Jul 28 10:35:04.813294 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:35:04.813072    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 10:35:04.813294 ip-10-0-131-88 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:35:04.813294 ip-10-0-131-88 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 28 10:35:04.813294 ip-10-0-131-88 kubenswrapper[2557]:  > pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-v62hx" podUID="44d000da-910c-4455-8e5f-207e721228aa" containerName="main" containerID="cri-o://f81af067d5476367d6533c63c56dd190a3345930fb6d80946c06d4c75782be3d"
Jul 28 10:35:04.813294 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:04.813117    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-v62hx" podUID="44d000da-910c-4455-8e5f-207e721228aa" containerName="main" containerID="cri-o://f81af067d5476367d6533c63c56dd190a3345930fb6d80946c06d4c75782be3d" gracePeriod=60
Jul 28 10:35:05.160485 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:05.160460    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-v62hx"
Jul 28 10:35:05.266810 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:05.266776    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44d000da-910c-4455-8e5f-207e721228aa-dshm\") pod \"44d000da-910c-4455-8e5f-207e721228aa\" (UID: \"44d000da-910c-4455-8e5f-207e721228aa\") "
Jul 28 10:35:05.267011 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:05.266827    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-649z7\" (UniqueName: \"kubernetes.io/projected/44d000da-910c-4455-8e5f-207e721228aa-kube-api-access-649z7\") pod \"44d000da-910c-4455-8e5f-207e721228aa\" (UID: \"44d000da-910c-4455-8e5f-207e721228aa\") "
Jul 28 10:35:05.267011 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:05.266845    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44d000da-910c-4455-8e5f-207e721228aa-home\") pod \"44d000da-910c-4455-8e5f-207e721228aa\" (UID: \"44d000da-910c-4455-8e5f-207e721228aa\") "
Jul 28 10:35:05.267011 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:05.266863    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44d000da-910c-4455-8e5f-207e721228aa-tls-certs\") pod \"44d000da-910c-4455-8e5f-207e721228aa\" (UID: \"44d000da-910c-4455-8e5f-207e721228aa\") "
Jul 28 10:35:05.267011 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:05.266988    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44d000da-910c-4455-8e5f-207e721228aa-tmp-dir\") pod \"44d000da-910c-4455-8e5f-207e721228aa\" (UID: \"44d000da-910c-4455-8e5f-207e721228aa\") "
Jul 28 10:35:05.267247 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:05.267093    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44d000da-910c-4455-8e5f-207e721228aa-model-cache\") pod \"44d000da-910c-4455-8e5f-207e721228aa\" (UID: \"44d000da-910c-4455-8e5f-207e721228aa\") "
Jul 28 10:35:05.267247 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:05.267131    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44d000da-910c-4455-8e5f-207e721228aa-home" (OuterVolumeSpecName: "home") pod "44d000da-910c-4455-8e5f-207e721228aa" (UID: "44d000da-910c-4455-8e5f-207e721228aa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:35:05.267317 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:05.267266    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44d000da-910c-4455-8e5f-207e721228aa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "44d000da-910c-4455-8e5f-207e721228aa" (UID: "44d000da-910c-4455-8e5f-207e721228aa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:35:05.267398 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:05.267368    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44d000da-910c-4455-8e5f-207e721228aa-model-cache" (OuterVolumeSpecName: "model-cache") pod "44d000da-910c-4455-8e5f-207e721228aa" (UID: "44d000da-910c-4455-8e5f-207e721228aa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:35:05.267446 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:05.267387    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44d000da-910c-4455-8e5f-207e721228aa-home\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:35:05.267446 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:05.267426    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44d000da-910c-4455-8e5f-207e721228aa-tmp-dir\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:35:05.269090 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:05.269070    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44d000da-910c-4455-8e5f-207e721228aa-dshm" (OuterVolumeSpecName: "dshm") pod "44d000da-910c-4455-8e5f-207e721228aa" (UID: "44d000da-910c-4455-8e5f-207e721228aa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:35:05.269166 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:05.269106    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44d000da-910c-4455-8e5f-207e721228aa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "44d000da-910c-4455-8e5f-207e721228aa" (UID: "44d000da-910c-4455-8e5f-207e721228aa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:35:05.269166 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:05.269141    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44d000da-910c-4455-8e5f-207e721228aa-kube-api-access-649z7" (OuterVolumeSpecName: "kube-api-access-649z7") pod "44d000da-910c-4455-8e5f-207e721228aa" (UID: "44d000da-910c-4455-8e5f-207e721228aa"). InnerVolumeSpecName "kube-api-access-649z7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:35:05.368932 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:05.368840    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44d000da-910c-4455-8e5f-207e721228aa-dshm\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:35:05.368932 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:05.368871    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-649z7\" (UniqueName: \"kubernetes.io/projected/44d000da-910c-4455-8e5f-207e721228aa-kube-api-access-649z7\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:35:05.368932 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:05.368884    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44d000da-910c-4455-8e5f-207e721228aa-tls-certs\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:35:05.368932 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:05.368897    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44d000da-910c-4455-8e5f-207e721228aa-model-cache\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:35:05.904979 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:05.904940    2557 generic.go:356] "Generic (PLEG): container finished" podID="44d000da-910c-4455-8e5f-207e721228aa" containerID="f81af067d5476367d6533c63c56dd190a3345930fb6d80946c06d4c75782be3d" exitCode=0
Jul 28 10:35:05.905415 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:05.904989    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-v62hx"
Jul 28 10:35:05.905415 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:05.905027    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-v62hx" event={"ID":"44d000da-910c-4455-8e5f-207e721228aa","Type":"ContainerDied","Data":"f81af067d5476367d6533c63c56dd190a3345930fb6d80946c06d4c75782be3d"}
Jul 28 10:35:05.905415 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:05.905080    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-v62hx" event={"ID":"44d000da-910c-4455-8e5f-207e721228aa","Type":"ContainerDied","Data":"15dd410b8850d8d90656128070120f05303c3a8d38ca919eb40d0d33f01d4bea"}
Jul 28 10:35:05.905415 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:05.905102    2557 scope.go:117] "RemoveContainer" containerID="f81af067d5476367d6533c63c56dd190a3345930fb6d80946c06d4c75782be3d"
Jul 28 10:35:05.914329 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:05.914311    2557 scope.go:117] "RemoveContainer" containerID="f81af067d5476367d6533c63c56dd190a3345930fb6d80946c06d4c75782be3d"
Jul 28 10:35:05.914633 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:35:05.914595    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f81af067d5476367d6533c63c56dd190a3345930fb6d80946c06d4c75782be3d\": container with ID starting with f81af067d5476367d6533c63c56dd190a3345930fb6d80946c06d4c75782be3d not found: ID does not exist" containerID="f81af067d5476367d6533c63c56dd190a3345930fb6d80946c06d4c75782be3d"
Jul 28 10:35:05.914687 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:05.914645    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f81af067d5476367d6533c63c56dd190a3345930fb6d80946c06d4c75782be3d"} err="failed to get container status \"f81af067d5476367d6533c63c56dd190a3345930fb6d80946c06d4c75782be3d\": rpc error: code = NotFound desc = could not find container \"f81af067d5476367d6533c63c56dd190a3345930fb6d80946c06d4c75782be3d\": container with ID starting with f81af067d5476367d6533c63c56dd190a3345930fb6d80946c06d4c75782be3d not found: ID does not exist"
Jul 28 10:35:05.926549 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:05.926524    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-v62hx"]
Jul 28 10:35:05.930484 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:05.930463    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-v62hx"]
Jul 28 10:35:06.580274 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:06.580240    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44d000da-910c-4455-8e5f-207e721228aa" path="/var/lib/kubelet/pods/44d000da-910c-4455-8e5f-207e721228aa/volumes"
Jul 28 10:35:38.018674 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:38.018632    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz"]
Jul 28 10:35:38.019187 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:38.019167    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44d000da-910c-4455-8e5f-207e721228aa" containerName="main"
Jul 28 10:35:38.019260 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:38.019190    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="44d000da-910c-4455-8e5f-207e721228aa" containerName="main"
Jul 28 10:35:38.019317 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:38.019279    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="44d000da-910c-4455-8e5f-207e721228aa" containerName="main"
Jul 28 10:35:38.021680 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:38.021657    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz"
Jul 28 10:35:38.023914 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:38.023891    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Jul 28 10:35:38.024496 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:38.024464    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-pgj7g\""
Jul 28 10:35:38.024590 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:38.024510    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Jul 28 10:35:38.024590 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:38.024562    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Jul 28 10:35:38.024909 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:38.024889    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-pbdqr\""
Jul 28 10:35:38.033403 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:38.033380    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz"]
Jul 28 10:35:38.153554 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:38.153513    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebbbac03-3a4f-4e36-b961-2a9dc1995eaf-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz\" (UID: \"ebbbac03-3a4f-4e36-b961-2a9dc1995eaf\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz"
Jul 28 10:35:38.153554 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:38.153555    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebbbac03-3a4f-4e36-b961-2a9dc1995eaf-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz\" (UID: \"ebbbac03-3a4f-4e36-b961-2a9dc1995eaf\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz"
Jul 28 10:35:38.153825 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:38.153580    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebbbac03-3a4f-4e36-b961-2a9dc1995eaf-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz\" (UID: \"ebbbac03-3a4f-4e36-b961-2a9dc1995eaf\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz"
Jul 28 10:35:38.153825 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:38.153688    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebbbac03-3a4f-4e36-b961-2a9dc1995eaf-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz\" (UID: \"ebbbac03-3a4f-4e36-b961-2a9dc1995eaf\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz"
Jul 28 10:35:38.153825 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:38.153715    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvcz6\" (UniqueName: \"kubernetes.io/projected/ebbbac03-3a4f-4e36-b961-2a9dc1995eaf-kube-api-access-kvcz6\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz\" (UID: \"ebbbac03-3a4f-4e36-b961-2a9dc1995eaf\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz"
Jul 28 10:35:38.153825 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:38.153744    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebbbac03-3a4f-4e36-b961-2a9dc1995eaf-home\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz\" (UID: \"ebbbac03-3a4f-4e36-b961-2a9dc1995eaf\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz"
Jul 28 10:35:38.251507 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:38.251475    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f98cgs6"]
Jul 28 10:35:38.253779 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:38.253759    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f98cgs6"
Jul 28 10:35:38.254255 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:38.254231    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebbbac03-3a4f-4e36-b961-2a9dc1995eaf-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz\" (UID: \"ebbbac03-3a4f-4e36-b961-2a9dc1995eaf\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz"
Jul 28 10:35:38.254355 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:38.254269    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvcz6\" (UniqueName: \"kubernetes.io/projected/ebbbac03-3a4f-4e36-b961-2a9dc1995eaf-kube-api-access-kvcz6\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz\" (UID: \"ebbbac03-3a4f-4e36-b961-2a9dc1995eaf\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz"
Jul 28 10:35:38.254355 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:38.254315    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebbbac03-3a4f-4e36-b961-2a9dc1995eaf-home\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz\" (UID: \"ebbbac03-3a4f-4e36-b961-2a9dc1995eaf\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz"
Jul 28 10:35:38.254468 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:38.254395    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebbbac03-3a4f-4e36-b961-2a9dc1995eaf-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz\" (UID: \"ebbbac03-3a4f-4e36-b961-2a9dc1995eaf\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz"
Jul 28 10:35:38.254468 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:38.254424    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebbbac03-3a4f-4e36-b961-2a9dc1995eaf-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz\" (UID: \"ebbbac03-3a4f-4e36-b961-2a9dc1995eaf\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz"
Jul 28 10:35:38.254468 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:38.254464    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebbbac03-3a4f-4e36-b961-2a9dc1995eaf-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz\" (UID: \"ebbbac03-3a4f-4e36-b961-2a9dc1995eaf\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz"
Jul 28 10:35:38.254759 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:38.254731    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebbbac03-3a4f-4e36-b961-2a9dc1995eaf-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz\" (UID: \"ebbbac03-3a4f-4e36-b961-2a9dc1995eaf\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz"
Jul 28 10:35:38.254759 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:38.254743    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebbbac03-3a4f-4e36-b961-2a9dc1995eaf-home\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz\" (UID: \"ebbbac03-3a4f-4e36-b961-2a9dc1995eaf\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz"
Jul 28 10:35:38.254939 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:38.254829    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebbbac03-3a4f-4e36-b961-2a9dc1995eaf-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz\" (UID: \"ebbbac03-3a4f-4e36-b961-2a9dc1995eaf\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz"
Jul 28 10:35:38.256213 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:38.256190    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-epp-sa-dockercfg-w5qnh\""
Jul 28 10:35:38.256839 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:38.256820    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebbbac03-3a4f-4e36-b961-2a9dc1995eaf-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz\" (UID: \"ebbbac03-3a4f-4e36-b961-2a9dc1995eaf\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz"
Jul 28 10:35:38.257011 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:38.256996    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebbbac03-3a4f-4e36-b961-2a9dc1995eaf-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz\" (UID: \"ebbbac03-3a4f-4e36-b961-2a9dc1995eaf\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz"
Jul 28 10:35:38.261834 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:38.261766    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvcz6\" (UniqueName: \"kubernetes.io/projected/ebbbac03-3a4f-4e36-b961-2a9dc1995eaf-kube-api-access-kvcz6\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz\" (UID: \"ebbbac03-3a4f-4e36-b961-2a9dc1995eaf\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz"
Jul 28 10:35:38.263555 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:38.263533    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f98cgs6"]
Jul 28 10:35:38.332791 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:38.332742    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz"
Jul 28 10:35:38.355515 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:38.355486    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb54f781-ebdf-497e-a3d5-9df5a7475d76-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f98cgs6\" (UID: \"cb54f781-ebdf-497e-a3d5-9df5a7475d76\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f98cgs6"
Jul 28 10:35:38.355687 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:38.355557    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn2hj\" (UniqueName: \"kubernetes.io/projected/cb54f781-ebdf-497e-a3d5-9df5a7475d76-kube-api-access-wn2hj\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f98cgs6\" (UID: \"cb54f781-ebdf-497e-a3d5-9df5a7475d76\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f98cgs6"
Jul 28 10:35:38.456470 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:38.456436    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn2hj\" (UniqueName: \"kubernetes.io/projected/cb54f781-ebdf-497e-a3d5-9df5a7475d76-kube-api-access-wn2hj\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f98cgs6\" (UID: \"cb54f781-ebdf-497e-a3d5-9df5a7475d76\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f98cgs6"
Jul 28 10:35:38.456667 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:38.456550    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb54f781-ebdf-497e-a3d5-9df5a7475d76-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f98cgs6\" (UID: \"cb54f781-ebdf-497e-a3d5-9df5a7475d76\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f98cgs6"
Jul 28 10:35:38.458989 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:38.458962    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb54f781-ebdf-497e-a3d5-9df5a7475d76-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f98cgs6\" (UID: \"cb54f781-ebdf-497e-a3d5-9df5a7475d76\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f98cgs6"
Jul 28 10:35:38.463773 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:38.463751    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn2hj\" (UniqueName: \"kubernetes.io/projected/cb54f781-ebdf-497e-a3d5-9df5a7475d76-kube-api-access-wn2hj\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f98cgs6\" (UID: \"cb54f781-ebdf-497e-a3d5-9df5a7475d76\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f98cgs6"
Jul 28 10:35:38.471762 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:38.471738    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz"]
Jul 28 10:35:38.473717 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:35:38.473690    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebbbac03_3a4f_4e36_b961_2a9dc1995eaf.slice/crio-182d97a2e785f36265561eb6f9034e51d1d9222d693c0c64bf3259bec209ddf5 WatchSource:0}: Error finding container 182d97a2e785f36265561eb6f9034e51d1d9222d693c0c64bf3259bec209ddf5: Status 404 returned error can't find the container with id 182d97a2e785f36265561eb6f9034e51d1d9222d693c0c64bf3259bec209ddf5
Jul 28 10:35:38.581451 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:38.581362    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f98cgs6"
Jul 28 10:35:38.718964 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:38.718941    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f98cgs6"]
Jul 28 10:35:38.721478 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:35:38.721456    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb54f781_ebdf_497e_a3d5_9df5a7475d76.slice/crio-b4a4d9464c70e08007a0daf44db1686c21a4984684d23be562287ab885395933 WatchSource:0}: Error finding container b4a4d9464c70e08007a0daf44db1686c21a4984684d23be562287ab885395933: Status 404 returned error can't find the container with id b4a4d9464c70e08007a0daf44db1686c21a4984684d23be562287ab885395933
Jul 28 10:35:39.048764 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:39.048728    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f98cgs6" event={"ID":"cb54f781-ebdf-497e-a3d5-9df5a7475d76","Type":"ContainerStarted","Data":"42c55ea96b635fda4ba07aa5e12504678080a426635aa7daa90c2d49131eb383"}
Jul 28 10:35:39.049219 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:39.048789    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f98cgs6" event={"ID":"cb54f781-ebdf-497e-a3d5-9df5a7475d76","Type":"ContainerStarted","Data":"b4a4d9464c70e08007a0daf44db1686c21a4984684d23be562287ab885395933"}
Jul 28 10:35:39.049219 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:39.048929    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f98cgs6"
Jul 28 10:35:39.050048 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:39.050024    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz" event={"ID":"ebbbac03-3a4f-4e36-b961-2a9dc1995eaf","Type":"ContainerStarted","Data":"182d97a2e785f36265561eb6f9034e51d1d9222d693c0c64bf3259bec209ddf5"}
Jul 28 10:35:39.069839 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:39.069777    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f98cgs6" podStartSLOduration=1.069759662 podStartE2EDuration="1.069759662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:35:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:35:39.068054728 +0000 UTC m=+1675.026823120" watchObservedRunningTime="2026-07-28 10:35:39.069759662 +0000 UTC m=+1675.028528043"
Jul 28 10:35:41.060433 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:41.060395    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz" event={"ID":"ebbbac03-3a4f-4e36-b961-2a9dc1995eaf","Type":"ContainerStarted","Data":"0a599dddb2a188eef30709bea6a8377b75c88d240f2cfafdb6711e55e0625c83"}
Jul 28 10:35:41.060932 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:41.060510    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz"
Jul 28 10:35:42.066959 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:42.066913    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz" event={"ID":"ebbbac03-3a4f-4e36-b961-2a9dc1995eaf","Type":"ContainerStarted","Data":"c8e99242c1ccda192aac65890d57fe81b340c3f194a933a87673d69ba232b7c3"}
Jul 28 10:35:42.087043 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:42.086987    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz" podStartSLOduration=3.129719927 podStartE2EDuration="5.086972646s" totalImagesPullingTime="1.957252719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:35:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:35:42.084870787 +0000 UTC m=+1678.043639177" watchObservedRunningTime="2026-07-28 10:35:42.086972646 +0000 UTC m=+1678.045741026"
Jul 28 10:35:42.333373 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:42.333277    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz"
Jul 28 10:35:48.333068 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:48.333031    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz"
Jul 28 10:35:48.345323 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:48.345297    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz"
Jul 28 10:35:49.107827 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:49.107797    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz"
Jul 28 10:35:50.114250 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:35:50.114224    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz"
Jul 28 10:36:10.056822 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:36:10.056746    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f98cgs6"
Jul 28 10:37:12.336109 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:12.336071    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz"]
Jul 28 10:37:12.345288 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:37:12.345239    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 10:37:12.345288 ip-10-0-131-88 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:37:12.345288 ip-10-0-131-88 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 28 10:37:12.345288 ip-10-0-131-88 kubenswrapper[2557]:  > containerID="c8e99242c1ccda192aac65890d57fe81b340c3f194a933a87673d69ba232b7c3" cmd=["/bin/sleep","15"]
Jul 28 10:37:12.345594 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:12.345299    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz" message="" err=<
Jul 28 10:37:12.345594 ip-10-0-131-88 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:37:12.345594 ip-10-0-131-88 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 28 10:37:12.345594 ip-10-0-131-88 kubenswrapper[2557]:  >
Jul 28 10:37:12.345594 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:37:12.345311    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 10:37:12.345594 ip-10-0-131-88 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:37:12.345594 ip-10-0-131-88 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 28 10:37:12.345594 ip-10-0-131-88 kubenswrapper[2557]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz" podUID="ebbbac03-3a4f-4e36-b961-2a9dc1995eaf" containerName="main" containerID="cri-o://c8e99242c1ccda192aac65890d57fe81b340c3f194a933a87673d69ba232b7c3"
Jul 28 10:37:12.345594 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:12.345363    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz" podUID="ebbbac03-3a4f-4e36-b961-2a9dc1995eaf" containerName="main" containerID="cri-o://c8e99242c1ccda192aac65890d57fe81b340c3f194a933a87673d69ba232b7c3" gracePeriod=60
Jul 28 10:37:12.352229 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:12.352182    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f98cgs6"]
Jul 28 10:37:12.561556 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:12.561521    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz" podUID="ebbbac03-3a4f-4e36-b961-2a9dc1995eaf" containerName="llm-d-routing-sidecar" containerID="cri-o://0a599dddb2a188eef30709bea6a8377b75c88d240f2cfafdb6711e55e0625c83" gracePeriod=60
Jul 28 10:37:12.805650 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:12.805627    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz"
Jul 28 10:37:12.864725 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:12.864687    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebbbac03-3a4f-4e36-b961-2a9dc1995eaf-tls-certs\") pod \"ebbbac03-3a4f-4e36-b961-2a9dc1995eaf\" (UID: \"ebbbac03-3a4f-4e36-b961-2a9dc1995eaf\") "
Jul 28 10:37:12.864725 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:12.864730    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebbbac03-3a4f-4e36-b961-2a9dc1995eaf-model-cache\") pod \"ebbbac03-3a4f-4e36-b961-2a9dc1995eaf\" (UID: \"ebbbac03-3a4f-4e36-b961-2a9dc1995eaf\") "
Jul 28 10:37:12.864924 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:12.864756    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebbbac03-3a4f-4e36-b961-2a9dc1995eaf-tmp-dir\") pod \"ebbbac03-3a4f-4e36-b961-2a9dc1995eaf\" (UID: \"ebbbac03-3a4f-4e36-b961-2a9dc1995eaf\") "
Jul 28 10:37:12.864924 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:12.864792    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebbbac03-3a4f-4e36-b961-2a9dc1995eaf-dshm\") pod \"ebbbac03-3a4f-4e36-b961-2a9dc1995eaf\" (UID: \"ebbbac03-3a4f-4e36-b961-2a9dc1995eaf\") "
Jul 28 10:37:12.864924 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:12.864914    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvcz6\" (UniqueName: \"kubernetes.io/projected/ebbbac03-3a4f-4e36-b961-2a9dc1995eaf-kube-api-access-kvcz6\") pod \"ebbbac03-3a4f-4e36-b961-2a9dc1995eaf\" (UID: \"ebbbac03-3a4f-4e36-b961-2a9dc1995eaf\") "
Jul 28 10:37:12.865041 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:12.865004    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebbbac03-3a4f-4e36-b961-2a9dc1995eaf-home\") pod \"ebbbac03-3a4f-4e36-b961-2a9dc1995eaf\" (UID: \"ebbbac03-3a4f-4e36-b961-2a9dc1995eaf\") "
Jul 28 10:37:12.865089 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:12.865047    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebbbac03-3a4f-4e36-b961-2a9dc1995eaf-model-cache" (OuterVolumeSpecName: "model-cache") pod "ebbbac03-3a4f-4e36-b961-2a9dc1995eaf" (UID: "ebbbac03-3a4f-4e36-b961-2a9dc1995eaf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:37:12.865089 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:12.865063    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebbbac03-3a4f-4e36-b961-2a9dc1995eaf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ebbbac03-3a4f-4e36-b961-2a9dc1995eaf" (UID: "ebbbac03-3a4f-4e36-b961-2a9dc1995eaf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:37:12.865276 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:12.865255    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebbbac03-3a4f-4e36-b961-2a9dc1995eaf-model-cache\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:37:12.865276 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:12.865274    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebbbac03-3a4f-4e36-b961-2a9dc1995eaf-tmp-dir\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:37:12.865478 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:12.865293    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebbbac03-3a4f-4e36-b961-2a9dc1995eaf-home" (OuterVolumeSpecName: "home") pod "ebbbac03-3a4f-4e36-b961-2a9dc1995eaf" (UID: "ebbbac03-3a4f-4e36-b961-2a9dc1995eaf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:37:12.867047 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:12.867019    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebbbac03-3a4f-4e36-b961-2a9dc1995eaf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ebbbac03-3a4f-4e36-b961-2a9dc1995eaf" (UID: "ebbbac03-3a4f-4e36-b961-2a9dc1995eaf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:37:12.867047 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:12.867025    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebbbac03-3a4f-4e36-b961-2a9dc1995eaf-dshm" (OuterVolumeSpecName: "dshm") pod "ebbbac03-3a4f-4e36-b961-2a9dc1995eaf" (UID: "ebbbac03-3a4f-4e36-b961-2a9dc1995eaf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:37:12.867187 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:12.867084    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebbbac03-3a4f-4e36-b961-2a9dc1995eaf-kube-api-access-kvcz6" (OuterVolumeSpecName: "kube-api-access-kvcz6") pod "ebbbac03-3a4f-4e36-b961-2a9dc1995eaf" (UID: "ebbbac03-3a4f-4e36-b961-2a9dc1995eaf"). InnerVolumeSpecName "kube-api-access-kvcz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:37:12.966036 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:12.965946    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebbbac03-3a4f-4e36-b961-2a9dc1995eaf-tls-certs\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:37:12.966036 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:12.965977    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebbbac03-3a4f-4e36-b961-2a9dc1995eaf-dshm\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:37:12.966036 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:12.965986    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvcz6\" (UniqueName: \"kubernetes.io/projected/ebbbac03-3a4f-4e36-b961-2a9dc1995eaf-kube-api-access-kvcz6\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:37:12.966036 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:12.965994    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebbbac03-3a4f-4e36-b961-2a9dc1995eaf-home\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:37:13.477467 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:13.477431    2557 generic.go:356] "Generic (PLEG): container finished" podID="ebbbac03-3a4f-4e36-b961-2a9dc1995eaf" containerID="c8e99242c1ccda192aac65890d57fe81b340c3f194a933a87673d69ba232b7c3" exitCode=0
Jul 28 10:37:13.477467 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:13.477455    2557 generic.go:356] "Generic (PLEG): container finished" podID="ebbbac03-3a4f-4e36-b961-2a9dc1995eaf" containerID="0a599dddb2a188eef30709bea6a8377b75c88d240f2cfafdb6711e55e0625c83" exitCode=0
Jul 28 10:37:13.478023 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:13.477479    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz" event={"ID":"ebbbac03-3a4f-4e36-b961-2a9dc1995eaf","Type":"ContainerDied","Data":"c8e99242c1ccda192aac65890d57fe81b340c3f194a933a87673d69ba232b7c3"}
Jul 28 10:37:13.478023 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:13.477494    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz"
Jul 28 10:37:13.478023 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:13.477523    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz" event={"ID":"ebbbac03-3a4f-4e36-b961-2a9dc1995eaf","Type":"ContainerDied","Data":"0a599dddb2a188eef30709bea6a8377b75c88d240f2cfafdb6711e55e0625c83"}
Jul 28 10:37:13.478023 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:13.477541    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz" event={"ID":"ebbbac03-3a4f-4e36-b961-2a9dc1995eaf","Type":"ContainerDied","Data":"182d97a2e785f36265561eb6f9034e51d1d9222d693c0c64bf3259bec209ddf5"}
Jul 28 10:37:13.478023 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:13.477560    2557 scope.go:117] "RemoveContainer" containerID="c8e99242c1ccda192aac65890d57fe81b340c3f194a933a87673d69ba232b7c3"
Jul 28 10:37:13.487144 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:13.487124    2557 scope.go:117] "RemoveContainer" containerID="0a599dddb2a188eef30709bea6a8377b75c88d240f2cfafdb6711e55e0625c83"
Jul 28 10:37:13.494628 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:13.494583    2557 scope.go:117] "RemoveContainer" containerID="c8e99242c1ccda192aac65890d57fe81b340c3f194a933a87673d69ba232b7c3"
Jul 28 10:37:13.494891 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:37:13.494873    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8e99242c1ccda192aac65890d57fe81b340c3f194a933a87673d69ba232b7c3\": container with ID starting with c8e99242c1ccda192aac65890d57fe81b340c3f194a933a87673d69ba232b7c3 not found: ID does not exist" containerID="c8e99242c1ccda192aac65890d57fe81b340c3f194a933a87673d69ba232b7c3"
Jul 28 10:37:13.494972 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:13.494905    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8e99242c1ccda192aac65890d57fe81b340c3f194a933a87673d69ba232b7c3"} err="failed to get container status \"c8e99242c1ccda192aac65890d57fe81b340c3f194a933a87673d69ba232b7c3\": rpc error: code = NotFound desc = could not find container \"c8e99242c1ccda192aac65890d57fe81b340c3f194a933a87673d69ba232b7c3\": container with ID starting with c8e99242c1ccda192aac65890d57fe81b340c3f194a933a87673d69ba232b7c3 not found: ID does not exist"
Jul 28 10:37:13.494972 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:13.494930    2557 scope.go:117] "RemoveContainer" containerID="0a599dddb2a188eef30709bea6a8377b75c88d240f2cfafdb6711e55e0625c83"
Jul 28 10:37:13.495180 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:37:13.495159    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a599dddb2a188eef30709bea6a8377b75c88d240f2cfafdb6711e55e0625c83\": container with ID starting with 0a599dddb2a188eef30709bea6a8377b75c88d240f2cfafdb6711e55e0625c83 not found: ID does not exist" containerID="0a599dddb2a188eef30709bea6a8377b75c88d240f2cfafdb6711e55e0625c83"
Jul 28 10:37:13.495229 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:13.495187    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a599dddb2a188eef30709bea6a8377b75c88d240f2cfafdb6711e55e0625c83"} err="failed to get container status \"0a599dddb2a188eef30709bea6a8377b75c88d240f2cfafdb6711e55e0625c83\": rpc error: code = NotFound desc = could not find container \"0a599dddb2a188eef30709bea6a8377b75c88d240f2cfafdb6711e55e0625c83\": container with ID starting with 0a599dddb2a188eef30709bea6a8377b75c88d240f2cfafdb6711e55e0625c83 not found: ID does not exist"
Jul 28 10:37:13.495229 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:13.495208    2557 scope.go:117] "RemoveContainer" containerID="c8e99242c1ccda192aac65890d57fe81b340c3f194a933a87673d69ba232b7c3"
Jul 28 10:37:13.495428 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:13.495410    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8e99242c1ccda192aac65890d57fe81b340c3f194a933a87673d69ba232b7c3"} err="failed to get container status \"c8e99242c1ccda192aac65890d57fe81b340c3f194a933a87673d69ba232b7c3\": rpc error: code = NotFound desc = could not find container \"c8e99242c1ccda192aac65890d57fe81b340c3f194a933a87673d69ba232b7c3\": container with ID starting with c8e99242c1ccda192aac65890d57fe81b340c3f194a933a87673d69ba232b7c3 not found: ID does not exist"
Jul 28 10:37:13.495499 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:13.495430    2557 scope.go:117] "RemoveContainer" containerID="0a599dddb2a188eef30709bea6a8377b75c88d240f2cfafdb6711e55e0625c83"
Jul 28 10:37:13.495668 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:13.495646    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a599dddb2a188eef30709bea6a8377b75c88d240f2cfafdb6711e55e0625c83"} err="failed to get container status \"0a599dddb2a188eef30709bea6a8377b75c88d240f2cfafdb6711e55e0625c83\": rpc error: code = NotFound desc = could not find container \"0a599dddb2a188eef30709bea6a8377b75c88d240f2cfafdb6711e55e0625c83\": container with ID starting with 0a599dddb2a188eef30709bea6a8377b75c88d240f2cfafdb6711e55e0625c83 not found: ID does not exist"
Jul 28 10:37:13.498452 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:13.498429    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz"]
Jul 28 10:37:13.502450 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:13.502421    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-pnwpz"]
Jul 28 10:37:14.580357 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:14.580315    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebbbac03-3a4f-4e36-b961-2a9dc1995eaf" path="/var/lib/kubelet/pods/ebbbac03-3a4f-4e36-b961-2a9dc1995eaf/volumes"
Jul 28 10:37:20.056179 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:20.056139    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f98cgs6" podUID="cb54f781-ebdf-497e-a3d5-9df5a7475d76" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 10:37:27.371567 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:27.371525    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f98cgs6" podUID="cb54f781-ebdf-497e-a3d5-9df5a7475d76" containerName="main" containerID="cri-o://42c55ea96b635fda4ba07aa5e12504678080a426635aa7daa90c2d49131eb383" gracePeriod=45
Jul 28 10:37:27.544499 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:27.544468    2557 generic.go:356] "Generic (PLEG): container finished" podID="cb54f781-ebdf-497e-a3d5-9df5a7475d76" containerID="42c55ea96b635fda4ba07aa5e12504678080a426635aa7daa90c2d49131eb383" exitCode=0
Jul 28 10:37:27.544631 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:27.544535    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f98cgs6" event={"ID":"cb54f781-ebdf-497e-a3d5-9df5a7475d76","Type":"ContainerDied","Data":"42c55ea96b635fda4ba07aa5e12504678080a426635aa7daa90c2d49131eb383"}
Jul 28 10:37:27.628690 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:27.628627    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f98cgs6"
Jul 28 10:37:27.693888 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:27.693849    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb54f781-ebdf-497e-a3d5-9df5a7475d76-tls-certs\") pod \"cb54f781-ebdf-497e-a3d5-9df5a7475d76\" (UID: \"cb54f781-ebdf-497e-a3d5-9df5a7475d76\") "
Jul 28 10:37:27.694060 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:27.693935    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wn2hj\" (UniqueName: \"kubernetes.io/projected/cb54f781-ebdf-497e-a3d5-9df5a7475d76-kube-api-access-wn2hj\") pod \"cb54f781-ebdf-497e-a3d5-9df5a7475d76\" (UID: \"cb54f781-ebdf-497e-a3d5-9df5a7475d76\") "
Jul 28 10:37:27.696019 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:27.695984    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb54f781-ebdf-497e-a3d5-9df5a7475d76-kube-api-access-wn2hj" (OuterVolumeSpecName: "kube-api-access-wn2hj") pod "cb54f781-ebdf-497e-a3d5-9df5a7475d76" (UID: "cb54f781-ebdf-497e-a3d5-9df5a7475d76"). InnerVolumeSpecName "kube-api-access-wn2hj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:37:27.696183 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:27.696160    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb54f781-ebdf-497e-a3d5-9df5a7475d76-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cb54f781-ebdf-497e-a3d5-9df5a7475d76" (UID: "cb54f781-ebdf-497e-a3d5-9df5a7475d76"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:37:27.795043 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:27.795004    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb54f781-ebdf-497e-a3d5-9df5a7475d76-tls-certs\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:37:27.795043 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:27.795035    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wn2hj\" (UniqueName: \"kubernetes.io/projected/cb54f781-ebdf-497e-a3d5-9df5a7475d76-kube-api-access-wn2hj\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:37:28.550683 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:28.550634    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f98cgs6" event={"ID":"cb54f781-ebdf-497e-a3d5-9df5a7475d76","Type":"ContainerDied","Data":"b4a4d9464c70e08007a0daf44db1686c21a4984684d23be562287ab885395933"}
Jul 28 10:37:28.550683 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:28.550684    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f98cgs6"
Jul 28 10:37:28.551152 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:28.550750    2557 scope.go:117] "RemoveContainer" containerID="42c55ea96b635fda4ba07aa5e12504678080a426635aa7daa90c2d49131eb383"
Jul 28 10:37:28.571752 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:28.571723    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f98cgs6"]
Jul 28 10:37:28.579866 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:28.579840    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f98cgs6"]
Jul 28 10:37:30.579997 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:37:30.579962    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb54f781-ebdf-497e-a3d5-9df5a7475d76" path="/var/lib/kubelet/pods/cb54f781-ebdf-497e-a3d5-9df5a7475d76/volumes"
Jul 28 10:38:14.103920 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:14.103884    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g"]
Jul 28 10:38:14.104370 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:14.104337    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebbbac03-3a4f-4e36-b961-2a9dc1995eaf" containerName="llm-d-routing-sidecar"
Jul 28 10:38:14.104370 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:14.104353    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebbbac03-3a4f-4e36-b961-2a9dc1995eaf" containerName="llm-d-routing-sidecar"
Jul 28 10:38:14.104459 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:14.104375    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebbbac03-3a4f-4e36-b961-2a9dc1995eaf" containerName="main"
Jul 28 10:38:14.104459 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:14.104385    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebbbac03-3a4f-4e36-b961-2a9dc1995eaf" containerName="main"
Jul 28 10:38:14.104459 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:14.104418    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb54f781-ebdf-497e-a3d5-9df5a7475d76" containerName="main"
Jul 28 10:38:14.104459 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:14.104428    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb54f781-ebdf-497e-a3d5-9df5a7475d76" containerName="main"
Jul 28 10:38:14.104614 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:14.104501    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebbbac03-3a4f-4e36-b961-2a9dc1995eaf" containerName="main"
Jul 28 10:38:14.104614 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:14.104512    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebbbac03-3a4f-4e36-b961-2a9dc1995eaf" containerName="llm-d-routing-sidecar"
Jul 28 10:38:14.104614 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:14.104525    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb54f781-ebdf-497e-a3d5-9df5a7475d76" containerName="main"
Jul 28 10:38:14.109652 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:14.109631    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g"
Jul 28 10:38:14.112541 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:14.112517    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-6xmnv\""
Jul 28 10:38:14.112685 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:14.112642    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-dockercfg-fx59x\""
Jul 28 10:38:14.113391 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:14.113367    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Jul 28 10:38:14.113527 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:14.113372    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Jul 28 10:38:14.113527 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:14.113461    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Jul 28 10:38:14.119085 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:14.119064    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g"]
Jul 28 10:38:14.198892 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:14.198853    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec81110b-f5db-422a-9c2f-180b509a3fce-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g\" (UID: \"ec81110b-f5db-422a-9c2f-180b509a3fce\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g"
Jul 28 10:38:14.198892 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:14.198891    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec81110b-f5db-422a-9c2f-180b509a3fce-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g\" (UID: \"ec81110b-f5db-422a-9c2f-180b509a3fce\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g"
Jul 28 10:38:14.199117 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:14.198915    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec81110b-f5db-422a-9c2f-180b509a3fce-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g\" (UID: \"ec81110b-f5db-422a-9c2f-180b509a3fce\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g"
Jul 28 10:38:14.199117 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:14.198956    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl69q\" (UniqueName: \"kubernetes.io/projected/ec81110b-f5db-422a-9c2f-180b509a3fce-kube-api-access-hl69q\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g\" (UID: \"ec81110b-f5db-422a-9c2f-180b509a3fce\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g"
Jul 28 10:38:14.199117 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:14.199088    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec81110b-f5db-422a-9c2f-180b509a3fce-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g\" (UID: \"ec81110b-f5db-422a-9c2f-180b509a3fce\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g"
Jul 28 10:38:14.199218 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:14.199149    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec81110b-f5db-422a-9c2f-180b509a3fce-home\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g\" (UID: \"ec81110b-f5db-422a-9c2f-180b509a3fce\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g"
Jul 28 10:38:14.300221 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:14.300169    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec81110b-f5db-422a-9c2f-180b509a3fce-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g\" (UID: \"ec81110b-f5db-422a-9c2f-180b509a3fce\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g"
Jul 28 10:38:14.300385 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:14.300339    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec81110b-f5db-422a-9c2f-180b509a3fce-home\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g\" (UID: \"ec81110b-f5db-422a-9c2f-180b509a3fce\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g"
Jul 28 10:38:14.300426 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:14.300384    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec81110b-f5db-422a-9c2f-180b509a3fce-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g\" (UID: \"ec81110b-f5db-422a-9c2f-180b509a3fce\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g"
Jul 28 10:38:14.300426 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:14.300411    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec81110b-f5db-422a-9c2f-180b509a3fce-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g\" (UID: \"ec81110b-f5db-422a-9c2f-180b509a3fce\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g"
Jul 28 10:38:14.300493 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:14.300451    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec81110b-f5db-422a-9c2f-180b509a3fce-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g\" (UID: \"ec81110b-f5db-422a-9c2f-180b509a3fce\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g"
Jul 28 10:38:14.300493 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:14.300477    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl69q\" (UniqueName: \"kubernetes.io/projected/ec81110b-f5db-422a-9c2f-180b509a3fce-kube-api-access-hl69q\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g\" (UID: \"ec81110b-f5db-422a-9c2f-180b509a3fce\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g"
Jul 28 10:38:14.300687 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:14.300667    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec81110b-f5db-422a-9c2f-180b509a3fce-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g\" (UID: \"ec81110b-f5db-422a-9c2f-180b509a3fce\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g"
Jul 28 10:38:14.300734 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:14.300715    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec81110b-f5db-422a-9c2f-180b509a3fce-home\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g\" (UID: \"ec81110b-f5db-422a-9c2f-180b509a3fce\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g"
Jul 28 10:38:14.300809 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:14.300791    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec81110b-f5db-422a-9c2f-180b509a3fce-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g\" (UID: \"ec81110b-f5db-422a-9c2f-180b509a3fce\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g"
Jul 28 10:38:14.302709 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:14.302686    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec81110b-f5db-422a-9c2f-180b509a3fce-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g\" (UID: \"ec81110b-f5db-422a-9c2f-180b509a3fce\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g"
Jul 28 10:38:14.302991 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:14.302970    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec81110b-f5db-422a-9c2f-180b509a3fce-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g\" (UID: \"ec81110b-f5db-422a-9c2f-180b509a3fce\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g"
Jul 28 10:38:14.308700 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:14.308678    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl69q\" (UniqueName: \"kubernetes.io/projected/ec81110b-f5db-422a-9c2f-180b509a3fce-kube-api-access-hl69q\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g\" (UID: \"ec81110b-f5db-422a-9c2f-180b509a3fce\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g"
Jul 28 10:38:14.339274 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:14.339237    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedsg4ln"]
Jul 28 10:38:14.351111 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:14.351074    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedsg4ln"]
Jul 28 10:38:14.351260 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:14.351203    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedsg4ln"
Jul 28 10:38:14.353403 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:14.353376    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-epp-sa-dockercfg-6lmn2\""
Jul 28 10:38:14.401362 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:14.401330    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/368fb701-39d6-4e29-a443-80832c4b83b0-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedsg4ln\" (UID: \"368fb701-39d6-4e29-a443-80832c4b83b0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedsg4ln"
Jul 28 10:38:14.401540 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:14.401386    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wp8h\" (UniqueName: \"kubernetes.io/projected/368fb701-39d6-4e29-a443-80832c4b83b0-kube-api-access-4wp8h\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedsg4ln\" (UID: \"368fb701-39d6-4e29-a443-80832c4b83b0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedsg4ln"
Jul 28 10:38:14.421937 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:14.421906    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g"
Jul 28 10:38:14.502138 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:14.502101    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/368fb701-39d6-4e29-a443-80832c4b83b0-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedsg4ln\" (UID: \"368fb701-39d6-4e29-a443-80832c4b83b0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedsg4ln"
Jul 28 10:38:14.502315 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:14.502165    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wp8h\" (UniqueName: \"kubernetes.io/projected/368fb701-39d6-4e29-a443-80832c4b83b0-kube-api-access-4wp8h\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedsg4ln\" (UID: \"368fb701-39d6-4e29-a443-80832c4b83b0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedsg4ln"
Jul 28 10:38:14.505795 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:14.505740    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/368fb701-39d6-4e29-a443-80832c4b83b0-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedsg4ln\" (UID: \"368fb701-39d6-4e29-a443-80832c4b83b0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedsg4ln"
Jul 28 10:38:14.513469 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:14.513441    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wp8h\" (UniqueName: \"kubernetes.io/projected/368fb701-39d6-4e29-a443-80832c4b83b0-kube-api-access-4wp8h\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedsg4ln\" (UID: \"368fb701-39d6-4e29-a443-80832c4b83b0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedsg4ln"
Jul 28 10:38:14.561525 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:14.561356    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g"]
Jul 28 10:38:14.564270 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:38:14.564246    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec81110b_f5db_422a_9c2f_180b509a3fce.slice/crio-010be224c4b4b161b51a5dadc57627bc308ece29fc30084e5a45a1497bf2dfab WatchSource:0}: Error finding container 010be224c4b4b161b51a5dadc57627bc308ece29fc30084e5a45a1497bf2dfab: Status 404 returned error can't find the container with id 010be224c4b4b161b51a5dadc57627bc308ece29fc30084e5a45a1497bf2dfab
Jul 28 10:38:14.663282 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:14.663248    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedsg4ln"
Jul 28 10:38:14.751488 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:14.751450    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g" event={"ID":"ec81110b-f5db-422a-9c2f-180b509a3fce","Type":"ContainerStarted","Data":"b052cc31f56c25b817704fb5ea88f1e6dabf4d3b8c0426ece7d4c1b25ea0f393"}
Jul 28 10:38:14.751688 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:14.751546    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g" event={"ID":"ec81110b-f5db-422a-9c2f-180b509a3fce","Type":"ContainerStarted","Data":"010be224c4b4b161b51a5dadc57627bc308ece29fc30084e5a45a1497bf2dfab"}
Jul 28 10:38:14.752332 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:14.752306    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g"
Jul 28 10:38:14.810677 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:14.810651    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedsg4ln"]
Jul 28 10:38:14.813213 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:38:14.813192    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod368fb701_39d6_4e29_a443_80832c4b83b0.slice/crio-ad9f4d3d051d009d9494677330771a7a57740c8117e899dabe8b6a2b2746310f WatchSource:0}: Error finding container ad9f4d3d051d009d9494677330771a7a57740c8117e899dabe8b6a2b2746310f: Status 404 returned error can't find the container with id ad9f4d3d051d009d9494677330771a7a57740c8117e899dabe8b6a2b2746310f
Jul 28 10:38:15.757962 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:15.757928    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedsg4ln" event={"ID":"368fb701-39d6-4e29-a443-80832c4b83b0","Type":"ContainerStarted","Data":"ab3a4ba94cfda79d9ca3dd732e9eb9814d3b7048e7bbcac87dde649def8bacc4"}
Jul 28 10:38:15.758426 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:15.757979    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedsg4ln" event={"ID":"368fb701-39d6-4e29-a443-80832c4b83b0","Type":"ContainerStarted","Data":"ad9f4d3d051d009d9494677330771a7a57740c8117e899dabe8b6a2b2746310f"}
Jul 28 10:38:15.758426 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:15.758052    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedsg4ln"
Jul 28 10:38:15.759816 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:15.759797    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g" event={"ID":"ec81110b-f5db-422a-9c2f-180b509a3fce","Type":"ContainerStarted","Data":"f5157d039d0cbd540b7290368407565284442d081fefb2a06b89bc35501fc6bf"}
Jul 28 10:38:15.775710 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:15.775660    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedsg4ln" podStartSLOduration=1.7756423319999999 podStartE2EDuration="1.775642332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:38:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:38:15.774403343 +0000 UTC m=+1831.733171744" watchObservedRunningTime="2026-07-28 10:38:15.775642332 +0000 UTC m=+1831.734410712"
Jul 28 10:38:15.793086 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:15.793042    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g" podStartSLOduration=1.79302937 podStartE2EDuration="1.79302937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:38:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:38:15.792203718 +0000 UTC m=+1831.750972099" watchObservedRunningTime="2026-07-28 10:38:15.79302937 +0000 UTC m=+1831.751797749"
Jul 28 10:38:16.422567 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:16.422533    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g"
Jul 28 10:38:24.422146 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:24.422104    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g"
Jul 28 10:38:24.440506 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:24.440473    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g"
Jul 28 10:38:24.441453 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:24.441432    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g"
Jul 28 10:38:24.816752 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:24.816722    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g"
Jul 28 10:38:46.767682 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:46.767634    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedsg4ln"
Jul 28 10:38:48.659151 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:48.659073    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g"]
Jul 28 10:38:48.666933 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:48.666898    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedsg4ln"]
Jul 28 10:38:48.679774 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:38:48.679450    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 10:38:48.679774 ip-10-0-131-88 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:38:48.679774 ip-10-0-131-88 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 28 10:38:48.679774 ip-10-0-131-88 kubenswrapper[2557]:  > containerID="f5157d039d0cbd540b7290368407565284442d081fefb2a06b89bc35501fc6bf" cmd=["/bin/sleep","15"]
Jul 28 10:38:48.679774 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:48.679518    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g" message="" err=<
Jul 28 10:38:48.679774 ip-10-0-131-88 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:38:48.679774 ip-10-0-131-88 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 28 10:38:48.679774 ip-10-0-131-88 kubenswrapper[2557]:  >
Jul 28 10:38:48.679774 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:38:48.679536    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 10:38:48.679774 ip-10-0-131-88 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:38:48.679774 ip-10-0-131-88 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 28 10:38:48.679774 ip-10-0-131-88 kubenswrapper[2557]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g" podUID="ec81110b-f5db-422a-9c2f-180b509a3fce" containerName="main" containerID="cri-o://f5157d039d0cbd540b7290368407565284442d081fefb2a06b89bc35501fc6bf"
Jul 28 10:38:48.679774 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:48.679610    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g" podUID="ec81110b-f5db-422a-9c2f-180b509a3fce" containerName="main" containerID="cri-o://f5157d039d0cbd540b7290368407565284442d081fefb2a06b89bc35501fc6bf" gracePeriod=60
Jul 28 10:38:48.806693 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:48.806416    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g" podUID="ec81110b-f5db-422a-9c2f-180b509a3fce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8001/health\": dial tcp 10.133.0.46:8001: connect: connection refused"
Jul 28 10:38:48.893911 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:48.893869    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g" podUID="ec81110b-f5db-422a-9c2f-180b509a3fce" containerName="llm-d-routing-sidecar" containerID="cri-o://b052cc31f56c25b817704fb5ea88f1e6dabf4d3b8c0426ece7d4c1b25ea0f393" gracePeriod=60
Jul 28 10:38:48.912720 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:48.912652    2557 generic.go:356] "Generic (PLEG): container finished" podID="ec81110b-f5db-422a-9c2f-180b509a3fce" containerID="f5157d039d0cbd540b7290368407565284442d081fefb2a06b89bc35501fc6bf" exitCode=0
Jul 28 10:38:48.912720 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:48.912695    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g" event={"ID":"ec81110b-f5db-422a-9c2f-180b509a3fce","Type":"ContainerDied","Data":"f5157d039d0cbd540b7290368407565284442d081fefb2a06b89bc35501fc6bf"}
Jul 28 10:38:49.141881 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:49.141853    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g"
Jul 28 10:38:49.218147 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:49.218047    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec81110b-f5db-422a-9c2f-180b509a3fce-tmp-dir\") pod \"ec81110b-f5db-422a-9c2f-180b509a3fce\" (UID: \"ec81110b-f5db-422a-9c2f-180b509a3fce\") "
Jul 28 10:38:49.218147 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:49.218123    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec81110b-f5db-422a-9c2f-180b509a3fce-model-cache\") pod \"ec81110b-f5db-422a-9c2f-180b509a3fce\" (UID: \"ec81110b-f5db-422a-9c2f-180b509a3fce\") "
Jul 28 10:38:49.218393 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:49.218200    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec81110b-f5db-422a-9c2f-180b509a3fce-dshm\") pod \"ec81110b-f5db-422a-9c2f-180b509a3fce\" (UID: \"ec81110b-f5db-422a-9c2f-180b509a3fce\") "
Jul 28 10:38:49.218393 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:49.218223    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hl69q\" (UniqueName: \"kubernetes.io/projected/ec81110b-f5db-422a-9c2f-180b509a3fce-kube-api-access-hl69q\") pod \"ec81110b-f5db-422a-9c2f-180b509a3fce\" (UID: \"ec81110b-f5db-422a-9c2f-180b509a3fce\") "
Jul 28 10:38:49.218393 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:49.218248    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec81110b-f5db-422a-9c2f-180b509a3fce-home\") pod \"ec81110b-f5db-422a-9c2f-180b509a3fce\" (UID: \"ec81110b-f5db-422a-9c2f-180b509a3fce\") "
Jul 28 10:38:49.218393 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:49.218276    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec81110b-f5db-422a-9c2f-180b509a3fce-tls-certs\") pod \"ec81110b-f5db-422a-9c2f-180b509a3fce\" (UID: \"ec81110b-f5db-422a-9c2f-180b509a3fce\") "
Jul 28 10:38:49.218393 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:49.218337    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec81110b-f5db-422a-9c2f-180b509a3fce-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ec81110b-f5db-422a-9c2f-180b509a3fce" (UID: "ec81110b-f5db-422a-9c2f-180b509a3fce"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:38:49.218682 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:49.218435    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec81110b-f5db-422a-9c2f-180b509a3fce-model-cache" (OuterVolumeSpecName: "model-cache") pod "ec81110b-f5db-422a-9c2f-180b509a3fce" (UID: "ec81110b-f5db-422a-9c2f-180b509a3fce"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:38:49.218682 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:49.218537    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec81110b-f5db-422a-9c2f-180b509a3fce-home" (OuterVolumeSpecName: "home") pod "ec81110b-f5db-422a-9c2f-180b509a3fce" (UID: "ec81110b-f5db-422a-9c2f-180b509a3fce"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:38:49.218682 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:49.218594    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec81110b-f5db-422a-9c2f-180b509a3fce-tmp-dir\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:38:49.218682 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:49.218640    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec81110b-f5db-422a-9c2f-180b509a3fce-model-cache\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:38:49.220709 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:49.220672    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec81110b-f5db-422a-9c2f-180b509a3fce-dshm" (OuterVolumeSpecName: "dshm") pod "ec81110b-f5db-422a-9c2f-180b509a3fce" (UID: "ec81110b-f5db-422a-9c2f-180b509a3fce"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:38:49.220885 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:49.220865    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec81110b-f5db-422a-9c2f-180b509a3fce-kube-api-access-hl69q" (OuterVolumeSpecName: "kube-api-access-hl69q") pod "ec81110b-f5db-422a-9c2f-180b509a3fce" (UID: "ec81110b-f5db-422a-9c2f-180b509a3fce"). InnerVolumeSpecName "kube-api-access-hl69q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:38:49.221029 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:49.220999    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec81110b-f5db-422a-9c2f-180b509a3fce-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ec81110b-f5db-422a-9c2f-180b509a3fce" (UID: "ec81110b-f5db-422a-9c2f-180b509a3fce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:38:49.319986 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:49.319954    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec81110b-f5db-422a-9c2f-180b509a3fce-dshm\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:38:49.319986 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:49.319979    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hl69q\" (UniqueName: \"kubernetes.io/projected/ec81110b-f5db-422a-9c2f-180b509a3fce-kube-api-access-hl69q\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:38:49.319986 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:49.319989    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec81110b-f5db-422a-9c2f-180b509a3fce-home\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:38:49.319986 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:49.319996    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec81110b-f5db-422a-9c2f-180b509a3fce-tls-certs\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:38:49.919044 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:49.919008    2557 generic.go:356] "Generic (PLEG): container finished" podID="ec81110b-f5db-422a-9c2f-180b509a3fce" containerID="b052cc31f56c25b817704fb5ea88f1e6dabf4d3b8c0426ece7d4c1b25ea0f393" exitCode=0
Jul 28 10:38:49.919468 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:49.919050    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g" event={"ID":"ec81110b-f5db-422a-9c2f-180b509a3fce","Type":"ContainerDied","Data":"b052cc31f56c25b817704fb5ea88f1e6dabf4d3b8c0426ece7d4c1b25ea0f393"}
Jul 28 10:38:49.919468 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:49.919113    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g" event={"ID":"ec81110b-f5db-422a-9c2f-180b509a3fce","Type":"ContainerDied","Data":"010be224c4b4b161b51a5dadc57627bc308ece29fc30084e5a45a1497bf2dfab"}
Jul 28 10:38:49.919468 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:49.919060    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g"
Jul 28 10:38:49.919468 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:49.919138    2557 scope.go:117] "RemoveContainer" containerID="f5157d039d0cbd540b7290368407565284442d081fefb2a06b89bc35501fc6bf"
Jul 28 10:38:49.928956 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:49.928937    2557 scope.go:117] "RemoveContainer" containerID="b052cc31f56c25b817704fb5ea88f1e6dabf4d3b8c0426ece7d4c1b25ea0f393"
Jul 28 10:38:49.936199 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:49.936183    2557 scope.go:117] "RemoveContainer" containerID="f5157d039d0cbd540b7290368407565284442d081fefb2a06b89bc35501fc6bf"
Jul 28 10:38:49.936424 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:38:49.936407    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5157d039d0cbd540b7290368407565284442d081fefb2a06b89bc35501fc6bf\": container with ID starting with f5157d039d0cbd540b7290368407565284442d081fefb2a06b89bc35501fc6bf not found: ID does not exist" containerID="f5157d039d0cbd540b7290368407565284442d081fefb2a06b89bc35501fc6bf"
Jul 28 10:38:49.936470 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:49.936435    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5157d039d0cbd540b7290368407565284442d081fefb2a06b89bc35501fc6bf"} err="failed to get container status \"f5157d039d0cbd540b7290368407565284442d081fefb2a06b89bc35501fc6bf\": rpc error: code = NotFound desc = could not find container \"f5157d039d0cbd540b7290368407565284442d081fefb2a06b89bc35501fc6bf\": container with ID starting with f5157d039d0cbd540b7290368407565284442d081fefb2a06b89bc35501fc6bf not found: ID does not exist"
Jul 28 10:38:49.936470 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:49.936453    2557 scope.go:117] "RemoveContainer" containerID="b052cc31f56c25b817704fb5ea88f1e6dabf4d3b8c0426ece7d4c1b25ea0f393"
Jul 28 10:38:49.936704 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:38:49.936685    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b052cc31f56c25b817704fb5ea88f1e6dabf4d3b8c0426ece7d4c1b25ea0f393\": container with ID starting with b052cc31f56c25b817704fb5ea88f1e6dabf4d3b8c0426ece7d4c1b25ea0f393 not found: ID does not exist" containerID="b052cc31f56c25b817704fb5ea88f1e6dabf4d3b8c0426ece7d4c1b25ea0f393"
Jul 28 10:38:49.936767 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:49.936708    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b052cc31f56c25b817704fb5ea88f1e6dabf4d3b8c0426ece7d4c1b25ea0f393"} err="failed to get container status \"b052cc31f56c25b817704fb5ea88f1e6dabf4d3b8c0426ece7d4c1b25ea0f393\": rpc error: code = NotFound desc = could not find container \"b052cc31f56c25b817704fb5ea88f1e6dabf4d3b8c0426ece7d4c1b25ea0f393\": container with ID starting with b052cc31f56c25b817704fb5ea88f1e6dabf4d3b8c0426ece7d4c1b25ea0f393 not found: ID does not exist"
Jul 28 10:38:49.944458 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:49.944432    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g"]
Jul 28 10:38:49.950208 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:49.950187    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-lfh8g"]
Jul 28 10:38:50.581243 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:50.581205    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec81110b-f5db-422a-9c2f-180b509a3fce" path="/var/lib/kubelet/pods/ec81110b-f5db-422a-9c2f-180b509a3fce/volumes"
Jul 28 10:38:56.767307 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:38:56.767271    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedsg4ln" podUID="368fb701-39d6-4e29-a443-80832c4b83b0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 10:39:03.699690 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:03.699628    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedsg4ln" podUID="368fb701-39d6-4e29-a443-80832c4b83b0" containerName="main" containerID="cri-o://ab3a4ba94cfda79d9ca3dd732e9eb9814d3b7048e7bbcac87dde649def8bacc4" gracePeriod=45
Jul 28 10:39:03.947494 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:03.947464    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedsg4ln"
Jul 28 10:39:03.983669 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:03.983553    2557 generic.go:356] "Generic (PLEG): container finished" podID="368fb701-39d6-4e29-a443-80832c4b83b0" containerID="ab3a4ba94cfda79d9ca3dd732e9eb9814d3b7048e7bbcac87dde649def8bacc4" exitCode=0
Jul 28 10:39:03.983669 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:03.983622    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedsg4ln"
Jul 28 10:39:03.983927 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:03.983626    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedsg4ln" event={"ID":"368fb701-39d6-4e29-a443-80832c4b83b0","Type":"ContainerDied","Data":"ab3a4ba94cfda79d9ca3dd732e9eb9814d3b7048e7bbcac87dde649def8bacc4"}
Jul 28 10:39:03.983927 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:03.983739    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedsg4ln" event={"ID":"368fb701-39d6-4e29-a443-80832c4b83b0","Type":"ContainerDied","Data":"ad9f4d3d051d009d9494677330771a7a57740c8117e899dabe8b6a2b2746310f"}
Jul 28 10:39:03.983927 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:03.983765    2557 scope.go:117] "RemoveContainer" containerID="ab3a4ba94cfda79d9ca3dd732e9eb9814d3b7048e7bbcac87dde649def8bacc4"
Jul 28 10:39:03.992640 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:03.992594    2557 scope.go:117] "RemoveContainer" containerID="ab3a4ba94cfda79d9ca3dd732e9eb9814d3b7048e7bbcac87dde649def8bacc4"
Jul 28 10:39:03.992954 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:39:03.992935    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab3a4ba94cfda79d9ca3dd732e9eb9814d3b7048e7bbcac87dde649def8bacc4\": container with ID starting with ab3a4ba94cfda79d9ca3dd732e9eb9814d3b7048e7bbcac87dde649def8bacc4 not found: ID does not exist" containerID="ab3a4ba94cfda79d9ca3dd732e9eb9814d3b7048e7bbcac87dde649def8bacc4"
Jul 28 10:39:03.992996 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:03.992963    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab3a4ba94cfda79d9ca3dd732e9eb9814d3b7048e7bbcac87dde649def8bacc4"} err="failed to get container status \"ab3a4ba94cfda79d9ca3dd732e9eb9814d3b7048e7bbcac87dde649def8bacc4\": rpc error: code = NotFound desc = could not find container \"ab3a4ba94cfda79d9ca3dd732e9eb9814d3b7048e7bbcac87dde649def8bacc4\": container with ID starting with ab3a4ba94cfda79d9ca3dd732e9eb9814d3b7048e7bbcac87dde649def8bacc4 not found: ID does not exist"
Jul 28 10:39:04.051470 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:04.051433    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/368fb701-39d6-4e29-a443-80832c4b83b0-tls-certs\") pod \"368fb701-39d6-4e29-a443-80832c4b83b0\" (UID: \"368fb701-39d6-4e29-a443-80832c4b83b0\") "
Jul 28 10:39:04.051688 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:04.051481    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wp8h\" (UniqueName: \"kubernetes.io/projected/368fb701-39d6-4e29-a443-80832c4b83b0-kube-api-access-4wp8h\") pod \"368fb701-39d6-4e29-a443-80832c4b83b0\" (UID: \"368fb701-39d6-4e29-a443-80832c4b83b0\") "
Jul 28 10:39:04.053650 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:04.053617    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/368fb701-39d6-4e29-a443-80832c4b83b0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "368fb701-39d6-4e29-a443-80832c4b83b0" (UID: "368fb701-39d6-4e29-a443-80832c4b83b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:39:04.053768 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:04.053648    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/368fb701-39d6-4e29-a443-80832c4b83b0-kube-api-access-4wp8h" (OuterVolumeSpecName: "kube-api-access-4wp8h") pod "368fb701-39d6-4e29-a443-80832c4b83b0" (UID: "368fb701-39d6-4e29-a443-80832c4b83b0"). InnerVolumeSpecName "kube-api-access-4wp8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:39:04.152751 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:04.152710    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/368fb701-39d6-4e29-a443-80832c4b83b0-tls-certs\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:39:04.152751 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:04.152743    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wp8h\" (UniqueName: \"kubernetes.io/projected/368fb701-39d6-4e29-a443-80832c4b83b0-kube-api-access-4wp8h\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:39:04.307993 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:04.307957    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedsg4ln"]
Jul 28 10:39:04.314078 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:04.314052    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedsg4ln"]
Jul 28 10:39:04.580783 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:04.580704    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="368fb701-39d6-4e29-a443-80832c4b83b0" path="/var/lib/kubelet/pods/368fb701-39d6-4e29-a443-80832c4b83b0/volumes"
Jul 28 10:39:14.598662 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:14.598574    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q"]
Jul 28 10:39:14.599065 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:14.599046    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="368fb701-39d6-4e29-a443-80832c4b83b0" containerName="main"
Jul 28 10:39:14.599119 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:14.599069    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="368fb701-39d6-4e29-a443-80832c4b83b0" containerName="main"
Jul 28 10:39:14.599119 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:14.599087    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec81110b-f5db-422a-9c2f-180b509a3fce" containerName="main"
Jul 28 10:39:14.599119 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:14.599096    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec81110b-f5db-422a-9c2f-180b509a3fce" containerName="main"
Jul 28 10:39:14.599119 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:14.599116    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec81110b-f5db-422a-9c2f-180b509a3fce" containerName="llm-d-routing-sidecar"
Jul 28 10:39:14.599306 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:14.599125    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec81110b-f5db-422a-9c2f-180b509a3fce" containerName="llm-d-routing-sidecar"
Jul 28 10:39:14.599306 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:14.599215    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec81110b-f5db-422a-9c2f-180b509a3fce" containerName="llm-d-routing-sidecar"
Jul 28 10:39:14.599306 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:14.599226    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec81110b-f5db-422a-9c2f-180b509a3fce" containerName="main"
Jul 28 10:39:14.599306 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:14.599237    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="368fb701-39d6-4e29-a443-80832c4b83b0" containerName="main"
Jul 28 10:39:14.604204 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:14.604180    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q"
Jul 28 10:39:14.607447 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:14.607419    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-hvlmd\""
Jul 28 10:39:14.607694 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:14.607461    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Jul 28 10:39:14.607843 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:14.607461    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Jul 28 10:39:14.607843 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:14.607507    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Jul 28 10:39:14.608027 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:14.607525    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Jul 28 10:39:14.616642 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:14.616615    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q"]
Jul 28 10:39:14.744680 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:14.744647    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dece5bc-1ed4-47fb-8844-909d8a54d0ab-tls-certs\") pod \"tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q\" (UID: \"1dece5bc-1ed4-47fb-8844-909d8a54d0ab\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q"
Jul 28 10:39:14.744844 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:14.744689    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89g4m\" (UniqueName: \"kubernetes.io/projected/1dece5bc-1ed4-47fb-8844-909d8a54d0ab-kube-api-access-89g4m\") pod \"tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q\" (UID: \"1dece5bc-1ed4-47fb-8844-909d8a54d0ab\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q"
Jul 28 10:39:14.744844 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:14.744712    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dece5bc-1ed4-47fb-8844-909d8a54d0ab-dshm\") pod \"tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q\" (UID: \"1dece5bc-1ed4-47fb-8844-909d8a54d0ab\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q"
Jul 28 10:39:14.744962 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:14.744839    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1dece5bc-1ed4-47fb-8844-909d8a54d0ab-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q\" (UID: \"1dece5bc-1ed4-47fb-8844-909d8a54d0ab\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q"
Jul 28 10:39:14.744962 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:14.744878    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dece5bc-1ed4-47fb-8844-909d8a54d0ab-model-cache\") pod \"tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q\" (UID: \"1dece5bc-1ed4-47fb-8844-909d8a54d0ab\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q"
Jul 28 10:39:14.744962 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:14.744901    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dece5bc-1ed4-47fb-8844-909d8a54d0ab-home\") pod \"tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q\" (UID: \"1dece5bc-1ed4-47fb-8844-909d8a54d0ab\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q"
Jul 28 10:39:14.744962 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:14.744937    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dece5bc-1ed4-47fb-8844-909d8a54d0ab-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q\" (UID: \"1dece5bc-1ed4-47fb-8844-909d8a54d0ab\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q"
Jul 28 10:39:14.826176 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:14.826138    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-95bbf4d44-7mbfw"]
Jul 28 10:39:14.830011 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:14.829986    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-95bbf4d44-7mbfw"
Jul 28 10:39:14.841763 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:14.841732    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-95bbf4d44-7mbfw"]
Jul 28 10:39:14.846331 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:14.846300    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dece5bc-1ed4-47fb-8844-909d8a54d0ab-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q\" (UID: \"1dece5bc-1ed4-47fb-8844-909d8a54d0ab\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q"
Jul 28 10:39:14.846467 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:14.846446    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dece5bc-1ed4-47fb-8844-909d8a54d0ab-tls-certs\") pod \"tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q\" (UID: \"1dece5bc-1ed4-47fb-8844-909d8a54d0ab\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q"
Jul 28 10:39:14.846540 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:14.846493    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89g4m\" (UniqueName: \"kubernetes.io/projected/1dece5bc-1ed4-47fb-8844-909d8a54d0ab-kube-api-access-89g4m\") pod \"tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q\" (UID: \"1dece5bc-1ed4-47fb-8844-909d8a54d0ab\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q"
Jul 28 10:39:14.846540 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:14.846531    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dece5bc-1ed4-47fb-8844-909d8a54d0ab-dshm\") pod \"tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q\" (UID: \"1dece5bc-1ed4-47fb-8844-909d8a54d0ab\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q"
Jul 28 10:39:14.846717 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:14.846613    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1dece5bc-1ed4-47fb-8844-909d8a54d0ab-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q\" (UID: \"1dece5bc-1ed4-47fb-8844-909d8a54d0ab\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q"
Jul 28 10:39:14.846717 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:14.846651    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dece5bc-1ed4-47fb-8844-909d8a54d0ab-model-cache\") pod \"tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q\" (UID: \"1dece5bc-1ed4-47fb-8844-909d8a54d0ab\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q"
Jul 28 10:39:14.846717 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:14.846684    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dece5bc-1ed4-47fb-8844-909d8a54d0ab-home\") pod \"tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q\" (UID: \"1dece5bc-1ed4-47fb-8844-909d8a54d0ab\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q"
Jul 28 10:39:14.846962 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:14.846748    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dece5bc-1ed4-47fb-8844-909d8a54d0ab-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q\" (UID: \"1dece5bc-1ed4-47fb-8844-909d8a54d0ab\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q"
Jul 28 10:39:14.847021 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:14.847000    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dece5bc-1ed4-47fb-8844-909d8a54d0ab-home\") pod \"tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q\" (UID: \"1dece5bc-1ed4-47fb-8844-909d8a54d0ab\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q"
Jul 28 10:39:14.847298 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:14.847278    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1dece5bc-1ed4-47fb-8844-909d8a54d0ab-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q\" (UID: \"1dece5bc-1ed4-47fb-8844-909d8a54d0ab\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q"
Jul 28 10:39:14.847446 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:14.847419    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dece5bc-1ed4-47fb-8844-909d8a54d0ab-model-cache\") pod \"tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q\" (UID: \"1dece5bc-1ed4-47fb-8844-909d8a54d0ab\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q"
Jul 28 10:39:14.849060 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:14.849006    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dece5bc-1ed4-47fb-8844-909d8a54d0ab-dshm\") pod \"tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q\" (UID: \"1dece5bc-1ed4-47fb-8844-909d8a54d0ab\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q"
Jul 28 10:39:14.850421 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:14.850400    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dece5bc-1ed4-47fb-8844-909d8a54d0ab-tls-certs\") pod \"tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q\" (UID: \"1dece5bc-1ed4-47fb-8844-909d8a54d0ab\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q"
Jul 28 10:39:14.863373 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:14.863348    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89g4m\" (UniqueName: \"kubernetes.io/projected/1dece5bc-1ed4-47fb-8844-909d8a54d0ab-kube-api-access-89g4m\") pod \"tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q\" (UID: \"1dece5bc-1ed4-47fb-8844-909d8a54d0ab\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q"
Jul 28 10:39:14.916903 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:14.916866    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q"
Jul 28 10:39:14.948040 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:14.948007    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6a4d122-4144-4d75-a1e8-edbf58757571-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-95bbf4d44-7mbfw\" (UID: \"d6a4d122-4144-4d75-a1e8-edbf58757571\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-95bbf4d44-7mbfw"
Jul 28 10:39:14.948211 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:14.948069    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6a4d122-4144-4d75-a1e8-edbf58757571-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-95bbf4d44-7mbfw\" (UID: \"d6a4d122-4144-4d75-a1e8-edbf58757571\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-95bbf4d44-7mbfw"
Jul 28 10:39:14.948211 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:14.948185    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6a4d122-4144-4d75-a1e8-edbf58757571-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-95bbf4d44-7mbfw\" (UID: \"d6a4d122-4144-4d75-a1e8-edbf58757571\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-95bbf4d44-7mbfw"
Jul 28 10:39:15.049713 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:15.049675    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6a4d122-4144-4d75-a1e8-edbf58757571-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-95bbf4d44-7mbfw\" (UID: \"d6a4d122-4144-4d75-a1e8-edbf58757571\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-95bbf4d44-7mbfw"
Jul 28 10:39:15.049910 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:15.049728    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6a4d122-4144-4d75-a1e8-edbf58757571-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-95bbf4d44-7mbfw\" (UID: \"d6a4d122-4144-4d75-a1e8-edbf58757571\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-95bbf4d44-7mbfw"
Jul 28 10:39:15.049910 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:15.049780    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6a4d122-4144-4d75-a1e8-edbf58757571-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-95bbf4d44-7mbfw\" (UID: \"d6a4d122-4144-4d75-a1e8-edbf58757571\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-95bbf4d44-7mbfw"
Jul 28 10:39:15.050137 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:15.050112    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6a4d122-4144-4d75-a1e8-edbf58757571-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-95bbf4d44-7mbfw\" (UID: \"d6a4d122-4144-4d75-a1e8-edbf58757571\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-95bbf4d44-7mbfw"
Jul 28 10:39:15.050210 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:15.050174    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6a4d122-4144-4d75-a1e8-edbf58757571-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-95bbf4d44-7mbfw\" (UID: \"d6a4d122-4144-4d75-a1e8-edbf58757571\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-95bbf4d44-7mbfw"
Jul 28 10:39:15.052734 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:15.052709    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6a4d122-4144-4d75-a1e8-edbf58757571-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-95bbf4d44-7mbfw\" (UID: \"d6a4d122-4144-4d75-a1e8-edbf58757571\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-95bbf4d44-7mbfw"
Jul 28 10:39:15.054287 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:15.054241    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q"]
Jul 28 10:39:15.056910 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:39:15.056887    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dece5bc_1ed4_47fb_8844_909d8a54d0ab.slice/crio-887d39abc70507ae8164b7ffaa8adc36a3b17e0d181671f249d7731ad811463e WatchSource:0}: Error finding container 887d39abc70507ae8164b7ffaa8adc36a3b17e0d181671f249d7731ad811463e: Status 404 returned error can't find the container with id 887d39abc70507ae8164b7ffaa8adc36a3b17e0d181671f249d7731ad811463e
Jul 28 10:39:15.143777 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:15.143743    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-95bbf4d44-7mbfw"
Jul 28 10:39:15.280167 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:15.280106    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-95bbf4d44-7mbfw"]
Jul 28 10:39:15.282312 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:39:15.282276    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6a4d122_4144_4d75_a1e8_edbf58757571.slice/crio-e9382becaeaa46959daf280a074b12e1932d71eddfbfc7cba27366b0c7cda8df WatchSource:0}: Error finding container e9382becaeaa46959daf280a074b12e1932d71eddfbfc7cba27366b0c7cda8df: Status 404 returned error can't find the container with id e9382becaeaa46959daf280a074b12e1932d71eddfbfc7cba27366b0c7cda8df
Jul 28 10:39:16.037975 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:16.037934    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q" event={"ID":"1dece5bc-1ed4-47fb-8844-909d8a54d0ab","Type":"ContainerStarted","Data":"faaabfcd21ca4c927f60cf110a3fe5f9bb6277a4945157262042ff885ffaf01c"}
Jul 28 10:39:16.038429 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:16.038006    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q" event={"ID":"1dece5bc-1ed4-47fb-8844-909d8a54d0ab","Type":"ContainerStarted","Data":"887d39abc70507ae8164b7ffaa8adc36a3b17e0d181671f249d7731ad811463e"}
Jul 28 10:39:16.039332 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:16.039304    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-95bbf4d44-7mbfw" event={"ID":"d6a4d122-4144-4d75-a1e8-edbf58757571","Type":"ContainerStarted","Data":"ac913c8871d49e13468c9098a73abfcbdffc07424974f47b8aa7e58ff2c9bd81"}
Jul 28 10:39:16.039442 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:16.039353    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-95bbf4d44-7mbfw" event={"ID":"d6a4d122-4144-4d75-a1e8-edbf58757571","Type":"ContainerStarted","Data":"e9382becaeaa46959daf280a074b12e1932d71eddfbfc7cba27366b0c7cda8df"}
Jul 28 10:39:19.056148 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:19.056117    2557 generic.go:356] "Generic (PLEG): container finished" podID="1dece5bc-1ed4-47fb-8844-909d8a54d0ab" containerID="faaabfcd21ca4c927f60cf110a3fe5f9bb6277a4945157262042ff885ffaf01c" exitCode=0
Jul 28 10:39:19.056511 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:19.056188    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q" event={"ID":"1dece5bc-1ed4-47fb-8844-909d8a54d0ab","Type":"ContainerDied","Data":"faaabfcd21ca4c927f60cf110a3fe5f9bb6277a4945157262042ff885ffaf01c"}
Jul 28 10:39:19.057817 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:19.057793    2557 generic.go:356] "Generic (PLEG): container finished" podID="d6a4d122-4144-4d75-a1e8-edbf58757571" containerID="ac913c8871d49e13468c9098a73abfcbdffc07424974f47b8aa7e58ff2c9bd81" exitCode=0
Jul 28 10:39:19.057918 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:19.057845    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-95bbf4d44-7mbfw" event={"ID":"d6a4d122-4144-4d75-a1e8-edbf58757571","Type":"ContainerDied","Data":"ac913c8871d49e13468c9098a73abfcbdffc07424974f47b8aa7e58ff2c9bd81"}
Jul 28 10:39:19.058879 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:19.058863    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 10:39:20.067542 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:20.067471    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q" event={"ID":"1dece5bc-1ed4-47fb-8844-909d8a54d0ab","Type":"ContainerStarted","Data":"07e7bb5c53e8db42e62b8eec76d76329b31cfa9300d64be5f94f7c939c78149c"}
Jul 28 10:39:20.092184 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:20.092120    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q" podStartSLOduration=2.481219661 podStartE2EDuration="6.092101988s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.610882327s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:39:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:39:20.091108982 +0000 UTC m=+1896.049877362" watchObservedRunningTime="2026-07-28 10:39:20.092101988 +0000 UTC m=+1896.050870366"
Jul 28 10:39:20.917711 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:20.917673    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q"
Jul 28 10:39:24.917972 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:24.917930    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q"
Jul 28 10:39:24.933248 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:24.933223    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q"
Jul 28 10:39:25.108182 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:25.108152    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q"
Jul 28 10:39:54.257272 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:54.257234    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-95bbf4d44-7mbfw" event={"ID":"d6a4d122-4144-4d75-a1e8-edbf58757571","Type":"ContainerStarted","Data":"e07c52f7040554b41585ecbe510c7a40d9f4192176d8bde3c85c23c74b669437"}
Jul 28 10:39:54.257727 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:54.257493    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-95bbf4d44-7mbfw"
Jul 28 10:39:54.275410 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:39:54.275351    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-95bbf4d44-7mbfw" podStartSLOduration=2.433570672 podStartE2EDuration="40.275335651s" totalImagesPullingTime="34.173794507s" totalInitContainerRuntime="3.667970472s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:39:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:39:54.274006813 +0000 UTC m=+1930.232775193" watchObservedRunningTime="2026-07-28 10:39:54.275335651 +0000 UTC m=+1930.234104030"
Jul 28 10:40:25.274143 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:25.274106    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-95bbf4d44-7mbfw"
Jul 28 10:40:26.501110 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:26.501085    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-cf47448b_tokenizer-clean-path-test-tokenizer-95bbf4d44-7mbfw_d6a4d122-4144-4d75-a1e8-edbf58757571/main/0.log"
Jul 28 10:40:26.718968 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:26.718929    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q"]
Jul 28 10:40:26.726527 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:26.726497    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-95bbf4d44-7mbfw"]
Jul 28 10:40:26.726748 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:26.726703    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-95bbf4d44-7mbfw" podUID="d6a4d122-4144-4d75-a1e8-edbf58757571" containerName="main" containerID="cri-o://e07c52f7040554b41585ecbe510c7a40d9f4192176d8bde3c85c23c74b669437" gracePeriod=30
Jul 28 10:40:26.729149 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:40:26.729117    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 10:40:26.729149 ip-10-0-131-88 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:40:26.729149 ip-10-0-131-88 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 28 10:40:26.729149 ip-10-0-131-88 kubenswrapper[2557]:  > containerID="07e7bb5c53e8db42e62b8eec76d76329b31cfa9300d64be5f94f7c939c78149c" cmd=["/bin/sleep","15"]
Jul 28 10:40:26.729419 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:26.729166    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q" message="" err=<
Jul 28 10:40:26.729419 ip-10-0-131-88 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:40:26.729419 ip-10-0-131-88 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 28 10:40:26.729419 ip-10-0-131-88 kubenswrapper[2557]:  >
Jul 28 10:40:26.729419 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:40:26.729180    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 10:40:26.729419 ip-10-0-131-88 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:40:26.729419 ip-10-0-131-88 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 28 10:40:26.729419 ip-10-0-131-88 kubenswrapper[2557]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q" podUID="1dece5bc-1ed4-47fb-8844-909d8a54d0ab" containerName="main" containerID="cri-o://07e7bb5c53e8db42e62b8eec76d76329b31cfa9300d64be5f94f7c939c78149c"
Jul 28 10:40:26.729419 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:26.729217    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q" podUID="1dece5bc-1ed4-47fb-8844-909d8a54d0ab" containerName="main" containerID="cri-o://07e7bb5c53e8db42e62b8eec76d76329b31cfa9300d64be5f94f7c939c78149c" gracePeriod=60
Jul 28 10:40:27.095264 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:27.095190    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q"
Jul 28 10:40:27.238848 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:27.238818    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1dece5bc-1ed4-47fb-8844-909d8a54d0ab-kserve-provision-location\") pod \"1dece5bc-1ed4-47fb-8844-909d8a54d0ab\" (UID: \"1dece5bc-1ed4-47fb-8844-909d8a54d0ab\") "
Jul 28 10:40:27.239005 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:27.238914    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-89g4m\" (UniqueName: \"kubernetes.io/projected/1dece5bc-1ed4-47fb-8844-909d8a54d0ab-kube-api-access-89g4m\") pod \"1dece5bc-1ed4-47fb-8844-909d8a54d0ab\" (UID: \"1dece5bc-1ed4-47fb-8844-909d8a54d0ab\") "
Jul 28 10:40:27.239005 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:27.238950    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dece5bc-1ed4-47fb-8844-909d8a54d0ab-home\") pod \"1dece5bc-1ed4-47fb-8844-909d8a54d0ab\" (UID: \"1dece5bc-1ed4-47fb-8844-909d8a54d0ab\") "
Jul 28 10:40:27.239005 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:27.238979    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dece5bc-1ed4-47fb-8844-909d8a54d0ab-tmp-dir\") pod \"1dece5bc-1ed4-47fb-8844-909d8a54d0ab\" (UID: \"1dece5bc-1ed4-47fb-8844-909d8a54d0ab\") "
Jul 28 10:40:27.239179 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:27.239012    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dece5bc-1ed4-47fb-8844-909d8a54d0ab-tls-certs\") pod \"1dece5bc-1ed4-47fb-8844-909d8a54d0ab\" (UID: \"1dece5bc-1ed4-47fb-8844-909d8a54d0ab\") "
Jul 28 10:40:27.239179 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:27.239039    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dece5bc-1ed4-47fb-8844-909d8a54d0ab-dshm\") pod \"1dece5bc-1ed4-47fb-8844-909d8a54d0ab\" (UID: \"1dece5bc-1ed4-47fb-8844-909d8a54d0ab\") "
Jul 28 10:40:27.239179 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:27.239153    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dece5bc-1ed4-47fb-8844-909d8a54d0ab-model-cache\") pod \"1dece5bc-1ed4-47fb-8844-909d8a54d0ab\" (UID: \"1dece5bc-1ed4-47fb-8844-909d8a54d0ab\") "
Jul 28 10:40:27.239369 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:27.239341    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dece5bc-1ed4-47fb-8844-909d8a54d0ab-home" (OuterVolumeSpecName: "home") pod "1dece5bc-1ed4-47fb-8844-909d8a54d0ab" (UID: "1dece5bc-1ed4-47fb-8844-909d8a54d0ab"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:40:27.239493 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:27.239469    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dece5bc-1ed4-47fb-8844-909d8a54d0ab-home\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:40:27.239698 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:27.239675    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dece5bc-1ed4-47fb-8844-909d8a54d0ab-model-cache" (OuterVolumeSpecName: "model-cache") pod "1dece5bc-1ed4-47fb-8844-909d8a54d0ab" (UID: "1dece5bc-1ed4-47fb-8844-909d8a54d0ab"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:40:27.239927 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:27.239895    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dece5bc-1ed4-47fb-8844-909d8a54d0ab-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1dece5bc-1ed4-47fb-8844-909d8a54d0ab" (UID: "1dece5bc-1ed4-47fb-8844-909d8a54d0ab"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:40:27.241441 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:27.241414    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dece5bc-1ed4-47fb-8844-909d8a54d0ab-kube-api-access-89g4m" (OuterVolumeSpecName: "kube-api-access-89g4m") pod "1dece5bc-1ed4-47fb-8844-909d8a54d0ab" (UID: "1dece5bc-1ed4-47fb-8844-909d8a54d0ab"). InnerVolumeSpecName "kube-api-access-89g4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:40:27.241638 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:27.241612    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dece5bc-1ed4-47fb-8844-909d8a54d0ab-dshm" (OuterVolumeSpecName: "dshm") pod "1dece5bc-1ed4-47fb-8844-909d8a54d0ab" (UID: "1dece5bc-1ed4-47fb-8844-909d8a54d0ab"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:40:27.241701 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:27.241648    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dece5bc-1ed4-47fb-8844-909d8a54d0ab-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1dece5bc-1ed4-47fb-8844-909d8a54d0ab" (UID: "1dece5bc-1ed4-47fb-8844-909d8a54d0ab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:40:27.287674 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:27.287648    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-95bbf4d44-7mbfw"
Jul 28 10:40:27.302887 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:27.302842    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dece5bc-1ed4-47fb-8844-909d8a54d0ab-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1dece5bc-1ed4-47fb-8844-909d8a54d0ab" (UID: "1dece5bc-1ed4-47fb-8844-909d8a54d0ab"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:40:27.339992 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:27.339957    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dece5bc-1ed4-47fb-8844-909d8a54d0ab-model-cache\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:40:27.339992 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:27.339985    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1dece5bc-1ed4-47fb-8844-909d8a54d0ab-kserve-provision-location\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:40:27.339992 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:27.339995    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89g4m\" (UniqueName: \"kubernetes.io/projected/1dece5bc-1ed4-47fb-8844-909d8a54d0ab-kube-api-access-89g4m\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:40:27.340211 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:27.340027    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dece5bc-1ed4-47fb-8844-909d8a54d0ab-tmp-dir\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:40:27.340211 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:27.340037    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dece5bc-1ed4-47fb-8844-909d8a54d0ab-tls-certs\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:40:27.340211 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:27.340045    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dece5bc-1ed4-47fb-8844-909d8a54d0ab-dshm\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:40:27.427126 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:27.427021    2557 generic.go:356] "Generic (PLEG): container finished" podID="1dece5bc-1ed4-47fb-8844-909d8a54d0ab" containerID="07e7bb5c53e8db42e62b8eec76d76329b31cfa9300d64be5f94f7c939c78149c" exitCode=0
Jul 28 10:40:27.427126 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:27.427084    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q"
Jul 28 10:40:27.427126 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:27.427109    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q" event={"ID":"1dece5bc-1ed4-47fb-8844-909d8a54d0ab","Type":"ContainerDied","Data":"07e7bb5c53e8db42e62b8eec76d76329b31cfa9300d64be5f94f7c939c78149c"}
Jul 28 10:40:27.427416 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:27.427193    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q" event={"ID":"1dece5bc-1ed4-47fb-8844-909d8a54d0ab","Type":"ContainerDied","Data":"887d39abc70507ae8164b7ffaa8adc36a3b17e0d181671f249d7731ad811463e"}
Jul 28 10:40:27.427416 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:27.427223    2557 scope.go:117] "RemoveContainer" containerID="07e7bb5c53e8db42e62b8eec76d76329b31cfa9300d64be5f94f7c939c78149c"
Jul 28 10:40:27.428798 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:27.428771    2557 generic.go:356] "Generic (PLEG): container finished" podID="d6a4d122-4144-4d75-a1e8-edbf58757571" containerID="e07c52f7040554b41585ecbe510c7a40d9f4192176d8bde3c85c23c74b669437" exitCode=0
Jul 28 10:40:27.428918 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:27.428813    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-95bbf4d44-7mbfw"
Jul 28 10:40:27.428918 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:27.428811    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-95bbf4d44-7mbfw" event={"ID":"d6a4d122-4144-4d75-a1e8-edbf58757571","Type":"ContainerDied","Data":"e07c52f7040554b41585ecbe510c7a40d9f4192176d8bde3c85c23c74b669437"}
Jul 28 10:40:27.429036 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:27.428938    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-95bbf4d44-7mbfw" event={"ID":"d6a4d122-4144-4d75-a1e8-edbf58757571","Type":"ContainerDied","Data":"e9382becaeaa46959daf280a074b12e1932d71eddfbfc7cba27366b0c7cda8df"}
Jul 28 10:40:27.436750 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:27.436662    2557 scope.go:117] "RemoveContainer" containerID="faaabfcd21ca4c927f60cf110a3fe5f9bb6277a4945157262042ff885ffaf01c"
Jul 28 10:40:27.440874 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:27.440843    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6a4d122-4144-4d75-a1e8-edbf58757571-tls-certs\") pod \"d6a4d122-4144-4d75-a1e8-edbf58757571\" (UID: \"d6a4d122-4144-4d75-a1e8-edbf58757571\") "
Jul 28 10:40:27.440959 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:27.440899    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6a4d122-4144-4d75-a1e8-edbf58757571-tmp-dir\") pod \"d6a4d122-4144-4d75-a1e8-edbf58757571\" (UID: \"d6a4d122-4144-4d75-a1e8-edbf58757571\") "
Jul 28 10:40:27.441008 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:27.440994    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6a4d122-4144-4d75-a1e8-edbf58757571-kserve-provision-location\") pod \"d6a4d122-4144-4d75-a1e8-edbf58757571\" (UID: \"d6a4d122-4144-4d75-a1e8-edbf58757571\") "
Jul 28 10:40:27.441187 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:27.441160    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6a4d122-4144-4d75-a1e8-edbf58757571-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d6a4d122-4144-4d75-a1e8-edbf58757571" (UID: "d6a4d122-4144-4d75-a1e8-edbf58757571"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:40:27.441248 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:27.441237    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6a4d122-4144-4d75-a1e8-edbf58757571-tmp-dir\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:40:27.442833 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:27.442808    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6a4d122-4144-4d75-a1e8-edbf58757571-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d6a4d122-4144-4d75-a1e8-edbf58757571" (UID: "d6a4d122-4144-4d75-a1e8-edbf58757571"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:40:27.444644 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:27.444621    2557 scope.go:117] "RemoveContainer" containerID="07e7bb5c53e8db42e62b8eec76d76329b31cfa9300d64be5f94f7c939c78149c"
Jul 28 10:40:27.444912 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:40:27.444894    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07e7bb5c53e8db42e62b8eec76d76329b31cfa9300d64be5f94f7c939c78149c\": container with ID starting with 07e7bb5c53e8db42e62b8eec76d76329b31cfa9300d64be5f94f7c939c78149c not found: ID does not exist" containerID="07e7bb5c53e8db42e62b8eec76d76329b31cfa9300d64be5f94f7c939c78149c"
Jul 28 10:40:27.444979 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:27.444921    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07e7bb5c53e8db42e62b8eec76d76329b31cfa9300d64be5f94f7c939c78149c"} err="failed to get container status \"07e7bb5c53e8db42e62b8eec76d76329b31cfa9300d64be5f94f7c939c78149c\": rpc error: code = NotFound desc = could not find container \"07e7bb5c53e8db42e62b8eec76d76329b31cfa9300d64be5f94f7c939c78149c\": container with ID starting with 07e7bb5c53e8db42e62b8eec76d76329b31cfa9300d64be5f94f7c939c78149c not found: ID does not exist"
Jul 28 10:40:27.444979 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:27.444938    2557 scope.go:117] "RemoveContainer" containerID="faaabfcd21ca4c927f60cf110a3fe5f9bb6277a4945157262042ff885ffaf01c"
Jul 28 10:40:27.445194 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:40:27.445179    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faaabfcd21ca4c927f60cf110a3fe5f9bb6277a4945157262042ff885ffaf01c\": container with ID starting with faaabfcd21ca4c927f60cf110a3fe5f9bb6277a4945157262042ff885ffaf01c not found: ID does not exist" containerID="faaabfcd21ca4c927f60cf110a3fe5f9bb6277a4945157262042ff885ffaf01c"
Jul 28 10:40:27.445251 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:27.445196    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faaabfcd21ca4c927f60cf110a3fe5f9bb6277a4945157262042ff885ffaf01c"} err="failed to get container status \"faaabfcd21ca4c927f60cf110a3fe5f9bb6277a4945157262042ff885ffaf01c\": rpc error: code = NotFound desc = could not find container \"faaabfcd21ca4c927f60cf110a3fe5f9bb6277a4945157262042ff885ffaf01c\": container with ID starting with faaabfcd21ca4c927f60cf110a3fe5f9bb6277a4945157262042ff885ffaf01c not found: ID does not exist"
Jul 28 10:40:27.445251 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:27.445208    2557 scope.go:117] "RemoveContainer" containerID="e07c52f7040554b41585ecbe510c7a40d9f4192176d8bde3c85c23c74b669437"
Jul 28 10:40:27.448593 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:27.448570    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q"]
Jul 28 10:40:27.452891 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:27.452868    2557 scope.go:117] "RemoveContainer" containerID="ac913c8871d49e13468c9098a73abfcbdffc07424974f47b8aa7e58ff2c9bd81"
Jul 28 10:40:27.456912 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:27.456890    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-9hd7q"]
Jul 28 10:40:27.461146 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:27.461126    2557 scope.go:117] "RemoveContainer" containerID="e07c52f7040554b41585ecbe510c7a40d9f4192176d8bde3c85c23c74b669437"
Jul 28 10:40:27.461434 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:40:27.461415    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e07c52f7040554b41585ecbe510c7a40d9f4192176d8bde3c85c23c74b669437\": container with ID starting with e07c52f7040554b41585ecbe510c7a40d9f4192176d8bde3c85c23c74b669437 not found: ID does not exist" containerID="e07c52f7040554b41585ecbe510c7a40d9f4192176d8bde3c85c23c74b669437"
Jul 28 10:40:27.461500 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:27.461440    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e07c52f7040554b41585ecbe510c7a40d9f4192176d8bde3c85c23c74b669437"} err="failed to get container status \"e07c52f7040554b41585ecbe510c7a40d9f4192176d8bde3c85c23c74b669437\": rpc error: code = NotFound desc = could not find container \"e07c52f7040554b41585ecbe510c7a40d9f4192176d8bde3c85c23c74b669437\": container with ID starting with e07c52f7040554b41585ecbe510c7a40d9f4192176d8bde3c85c23c74b669437 not found: ID does not exist"
Jul 28 10:40:27.461500 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:27.461459    2557 scope.go:117] "RemoveContainer" containerID="ac913c8871d49e13468c9098a73abfcbdffc07424974f47b8aa7e58ff2c9bd81"
Jul 28 10:40:27.461720 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:40:27.461705    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac913c8871d49e13468c9098a73abfcbdffc07424974f47b8aa7e58ff2c9bd81\": container with ID starting with ac913c8871d49e13468c9098a73abfcbdffc07424974f47b8aa7e58ff2c9bd81 not found: ID does not exist" containerID="ac913c8871d49e13468c9098a73abfcbdffc07424974f47b8aa7e58ff2c9bd81"
Jul 28 10:40:27.461772 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:27.461724    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac913c8871d49e13468c9098a73abfcbdffc07424974f47b8aa7e58ff2c9bd81"} err="failed to get container status \"ac913c8871d49e13468c9098a73abfcbdffc07424974f47b8aa7e58ff2c9bd81\": rpc error: code = NotFound desc = could not find container \"ac913c8871d49e13468c9098a73abfcbdffc07424974f47b8aa7e58ff2c9bd81\": container with ID starting with ac913c8871d49e13468c9098a73abfcbdffc07424974f47b8aa7e58ff2c9bd81 not found: ID does not exist"
Jul 28 10:40:27.501437 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:27.501390    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6a4d122-4144-4d75-a1e8-edbf58757571-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d6a4d122-4144-4d75-a1e8-edbf58757571" (UID: "d6a4d122-4144-4d75-a1e8-edbf58757571"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:40:27.542424 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:27.542382    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6a4d122-4144-4d75-a1e8-edbf58757571-kserve-provision-location\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:40:27.542424 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:27.542417    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6a4d122-4144-4d75-a1e8-edbf58757571-tls-certs\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:40:27.749461 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:27.749426    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-95bbf4d44-7mbfw"]
Jul 28 10:40:27.751854 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:27.751828    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-95bbf4d44-7mbfw"]
Jul 28 10:40:28.580614 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:28.580572    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1dece5bc-1ed4-47fb-8844-909d8a54d0ab" path="/var/lib/kubelet/pods/1dece5bc-1ed4-47fb-8844-909d8a54d0ab/volumes"
Jul 28 10:40:28.581053 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:40:28.581039    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6a4d122-4144-4d75-a1e8-edbf58757571" path="/var/lib/kubelet/pods/d6a4d122-4144-4d75-a1e8-edbf58757571/volumes"
Jul 28 10:41:02.680677 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:02.680644    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq"]
Jul 28 10:41:02.681157 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:02.680974    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6a4d122-4144-4d75-a1e8-edbf58757571" containerName="main"
Jul 28 10:41:02.681157 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:02.680985    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6a4d122-4144-4d75-a1e8-edbf58757571" containerName="main"
Jul 28 10:41:02.681157 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:02.680994    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dece5bc-1ed4-47fb-8844-909d8a54d0ab" containerName="storage-initializer"
Jul 28 10:41:02.681157 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:02.681001    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dece5bc-1ed4-47fb-8844-909d8a54d0ab" containerName="storage-initializer"
Jul 28 10:41:02.681157 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:02.681008    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6a4d122-4144-4d75-a1e8-edbf58757571" containerName="storage-initializer"
Jul 28 10:41:02.681157 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:02.681014    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6a4d122-4144-4d75-a1e8-edbf58757571" containerName="storage-initializer"
Jul 28 10:41:02.681157 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:02.681024    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dece5bc-1ed4-47fb-8844-909d8a54d0ab" containerName="main"
Jul 28 10:41:02.681157 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:02.681029    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dece5bc-1ed4-47fb-8844-909d8a54d0ab" containerName="main"
Jul 28 10:41:02.681157 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:02.681077    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dece5bc-1ed4-47fb-8844-909d8a54d0ab" containerName="main"
Jul 28 10:41:02.681157 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:02.681086    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6a4d122-4144-4d75-a1e8-edbf58757571" containerName="main"
Jul 28 10:41:02.684021 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:02.683999    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq"
Jul 28 10:41:02.686282 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:02.686231    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Jul 28 10:41:02.686742 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:02.686718    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-7r99c\""
Jul 28 10:41:02.686871 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:02.686719    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Jul 28 10:41:02.686871 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:02.686718    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Jul 28 10:41:02.686871 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:02.686719    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Jul 28 10:41:02.695627 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:02.695585    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq"]
Jul 28 10:41:02.754315 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:02.754284    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbae359a-cb0a-4c23-859a-4b57eb0b8519-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq\" (UID: \"dbae359a-cb0a-4c23-859a-4b57eb0b8519\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq"
Jul 28 10:41:02.754496 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:02.754332    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbae359a-cb0a-4c23-859a-4b57eb0b8519-tls-certs\") pod \"tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq\" (UID: \"dbae359a-cb0a-4c23-859a-4b57eb0b8519\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq"
Jul 28 10:41:02.754496 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:02.754352    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkntg\" (UniqueName: \"kubernetes.io/projected/dbae359a-cb0a-4c23-859a-4b57eb0b8519-kube-api-access-tkntg\") pod \"tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq\" (UID: \"dbae359a-cb0a-4c23-859a-4b57eb0b8519\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq"
Jul 28 10:41:02.754496 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:02.754443    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbae359a-cb0a-4c23-859a-4b57eb0b8519-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq\" (UID: \"dbae359a-cb0a-4c23-859a-4b57eb0b8519\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq"
Jul 28 10:41:02.754496 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:02.754480    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbae359a-cb0a-4c23-859a-4b57eb0b8519-home\") pod \"tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq\" (UID: \"dbae359a-cb0a-4c23-859a-4b57eb0b8519\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq"
Jul 28 10:41:02.754670 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:02.754515    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbae359a-cb0a-4c23-859a-4b57eb0b8519-dshm\") pod \"tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq\" (UID: \"dbae359a-cb0a-4c23-859a-4b57eb0b8519\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq"
Jul 28 10:41:02.754670 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:02.754540    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbae359a-cb0a-4c23-859a-4b57eb0b8519-model-cache\") pod \"tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq\" (UID: \"dbae359a-cb0a-4c23-859a-4b57eb0b8519\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq"
Jul 28 10:41:02.855758 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:02.855720    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbae359a-cb0a-4c23-859a-4b57eb0b8519-tls-certs\") pod \"tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq\" (UID: \"dbae359a-cb0a-4c23-859a-4b57eb0b8519\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq"
Jul 28 10:41:02.855758 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:02.855758    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkntg\" (UniqueName: \"kubernetes.io/projected/dbae359a-cb0a-4c23-859a-4b57eb0b8519-kube-api-access-tkntg\") pod \"tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq\" (UID: \"dbae359a-cb0a-4c23-859a-4b57eb0b8519\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq"
Jul 28 10:41:02.855977 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:02.855789    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbae359a-cb0a-4c23-859a-4b57eb0b8519-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq\" (UID: \"dbae359a-cb0a-4c23-859a-4b57eb0b8519\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq"
Jul 28 10:41:02.855977 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:02.855920    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbae359a-cb0a-4c23-859a-4b57eb0b8519-home\") pod \"tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq\" (UID: \"dbae359a-cb0a-4c23-859a-4b57eb0b8519\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq"
Jul 28 10:41:02.856113 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:02.855982    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbae359a-cb0a-4c23-859a-4b57eb0b8519-dshm\") pod \"tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq\" (UID: \"dbae359a-cb0a-4c23-859a-4b57eb0b8519\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq"
Jul 28 10:41:02.856113 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:02.856026    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbae359a-cb0a-4c23-859a-4b57eb0b8519-model-cache\") pod \"tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq\" (UID: \"dbae359a-cb0a-4c23-859a-4b57eb0b8519\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq"
Jul 28 10:41:02.856216 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:02.856114    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbae359a-cb0a-4c23-859a-4b57eb0b8519-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq\" (UID: \"dbae359a-cb0a-4c23-859a-4b57eb0b8519\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq"
Jul 28 10:41:02.856216 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:02.856117    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbae359a-cb0a-4c23-859a-4b57eb0b8519-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq\" (UID: \"dbae359a-cb0a-4c23-859a-4b57eb0b8519\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq"
Jul 28 10:41:02.856327 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:02.856248    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbae359a-cb0a-4c23-859a-4b57eb0b8519-home\") pod \"tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq\" (UID: \"dbae359a-cb0a-4c23-859a-4b57eb0b8519\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq"
Jul 28 10:41:02.856388 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:02.856339    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbae359a-cb0a-4c23-859a-4b57eb0b8519-model-cache\") pod \"tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq\" (UID: \"dbae359a-cb0a-4c23-859a-4b57eb0b8519\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq"
Jul 28 10:41:02.856388 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:02.856373    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbae359a-cb0a-4c23-859a-4b57eb0b8519-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq\" (UID: \"dbae359a-cb0a-4c23-859a-4b57eb0b8519\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq"
Jul 28 10:41:02.858221 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:02.858198    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbae359a-cb0a-4c23-859a-4b57eb0b8519-dshm\") pod \"tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq\" (UID: \"dbae359a-cb0a-4c23-859a-4b57eb0b8519\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq"
Jul 28 10:41:02.858327 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:02.858310    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbae359a-cb0a-4c23-859a-4b57eb0b8519-tls-certs\") pod \"tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq\" (UID: \"dbae359a-cb0a-4c23-859a-4b57eb0b8519\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq"
Jul 28 10:41:02.863939 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:02.863923    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkntg\" (UniqueName: \"kubernetes.io/projected/dbae359a-cb0a-4c23-859a-4b57eb0b8519-kube-api-access-tkntg\") pod \"tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq\" (UID: \"dbae359a-cb0a-4c23-859a-4b57eb0b8519\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq"
Jul 28 10:41:02.937972 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:02.937897    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77c59bd55d-zdfk5"]
Jul 28 10:41:02.941068 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:02.941045    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77c59bd55d-zdfk5"
Jul 28 10:41:02.953050 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:02.953026    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77c59bd55d-zdfk5"]
Jul 28 10:41:02.994934 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:02.994903    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq"
Jul 28 10:41:03.058385 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:03.058342    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f21acf6-24f2-4944-afff-22b4a1dfe27e-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-77c59bd55d-zdfk5\" (UID: \"4f21acf6-24f2-4944-afff-22b4a1dfe27e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77c59bd55d-zdfk5"
Jul 28 10:41:03.058541 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:03.058436    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f21acf6-24f2-4944-afff-22b4a1dfe27e-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-77c59bd55d-zdfk5\" (UID: \"4f21acf6-24f2-4944-afff-22b4a1dfe27e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77c59bd55d-zdfk5"
Jul 28 10:41:03.058541 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:03.058480    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f21acf6-24f2-4944-afff-22b4a1dfe27e-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-77c59bd55d-zdfk5\" (UID: \"4f21acf6-24f2-4944-afff-22b4a1dfe27e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77c59bd55d-zdfk5"
Jul 28 10:41:03.129737 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:03.129697    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq"]
Jul 28 10:41:03.132000 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:41:03.131969    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbae359a_cb0a_4c23_859a_4b57eb0b8519.slice/crio-26d649ecc13066a9fba3c185d886bc52521f8883c93e62a5d4418fc738aaa80b WatchSource:0}: Error finding container 26d649ecc13066a9fba3c185d886bc52521f8883c93e62a5d4418fc738aaa80b: Status 404 returned error can't find the container with id 26d649ecc13066a9fba3c185d886bc52521f8883c93e62a5d4418fc738aaa80b
Jul 28 10:41:03.159529 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:03.159506    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f21acf6-24f2-4944-afff-22b4a1dfe27e-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-77c59bd55d-zdfk5\" (UID: \"4f21acf6-24f2-4944-afff-22b4a1dfe27e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77c59bd55d-zdfk5"
Jul 28 10:41:03.159658 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:03.159556    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f21acf6-24f2-4944-afff-22b4a1dfe27e-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-77c59bd55d-zdfk5\" (UID: \"4f21acf6-24f2-4944-afff-22b4a1dfe27e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77c59bd55d-zdfk5"
Jul 28 10:41:03.159711 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:03.159651    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f21acf6-24f2-4944-afff-22b4a1dfe27e-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-77c59bd55d-zdfk5\" (UID: \"4f21acf6-24f2-4944-afff-22b4a1dfe27e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77c59bd55d-zdfk5"
Jul 28 10:41:03.159873 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:03.159857    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f21acf6-24f2-4944-afff-22b4a1dfe27e-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-77c59bd55d-zdfk5\" (UID: \"4f21acf6-24f2-4944-afff-22b4a1dfe27e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77c59bd55d-zdfk5"
Jul 28 10:41:03.159925 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:03.159906    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f21acf6-24f2-4944-afff-22b4a1dfe27e-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-77c59bd55d-zdfk5\" (UID: \"4f21acf6-24f2-4944-afff-22b4a1dfe27e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77c59bd55d-zdfk5"
Jul 28 10:41:03.161925 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:03.161909    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f21acf6-24f2-4944-afff-22b4a1dfe27e-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-77c59bd55d-zdfk5\" (UID: \"4f21acf6-24f2-4944-afff-22b4a1dfe27e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77c59bd55d-zdfk5"
Jul 28 10:41:03.251226 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:03.251199    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77c59bd55d-zdfk5"
Jul 28 10:41:03.383887 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:03.383757    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77c59bd55d-zdfk5"]
Jul 28 10:41:03.385906 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:41:03.385878    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f21acf6_24f2_4944_afff_22b4a1dfe27e.slice/crio-502313fcaed1ad68cf513ca1b7bae251d7a1e31470e60814f288118e003391ef WatchSource:0}: Error finding container 502313fcaed1ad68cf513ca1b7bae251d7a1e31470e60814f288118e003391ef: Status 404 returned error can't find the container with id 502313fcaed1ad68cf513ca1b7bae251d7a1e31470e60814f288118e003391ef
Jul 28 10:41:03.600041 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:03.599916    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq" event={"ID":"dbae359a-cb0a-4c23-859a-4b57eb0b8519","Type":"ContainerStarted","Data":"d2211f213e435daf3e84ceb481a9b6c15cc0cb18f2db2521a73165634aa73bdf"}
Jul 28 10:41:03.600041 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:03.599984    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq" event={"ID":"dbae359a-cb0a-4c23-859a-4b57eb0b8519","Type":"ContainerStarted","Data":"26d649ecc13066a9fba3c185d886bc52521f8883c93e62a5d4418fc738aaa80b"}
Jul 28 10:41:03.601880 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:03.601849    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77c59bd55d-zdfk5" event={"ID":"4f21acf6-24f2-4944-afff-22b4a1dfe27e","Type":"ContainerStarted","Data":"3a13e282b9a0c94bad174a18b03146306e048b22d5122051dee40a5a2e0cb755"}
Jul 28 10:41:03.602002 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:03.601902    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77c59bd55d-zdfk5" event={"ID":"4f21acf6-24f2-4944-afff-22b4a1dfe27e","Type":"ContainerStarted","Data":"502313fcaed1ad68cf513ca1b7bae251d7a1e31470e60814f288118e003391ef"}
Jul 28 10:41:06.617914 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:06.617880    2557 generic.go:356] "Generic (PLEG): container finished" podID="4f21acf6-24f2-4944-afff-22b4a1dfe27e" containerID="3a13e282b9a0c94bad174a18b03146306e048b22d5122051dee40a5a2e0cb755" exitCode=0
Jul 28 10:41:06.618287 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:06.617937    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77c59bd55d-zdfk5" event={"ID":"4f21acf6-24f2-4944-afff-22b4a1dfe27e","Type":"ContainerDied","Data":"3a13e282b9a0c94bad174a18b03146306e048b22d5122051dee40a5a2e0cb755"}
Jul 28 10:41:07.624339 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:07.624299    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77c59bd55d-zdfk5" event={"ID":"4f21acf6-24f2-4944-afff-22b4a1dfe27e","Type":"ContainerStarted","Data":"ca6878e4e25abddae66833a8363ef58f2d03de22a2a9103f7e1366a03656dc67"}
Jul 28 10:41:07.624872 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:07.624536    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77c59bd55d-zdfk5"
Jul 28 10:41:07.625782 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:07.625760    2557 generic.go:356] "Generic (PLEG): container finished" podID="dbae359a-cb0a-4c23-859a-4b57eb0b8519" containerID="d2211f213e435daf3e84ceb481a9b6c15cc0cb18f2db2521a73165634aa73bdf" exitCode=0
Jul 28 10:41:07.625922 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:07.625813    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq" event={"ID":"dbae359a-cb0a-4c23-859a-4b57eb0b8519","Type":"ContainerDied","Data":"d2211f213e435daf3e84ceb481a9b6c15cc0cb18f2db2521a73165634aa73bdf"}
Jul 28 10:41:07.644388 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:07.644323    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77c59bd55d-zdfk5" podStartSLOduration=2.508303774 podStartE2EDuration="5.644300632s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.135996858s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:41:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:41:07.640931515 +0000 UTC m=+2003.599699893" watchObservedRunningTime="2026-07-28 10:41:07.644300632 +0000 UTC m=+2003.603069014"
Jul 28 10:41:08.632540 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:08.632497    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq" event={"ID":"dbae359a-cb0a-4c23-859a-4b57eb0b8519","Type":"ContainerStarted","Data":"5f546009a690a3bb1a276e389ee7eea2f895759b21e0d4eb353b3437a5c15c60"}
Jul 28 10:41:08.653003 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:08.652943    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq" podStartSLOduration=3.197693885 podStartE2EDuration="6.652920239s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.455226354s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:41:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:41:08.650493619 +0000 UTC m=+2004.609261997" watchObservedRunningTime="2026-07-28 10:41:08.652920239 +0000 UTC m=+2004.611688618"
Jul 28 10:41:08.995788 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:08.995681    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq"
Jul 28 10:41:12.995263 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:12.995225    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq"
Jul 28 10:41:13.007974 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:13.007949    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq"
Jul 28 10:41:13.668295 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:13.668265    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq"
Jul 28 10:41:38.644249 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:38.644215    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77c59bd55d-zdfk5"
Jul 28 10:41:40.039704 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:40.039670    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_tokenizer-migration-path-test-tokenizer-77c59bd55d-zdfk5_4f21acf6-24f2-4944-afff-22b4a1dfe27e/main/0.log"
Jul 28 10:41:40.291979 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:40.291901    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77c59bd55d-zdfk5"]
Jul 28 10:41:40.292122 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:40.292097    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77c59bd55d-zdfk5" podUID="4f21acf6-24f2-4944-afff-22b4a1dfe27e" containerName="main" containerID="cri-o://ca6878e4e25abddae66833a8363ef58f2d03de22a2a9103f7e1366a03656dc67" gracePeriod=30
Jul 28 10:41:40.319322 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:40.319289    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq"]
Jul 28 10:41:40.328339 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:41:40.328301    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 10:41:40.328339 ip-10-0-131-88 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:41:40.328339 ip-10-0-131-88 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 28 10:41:40.328339 ip-10-0-131-88 kubenswrapper[2557]:  > containerID="5f546009a690a3bb1a276e389ee7eea2f895759b21e0d4eb353b3437a5c15c60" cmd=["/bin/sleep","15"]
Jul 28 10:41:40.328707 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:40.328354    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq" message="" err=<
Jul 28 10:41:40.328707 ip-10-0-131-88 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:41:40.328707 ip-10-0-131-88 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 28 10:41:40.328707 ip-10-0-131-88 kubenswrapper[2557]:  >
Jul 28 10:41:40.328707 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:41:40.328369    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 10:41:40.328707 ip-10-0-131-88 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:41:40.328707 ip-10-0-131-88 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 28 10:41:40.328707 ip-10-0-131-88 kubenswrapper[2557]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq" podUID="dbae359a-cb0a-4c23-859a-4b57eb0b8519" containerName="main" containerID="cri-o://5f546009a690a3bb1a276e389ee7eea2f895759b21e0d4eb353b3437a5c15c60"
Jul 28 10:41:40.328707 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:40.328419    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq" podUID="dbae359a-cb0a-4c23-859a-4b57eb0b8519" containerName="main" containerID="cri-o://5f546009a690a3bb1a276e389ee7eea2f895759b21e0d4eb353b3437a5c15c60" gracePeriod=60
Jul 28 10:41:40.735521 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:40.735494    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq"
Jul 28 10:41:40.794557 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:40.794531    2557 generic.go:356] "Generic (PLEG): container finished" podID="dbae359a-cb0a-4c23-859a-4b57eb0b8519" containerID="5f546009a690a3bb1a276e389ee7eea2f895759b21e0d4eb353b3437a5c15c60" exitCode=0
Jul 28 10:41:40.794719 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:40.794583    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq"
Jul 28 10:41:40.794719 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:40.794631    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq" event={"ID":"dbae359a-cb0a-4c23-859a-4b57eb0b8519","Type":"ContainerDied","Data":"5f546009a690a3bb1a276e389ee7eea2f895759b21e0d4eb353b3437a5c15c60"}
Jul 28 10:41:40.794719 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:40.794688    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq" event={"ID":"dbae359a-cb0a-4c23-859a-4b57eb0b8519","Type":"ContainerDied","Data":"26d649ecc13066a9fba3c185d886bc52521f8883c93e62a5d4418fc738aaa80b"}
Jul 28 10:41:40.794719 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:40.794719    2557 scope.go:117] "RemoveContainer" containerID="5f546009a690a3bb1a276e389ee7eea2f895759b21e0d4eb353b3437a5c15c60"
Jul 28 10:41:40.796780 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:40.796754    2557 generic.go:356] "Generic (PLEG): container finished" podID="4f21acf6-24f2-4944-afff-22b4a1dfe27e" containerID="ca6878e4e25abddae66833a8363ef58f2d03de22a2a9103f7e1366a03656dc67" exitCode=0
Jul 28 10:41:40.796891 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:40.796801    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77c59bd55d-zdfk5" event={"ID":"4f21acf6-24f2-4944-afff-22b4a1dfe27e","Type":"ContainerDied","Data":"ca6878e4e25abddae66833a8363ef58f2d03de22a2a9103f7e1366a03656dc67"}
Jul 28 10:41:40.805342 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:40.805325    2557 scope.go:117] "RemoveContainer" containerID="d2211f213e435daf3e84ceb481a9b6c15cc0cb18f2db2521a73165634aa73bdf"
Jul 28 10:41:40.813429 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:40.813408    2557 scope.go:117] "RemoveContainer" containerID="5f546009a690a3bb1a276e389ee7eea2f895759b21e0d4eb353b3437a5c15c60"
Jul 28 10:41:40.813720 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:41:40.813700    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f546009a690a3bb1a276e389ee7eea2f895759b21e0d4eb353b3437a5c15c60\": container with ID starting with 5f546009a690a3bb1a276e389ee7eea2f895759b21e0d4eb353b3437a5c15c60 not found: ID does not exist" containerID="5f546009a690a3bb1a276e389ee7eea2f895759b21e0d4eb353b3437a5c15c60"
Jul 28 10:41:40.813805 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:40.813730    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f546009a690a3bb1a276e389ee7eea2f895759b21e0d4eb353b3437a5c15c60"} err="failed to get container status \"5f546009a690a3bb1a276e389ee7eea2f895759b21e0d4eb353b3437a5c15c60\": rpc error: code = NotFound desc = could not find container \"5f546009a690a3bb1a276e389ee7eea2f895759b21e0d4eb353b3437a5c15c60\": container with ID starting with 5f546009a690a3bb1a276e389ee7eea2f895759b21e0d4eb353b3437a5c15c60 not found: ID does not exist"
Jul 28 10:41:40.813805 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:40.813748    2557 scope.go:117] "RemoveContainer" containerID="d2211f213e435daf3e84ceb481a9b6c15cc0cb18f2db2521a73165634aa73bdf"
Jul 28 10:41:40.813908 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:40.813881    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbae359a-cb0a-4c23-859a-4b57eb0b8519-kserve-provision-location\") pod \"dbae359a-cb0a-4c23-859a-4b57eb0b8519\" (UID: \"dbae359a-cb0a-4c23-859a-4b57eb0b8519\") "
Jul 28 10:41:40.813957 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:40.813917    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbae359a-cb0a-4c23-859a-4b57eb0b8519-tls-certs\") pod \"dbae359a-cb0a-4c23-859a-4b57eb0b8519\" (UID: \"dbae359a-cb0a-4c23-859a-4b57eb0b8519\") "
Jul 28 10:41:40.813957 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:40.813949    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbae359a-cb0a-4c23-859a-4b57eb0b8519-dshm\") pod \"dbae359a-cb0a-4c23-859a-4b57eb0b8519\" (UID: \"dbae359a-cb0a-4c23-859a-4b57eb0b8519\") "
Jul 28 10:41:40.814435 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:40.813995    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbae359a-cb0a-4c23-859a-4b57eb0b8519-tmp-dir\") pod \"dbae359a-cb0a-4c23-859a-4b57eb0b8519\" (UID: \"dbae359a-cb0a-4c23-859a-4b57eb0b8519\") "
Jul 28 10:41:40.814435 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:40.814052    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbae359a-cb0a-4c23-859a-4b57eb0b8519-model-cache\") pod \"dbae359a-cb0a-4c23-859a-4b57eb0b8519\" (UID: \"dbae359a-cb0a-4c23-859a-4b57eb0b8519\") "
Jul 28 10:41:40.814435 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:41:40.814049    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2211f213e435daf3e84ceb481a9b6c15cc0cb18f2db2521a73165634aa73bdf\": container with ID starting with d2211f213e435daf3e84ceb481a9b6c15cc0cb18f2db2521a73165634aa73bdf not found: ID does not exist" containerID="d2211f213e435daf3e84ceb481a9b6c15cc0cb18f2db2521a73165634aa73bdf"
Jul 28 10:41:40.814435 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:40.814082    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2211f213e435daf3e84ceb481a9b6c15cc0cb18f2db2521a73165634aa73bdf"} err="failed to get container status \"d2211f213e435daf3e84ceb481a9b6c15cc0cb18f2db2521a73165634aa73bdf\": rpc error: code = NotFound desc = could not find container \"d2211f213e435daf3e84ceb481a9b6c15cc0cb18f2db2521a73165634aa73bdf\": container with ID starting with d2211f213e435daf3e84ceb481a9b6c15cc0cb18f2db2521a73165634aa73bdf not found: ID does not exist"
Jul 28 10:41:40.814435 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:40.814108    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbae359a-cb0a-4c23-859a-4b57eb0b8519-home\") pod \"dbae359a-cb0a-4c23-859a-4b57eb0b8519\" (UID: \"dbae359a-cb0a-4c23-859a-4b57eb0b8519\") "
Jul 28 10:41:40.814435 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:40.814161    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tkntg\" (UniqueName: \"kubernetes.io/projected/dbae359a-cb0a-4c23-859a-4b57eb0b8519-kube-api-access-tkntg\") pod \"dbae359a-cb0a-4c23-859a-4b57eb0b8519\" (UID: \"dbae359a-cb0a-4c23-859a-4b57eb0b8519\") "
Jul 28 10:41:40.814435 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:40.814337    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbae359a-cb0a-4c23-859a-4b57eb0b8519-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dbae359a-cb0a-4c23-859a-4b57eb0b8519" (UID: "dbae359a-cb0a-4c23-859a-4b57eb0b8519"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:41:40.814435 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:40.814355    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbae359a-cb0a-4c23-859a-4b57eb0b8519-model-cache" (OuterVolumeSpecName: "model-cache") pod "dbae359a-cb0a-4c23-859a-4b57eb0b8519" (UID: "dbae359a-cb0a-4c23-859a-4b57eb0b8519"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:41:40.814833 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:40.814486    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbae359a-cb0a-4c23-859a-4b57eb0b8519-home" (OuterVolumeSpecName: "home") pod "dbae359a-cb0a-4c23-859a-4b57eb0b8519" (UID: "dbae359a-cb0a-4c23-859a-4b57eb0b8519"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:41:40.814833 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:40.814527    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbae359a-cb0a-4c23-859a-4b57eb0b8519-tmp-dir\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:41:40.814833 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:40.814566    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbae359a-cb0a-4c23-859a-4b57eb0b8519-model-cache\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:41:40.816440 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:40.816413    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbae359a-cb0a-4c23-859a-4b57eb0b8519-dshm" (OuterVolumeSpecName: "dshm") pod "dbae359a-cb0a-4c23-859a-4b57eb0b8519" (UID: "dbae359a-cb0a-4c23-859a-4b57eb0b8519"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:41:40.816533 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:40.816440    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbae359a-cb0a-4c23-859a-4b57eb0b8519-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dbae359a-cb0a-4c23-859a-4b57eb0b8519" (UID: "dbae359a-cb0a-4c23-859a-4b57eb0b8519"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:41:40.817986 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:40.817963    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbae359a-cb0a-4c23-859a-4b57eb0b8519-kube-api-access-tkntg" (OuterVolumeSpecName: "kube-api-access-tkntg") pod "dbae359a-cb0a-4c23-859a-4b57eb0b8519" (UID: "dbae359a-cb0a-4c23-859a-4b57eb0b8519"). InnerVolumeSpecName "kube-api-access-tkntg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:41:40.844403 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:40.844373    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77c59bd55d-zdfk5"
Jul 28 10:41:40.877169 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:40.877129    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbae359a-cb0a-4c23-859a-4b57eb0b8519-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dbae359a-cb0a-4c23-859a-4b57eb0b8519" (UID: "dbae359a-cb0a-4c23-859a-4b57eb0b8519"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:41:40.915202 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:40.915172    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f21acf6-24f2-4944-afff-22b4a1dfe27e-kserve-provision-location\") pod \"4f21acf6-24f2-4944-afff-22b4a1dfe27e\" (UID: \"4f21acf6-24f2-4944-afff-22b4a1dfe27e\") "
Jul 28 10:41:40.915386 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:40.915230    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f21acf6-24f2-4944-afff-22b4a1dfe27e-tls-certs\") pod \"4f21acf6-24f2-4944-afff-22b4a1dfe27e\" (UID: \"4f21acf6-24f2-4944-afff-22b4a1dfe27e\") "
Jul 28 10:41:40.915386 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:40.915264    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f21acf6-24f2-4944-afff-22b4a1dfe27e-tmp-dir\") pod \"4f21acf6-24f2-4944-afff-22b4a1dfe27e\" (UID: \"4f21acf6-24f2-4944-afff-22b4a1dfe27e\") "
Jul 28 10:41:40.915386 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:40.915384    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbae359a-cb0a-4c23-859a-4b57eb0b8519-dshm\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:41:40.915572 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:40.915393    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbae359a-cb0a-4c23-859a-4b57eb0b8519-home\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:41:40.915572 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:40.915402    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkntg\" (UniqueName: \"kubernetes.io/projected/dbae359a-cb0a-4c23-859a-4b57eb0b8519-kube-api-access-tkntg\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:41:40.915572 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:40.915410    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbae359a-cb0a-4c23-859a-4b57eb0b8519-kserve-provision-location\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:41:40.915572 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:40.915418    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbae359a-cb0a-4c23-859a-4b57eb0b8519-tls-certs\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:41:40.915820 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:40.915666    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f21acf6-24f2-4944-afff-22b4a1dfe27e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4f21acf6-24f2-4944-afff-22b4a1dfe27e" (UID: "4f21acf6-24f2-4944-afff-22b4a1dfe27e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:41:40.917278 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:40.917255    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f21acf6-24f2-4944-afff-22b4a1dfe27e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4f21acf6-24f2-4944-afff-22b4a1dfe27e" (UID: "4f21acf6-24f2-4944-afff-22b4a1dfe27e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:41:40.975923 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:40.975879    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f21acf6-24f2-4944-afff-22b4a1dfe27e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4f21acf6-24f2-4944-afff-22b4a1dfe27e" (UID: "4f21acf6-24f2-4944-afff-22b4a1dfe27e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:41:41.015828 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:41.015794    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f21acf6-24f2-4944-afff-22b4a1dfe27e-tls-certs\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:41:41.015828 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:41.015821    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f21acf6-24f2-4944-afff-22b4a1dfe27e-tmp-dir\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:41:41.015828 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:41.015833    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f21acf6-24f2-4944-afff-22b4a1dfe27e-kserve-provision-location\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:41:41.117653 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:41.117621    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq"]
Jul 28 10:41:41.121759 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:41.121733    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq"]
Jul 28 10:41:41.658118 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:41.658071    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-bghqq" podUID="dbae359a-cb0a-4c23-859a-4b57eb0b8519" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": context deadline exceeded"
Jul 28 10:41:41.803249 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:41.803217    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77c59bd55d-zdfk5"
Jul 28 10:41:41.803249 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:41.803234    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77c59bd55d-zdfk5" event={"ID":"4f21acf6-24f2-4944-afff-22b4a1dfe27e","Type":"ContainerDied","Data":"502313fcaed1ad68cf513ca1b7bae251d7a1e31470e60814f288118e003391ef"}
Jul 28 10:41:41.803490 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:41.803298    2557 scope.go:117] "RemoveContainer" containerID="ca6878e4e25abddae66833a8363ef58f2d03de22a2a9103f7e1366a03656dc67"
Jul 28 10:41:41.812017 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:41.812001    2557 scope.go:117] "RemoveContainer" containerID="3a13e282b9a0c94bad174a18b03146306e048b22d5122051dee40a5a2e0cb755"
Jul 28 10:41:41.830536 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:41.825422    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77c59bd55d-zdfk5"]
Jul 28 10:41:41.830536 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:41.829842    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77c59bd55d-zdfk5"]
Jul 28 10:41:42.580331 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:42.580294    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f21acf6-24f2-4944-afff-22b4a1dfe27e" path="/var/lib/kubelet/pods/4f21acf6-24f2-4944-afff-22b4a1dfe27e/volumes"
Jul 28 10:41:42.580759 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:41:42.580745    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbae359a-cb0a-4c23-859a-4b57eb0b8519" path="/var/lib/kubelet/pods/dbae359a-cb0a-4c23-859a-4b57eb0b8519/volumes"
Jul 28 10:42:16.085073 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:16.084997    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-f2jv8"]
Jul 28 10:42:16.085530 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:16.085375    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f21acf6-24f2-4944-afff-22b4a1dfe27e" containerName="storage-initializer"
Jul 28 10:42:16.085530 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:16.085389    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f21acf6-24f2-4944-afff-22b4a1dfe27e" containerName="storage-initializer"
Jul 28 10:42:16.085530 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:16.085397    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f21acf6-24f2-4944-afff-22b4a1dfe27e" containerName="main"
Jul 28 10:42:16.085530 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:16.085403    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f21acf6-24f2-4944-afff-22b4a1dfe27e" containerName="main"
Jul 28 10:42:16.085530 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:16.085409    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbae359a-cb0a-4c23-859a-4b57eb0b8519" containerName="main"
Jul 28 10:42:16.085530 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:16.085415    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbae359a-cb0a-4c23-859a-4b57eb0b8519" containerName="main"
Jul 28 10:42:16.085530 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:16.085425    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbae359a-cb0a-4c23-859a-4b57eb0b8519" containerName="storage-initializer"
Jul 28 10:42:16.085530 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:16.085430    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbae359a-cb0a-4c23-859a-4b57eb0b8519" containerName="storage-initializer"
Jul 28 10:42:16.085530 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:16.085482    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbae359a-cb0a-4c23-859a-4b57eb0b8519" containerName="main"
Jul 28 10:42:16.085530 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:16.085490    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f21acf6-24f2-4944-afff-22b4a1dfe27e" containerName="main"
Jul 28 10:42:16.088732 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:16.088716    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-f2jv8"
Jul 28 10:42:16.090840 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:16.090820    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-5lckb\""
Jul 28 10:42:16.091463 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:16.091446    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Jul 28 10:42:16.091766 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:16.091751    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Jul 28 10:42:16.091840 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:16.091779    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Jul 28 10:42:16.101144 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:16.101122    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-f2jv8"]
Jul 28 10:42:16.131891 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:16.131854    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2b316c6-0ccd-4002-b5d1-522ec810c062-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-f2jv8\" (UID: \"c2b316c6-0ccd-4002-b5d1-522ec810c062\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-f2jv8"
Jul 28 10:42:16.132046 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:16.131895    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2b316c6-0ccd-4002-b5d1-522ec810c062-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-f2jv8\" (UID: \"c2b316c6-0ccd-4002-b5d1-522ec810c062\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-f2jv8"
Jul 28 10:42:16.132046 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:16.131965    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2b316c6-0ccd-4002-b5d1-522ec810c062-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-f2jv8\" (UID: \"c2b316c6-0ccd-4002-b5d1-522ec810c062\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-f2jv8"
Jul 28 10:42:16.132046 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:16.132005    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2b316c6-0ccd-4002-b5d1-522ec810c062-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-f2jv8\" (UID: \"c2b316c6-0ccd-4002-b5d1-522ec810c062\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-f2jv8"
Jul 28 10:42:16.132167 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:16.132069    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2b316c6-0ccd-4002-b5d1-522ec810c062-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-f2jv8\" (UID: \"c2b316c6-0ccd-4002-b5d1-522ec810c062\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-f2jv8"
Jul 28 10:42:16.132167 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:16.132101    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh59c\" (UniqueName: \"kubernetes.io/projected/c2b316c6-0ccd-4002-b5d1-522ec810c062-kube-api-access-zh59c\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-f2jv8\" (UID: \"c2b316c6-0ccd-4002-b5d1-522ec810c062\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-f2jv8"
Jul 28 10:42:16.233064 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:16.233026    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2b316c6-0ccd-4002-b5d1-522ec810c062-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-f2jv8\" (UID: \"c2b316c6-0ccd-4002-b5d1-522ec810c062\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-f2jv8"
Jul 28 10:42:16.233064 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:16.233068    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2b316c6-0ccd-4002-b5d1-522ec810c062-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-f2jv8\" (UID: \"c2b316c6-0ccd-4002-b5d1-522ec810c062\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-f2jv8"
Jul 28 10:42:16.233300 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:16.233137    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2b316c6-0ccd-4002-b5d1-522ec810c062-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-f2jv8\" (UID: \"c2b316c6-0ccd-4002-b5d1-522ec810c062\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-f2jv8"
Jul 28 10:42:16.233300 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:16.233168    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh59c\" (UniqueName: \"kubernetes.io/projected/c2b316c6-0ccd-4002-b5d1-522ec810c062-kube-api-access-zh59c\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-f2jv8\" (UID: \"c2b316c6-0ccd-4002-b5d1-522ec810c062\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-f2jv8"
Jul 28 10:42:16.233300 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:16.233189    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2b316c6-0ccd-4002-b5d1-522ec810c062-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-f2jv8\" (UID: \"c2b316c6-0ccd-4002-b5d1-522ec810c062\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-f2jv8"
Jul 28 10:42:16.233300 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:16.233213    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2b316c6-0ccd-4002-b5d1-522ec810c062-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-f2jv8\" (UID: \"c2b316c6-0ccd-4002-b5d1-522ec810c062\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-f2jv8"
Jul 28 10:42:16.233595 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:16.233568    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2b316c6-0ccd-4002-b5d1-522ec810c062-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-f2jv8\" (UID: \"c2b316c6-0ccd-4002-b5d1-522ec810c062\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-f2jv8"
Jul 28 10:42:16.233696 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:16.233583    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2b316c6-0ccd-4002-b5d1-522ec810c062-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-f2jv8\" (UID: \"c2b316c6-0ccd-4002-b5d1-522ec810c062\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-f2jv8"
Jul 28 10:42:16.233696 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:16.233674    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2b316c6-0ccd-4002-b5d1-522ec810c062-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-f2jv8\" (UID: \"c2b316c6-0ccd-4002-b5d1-522ec810c062\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-f2jv8"
Jul 28 10:42:16.235480 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:16.235455    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2b316c6-0ccd-4002-b5d1-522ec810c062-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-f2jv8\" (UID: \"c2b316c6-0ccd-4002-b5d1-522ec810c062\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-f2jv8"
Jul 28 10:42:16.235619 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:16.235584    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2b316c6-0ccd-4002-b5d1-522ec810c062-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-f2jv8\" (UID: \"c2b316c6-0ccd-4002-b5d1-522ec810c062\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-f2jv8"
Jul 28 10:42:16.241868 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:16.241843    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh59c\" (UniqueName: \"kubernetes.io/projected/c2b316c6-0ccd-4002-b5d1-522ec810c062-kube-api-access-zh59c\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-f2jv8\" (UID: \"c2b316c6-0ccd-4002-b5d1-522ec810c062\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-f2jv8"
Jul 28 10:42:16.398758 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:16.398662    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-f2jv8"
Jul 28 10:42:16.531870 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:16.531841    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-f2jv8"]
Jul 28 10:42:16.534259 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:42:16.534229    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2b316c6_0ccd_4002_b5d1_522ec810c062.slice/crio-d3d5eba9f1b853304e16221f854690e520ffc98890d39f3e94c2e09ea57f88af WatchSource:0}: Error finding container d3d5eba9f1b853304e16221f854690e520ffc98890d39f3e94c2e09ea57f88af: Status 404 returned error can't find the container with id d3d5eba9f1b853304e16221f854690e520ffc98890d39f3e94c2e09ea57f88af
Jul 28 10:42:16.965562 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:16.965528    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-f2jv8" event={"ID":"c2b316c6-0ccd-4002-b5d1-522ec810c062","Type":"ContainerStarted","Data":"7af019fde0ba0c3b75f1443dd19e8aac28233ce4ac1da7bcc780698dcd9e15d2"}
Jul 28 10:42:16.965745 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:16.965581    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-f2jv8" event={"ID":"c2b316c6-0ccd-4002-b5d1-522ec810c062","Type":"ContainerStarted","Data":"d3d5eba9f1b853304e16221f854690e520ffc98890d39f3e94c2e09ea57f88af"}
Jul 28 10:42:16.986030 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:16.985978    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-f2jv8" podStartSLOduration=0.985962771 podStartE2EDuration="985.962771ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:42:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:42:16.983632068 +0000 UTC m=+2072.942400463" watchObservedRunningTime="2026-07-28 10:42:16.985962771 +0000 UTC m=+2072.944731149"
Jul 28 10:42:17.399712 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:17.399674    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-f2jv8"
Jul 28 10:42:26.398991 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:26.398950    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-f2jv8"
Jul 28 10:42:26.411584 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:26.411556    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-f2jv8"
Jul 28 10:42:27.020639 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:27.020587    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-f2jv8"
Jul 28 10:42:49.454286 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:49.454249    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-f2jv8"]
Jul 28 10:42:49.465883 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:42:49.465819    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 10:42:49.465883 ip-10-0-131-88 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:42:49.465883 ip-10-0-131-88 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 28 10:42:49.465883 ip-10-0-131-88 kubenswrapper[2557]:  > containerID="7af019fde0ba0c3b75f1443dd19e8aac28233ce4ac1da7bcc780698dcd9e15d2" cmd=["/bin/sleep","15"]
Jul 28 10:42:49.466225 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:49.465894    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-f2jv8" message="" err=<
Jul 28 10:42:49.466225 ip-10-0-131-88 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:42:49.466225 ip-10-0-131-88 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 28 10:42:49.466225 ip-10-0-131-88 kubenswrapper[2557]:  >
Jul 28 10:42:49.466225 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:42:49.465910    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 10:42:49.466225 ip-10-0-131-88 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:42:49.466225 ip-10-0-131-88 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 28 10:42:49.466225 ip-10-0-131-88 kubenswrapper[2557]:  > pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-f2jv8" podUID="c2b316c6-0ccd-4002-b5d1-522ec810c062" containerName="main" containerID="cri-o://7af019fde0ba0c3b75f1443dd19e8aac28233ce4ac1da7bcc780698dcd9e15d2"
Jul 28 10:42:49.466225 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:49.465956    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-f2jv8" podUID="c2b316c6-0ccd-4002-b5d1-522ec810c062" containerName="main" containerID="cri-o://7af019fde0ba0c3b75f1443dd19e8aac28233ce4ac1da7bcc780698dcd9e15d2" gracePeriod=60
Jul 28 10:42:49.827247 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:49.827222    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-f2jv8"
Jul 28 10:42:49.931140 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:49.931105    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2b316c6-0ccd-4002-b5d1-522ec810c062-tls-certs\") pod \"c2b316c6-0ccd-4002-b5d1-522ec810c062\" (UID: \"c2b316c6-0ccd-4002-b5d1-522ec810c062\") "
Jul 28 10:42:49.931323 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:49.931155    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2b316c6-0ccd-4002-b5d1-522ec810c062-home\") pod \"c2b316c6-0ccd-4002-b5d1-522ec810c062\" (UID: \"c2b316c6-0ccd-4002-b5d1-522ec810c062\") "
Jul 28 10:42:49.931323 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:49.931221    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2b316c6-0ccd-4002-b5d1-522ec810c062-model-cache\") pod \"c2b316c6-0ccd-4002-b5d1-522ec810c062\" (UID: \"c2b316c6-0ccd-4002-b5d1-522ec810c062\") "
Jul 28 10:42:49.931323 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:49.931259    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2b316c6-0ccd-4002-b5d1-522ec810c062-dshm\") pod \"c2b316c6-0ccd-4002-b5d1-522ec810c062\" (UID: \"c2b316c6-0ccd-4002-b5d1-522ec810c062\") "
Jul 28 10:42:49.931323 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:49.931282    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2b316c6-0ccd-4002-b5d1-522ec810c062-tmp-dir\") pod \"c2b316c6-0ccd-4002-b5d1-522ec810c062\" (UID: \"c2b316c6-0ccd-4002-b5d1-522ec810c062\") "
Jul 28 10:42:49.931549 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:49.931341    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zh59c\" (UniqueName: \"kubernetes.io/projected/c2b316c6-0ccd-4002-b5d1-522ec810c062-kube-api-access-zh59c\") pod \"c2b316c6-0ccd-4002-b5d1-522ec810c062\" (UID: \"c2b316c6-0ccd-4002-b5d1-522ec810c062\") "
Jul 28 10:42:49.931549 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:49.931404    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2b316c6-0ccd-4002-b5d1-522ec810c062-home" (OuterVolumeSpecName: "home") pod "c2b316c6-0ccd-4002-b5d1-522ec810c062" (UID: "c2b316c6-0ccd-4002-b5d1-522ec810c062"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:42:49.931549 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:49.931442    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2b316c6-0ccd-4002-b5d1-522ec810c062-model-cache" (OuterVolumeSpecName: "model-cache") pod "c2b316c6-0ccd-4002-b5d1-522ec810c062" (UID: "c2b316c6-0ccd-4002-b5d1-522ec810c062"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:42:49.931735 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:49.931581    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2b316c6-0ccd-4002-b5d1-522ec810c062-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c2b316c6-0ccd-4002-b5d1-522ec810c062" (UID: "c2b316c6-0ccd-4002-b5d1-522ec810c062"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:42:49.931735 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:49.931708    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2b316c6-0ccd-4002-b5d1-522ec810c062-home\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:42:49.931735 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:49.931729    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2b316c6-0ccd-4002-b5d1-522ec810c062-model-cache\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:42:49.931907 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:49.931741    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2b316c6-0ccd-4002-b5d1-522ec810c062-tmp-dir\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:42:49.933462 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:49.933435    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2b316c6-0ccd-4002-b5d1-522ec810c062-dshm" (OuterVolumeSpecName: "dshm") pod "c2b316c6-0ccd-4002-b5d1-522ec810c062" (UID: "c2b316c6-0ccd-4002-b5d1-522ec810c062"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:42:49.933462 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:49.933454    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2b316c6-0ccd-4002-b5d1-522ec810c062-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c2b316c6-0ccd-4002-b5d1-522ec810c062" (UID: "c2b316c6-0ccd-4002-b5d1-522ec810c062"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:42:49.933615 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:49.933498    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2b316c6-0ccd-4002-b5d1-522ec810c062-kube-api-access-zh59c" (OuterVolumeSpecName: "kube-api-access-zh59c") pod "c2b316c6-0ccd-4002-b5d1-522ec810c062" (UID: "c2b316c6-0ccd-4002-b5d1-522ec810c062"). InnerVolumeSpecName "kube-api-access-zh59c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:42:50.032862 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:50.032818    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh59c\" (UniqueName: \"kubernetes.io/projected/c2b316c6-0ccd-4002-b5d1-522ec810c062-kube-api-access-zh59c\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:42:50.032862 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:50.032855    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2b316c6-0ccd-4002-b5d1-522ec810c062-tls-certs\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:42:50.032862 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:50.032869    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2b316c6-0ccd-4002-b5d1-522ec810c062-dshm\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:42:50.117868 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:50.117832    2557 generic.go:356] "Generic (PLEG): container finished" podID="c2b316c6-0ccd-4002-b5d1-522ec810c062" containerID="7af019fde0ba0c3b75f1443dd19e8aac28233ce4ac1da7bcc780698dcd9e15d2" exitCode=0
Jul 28 10:42:50.118052 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:50.117896    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-f2jv8"
Jul 28 10:42:50.118052 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:50.117921    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-f2jv8" event={"ID":"c2b316c6-0ccd-4002-b5d1-522ec810c062","Type":"ContainerDied","Data":"7af019fde0ba0c3b75f1443dd19e8aac28233ce4ac1da7bcc780698dcd9e15d2"}
Jul 28 10:42:50.118052 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:50.118009    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-f2jv8" event={"ID":"c2b316c6-0ccd-4002-b5d1-522ec810c062","Type":"ContainerDied","Data":"d3d5eba9f1b853304e16221f854690e520ffc98890d39f3e94c2e09ea57f88af"}
Jul 28 10:42:50.118052 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:50.118037    2557 scope.go:117] "RemoveContainer" containerID="7af019fde0ba0c3b75f1443dd19e8aac28233ce4ac1da7bcc780698dcd9e15d2"
Jul 28 10:42:50.127233 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:50.127216    2557 scope.go:117] "RemoveContainer" containerID="7af019fde0ba0c3b75f1443dd19e8aac28233ce4ac1da7bcc780698dcd9e15d2"
Jul 28 10:42:50.127475 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:42:50.127454    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7af019fde0ba0c3b75f1443dd19e8aac28233ce4ac1da7bcc780698dcd9e15d2\": container with ID starting with 7af019fde0ba0c3b75f1443dd19e8aac28233ce4ac1da7bcc780698dcd9e15d2 not found: ID does not exist" containerID="7af019fde0ba0c3b75f1443dd19e8aac28233ce4ac1da7bcc780698dcd9e15d2"
Jul 28 10:42:50.127555 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:50.127487    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7af019fde0ba0c3b75f1443dd19e8aac28233ce4ac1da7bcc780698dcd9e15d2"} err="failed to get container status \"7af019fde0ba0c3b75f1443dd19e8aac28233ce4ac1da7bcc780698dcd9e15d2\": rpc error: code = NotFound desc = could not find container \"7af019fde0ba0c3b75f1443dd19e8aac28233ce4ac1da7bcc780698dcd9e15d2\": container with ID starting with 7af019fde0ba0c3b75f1443dd19e8aac28233ce4ac1da7bcc780698dcd9e15d2 not found: ID does not exist"
Jul 28 10:42:50.144837 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:50.144808    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-f2jv8"]
Jul 28 10:42:50.147275 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:50.147250    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-f2jv8"]
Jul 28 10:42:50.579727 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:42:50.579691    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2b316c6-0ccd-4002-b5d1-522ec810c062" path="/var/lib/kubelet/pods/c2b316c6-0ccd-4002-b5d1-522ec810c062/volumes"
Jul 28 10:43:20.589172 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:43:20.589136    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gx56p"]
Jul 28 10:43:20.589694 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:43:20.589448    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2b316c6-0ccd-4002-b5d1-522ec810c062" containerName="main"
Jul 28 10:43:20.589694 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:43:20.589459    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2b316c6-0ccd-4002-b5d1-522ec810c062" containerName="main"
Jul 28 10:43:20.589694 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:43:20.589522    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2b316c6-0ccd-4002-b5d1-522ec810c062" containerName="main"
Jul 28 10:43:20.591442 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:43:20.591425    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gx56p"
Jul 28 10:43:20.594505 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:43:20.594485    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-m2p8x\""
Jul 28 10:43:20.594645 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:43:20.594532    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Jul 28 10:43:20.594645 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:43:20.594588    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Jul 28 10:43:20.594645 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:43:20.594588    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Jul 28 10:43:20.603339 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:43:20.603315    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gx56p"]
Jul 28 10:43:20.698188 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:43:20.698149    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32f872b9-eb4a-4775-81c0-8a5aeb1f5140-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gx56p\" (UID: \"32f872b9-eb4a-4775-81c0-8a5aeb1f5140\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gx56p"
Jul 28 10:43:20.698388 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:43:20.698196    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6bqf\" (UniqueName: \"kubernetes.io/projected/32f872b9-eb4a-4775-81c0-8a5aeb1f5140-kube-api-access-k6bqf\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gx56p\" (UID: \"32f872b9-eb4a-4775-81c0-8a5aeb1f5140\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gx56p"
Jul 28 10:43:20.698388 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:43:20.698226    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32f872b9-eb4a-4775-81c0-8a5aeb1f5140-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gx56p\" (UID: \"32f872b9-eb4a-4775-81c0-8a5aeb1f5140\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gx56p"
Jul 28 10:43:20.698388 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:43:20.698255    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32f872b9-eb4a-4775-81c0-8a5aeb1f5140-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gx56p\" (UID: \"32f872b9-eb4a-4775-81c0-8a5aeb1f5140\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gx56p"
Jul 28 10:43:20.698388 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:43:20.698287    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32f872b9-eb4a-4775-81c0-8a5aeb1f5140-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gx56p\" (UID: \"32f872b9-eb4a-4775-81c0-8a5aeb1f5140\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gx56p"
Jul 28 10:43:20.698388 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:43:20.698313    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32f872b9-eb4a-4775-81c0-8a5aeb1f5140-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gx56p\" (UID: \"32f872b9-eb4a-4775-81c0-8a5aeb1f5140\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gx56p"
Jul 28 10:43:20.799120 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:43:20.799070    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32f872b9-eb4a-4775-81c0-8a5aeb1f5140-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gx56p\" (UID: \"32f872b9-eb4a-4775-81c0-8a5aeb1f5140\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gx56p"
Jul 28 10:43:20.799120 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:43:20.799122    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6bqf\" (UniqueName: \"kubernetes.io/projected/32f872b9-eb4a-4775-81c0-8a5aeb1f5140-kube-api-access-k6bqf\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gx56p\" (UID: \"32f872b9-eb4a-4775-81c0-8a5aeb1f5140\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gx56p"
Jul 28 10:43:20.799400 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:43:20.799215    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32f872b9-eb4a-4775-81c0-8a5aeb1f5140-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gx56p\" (UID: \"32f872b9-eb4a-4775-81c0-8a5aeb1f5140\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gx56p"
Jul 28 10:43:20.799400 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:43:20.799242    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32f872b9-eb4a-4775-81c0-8a5aeb1f5140-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gx56p\" (UID: \"32f872b9-eb4a-4775-81c0-8a5aeb1f5140\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gx56p"
Jul 28 10:43:20.799400 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:43:20.799274    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32f872b9-eb4a-4775-81c0-8a5aeb1f5140-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gx56p\" (UID: \"32f872b9-eb4a-4775-81c0-8a5aeb1f5140\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gx56p"
Jul 28 10:43:20.799400 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:43:20.799297    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32f872b9-eb4a-4775-81c0-8a5aeb1f5140-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gx56p\" (UID: \"32f872b9-eb4a-4775-81c0-8a5aeb1f5140\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gx56p"
Jul 28 10:43:20.799743 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:43:20.799710    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32f872b9-eb4a-4775-81c0-8a5aeb1f5140-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gx56p\" (UID: \"32f872b9-eb4a-4775-81c0-8a5aeb1f5140\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gx56p"
Jul 28 10:43:20.799861 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:43:20.799757    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32f872b9-eb4a-4775-81c0-8a5aeb1f5140-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gx56p\" (UID: \"32f872b9-eb4a-4775-81c0-8a5aeb1f5140\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gx56p"
Jul 28 10:43:20.799861 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:43:20.799786    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32f872b9-eb4a-4775-81c0-8a5aeb1f5140-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gx56p\" (UID: \"32f872b9-eb4a-4775-81c0-8a5aeb1f5140\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gx56p"
Jul 28 10:43:20.801450 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:43:20.801424    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32f872b9-eb4a-4775-81c0-8a5aeb1f5140-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gx56p\" (UID: \"32f872b9-eb4a-4775-81c0-8a5aeb1f5140\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gx56p"
Jul 28 10:43:20.801773 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:43:20.801752    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32f872b9-eb4a-4775-81c0-8a5aeb1f5140-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gx56p\" (UID: \"32f872b9-eb4a-4775-81c0-8a5aeb1f5140\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gx56p"
Jul 28 10:43:20.810099 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:43:20.810074    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6bqf\" (UniqueName: \"kubernetes.io/projected/32f872b9-eb4a-4775-81c0-8a5aeb1f5140-kube-api-access-k6bqf\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gx56p\" (UID: \"32f872b9-eb4a-4775-81c0-8a5aeb1f5140\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gx56p"
Jul 28 10:43:20.901862 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:43:20.901772    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gx56p"
Jul 28 10:43:21.037991 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:43:21.037965    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gx56p"]
Jul 28 10:43:21.039870 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:43:21.039848    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32f872b9_eb4a_4775_81c0_8a5aeb1f5140.slice/crio-a1d893277fd2fc4f4be7021f2a0cf83a74ac0a165061a403a1159eb4b16e9d54 WatchSource:0}: Error finding container a1d893277fd2fc4f4be7021f2a0cf83a74ac0a165061a403a1159eb4b16e9d54: Status 404 returned error can't find the container with id a1d893277fd2fc4f4be7021f2a0cf83a74ac0a165061a403a1159eb4b16e9d54
Jul 28 10:43:21.256169 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:43:21.256137    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gx56p" event={"ID":"32f872b9-eb4a-4775-81c0-8a5aeb1f5140","Type":"ContainerStarted","Data":"117dedfab5190d3703968a8cf2261e09c22c296508cf7b952d3f64ac0e088deb"}
Jul 28 10:43:21.256554 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:43:21.256186    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gx56p" event={"ID":"32f872b9-eb4a-4775-81c0-8a5aeb1f5140","Type":"ContainerStarted","Data":"a1d893277fd2fc4f4be7021f2a0cf83a74ac0a165061a403a1159eb4b16e9d54"}
Jul 28 10:43:21.275899 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:43:21.275842    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gx56p" podStartSLOduration=1.2758201 podStartE2EDuration="1.2758201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:43:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:43:21.274711407 +0000 UTC m=+2137.233479789" watchObservedRunningTime="2026-07-28 10:43:21.2758201 +0000 UTC m=+2137.234588480"
Jul 28 10:43:21.902044 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:43:21.901989    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gx56p"
Jul 28 10:43:30.902791 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:43:30.902755    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gx56p"
Jul 28 10:43:30.915351 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:43:30.915319    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gx56p"
Jul 28 10:43:31.310498 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:43:31.310468    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gx56p"
Jul 28 10:44:01.627075 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:01.627032    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5w7mq"]
Jul 28 10:44:01.629286 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:01.629267    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5w7mq"
Jul 28 10:44:01.632272 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:01.632244    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa-dockercfg-992lj\""
Jul 28 10:44:01.632401 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:01.632306    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Jul 28 10:44:01.639577 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:01.639544    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5w7mq"]
Jul 28 10:44:01.657868 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:01.657805    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfnf8\" (UniqueName: \"kubernetes.io/projected/5a307ff7-d5ea-4b58-b2ee-77429ebdba63-kube-api-access-mfnf8\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5w7mq\" (UID: \"5a307ff7-d5ea-4b58-b2ee-77429ebdba63\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5w7mq"
Jul 28 10:44:01.658036 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:01.657891    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a307ff7-d5ea-4b58-b2ee-77429ebdba63-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5w7mq\" (UID: \"5a307ff7-d5ea-4b58-b2ee-77429ebdba63\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5w7mq"
Jul 28 10:44:01.758716 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:01.758678    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfnf8\" (UniqueName: \"kubernetes.io/projected/5a307ff7-d5ea-4b58-b2ee-77429ebdba63-kube-api-access-mfnf8\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5w7mq\" (UID: \"5a307ff7-d5ea-4b58-b2ee-77429ebdba63\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5w7mq"
Jul 28 10:44:01.758894 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:01.758731    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a307ff7-d5ea-4b58-b2ee-77429ebdba63-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5w7mq\" (UID: \"5a307ff7-d5ea-4b58-b2ee-77429ebdba63\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5w7mq"
Jul 28 10:44:01.761260 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:01.761230    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a307ff7-d5ea-4b58-b2ee-77429ebdba63-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5w7mq\" (UID: \"5a307ff7-d5ea-4b58-b2ee-77429ebdba63\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5w7mq"
Jul 28 10:44:01.765973 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:01.765952    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfnf8\" (UniqueName: \"kubernetes.io/projected/5a307ff7-d5ea-4b58-b2ee-77429ebdba63-kube-api-access-mfnf8\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5w7mq\" (UID: \"5a307ff7-d5ea-4b58-b2ee-77429ebdba63\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5w7mq"
Jul 28 10:44:01.947138 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:01.947043    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5w7mq"
Jul 28 10:44:02.075118 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:02.075089    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5w7mq"]
Jul 28 10:44:02.077833 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:44:02.077808    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a307ff7_d5ea_4b58_b2ee_77429ebdba63.slice/crio-82b7889f687c0789d6e76f379e40960d8a34314be56d197f7d48ad8199fbffb0 WatchSource:0}: Error finding container 82b7889f687c0789d6e76f379e40960d8a34314be56d197f7d48ad8199fbffb0: Status 404 returned error can't find the container with id 82b7889f687c0789d6e76f379e40960d8a34314be56d197f7d48ad8199fbffb0
Jul 28 10:44:02.444023 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:02.443986    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5w7mq" event={"ID":"5a307ff7-d5ea-4b58-b2ee-77429ebdba63","Type":"ContainerStarted","Data":"279fe622b23dbfaee906b00956f44171e89bd4e155ad0968989d69fdf56263a9"}
Jul 28 10:44:02.444211 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:02.444050    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5w7mq" event={"ID":"5a307ff7-d5ea-4b58-b2ee-77429ebdba63","Type":"ContainerStarted","Data":"82b7889f687c0789d6e76f379e40960d8a34314be56d197f7d48ad8199fbffb0"}
Jul 28 10:44:02.444211 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:02.444091    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5w7mq"
Jul 28 10:44:02.462055 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:02.461996    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5w7mq" podStartSLOduration=1.461976612 podStartE2EDuration="1.461976612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:44:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:44:02.460082727 +0000 UTC m=+2178.418851117" watchObservedRunningTime="2026-07-28 10:44:02.461976612 +0000 UTC m=+2178.420744991"
Jul 28 10:44:33.450276 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:33.450244    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5w7mq"
Jul 28 10:44:34.791071 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:34.791027    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5w7mq"]
Jul 28 10:44:43.450921 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:43.450883    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5w7mq" podUID="5a307ff7-d5ea-4b58-b2ee-77429ebdba63" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 10:44:49.810708 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:49.810657    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5w7mq" podUID="5a307ff7-d5ea-4b58-b2ee-77429ebdba63" containerName="main" containerID="cri-o://279fe622b23dbfaee906b00956f44171e89bd4e155ad0968989d69fdf56263a9" gracePeriod=45
Jul 28 10:44:50.047472 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:50.047447    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5w7mq"
Jul 28 10:44:50.091962 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:50.091878    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a307ff7-d5ea-4b58-b2ee-77429ebdba63-tls-certs\") pod \"5a307ff7-d5ea-4b58-b2ee-77429ebdba63\" (UID: \"5a307ff7-d5ea-4b58-b2ee-77429ebdba63\") "
Jul 28 10:44:50.092136 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:50.091981    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mfnf8\" (UniqueName: \"kubernetes.io/projected/5a307ff7-d5ea-4b58-b2ee-77429ebdba63-kube-api-access-mfnf8\") pod \"5a307ff7-d5ea-4b58-b2ee-77429ebdba63\" (UID: \"5a307ff7-d5ea-4b58-b2ee-77429ebdba63\") "
Jul 28 10:44:50.094196 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:50.094157    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a307ff7-d5ea-4b58-b2ee-77429ebdba63-kube-api-access-mfnf8" (OuterVolumeSpecName: "kube-api-access-mfnf8") pod "5a307ff7-d5ea-4b58-b2ee-77429ebdba63" (UID: "5a307ff7-d5ea-4b58-b2ee-77429ebdba63"). InnerVolumeSpecName "kube-api-access-mfnf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:44:50.094196 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:50.094167    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a307ff7-d5ea-4b58-b2ee-77429ebdba63-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5a307ff7-d5ea-4b58-b2ee-77429ebdba63" (UID: "5a307ff7-d5ea-4b58-b2ee-77429ebdba63"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:44:50.192695 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:50.192659    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfnf8\" (UniqueName: \"kubernetes.io/projected/5a307ff7-d5ea-4b58-b2ee-77429ebdba63-kube-api-access-mfnf8\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:44:50.192695 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:50.192687    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a307ff7-d5ea-4b58-b2ee-77429ebdba63-tls-certs\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:44:50.660784 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:50.660747    2557 generic.go:356] "Generic (PLEG): container finished" podID="5a307ff7-d5ea-4b58-b2ee-77429ebdba63" containerID="279fe622b23dbfaee906b00956f44171e89bd4e155ad0968989d69fdf56263a9" exitCode=0
Jul 28 10:44:50.660982 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:50.660795    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5w7mq"
Jul 28 10:44:50.660982 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:50.660796    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5w7mq" event={"ID":"5a307ff7-d5ea-4b58-b2ee-77429ebdba63","Type":"ContainerDied","Data":"279fe622b23dbfaee906b00956f44171e89bd4e155ad0968989d69fdf56263a9"}
Jul 28 10:44:50.660982 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:50.660853    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5w7mq" event={"ID":"5a307ff7-d5ea-4b58-b2ee-77429ebdba63","Type":"ContainerDied","Data":"82b7889f687c0789d6e76f379e40960d8a34314be56d197f7d48ad8199fbffb0"}
Jul 28 10:44:50.660982 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:50.660881    2557 scope.go:117] "RemoveContainer" containerID="279fe622b23dbfaee906b00956f44171e89bd4e155ad0968989d69fdf56263a9"
Jul 28 10:44:50.670660 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:50.670640    2557 scope.go:117] "RemoveContainer" containerID="279fe622b23dbfaee906b00956f44171e89bd4e155ad0968989d69fdf56263a9"
Jul 28 10:44:50.671021 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:44:50.670993    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"279fe622b23dbfaee906b00956f44171e89bd4e155ad0968989d69fdf56263a9\": container with ID starting with 279fe622b23dbfaee906b00956f44171e89bd4e155ad0968989d69fdf56263a9 not found: ID does not exist" containerID="279fe622b23dbfaee906b00956f44171e89bd4e155ad0968989d69fdf56263a9"
Jul 28 10:44:50.671113 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:50.671031    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"279fe622b23dbfaee906b00956f44171e89bd4e155ad0968989d69fdf56263a9"} err="failed to get container status \"279fe622b23dbfaee906b00956f44171e89bd4e155ad0968989d69fdf56263a9\": rpc error: code = NotFound desc = could not find container \"279fe622b23dbfaee906b00956f44171e89bd4e155ad0968989d69fdf56263a9\": container with ID starting with 279fe622b23dbfaee906b00956f44171e89bd4e155ad0968989d69fdf56263a9 not found: ID does not exist"
Jul 28 10:44:50.678239 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:50.678214    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5w7mq"]
Jul 28 10:44:50.682476 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:50.682452    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5w7mq"]
Jul 28 10:44:52.580179 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:52.580146    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a307ff7-d5ea-4b58-b2ee-77429ebdba63" path="/var/lib/kubelet/pods/5a307ff7-d5ea-4b58-b2ee-77429ebdba63/volumes"
Jul 28 10:44:54.528101 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:54.528069    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gx56p"]
Jul 28 10:44:54.537054 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:44:54.537018    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 10:44:54.537054 ip-10-0-131-88 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:44:54.537054 ip-10-0-131-88 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 28 10:44:54.537054 ip-10-0-131-88 kubenswrapper[2557]:  > containerID="117dedfab5190d3703968a8cf2261e09c22c296508cf7b952d3f64ac0e088deb" cmd=["/bin/sleep","15"]
Jul 28 10:44:54.537395 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:54.537068    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gx56p" message="" err=<
Jul 28 10:44:54.537395 ip-10-0-131-88 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:44:54.537395 ip-10-0-131-88 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 28 10:44:54.537395 ip-10-0-131-88 kubenswrapper[2557]:  >
Jul 28 10:44:54.537395 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:44:54.537080    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 10:44:54.537395 ip-10-0-131-88 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:44:54.537395 ip-10-0-131-88 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 28 10:44:54.537395 ip-10-0-131-88 kubenswrapper[2557]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gx56p" podUID="32f872b9-eb4a-4775-81c0-8a5aeb1f5140" containerName="main" containerID="cri-o://117dedfab5190d3703968a8cf2261e09c22c296508cf7b952d3f64ac0e088deb"
Jul 28 10:44:54.537395 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:54.537115    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gx56p" podUID="32f872b9-eb4a-4775-81c0-8a5aeb1f5140" containerName="main" containerID="cri-o://117dedfab5190d3703968a8cf2261e09c22c296508cf7b952d3f64ac0e088deb" gracePeriod=60
Jul 28 10:44:54.682940 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:54.682909    2557 generic.go:356] "Generic (PLEG): container finished" podID="32f872b9-eb4a-4775-81c0-8a5aeb1f5140" containerID="117dedfab5190d3703968a8cf2261e09c22c296508cf7b952d3f64ac0e088deb" exitCode=0
Jul 28 10:44:54.683129 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:54.682967    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gx56p" event={"ID":"32f872b9-eb4a-4775-81c0-8a5aeb1f5140","Type":"ContainerDied","Data":"117dedfab5190d3703968a8cf2261e09c22c296508cf7b952d3f64ac0e088deb"}
Jul 28 10:44:55.296382 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:55.296355    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gx56p"
Jul 28 10:44:55.340448 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:55.340417    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6bqf\" (UniqueName: \"kubernetes.io/projected/32f872b9-eb4a-4775-81c0-8a5aeb1f5140-kube-api-access-k6bqf\") pod \"32f872b9-eb4a-4775-81c0-8a5aeb1f5140\" (UID: \"32f872b9-eb4a-4775-81c0-8a5aeb1f5140\") "
Jul 28 10:44:55.340621 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:55.340460    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32f872b9-eb4a-4775-81c0-8a5aeb1f5140-model-cache\") pod \"32f872b9-eb4a-4775-81c0-8a5aeb1f5140\" (UID: \"32f872b9-eb4a-4775-81c0-8a5aeb1f5140\") "
Jul 28 10:44:55.340621 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:55.340486    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32f872b9-eb4a-4775-81c0-8a5aeb1f5140-dshm\") pod \"32f872b9-eb4a-4775-81c0-8a5aeb1f5140\" (UID: \"32f872b9-eb4a-4775-81c0-8a5aeb1f5140\") "
Jul 28 10:44:55.340621 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:55.340512    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32f872b9-eb4a-4775-81c0-8a5aeb1f5140-tls-certs\") pod \"32f872b9-eb4a-4775-81c0-8a5aeb1f5140\" (UID: \"32f872b9-eb4a-4775-81c0-8a5aeb1f5140\") "
Jul 28 10:44:55.340621 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:55.340534    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32f872b9-eb4a-4775-81c0-8a5aeb1f5140-tmp-dir\") pod \"32f872b9-eb4a-4775-81c0-8a5aeb1f5140\" (UID: \"32f872b9-eb4a-4775-81c0-8a5aeb1f5140\") "
Jul 28 10:44:55.340621 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:55.340563    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32f872b9-eb4a-4775-81c0-8a5aeb1f5140-home\") pod \"32f872b9-eb4a-4775-81c0-8a5aeb1f5140\" (UID: \"32f872b9-eb4a-4775-81c0-8a5aeb1f5140\") "
Jul 28 10:44:55.340898 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:55.340754    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32f872b9-eb4a-4775-81c0-8a5aeb1f5140-model-cache" (OuterVolumeSpecName: "model-cache") pod "32f872b9-eb4a-4775-81c0-8a5aeb1f5140" (UID: "32f872b9-eb4a-4775-81c0-8a5aeb1f5140"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:44:55.340962 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:55.340889    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32f872b9-eb4a-4775-81c0-8a5aeb1f5140-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "32f872b9-eb4a-4775-81c0-8a5aeb1f5140" (UID: "32f872b9-eb4a-4775-81c0-8a5aeb1f5140"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:44:55.340962 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:55.340900    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32f872b9-eb4a-4775-81c0-8a5aeb1f5140-model-cache\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:44:55.341039 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:55.340964    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32f872b9-eb4a-4775-81c0-8a5aeb1f5140-home" (OuterVolumeSpecName: "home") pod "32f872b9-eb4a-4775-81c0-8a5aeb1f5140" (UID: "32f872b9-eb4a-4775-81c0-8a5aeb1f5140"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:44:55.342670 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:55.342642    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32f872b9-eb4a-4775-81c0-8a5aeb1f5140-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "32f872b9-eb4a-4775-81c0-8a5aeb1f5140" (UID: "32f872b9-eb4a-4775-81c0-8a5aeb1f5140"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:44:55.342790 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:55.342694    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32f872b9-eb4a-4775-81c0-8a5aeb1f5140-kube-api-access-k6bqf" (OuterVolumeSpecName: "kube-api-access-k6bqf") pod "32f872b9-eb4a-4775-81c0-8a5aeb1f5140" (UID: "32f872b9-eb4a-4775-81c0-8a5aeb1f5140"). InnerVolumeSpecName "kube-api-access-k6bqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:44:55.343069 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:55.343045    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32f872b9-eb4a-4775-81c0-8a5aeb1f5140-dshm" (OuterVolumeSpecName: "dshm") pod "32f872b9-eb4a-4775-81c0-8a5aeb1f5140" (UID: "32f872b9-eb4a-4775-81c0-8a5aeb1f5140"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:44:55.442185 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:55.442088    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6bqf\" (UniqueName: \"kubernetes.io/projected/32f872b9-eb4a-4775-81c0-8a5aeb1f5140-kube-api-access-k6bqf\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:44:55.442185 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:55.442131    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32f872b9-eb4a-4775-81c0-8a5aeb1f5140-dshm\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:44:55.442185 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:55.442143    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32f872b9-eb4a-4775-81c0-8a5aeb1f5140-tls-certs\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:44:55.442185 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:55.442153    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32f872b9-eb4a-4775-81c0-8a5aeb1f5140-tmp-dir\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:44:55.442185 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:55.442160    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32f872b9-eb4a-4775-81c0-8a5aeb1f5140-home\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:44:55.688075 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:55.688044    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gx56p"
Jul 28 10:44:55.688075 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:55.688056    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gx56p" event={"ID":"32f872b9-eb4a-4775-81c0-8a5aeb1f5140","Type":"ContainerDied","Data":"a1d893277fd2fc4f4be7021f2a0cf83a74ac0a165061a403a1159eb4b16e9d54"}
Jul 28 10:44:55.688642 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:55.688115    2557 scope.go:117] "RemoveContainer" containerID="117dedfab5190d3703968a8cf2261e09c22c296508cf7b952d3f64ac0e088deb"
Jul 28 10:44:55.709143 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:55.709113    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gx56p"]
Jul 28 10:44:55.711461 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:55.711436    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gx56p"]
Jul 28 10:44:56.579738 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:44:56.579704    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32f872b9-eb4a-4775-81c0-8a5aeb1f5140" path="/var/lib/kubelet/pods/32f872b9-eb4a-4775-81c0-8a5aeb1f5140/volumes"
Jul 28 10:45:32.994928 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:45:32.994889    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bn8m2"]
Jul 28 10:45:32.995339 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:45:32.995202    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a307ff7-d5ea-4b58-b2ee-77429ebdba63" containerName="main"
Jul 28 10:45:32.995339 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:45:32.995213    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a307ff7-d5ea-4b58-b2ee-77429ebdba63" containerName="main"
Jul 28 10:45:32.995339 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:45:32.995228    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32f872b9-eb4a-4775-81c0-8a5aeb1f5140" containerName="main"
Jul 28 10:45:32.995339 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:45:32.995233    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="32f872b9-eb4a-4775-81c0-8a5aeb1f5140" containerName="main"
Jul 28 10:45:32.995339 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:45:32.995292    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a307ff7-d5ea-4b58-b2ee-77429ebdba63" containerName="main"
Jul 28 10:45:32.995339 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:45:32.995301    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="32f872b9-eb4a-4775-81c0-8a5aeb1f5140" containerName="main"
Jul 28 10:45:32.998216 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:45:32.998195    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bn8m2"
Jul 28 10:45:33.000827 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:45:33.000806    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Jul 28 10:45:33.000965 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:45:33.000891    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Jul 28 10:45:33.000965 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:45:33.000946    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-k49t4\""
Jul 28 10:45:33.000965 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:45:33.000954    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Jul 28 10:45:33.009289 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:45:33.009269    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bn8m2"]
Jul 28 10:45:33.168173 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:45:33.168135    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbff8e33-fc6e-467d-bb19-84b40e7ae310-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bn8m2\" (UID: \"dbff8e33-fc6e-467d-bb19-84b40e7ae310\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bn8m2"
Jul 28 10:45:33.168173 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:45:33.168177    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp5pj\" (UniqueName: \"kubernetes.io/projected/dbff8e33-fc6e-467d-bb19-84b40e7ae310-kube-api-access-hp5pj\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bn8m2\" (UID: \"dbff8e33-fc6e-467d-bb19-84b40e7ae310\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bn8m2"
Jul 28 10:45:33.168395 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:45:33.168201    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbff8e33-fc6e-467d-bb19-84b40e7ae310-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bn8m2\" (UID: \"dbff8e33-fc6e-467d-bb19-84b40e7ae310\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bn8m2"
Jul 28 10:45:33.168395 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:45:33.168287    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbff8e33-fc6e-467d-bb19-84b40e7ae310-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bn8m2\" (UID: \"dbff8e33-fc6e-467d-bb19-84b40e7ae310\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bn8m2"
Jul 28 10:45:33.168395 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:45:33.168361    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbff8e33-fc6e-467d-bb19-84b40e7ae310-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bn8m2\" (UID: \"dbff8e33-fc6e-467d-bb19-84b40e7ae310\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bn8m2"
Jul 28 10:45:33.168494 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:45:33.168408    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbff8e33-fc6e-467d-bb19-84b40e7ae310-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bn8m2\" (UID: \"dbff8e33-fc6e-467d-bb19-84b40e7ae310\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bn8m2"
Jul 28 10:45:33.269097 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:45:33.269066    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbff8e33-fc6e-467d-bb19-84b40e7ae310-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bn8m2\" (UID: \"dbff8e33-fc6e-467d-bb19-84b40e7ae310\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bn8m2"
Jul 28 10:45:33.269294 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:45:33.269134    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbff8e33-fc6e-467d-bb19-84b40e7ae310-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bn8m2\" (UID: \"dbff8e33-fc6e-467d-bb19-84b40e7ae310\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bn8m2"
Jul 28 10:45:33.269294 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:45:33.269154    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbff8e33-fc6e-467d-bb19-84b40e7ae310-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bn8m2\" (UID: \"dbff8e33-fc6e-467d-bb19-84b40e7ae310\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bn8m2"
Jul 28 10:45:33.269294 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:45:33.269173    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp5pj\" (UniqueName: \"kubernetes.io/projected/dbff8e33-fc6e-467d-bb19-84b40e7ae310-kube-api-access-hp5pj\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bn8m2\" (UID: \"dbff8e33-fc6e-467d-bb19-84b40e7ae310\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bn8m2"
Jul 28 10:45:33.269294 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:45:33.269196    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbff8e33-fc6e-467d-bb19-84b40e7ae310-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bn8m2\" (UID: \"dbff8e33-fc6e-467d-bb19-84b40e7ae310\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bn8m2"
Jul 28 10:45:33.269294 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:45:33.269216    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbff8e33-fc6e-467d-bb19-84b40e7ae310-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bn8m2\" (UID: \"dbff8e33-fc6e-467d-bb19-84b40e7ae310\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bn8m2"
Jul 28 10:45:33.269584 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:45:33.269490    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbff8e33-fc6e-467d-bb19-84b40e7ae310-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bn8m2\" (UID: \"dbff8e33-fc6e-467d-bb19-84b40e7ae310\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bn8m2"
Jul 28 10:45:33.269584 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:45:33.269561    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbff8e33-fc6e-467d-bb19-84b40e7ae310-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bn8m2\" (UID: \"dbff8e33-fc6e-467d-bb19-84b40e7ae310\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bn8m2"
Jul 28 10:45:33.269699 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:45:33.269616    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbff8e33-fc6e-467d-bb19-84b40e7ae310-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bn8m2\" (UID: \"dbff8e33-fc6e-467d-bb19-84b40e7ae310\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bn8m2"
Jul 28 10:45:33.271489 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:45:33.271472    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbff8e33-fc6e-467d-bb19-84b40e7ae310-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bn8m2\" (UID: \"dbff8e33-fc6e-467d-bb19-84b40e7ae310\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bn8m2"
Jul 28 10:45:33.271798 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:45:33.271777    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbff8e33-fc6e-467d-bb19-84b40e7ae310-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bn8m2\" (UID: \"dbff8e33-fc6e-467d-bb19-84b40e7ae310\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bn8m2"
Jul 28 10:45:33.277933 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:45:33.277914    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp5pj\" (UniqueName: \"kubernetes.io/projected/dbff8e33-fc6e-467d-bb19-84b40e7ae310-kube-api-access-hp5pj\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bn8m2\" (UID: \"dbff8e33-fc6e-467d-bb19-84b40e7ae310\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bn8m2"
Jul 28 10:45:33.308346 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:45:33.308308    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bn8m2"
Jul 28 10:45:33.443671 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:45:33.443640    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bn8m2"]
Jul 28 10:45:33.445935 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:45:33.445902    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbff8e33_fc6e_467d_bb19_84b40e7ae310.slice/crio-e93bb23bbcd37373830c7c85a4427fa34f065b2ede6a81eee2a06cb1e8e9a040 WatchSource:0}: Error finding container e93bb23bbcd37373830c7c85a4427fa34f065b2ede6a81eee2a06cb1e8e9a040: Status 404 returned error can't find the container with id e93bb23bbcd37373830c7c85a4427fa34f065b2ede6a81eee2a06cb1e8e9a040
Jul 28 10:45:33.855156 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:45:33.855112    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bn8m2" event={"ID":"dbff8e33-fc6e-467d-bb19-84b40e7ae310","Type":"ContainerStarted","Data":"563775052ec4379e86e304fcf4383ac495cfc8a24a756f4a1b760349be5e282f"}
Jul 28 10:45:33.855348 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:45:33.855171    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bn8m2" event={"ID":"dbff8e33-fc6e-467d-bb19-84b40e7ae310","Type":"ContainerStarted","Data":"e93bb23bbcd37373830c7c85a4427fa34f065b2ede6a81eee2a06cb1e8e9a040"}
Jul 28 10:45:33.874970 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:45:33.874898    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bn8m2" podStartSLOduration=1.874876569 podStartE2EDuration="1.874876569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:45:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:45:33.874017656 +0000 UTC m=+2269.832786038" watchObservedRunningTime="2026-07-28 10:45:33.874876569 +0000 UTC m=+2269.833644950"
Jul 28 10:45:34.309238 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:45:34.309191    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bn8m2"
Jul 28 10:45:43.308934 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:45:43.308895    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bn8m2"
Jul 28 10:45:43.321332 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:45:43.321303    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bn8m2"
Jul 28 10:45:43.910273 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:45:43.910237    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bn8m2"
Jul 28 10:46:10.935287 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:46:10.935252    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetr5j9"]
Jul 28 10:46:10.939925 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:46:10.939905    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetr5j9"
Jul 28 10:46:10.942305 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:46:10.942287    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Jul 28 10:46:10.942404 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:46:10.942285    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa-dockercfg-vvgxs\""
Jul 28 10:46:10.951837 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:46:10.951251    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetr5j9"]
Jul 28 10:46:11.103288 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:46:11.103242    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m85jk\" (UniqueName: \"kubernetes.io/projected/0e45a5f8-6dcf-42c3-a462-e2cc2517fda9-kube-api-access-m85jk\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetr5j9\" (UID: \"0e45a5f8-6dcf-42c3-a462-e2cc2517fda9\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetr5j9"
Jul 28 10:46:11.103473 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:46:11.103375    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e45a5f8-6dcf-42c3-a462-e2cc2517fda9-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetr5j9\" (UID: \"0e45a5f8-6dcf-42c3-a462-e2cc2517fda9\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetr5j9"
Jul 28 10:46:11.204596 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:46:11.204498    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m85jk\" (UniqueName: \"kubernetes.io/projected/0e45a5f8-6dcf-42c3-a462-e2cc2517fda9-kube-api-access-m85jk\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetr5j9\" (UID: \"0e45a5f8-6dcf-42c3-a462-e2cc2517fda9\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetr5j9"
Jul 28 10:46:11.204802 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:46:11.204621    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e45a5f8-6dcf-42c3-a462-e2cc2517fda9-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetr5j9\" (UID: \"0e45a5f8-6dcf-42c3-a462-e2cc2517fda9\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetr5j9"
Jul 28 10:46:11.207157 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:46:11.207124    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e45a5f8-6dcf-42c3-a462-e2cc2517fda9-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetr5j9\" (UID: \"0e45a5f8-6dcf-42c3-a462-e2cc2517fda9\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetr5j9"
Jul 28 10:46:11.211685 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:46:11.211667    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m85jk\" (UniqueName: \"kubernetes.io/projected/0e45a5f8-6dcf-42c3-a462-e2cc2517fda9-kube-api-access-m85jk\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetr5j9\" (UID: \"0e45a5f8-6dcf-42c3-a462-e2cc2517fda9\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetr5j9"
Jul 28 10:46:11.251330 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:46:11.251294    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetr5j9"
Jul 28 10:46:11.385788 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:46:11.385750    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetr5j9"]
Jul 28 10:46:11.389112 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:46:11.389087    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e45a5f8_6dcf_42c3_a462_e2cc2517fda9.slice/crio-34e33b2a4a734ea52083f215f695f362d7a2f95ba3a4673b8eae8cbe74e5c7da WatchSource:0}: Error finding container 34e33b2a4a734ea52083f215f695f362d7a2f95ba3a4673b8eae8cbe74e5c7da: Status 404 returned error can't find the container with id 34e33b2a4a734ea52083f215f695f362d7a2f95ba3a4673b8eae8cbe74e5c7da
Jul 28 10:46:12.018468 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:46:12.018435    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetr5j9" event={"ID":"0e45a5f8-6dcf-42c3-a462-e2cc2517fda9","Type":"ContainerStarted","Data":"d220fdd8d00cf22612bb77250640f37185c696ea65064d090cc1cdaa21a604da"}
Jul 28 10:46:12.018870 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:46:12.018485    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetr5j9" event={"ID":"0e45a5f8-6dcf-42c3-a462-e2cc2517fda9","Type":"ContainerStarted","Data":"34e33b2a4a734ea52083f215f695f362d7a2f95ba3a4673b8eae8cbe74e5c7da"}
Jul 28 10:46:12.018870 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:46:12.018555    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetr5j9"
Jul 28 10:46:12.038521 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:46:12.038469    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetr5j9" podStartSLOduration=2.038453009 podStartE2EDuration="2.038453009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:46:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:46:12.037292461 +0000 UTC m=+2307.996060845" watchObservedRunningTime="2026-07-28 10:46:12.038453009 +0000 UTC m=+2307.997221386"
Jul 28 10:46:43.024928 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:46:43.024854    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetr5j9"
Jul 28 10:46:44.387175 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:46:44.387143    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetr5j9"]
Jul 28 10:46:53.025111 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:46:53.025070    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetr5j9" podUID="0e45a5f8-6dcf-42c3-a462-e2cc2517fda9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 10:46:59.407472 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:46:59.407435    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetr5j9" podUID="0e45a5f8-6dcf-42c3-a462-e2cc2517fda9" containerName="main" containerID="cri-o://d220fdd8d00cf22612bb77250640f37185c696ea65064d090cc1cdaa21a604da" gracePeriod=45
Jul 28 10:46:59.435127 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:46:59.435097    2557 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e45a5f8_6dcf_42c3_a462_e2cc2517fda9.slice/crio-d220fdd8d00cf22612bb77250640f37185c696ea65064d090cc1cdaa21a604da.scope\": RecentStats: unable to find data in memory cache]"
Jul 28 10:46:59.660417 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:46:59.660357    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetr5j9"
Jul 28 10:46:59.839768 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:46:59.839733    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m85jk\" (UniqueName: \"kubernetes.io/projected/0e45a5f8-6dcf-42c3-a462-e2cc2517fda9-kube-api-access-m85jk\") pod \"0e45a5f8-6dcf-42c3-a462-e2cc2517fda9\" (UID: \"0e45a5f8-6dcf-42c3-a462-e2cc2517fda9\") "
Jul 28 10:46:59.839961 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:46:59.839776    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e45a5f8-6dcf-42c3-a462-e2cc2517fda9-tls-certs\") pod \"0e45a5f8-6dcf-42c3-a462-e2cc2517fda9\" (UID: \"0e45a5f8-6dcf-42c3-a462-e2cc2517fda9\") "
Jul 28 10:46:59.841958 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:46:59.841922    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e45a5f8-6dcf-42c3-a462-e2cc2517fda9-kube-api-access-m85jk" (OuterVolumeSpecName: "kube-api-access-m85jk") pod "0e45a5f8-6dcf-42c3-a462-e2cc2517fda9" (UID: "0e45a5f8-6dcf-42c3-a462-e2cc2517fda9"). InnerVolumeSpecName "kube-api-access-m85jk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:46:59.841958 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:46:59.841937    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e45a5f8-6dcf-42c3-a462-e2cc2517fda9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0e45a5f8-6dcf-42c3-a462-e2cc2517fda9" (UID: "0e45a5f8-6dcf-42c3-a462-e2cc2517fda9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:46:59.941369 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:46:59.941284    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m85jk\" (UniqueName: \"kubernetes.io/projected/0e45a5f8-6dcf-42c3-a462-e2cc2517fda9-kube-api-access-m85jk\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:46:59.941369 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:46:59.941315    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e45a5f8-6dcf-42c3-a462-e2cc2517fda9-tls-certs\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:47:00.233555 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:00.233466    2557 generic.go:356] "Generic (PLEG): container finished" podID="0e45a5f8-6dcf-42c3-a462-e2cc2517fda9" containerID="d220fdd8d00cf22612bb77250640f37185c696ea65064d090cc1cdaa21a604da" exitCode=0
Jul 28 10:47:00.233555 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:00.233511    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetr5j9"
Jul 28 10:47:00.233798 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:00.233550    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetr5j9" event={"ID":"0e45a5f8-6dcf-42c3-a462-e2cc2517fda9","Type":"ContainerDied","Data":"d220fdd8d00cf22612bb77250640f37185c696ea65064d090cc1cdaa21a604da"}
Jul 28 10:47:00.233798 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:00.233625    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetr5j9" event={"ID":"0e45a5f8-6dcf-42c3-a462-e2cc2517fda9","Type":"ContainerDied","Data":"34e33b2a4a734ea52083f215f695f362d7a2f95ba3a4673b8eae8cbe74e5c7da"}
Jul 28 10:47:00.233798 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:00.233650    2557 scope.go:117] "RemoveContainer" containerID="d220fdd8d00cf22612bb77250640f37185c696ea65064d090cc1cdaa21a604da"
Jul 28 10:47:00.242199 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:00.242184    2557 scope.go:117] "RemoveContainer" containerID="d220fdd8d00cf22612bb77250640f37185c696ea65064d090cc1cdaa21a604da"
Jul 28 10:47:00.242467 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:47:00.242445    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d220fdd8d00cf22612bb77250640f37185c696ea65064d090cc1cdaa21a604da\": container with ID starting with d220fdd8d00cf22612bb77250640f37185c696ea65064d090cc1cdaa21a604da not found: ID does not exist" containerID="d220fdd8d00cf22612bb77250640f37185c696ea65064d090cc1cdaa21a604da"
Jul 28 10:47:00.242517 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:00.242478    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d220fdd8d00cf22612bb77250640f37185c696ea65064d090cc1cdaa21a604da"} err="failed to get container status \"d220fdd8d00cf22612bb77250640f37185c696ea65064d090cc1cdaa21a604da\": rpc error: code = NotFound desc = could not find container \"d220fdd8d00cf22612bb77250640f37185c696ea65064d090cc1cdaa21a604da\": container with ID starting with d220fdd8d00cf22612bb77250640f37185c696ea65064d090cc1cdaa21a604da not found: ID does not exist"
Jul 28 10:47:00.253705 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:00.253677    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetr5j9"]
Jul 28 10:47:00.258481 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:00.258459    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetr5j9"]
Jul 28 10:47:00.580590 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:00.580555    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e45a5f8-6dcf-42c3-a462-e2cc2517fda9" path="/var/lib/kubelet/pods/0e45a5f8-6dcf-42c3-a462-e2cc2517fda9/volumes"
Jul 28 10:47:04.541719 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:04.541681    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bn8m2"]
Jul 28 10:47:04.551105 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:47:04.551064    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 10:47:04.551105 ip-10-0-131-88 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:47:04.551105 ip-10-0-131-88 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 28 10:47:04.551105 ip-10-0-131-88 kubenswrapper[2557]:  > containerID="563775052ec4379e86e304fcf4383ac495cfc8a24a756f4a1b760349be5e282f" cmd=["/bin/sleep","15"]
Jul 28 10:47:04.551441 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:04.551117    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bn8m2" message="" err=<
Jul 28 10:47:04.551441 ip-10-0-131-88 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:47:04.551441 ip-10-0-131-88 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 28 10:47:04.551441 ip-10-0-131-88 kubenswrapper[2557]:  >
Jul 28 10:47:04.551441 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:47:04.551131    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 10:47:04.551441 ip-10-0-131-88 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:47:04.551441 ip-10-0-131-88 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 28 10:47:04.551441 ip-10-0-131-88 kubenswrapper[2557]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bn8m2" podUID="dbff8e33-fc6e-467d-bb19-84b40e7ae310" containerName="main" containerID="cri-o://563775052ec4379e86e304fcf4383ac495cfc8a24a756f4a1b760349be5e282f"
Jul 28 10:47:04.551441 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:04.551164    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bn8m2" podUID="dbff8e33-fc6e-467d-bb19-84b40e7ae310" containerName="main" containerID="cri-o://563775052ec4379e86e304fcf4383ac495cfc8a24a756f4a1b760349be5e282f" gracePeriod=60
Jul 28 10:47:04.908885 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:04.908857    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bn8m2"
Jul 28 10:47:05.087561 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:05.087463    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbff8e33-fc6e-467d-bb19-84b40e7ae310-model-cache\") pod \"dbff8e33-fc6e-467d-bb19-84b40e7ae310\" (UID: \"dbff8e33-fc6e-467d-bb19-84b40e7ae310\") "
Jul 28 10:47:05.087561 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:05.087516    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbff8e33-fc6e-467d-bb19-84b40e7ae310-tmp-dir\") pod \"dbff8e33-fc6e-467d-bb19-84b40e7ae310\" (UID: \"dbff8e33-fc6e-467d-bb19-84b40e7ae310\") "
Jul 28 10:47:05.087561 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:05.087540    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbff8e33-fc6e-467d-bb19-84b40e7ae310-dshm\") pod \"dbff8e33-fc6e-467d-bb19-84b40e7ae310\" (UID: \"dbff8e33-fc6e-467d-bb19-84b40e7ae310\") "
Jul 28 10:47:05.087883 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:05.087566    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbff8e33-fc6e-467d-bb19-84b40e7ae310-tls-certs\") pod \"dbff8e33-fc6e-467d-bb19-84b40e7ae310\" (UID: \"dbff8e33-fc6e-467d-bb19-84b40e7ae310\") "
Jul 28 10:47:05.087883 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:05.087636    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hp5pj\" (UniqueName: \"kubernetes.io/projected/dbff8e33-fc6e-467d-bb19-84b40e7ae310-kube-api-access-hp5pj\") pod \"dbff8e33-fc6e-467d-bb19-84b40e7ae310\" (UID: \"dbff8e33-fc6e-467d-bb19-84b40e7ae310\") "
Jul 28 10:47:05.087883 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:05.087699    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbff8e33-fc6e-467d-bb19-84b40e7ae310-home\") pod \"dbff8e33-fc6e-467d-bb19-84b40e7ae310\" (UID: \"dbff8e33-fc6e-467d-bb19-84b40e7ae310\") "
Jul 28 10:47:05.087883 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:05.087841    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbff8e33-fc6e-467d-bb19-84b40e7ae310-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dbff8e33-fc6e-467d-bb19-84b40e7ae310" (UID: "dbff8e33-fc6e-467d-bb19-84b40e7ae310"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:47:05.087883 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:05.087860    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbff8e33-fc6e-467d-bb19-84b40e7ae310-model-cache" (OuterVolumeSpecName: "model-cache") pod "dbff8e33-fc6e-467d-bb19-84b40e7ae310" (UID: "dbff8e33-fc6e-467d-bb19-84b40e7ae310"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:47:05.088135 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:05.088007    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbff8e33-fc6e-467d-bb19-84b40e7ae310-home" (OuterVolumeSpecName: "home") pod "dbff8e33-fc6e-467d-bb19-84b40e7ae310" (UID: "dbff8e33-fc6e-467d-bb19-84b40e7ae310"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:47:05.088135 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:05.088079    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbff8e33-fc6e-467d-bb19-84b40e7ae310-tmp-dir\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:47:05.088135 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:05.088101    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbff8e33-fc6e-467d-bb19-84b40e7ae310-home\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:47:05.088135 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:05.088114    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbff8e33-fc6e-467d-bb19-84b40e7ae310-model-cache\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:47:05.089788 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:05.089762    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbff8e33-fc6e-467d-bb19-84b40e7ae310-dshm" (OuterVolumeSpecName: "dshm") pod "dbff8e33-fc6e-467d-bb19-84b40e7ae310" (UID: "dbff8e33-fc6e-467d-bb19-84b40e7ae310"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:47:05.089886 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:05.089821    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbff8e33-fc6e-467d-bb19-84b40e7ae310-kube-api-access-hp5pj" (OuterVolumeSpecName: "kube-api-access-hp5pj") pod "dbff8e33-fc6e-467d-bb19-84b40e7ae310" (UID: "dbff8e33-fc6e-467d-bb19-84b40e7ae310"). InnerVolumeSpecName "kube-api-access-hp5pj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:47:05.089886 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:05.089833    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbff8e33-fc6e-467d-bb19-84b40e7ae310-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dbff8e33-fc6e-467d-bb19-84b40e7ae310" (UID: "dbff8e33-fc6e-467d-bb19-84b40e7ae310"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:47:05.189442 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:05.189408    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbff8e33-fc6e-467d-bb19-84b40e7ae310-dshm\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:47:05.189442 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:05.189436    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbff8e33-fc6e-467d-bb19-84b40e7ae310-tls-certs\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:47:05.189442 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:05.189447    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hp5pj\" (UniqueName: \"kubernetes.io/projected/dbff8e33-fc6e-467d-bb19-84b40e7ae310-kube-api-access-hp5pj\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:47:05.257698 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:05.257644    2557 generic.go:356] "Generic (PLEG): container finished" podID="dbff8e33-fc6e-467d-bb19-84b40e7ae310" containerID="563775052ec4379e86e304fcf4383ac495cfc8a24a756f4a1b760349be5e282f" exitCode=0
Jul 28 10:47:05.257698 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:05.257692    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bn8m2"
Jul 28 10:47:05.257902 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:05.257733    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bn8m2" event={"ID":"dbff8e33-fc6e-467d-bb19-84b40e7ae310","Type":"ContainerDied","Data":"563775052ec4379e86e304fcf4383ac495cfc8a24a756f4a1b760349be5e282f"}
Jul 28 10:47:05.257902 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:05.257789    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bn8m2" event={"ID":"dbff8e33-fc6e-467d-bb19-84b40e7ae310","Type":"ContainerDied","Data":"e93bb23bbcd37373830c7c85a4427fa34f065b2ede6a81eee2a06cb1e8e9a040"}
Jul 28 10:47:05.257902 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:05.257813    2557 scope.go:117] "RemoveContainer" containerID="563775052ec4379e86e304fcf4383ac495cfc8a24a756f4a1b760349be5e282f"
Jul 28 10:47:05.267050 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:05.267029    2557 scope.go:117] "RemoveContainer" containerID="563775052ec4379e86e304fcf4383ac495cfc8a24a756f4a1b760349be5e282f"
Jul 28 10:47:05.267352 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:47:05.267330    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"563775052ec4379e86e304fcf4383ac495cfc8a24a756f4a1b760349be5e282f\": container with ID starting with 563775052ec4379e86e304fcf4383ac495cfc8a24a756f4a1b760349be5e282f not found: ID does not exist" containerID="563775052ec4379e86e304fcf4383ac495cfc8a24a756f4a1b760349be5e282f"
Jul 28 10:47:05.267434 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:05.267365    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"563775052ec4379e86e304fcf4383ac495cfc8a24a756f4a1b760349be5e282f"} err="failed to get container status \"563775052ec4379e86e304fcf4383ac495cfc8a24a756f4a1b760349be5e282f\": rpc error: code = NotFound desc = could not find container \"563775052ec4379e86e304fcf4383ac495cfc8a24a756f4a1b760349be5e282f\": container with ID starting with 563775052ec4379e86e304fcf4383ac495cfc8a24a756f4a1b760349be5e282f not found: ID does not exist"
Jul 28 10:47:05.280035 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:05.280005    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bn8m2"]
Jul 28 10:47:05.284311 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:05.284288    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bn8m2"]
Jul 28 10:47:05.900447 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:05.900399    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bn8m2" podUID="dbff8e33-fc6e-467d-bb19-84b40e7ae310" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 28 10:47:06.579964 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:06.579933    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbff8e33-fc6e-467d-bb19-84b40e7ae310" path="/var/lib/kubelet/pods/dbff8e33-fc6e-467d-bb19-84b40e7ae310/volumes"
Jul 28 10:47:28.802928 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:28.802892    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk"]
Jul 28 10:47:28.803323 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:28.803213    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e45a5f8-6dcf-42c3-a462-e2cc2517fda9" containerName="main"
Jul 28 10:47:28.803323 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:28.803226    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e45a5f8-6dcf-42c3-a462-e2cc2517fda9" containerName="main"
Jul 28 10:47:28.803323 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:28.803237    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbff8e33-fc6e-467d-bb19-84b40e7ae310" containerName="main"
Jul 28 10:47:28.803323 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:28.803242    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbff8e33-fc6e-467d-bb19-84b40e7ae310" containerName="main"
Jul 28 10:47:28.803323 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:28.803299    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e45a5f8-6dcf-42c3-a462-e2cc2517fda9" containerName="main"
Jul 28 10:47:28.803323 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:28.803307    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbff8e33-fc6e-467d-bb19-84b40e7ae310" containerName="main"
Jul 28 10:47:28.806338 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:28.806309    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk"
Jul 28 10:47:28.808575 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:28.808555    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"seaweedfs-s3-creds\""
Jul 28 10:47:28.809167 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:28.809149    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 10:47:28.809273 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:28.809254    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-jws88\""
Jul 28 10:47:28.809348 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:28.809272    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Jul 28 10:47:28.809425 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:28.809409    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Jul 28 10:47:28.809485 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:28.809445    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Jul 28 10:47:28.818014 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:28.817992    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk"]
Jul 28 10:47:28.894119 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:28.894082    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/622793e7-f6fd-4d0d-8a03-59450e296aff-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk\" (UID: \"622793e7-f6fd-4d0d-8a03-59450e296aff\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk"
Jul 28 10:47:28.894317 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:28.894146    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/622793e7-f6fd-4d0d-8a03-59450e296aff-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk\" (UID: \"622793e7-f6fd-4d0d-8a03-59450e296aff\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk"
Jul 28 10:47:28.894317 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:28.894215    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/622793e7-f6fd-4d0d-8a03-59450e296aff-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk\" (UID: \"622793e7-f6fd-4d0d-8a03-59450e296aff\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk"
Jul 28 10:47:28.894317 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:28.894265    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jc46\" (UniqueName: \"kubernetes.io/projected/622793e7-f6fd-4d0d-8a03-59450e296aff-kube-api-access-2jc46\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk\" (UID: \"622793e7-f6fd-4d0d-8a03-59450e296aff\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk"
Jul 28 10:47:28.894317 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:28.894290    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/622793e7-f6fd-4d0d-8a03-59450e296aff-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk\" (UID: \"622793e7-f6fd-4d0d-8a03-59450e296aff\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk"
Jul 28 10:47:28.894471 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:28.894335    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/622793e7-f6fd-4d0d-8a03-59450e296aff-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk\" (UID: \"622793e7-f6fd-4d0d-8a03-59450e296aff\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk"
Jul 28 10:47:28.894471 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:28.894393    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/622793e7-f6fd-4d0d-8a03-59450e296aff-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk\" (UID: \"622793e7-f6fd-4d0d-8a03-59450e296aff\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk"
Jul 28 10:47:28.894471 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:28.894441    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/622793e7-f6fd-4d0d-8a03-59450e296aff-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk\" (UID: \"622793e7-f6fd-4d0d-8a03-59450e296aff\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk"
Jul 28 10:47:28.995149 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:28.995108    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/622793e7-f6fd-4d0d-8a03-59450e296aff-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk\" (UID: \"622793e7-f6fd-4d0d-8a03-59450e296aff\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk"
Jul 28 10:47:28.995330 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:28.995164    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/622793e7-f6fd-4d0d-8a03-59450e296aff-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk\" (UID: \"622793e7-f6fd-4d0d-8a03-59450e296aff\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk"
Jul 28 10:47:28.995330 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:28.995283    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/622793e7-f6fd-4d0d-8a03-59450e296aff-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk\" (UID: \"622793e7-f6fd-4d0d-8a03-59450e296aff\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk"
Jul 28 10:47:28.995458 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:28.995329    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jc46\" (UniqueName: \"kubernetes.io/projected/622793e7-f6fd-4d0d-8a03-59450e296aff-kube-api-access-2jc46\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk\" (UID: \"622793e7-f6fd-4d0d-8a03-59450e296aff\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk"
Jul 28 10:47:28.995458 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:28.995447    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/622793e7-f6fd-4d0d-8a03-59450e296aff-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk\" (UID: \"622793e7-f6fd-4d0d-8a03-59450e296aff\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk"
Jul 28 10:47:28.995578 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:28.995502    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/622793e7-f6fd-4d0d-8a03-59450e296aff-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk\" (UID: \"622793e7-f6fd-4d0d-8a03-59450e296aff\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk"
Jul 28 10:47:28.995578 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:28.995506    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/622793e7-f6fd-4d0d-8a03-59450e296aff-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk\" (UID: \"622793e7-f6fd-4d0d-8a03-59450e296aff\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk"
Jul 28 10:47:28.995578 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:28.995561    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/622793e7-f6fd-4d0d-8a03-59450e296aff-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk\" (UID: \"622793e7-f6fd-4d0d-8a03-59450e296aff\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk"
Jul 28 10:47:28.995765 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:28.995569    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/622793e7-f6fd-4d0d-8a03-59450e296aff-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk\" (UID: \"622793e7-f6fd-4d0d-8a03-59450e296aff\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk"
Jul 28 10:47:28.995765 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:28.995670    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/622793e7-f6fd-4d0d-8a03-59450e296aff-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk\" (UID: \"622793e7-f6fd-4d0d-8a03-59450e296aff\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk"
Jul 28 10:47:28.995765 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:28.995672    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/622793e7-f6fd-4d0d-8a03-59450e296aff-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk\" (UID: \"622793e7-f6fd-4d0d-8a03-59450e296aff\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk"
Jul 28 10:47:28.996030 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:28.996006    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/622793e7-f6fd-4d0d-8a03-59450e296aff-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk\" (UID: \"622793e7-f6fd-4d0d-8a03-59450e296aff\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk"
Jul 28 10:47:28.996263 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:28.996245    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/622793e7-f6fd-4d0d-8a03-59450e296aff-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk\" (UID: \"622793e7-f6fd-4d0d-8a03-59450e296aff\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk"
Jul 28 10:47:28.997749 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:28.997731    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/622793e7-f6fd-4d0d-8a03-59450e296aff-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk\" (UID: \"622793e7-f6fd-4d0d-8a03-59450e296aff\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk"
Jul 28 10:47:28.997933 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:28.997915    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/622793e7-f6fd-4d0d-8a03-59450e296aff-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk\" (UID: \"622793e7-f6fd-4d0d-8a03-59450e296aff\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk"
Jul 28 10:47:29.005201 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:29.005175    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jc46\" (UniqueName: \"kubernetes.io/projected/622793e7-f6fd-4d0d-8a03-59450e296aff-kube-api-access-2jc46\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk\" (UID: \"622793e7-f6fd-4d0d-8a03-59450e296aff\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk"
Jul 28 10:47:29.117012 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:29.116920    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk"
Jul 28 10:47:29.257816 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:29.257649    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk"]
Jul 28 10:47:29.259921 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:47:29.259897    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod622793e7_f6fd_4d0d_8a03_59450e296aff.slice/crio-3e8ceb4b6df887d4447ce3dcb240c0e317cca9d31c7f3c2e2a6a5e0ef1582c36 WatchSource:0}: Error finding container 3e8ceb4b6df887d4447ce3dcb240c0e317cca9d31c7f3c2e2a6a5e0ef1582c36: Status 404 returned error can't find the container with id 3e8ceb4b6df887d4447ce3dcb240c0e317cca9d31c7f3c2e2a6a5e0ef1582c36
Jul 28 10:47:29.363674 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:29.363628    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk" event={"ID":"622793e7-f6fd-4d0d-8a03-59450e296aff","Type":"ContainerStarted","Data":"f081c549bbb06de6afc840296792b83a4f1caf0e74847c2a89673309f3661bbe"}
Jul 28 10:47:29.363850 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:29.363688    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk" event={"ID":"622793e7-f6fd-4d0d-8a03-59450e296aff","Type":"ContainerStarted","Data":"3e8ceb4b6df887d4447ce3dcb240c0e317cca9d31c7f3c2e2a6a5e0ef1582c36"}
Jul 28 10:47:34.393561 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:34.393529    2557 generic.go:356] "Generic (PLEG): container finished" podID="622793e7-f6fd-4d0d-8a03-59450e296aff" containerID="f081c549bbb06de6afc840296792b83a4f1caf0e74847c2a89673309f3661bbe" exitCode=0
Jul 28 10:47:34.393992 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:34.393576    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk" event={"ID":"622793e7-f6fd-4d0d-8a03-59450e296aff","Type":"ContainerDied","Data":"f081c549bbb06de6afc840296792b83a4f1caf0e74847c2a89673309f3661bbe"}
Jul 28 10:47:35.399980 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:35.399946    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk" event={"ID":"622793e7-f6fd-4d0d-8a03-59450e296aff","Type":"ContainerStarted","Data":"7427c708f37fc73acca4460813774330729ab727f1b71e760b685a7041bd122e"}
Jul 28 10:47:35.421550 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:35.421487    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk" podStartSLOduration=2.381260094 podStartE2EDuration="7.421464928s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.040204834s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:47:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:47:35.420980195 +0000 UTC m=+2391.379748599" watchObservedRunningTime="2026-07-28 10:47:35.421464928 +0000 UTC m=+2391.380233308"
Jul 28 10:47:36.118043 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:36.118004    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk"
Jul 28 10:47:39.117926 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:39.117890    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk"
Jul 28 10:47:39.119482 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:39.119451    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk" podUID="622793e7-f6fd-4d0d-8a03-59450e296aff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Jul 28 10:47:49.118258 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:49.118205    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk" podUID="622793e7-f6fd-4d0d-8a03-59450e296aff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Jul 28 10:47:59.118288 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:47:59.118245    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk" podUID="622793e7-f6fd-4d0d-8a03-59450e296aff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Jul 28 10:48:09.118420 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:48:09.118333    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk" podUID="622793e7-f6fd-4d0d-8a03-59450e296aff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Jul 28 10:48:19.117946 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:48:19.117895    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk" podUID="622793e7-f6fd-4d0d-8a03-59450e296aff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Jul 28 10:48:29.117672 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:48:29.117626    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk" podUID="622793e7-f6fd-4d0d-8a03-59450e296aff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Jul 28 10:48:39.117670 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:48:39.117620    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk" podUID="622793e7-f6fd-4d0d-8a03-59450e296aff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Jul 28 10:48:49.118210 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:48:49.118156    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk" podUID="622793e7-f6fd-4d0d-8a03-59450e296aff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Jul 28 10:48:59.127653 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:48:59.127591    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk"
Jul 28 10:48:59.135327 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:48:59.135297    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk"
Jul 28 10:49:24.457084 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:24.457042    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk"]
Jul 28 10:49:39.479937 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:39.479844    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk" podUID="622793e7-f6fd-4d0d-8a03-59450e296aff" containerName="main" containerID="cri-o://7427c708f37fc73acca4460813774330729ab727f1b71e760b685a7041bd122e" gracePeriod=45
Jul 28 10:49:40.132129 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:40.132102    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk"
Jul 28 10:49:40.203391 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:40.203301    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/622793e7-f6fd-4d0d-8a03-59450e296aff-dshm\") pod \"622793e7-f6fd-4d0d-8a03-59450e296aff\" (UID: \"622793e7-f6fd-4d0d-8a03-59450e296aff\") "
Jul 28 10:49:40.203391 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:40.203344    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/622793e7-f6fd-4d0d-8a03-59450e296aff-cabundle-cert\") pod \"622793e7-f6fd-4d0d-8a03-59450e296aff\" (UID: \"622793e7-f6fd-4d0d-8a03-59450e296aff\") "
Jul 28 10:49:40.203391 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:40.203387    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/622793e7-f6fd-4d0d-8a03-59450e296aff-model-cache\") pod \"622793e7-f6fd-4d0d-8a03-59450e296aff\" (UID: \"622793e7-f6fd-4d0d-8a03-59450e296aff\") "
Jul 28 10:49:40.203665 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:40.203413    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/622793e7-f6fd-4d0d-8a03-59450e296aff-tmp-dir\") pod \"622793e7-f6fd-4d0d-8a03-59450e296aff\" (UID: \"622793e7-f6fd-4d0d-8a03-59450e296aff\") "
Jul 28 10:49:40.203665 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:40.203479    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2jc46\" (UniqueName: \"kubernetes.io/projected/622793e7-f6fd-4d0d-8a03-59450e296aff-kube-api-access-2jc46\") pod \"622793e7-f6fd-4d0d-8a03-59450e296aff\" (UID: \"622793e7-f6fd-4d0d-8a03-59450e296aff\") "
Jul 28 10:49:40.203665 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:40.203559    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/622793e7-f6fd-4d0d-8a03-59450e296aff-kserve-provision-location\") pod \"622793e7-f6fd-4d0d-8a03-59450e296aff\" (UID: \"622793e7-f6fd-4d0d-8a03-59450e296aff\") "
Jul 28 10:49:40.203665 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:40.203589    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/622793e7-f6fd-4d0d-8a03-59450e296aff-home\") pod \"622793e7-f6fd-4d0d-8a03-59450e296aff\" (UID: \"622793e7-f6fd-4d0d-8a03-59450e296aff\") "
Jul 28 10:49:40.203885 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:40.203674    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/622793e7-f6fd-4d0d-8a03-59450e296aff-tls-certs\") pod \"622793e7-f6fd-4d0d-8a03-59450e296aff\" (UID: \"622793e7-f6fd-4d0d-8a03-59450e296aff\") "
Jul 28 10:49:40.203885 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:40.203737    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/622793e7-f6fd-4d0d-8a03-59450e296aff-model-cache" (OuterVolumeSpecName: "model-cache") pod "622793e7-f6fd-4d0d-8a03-59450e296aff" (UID: "622793e7-f6fd-4d0d-8a03-59450e296aff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:49:40.203885 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:40.203874    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/622793e7-f6fd-4d0d-8a03-59450e296aff-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "622793e7-f6fd-4d0d-8a03-59450e296aff" (UID: "622793e7-f6fd-4d0d-8a03-59450e296aff"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:49:40.204053 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:40.204002    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/622793e7-f6fd-4d0d-8a03-59450e296aff-cabundle-cert\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:49:40.204053 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:40.204019    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/622793e7-f6fd-4d0d-8a03-59450e296aff-model-cache\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:49:40.204053 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:40.204044    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/622793e7-f6fd-4d0d-8a03-59450e296aff-home" (OuterVolumeSpecName: "home") pod "622793e7-f6fd-4d0d-8a03-59450e296aff" (UID: "622793e7-f6fd-4d0d-8a03-59450e296aff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:49:40.206144 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:40.206098    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/622793e7-f6fd-4d0d-8a03-59450e296aff-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "622793e7-f6fd-4d0d-8a03-59450e296aff" (UID: "622793e7-f6fd-4d0d-8a03-59450e296aff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:49:40.206276 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:40.206175    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/622793e7-f6fd-4d0d-8a03-59450e296aff-dshm" (OuterVolumeSpecName: "dshm") pod "622793e7-f6fd-4d0d-8a03-59450e296aff" (UID: "622793e7-f6fd-4d0d-8a03-59450e296aff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:49:40.206276 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:40.206190    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/622793e7-f6fd-4d0d-8a03-59450e296aff-kube-api-access-2jc46" (OuterVolumeSpecName: "kube-api-access-2jc46") pod "622793e7-f6fd-4d0d-8a03-59450e296aff" (UID: "622793e7-f6fd-4d0d-8a03-59450e296aff"). InnerVolumeSpecName "kube-api-access-2jc46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:49:40.220845 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:40.220799    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/622793e7-f6fd-4d0d-8a03-59450e296aff-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "622793e7-f6fd-4d0d-8a03-59450e296aff" (UID: "622793e7-f6fd-4d0d-8a03-59450e296aff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:49:40.268443 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:40.268395    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/622793e7-f6fd-4d0d-8a03-59450e296aff-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "622793e7-f6fd-4d0d-8a03-59450e296aff" (UID: "622793e7-f6fd-4d0d-8a03-59450e296aff"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:49:40.304829 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:40.304796    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/622793e7-f6fd-4d0d-8a03-59450e296aff-tls-certs\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:49:40.304829 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:40.304824    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/622793e7-f6fd-4d0d-8a03-59450e296aff-dshm\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:49:40.304829 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:40.304836    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/622793e7-f6fd-4d0d-8a03-59450e296aff-tmp-dir\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:49:40.305057 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:40.304845    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jc46\" (UniqueName: \"kubernetes.io/projected/622793e7-f6fd-4d0d-8a03-59450e296aff-kube-api-access-2jc46\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:49:40.305057 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:40.304855    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/622793e7-f6fd-4d0d-8a03-59450e296aff-kserve-provision-location\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:49:40.305057 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:40.304863    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/622793e7-f6fd-4d0d-8a03-59450e296aff-home\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:49:40.990699 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:40.990588    2557 generic.go:356] "Generic (PLEG): container finished" podID="622793e7-f6fd-4d0d-8a03-59450e296aff" containerID="7427c708f37fc73acca4460813774330729ab727f1b71e760b685a7041bd122e" exitCode=0
Jul 28 10:49:40.990699 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:40.990630    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk" event={"ID":"622793e7-f6fd-4d0d-8a03-59450e296aff","Type":"ContainerDied","Data":"7427c708f37fc73acca4460813774330729ab727f1b71e760b685a7041bd122e"}
Jul 28 10:49:40.990699 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:40.990676    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk"
Jul 28 10:49:40.991280 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:40.990712    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk" event={"ID":"622793e7-f6fd-4d0d-8a03-59450e296aff","Type":"ContainerDied","Data":"3e8ceb4b6df887d4447ce3dcb240c0e317cca9d31c7f3c2e2a6a5e0ef1582c36"}
Jul 28 10:49:40.991280 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:40.990735    2557 scope.go:117] "RemoveContainer" containerID="7427c708f37fc73acca4460813774330729ab727f1b71e760b685a7041bd122e"
Jul 28 10:49:40.999313 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:40.999295    2557 scope.go:117] "RemoveContainer" containerID="f081c549bbb06de6afc840296792b83a4f1caf0e74847c2a89673309f3661bbe"
Jul 28 10:49:41.007787 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:41.007767    2557 scope.go:117] "RemoveContainer" containerID="7427c708f37fc73acca4460813774330729ab727f1b71e760b685a7041bd122e"
Jul 28 10:49:41.008056 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:49:41.008036    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7427c708f37fc73acca4460813774330729ab727f1b71e760b685a7041bd122e\": container with ID starting with 7427c708f37fc73acca4460813774330729ab727f1b71e760b685a7041bd122e not found: ID does not exist" containerID="7427c708f37fc73acca4460813774330729ab727f1b71e760b685a7041bd122e"
Jul 28 10:49:41.008131 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:41.008068    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7427c708f37fc73acca4460813774330729ab727f1b71e760b685a7041bd122e"} err="failed to get container status \"7427c708f37fc73acca4460813774330729ab727f1b71e760b685a7041bd122e\": rpc error: code = NotFound desc = could not find container \"7427c708f37fc73acca4460813774330729ab727f1b71e760b685a7041bd122e\": container with ID starting with 7427c708f37fc73acca4460813774330729ab727f1b71e760b685a7041bd122e not found: ID does not exist"
Jul 28 10:49:41.008131 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:41.008095    2557 scope.go:117] "RemoveContainer" containerID="f081c549bbb06de6afc840296792b83a4f1caf0e74847c2a89673309f3661bbe"
Jul 28 10:49:41.008310 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:41.008290    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk"]
Jul 28 10:49:41.008351 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:49:41.008316    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f081c549bbb06de6afc840296792b83a4f1caf0e74847c2a89673309f3661bbe\": container with ID starting with f081c549bbb06de6afc840296792b83a4f1caf0e74847c2a89673309f3661bbe not found: ID does not exist" containerID="f081c549bbb06de6afc840296792b83a4f1caf0e74847c2a89673309f3661bbe"
Jul 28 10:49:41.008351 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:41.008336    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f081c549bbb06de6afc840296792b83a4f1caf0e74847c2a89673309f3661bbe"} err="failed to get container status \"f081c549bbb06de6afc840296792b83a4f1caf0e74847c2a89673309f3661bbe\": rpc error: code = NotFound desc = could not find container \"f081c549bbb06de6afc840296792b83a4f1caf0e74847c2a89673309f3661bbe\": container with ID starting with f081c549bbb06de6afc840296792b83a4f1caf0e74847c2a89673309f3661bbe not found: ID does not exist"
Jul 28 10:49:41.011002 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:41.010979    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk"]
Jul 28 10:49:41.128640 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:41.128572    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-676ccb4577-ddjmk" podUID="622793e7-f6fd-4d0d-8a03-59450e296aff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 28 10:49:42.580489 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:42.580457    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="622793e7-f6fd-4d0d-8a03-59450e296aff" path="/var/lib/kubelet/pods/622793e7-f6fd-4d0d-8a03-59450e296aff/volumes"
Jul 28 10:49:59.033457 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:59.033422    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelkknw"]
Jul 28 10:49:59.034053 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:59.033774    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="622793e7-f6fd-4d0d-8a03-59450e296aff" containerName="storage-initializer"
Jul 28 10:49:59.034053 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:59.033787    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="622793e7-f6fd-4d0d-8a03-59450e296aff" containerName="storage-initializer"
Jul 28 10:49:59.034053 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:59.033795    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="622793e7-f6fd-4d0d-8a03-59450e296aff" containerName="main"
Jul 28 10:49:59.034053 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:59.033802    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="622793e7-f6fd-4d0d-8a03-59450e296aff" containerName="main"
Jul 28 10:49:59.034053 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:59.033857    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="622793e7-f6fd-4d0d-8a03-59450e296aff" containerName="main"
Jul 28 10:49:59.037683 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:59.037663    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelkknw"
Jul 28 10:49:59.040334 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:59.040248    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Jul 28 10:49:59.040334 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:59.040270    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-27jsr\""
Jul 28 10:49:59.040334 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:59.040283    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-rgrdd\""
Jul 28 10:49:59.040334 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:59.040299    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Jul 28 10:49:59.040726 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:59.040370    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Jul 28 10:49:59.046055 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:59.046029    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelkknw"]
Jul 28 10:49:59.170939 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:59.170902    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw9xb\" (UniqueName: \"kubernetes.io/projected/ba1846e6-878c-40e2-a165-2b10b9b4f463-kube-api-access-bw9xb\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelkknw\" (UID: \"ba1846e6-878c-40e2-a165-2b10b9b4f463\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelkknw"
Jul 28 10:49:59.171133 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:59.171001    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba1846e6-878c-40e2-a165-2b10b9b4f463-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelkknw\" (UID: \"ba1846e6-878c-40e2-a165-2b10b9b4f463\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelkknw"
Jul 28 10:49:59.271844 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:59.271803    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba1846e6-878c-40e2-a165-2b10b9b4f463-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelkknw\" (UID: \"ba1846e6-878c-40e2-a165-2b10b9b4f463\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelkknw"
Jul 28 10:49:59.272029 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:59.271877    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw9xb\" (UniqueName: \"kubernetes.io/projected/ba1846e6-878c-40e2-a165-2b10b9b4f463-kube-api-access-bw9xb\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelkknw\" (UID: \"ba1846e6-878c-40e2-a165-2b10b9b4f463\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelkknw"
Jul 28 10:49:59.274503 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:59.274474    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba1846e6-878c-40e2-a165-2b10b9b4f463-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelkknw\" (UID: \"ba1846e6-878c-40e2-a165-2b10b9b4f463\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelkknw"
Jul 28 10:49:59.279440 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:59.279415    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw9xb\" (UniqueName: \"kubernetes.io/projected/ba1846e6-878c-40e2-a165-2b10b9b4f463-kube-api-access-bw9xb\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelkknw\" (UID: \"ba1846e6-878c-40e2-a165-2b10b9b4f463\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelkknw"
Jul 28 10:49:59.350739 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:59.350652    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelkknw"
Jul 28 10:49:59.485770 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:49:59.485744    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelkknw"]
Jul 28 10:49:59.488182 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:49:59.488155    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba1846e6_878c_40e2_a165_2b10b9b4f463.slice/crio-6e9e30686086dc0a1d902bcb182dbeeaf67b4c394611382a97937131f5eb8630 WatchSource:0}: Error finding container 6e9e30686086dc0a1d902bcb182dbeeaf67b4c394611382a97937131f5eb8630: Status 404 returned error can't find the container with id 6e9e30686086dc0a1d902bcb182dbeeaf67b4c394611382a97937131f5eb8630
Jul 28 10:50:00.079392 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:50:00.079356    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelkknw" event={"ID":"ba1846e6-878c-40e2-a165-2b10b9b4f463","Type":"ContainerStarted","Data":"26eda020bf5abbc1896eb5209c7c24b220a4523c748350eb70b12f95acc4ae39"}
Jul 28 10:50:00.079811 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:50:00.079404    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelkknw" event={"ID":"ba1846e6-878c-40e2-a165-2b10b9b4f463","Type":"ContainerStarted","Data":"6e9e30686086dc0a1d902bcb182dbeeaf67b4c394611382a97937131f5eb8630"}
Jul 28 10:50:00.079811 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:50:00.079438    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelkknw"
Jul 28 10:50:00.096970 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:50:00.096917    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelkknw" podStartSLOduration=1.096901462 podStartE2EDuration="1.096901462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:49:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:50:00.09510861 +0000 UTC m=+2536.053876974" watchObservedRunningTime="2026-07-28 10:50:00.096901462 +0000 UTC m=+2536.055669856"
Jul 28 10:50:31.085501 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:50:31.085471    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelkknw"
Jul 28 10:51:31.382108 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:51:31.382075    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelkknw"]
Jul 28 10:51:41.085454 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:51:41.085414    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelkknw" podUID="ba1846e6-878c-40e2-a165-2b10b9b4f463" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 10:51:46.405675 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:51:46.405631    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelkknw" podUID="ba1846e6-878c-40e2-a165-2b10b9b4f463" containerName="main" containerID="cri-o://26eda020bf5abbc1896eb5209c7c24b220a4523c748350eb70b12f95acc4ae39" gracePeriod=45
Jul 28 10:51:46.540860 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:51:46.540828    2557 generic.go:356] "Generic (PLEG): container finished" podID="ba1846e6-878c-40e2-a165-2b10b9b4f463" containerID="26eda020bf5abbc1896eb5209c7c24b220a4523c748350eb70b12f95acc4ae39" exitCode=0
Jul 28 10:51:46.541013 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:51:46.540889    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelkknw" event={"ID":"ba1846e6-878c-40e2-a165-2b10b9b4f463","Type":"ContainerDied","Data":"26eda020bf5abbc1896eb5209c7c24b220a4523c748350eb70b12f95acc4ae39"}
Jul 28 10:51:46.658707 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:51:46.658650    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelkknw"
Jul 28 10:51:46.736042 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:51:46.736008    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba1846e6-878c-40e2-a165-2b10b9b4f463-tls-certs\") pod \"ba1846e6-878c-40e2-a165-2b10b9b4f463\" (UID: \"ba1846e6-878c-40e2-a165-2b10b9b4f463\") "
Jul 28 10:51:46.736235 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:51:46.736073    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bw9xb\" (UniqueName: \"kubernetes.io/projected/ba1846e6-878c-40e2-a165-2b10b9b4f463-kube-api-access-bw9xb\") pod \"ba1846e6-878c-40e2-a165-2b10b9b4f463\" (UID: \"ba1846e6-878c-40e2-a165-2b10b9b4f463\") "
Jul 28 10:51:46.738156 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:51:46.738122    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba1846e6-878c-40e2-a165-2b10b9b4f463-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ba1846e6-878c-40e2-a165-2b10b9b4f463" (UID: "ba1846e6-878c-40e2-a165-2b10b9b4f463"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:51:46.738295 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:51:46.738246    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba1846e6-878c-40e2-a165-2b10b9b4f463-kube-api-access-bw9xb" (OuterVolumeSpecName: "kube-api-access-bw9xb") pod "ba1846e6-878c-40e2-a165-2b10b9b4f463" (UID: "ba1846e6-878c-40e2-a165-2b10b9b4f463"). InnerVolumeSpecName "kube-api-access-bw9xb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:51:46.836833 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:51:46.836798    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba1846e6-878c-40e2-a165-2b10b9b4f463-tls-certs\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:51:46.836833 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:51:46.836827    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw9xb\" (UniqueName: \"kubernetes.io/projected/ba1846e6-878c-40e2-a165-2b10b9b4f463-kube-api-access-bw9xb\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:51:47.546756 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:51:47.546719    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelkknw" event={"ID":"ba1846e6-878c-40e2-a165-2b10b9b4f463","Type":"ContainerDied","Data":"6e9e30686086dc0a1d902bcb182dbeeaf67b4c394611382a97937131f5eb8630"}
Jul 28 10:51:47.546756 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:51:47.546758    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelkknw"
Jul 28 10:51:47.547259 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:51:47.546792    2557 scope.go:117] "RemoveContainer" containerID="26eda020bf5abbc1896eb5209c7c24b220a4523c748350eb70b12f95acc4ae39"
Jul 28 10:51:47.566453 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:51:47.566421    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelkknw"]
Jul 28 10:51:47.572445 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:51:47.572418    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelkknw"]
Jul 28 10:51:48.579878 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:51:48.579847    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba1846e6-878c-40e2-a165-2b10b9b4f463" path="/var/lib/kubelet/pods/ba1846e6-878c-40e2-a165-2b10b9b4f463/volumes"
Jul 28 10:51:54.499402 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:51:54.499368    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-46bff"]
Jul 28 10:51:54.499815 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:51:54.499690    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba1846e6-878c-40e2-a165-2b10b9b4f463" containerName="main"
Jul 28 10:51:54.499815 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:51:54.499701    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba1846e6-878c-40e2-a165-2b10b9b4f463" containerName="main"
Jul 28 10:51:54.499815 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:51:54.499759    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba1846e6-878c-40e2-a165-2b10b9b4f463" containerName="main"
Jul 28 10:51:54.503707 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:51:54.503690    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-46bff"
Jul 28 10:51:54.506064 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:51:54.506042    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"openshift-service-ca.crt\""
Jul 28 10:51:54.506199 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:51:54.506067    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"default-dockercfg-khtqv\""
Jul 28 10:51:54.506199 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:51:54.506083    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"kube-root-ca.crt\""
Jul 28 10:51:54.509915 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:51:54.509893    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-46bff"]
Jul 28 10:51:54.599713 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:51:54.599678    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2xgj\" (UniqueName: \"kubernetes.io/projected/2f39d075-5b82-477f-ab02-05583c9e2f1f-kube-api-access-b2xgj\") pod \"e2e-pvc-model-download-46bff\" (UID: \"2f39d075-5b82-477f-ab02-05583c9e2f1f\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-46bff"
Jul 28 10:51:54.700985 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:51:54.700946    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2xgj\" (UniqueName: \"kubernetes.io/projected/2f39d075-5b82-477f-ab02-05583c9e2f1f-kube-api-access-b2xgj\") pod \"e2e-pvc-model-download-46bff\" (UID: \"2f39d075-5b82-477f-ab02-05583c9e2f1f\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-46bff"
Jul 28 10:51:54.709939 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:51:54.709912    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2xgj\" (UniqueName: \"kubernetes.io/projected/2f39d075-5b82-477f-ab02-05583c9e2f1f-kube-api-access-b2xgj\") pod \"e2e-pvc-model-download-46bff\" (UID: \"2f39d075-5b82-477f-ab02-05583c9e2f1f\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-46bff"
Jul 28 10:52:00.048253 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:00.048221    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-b20895cc-2303-4bdd-89d4-b32b3d2b54f1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09013b836ca500b90\") pod \"e2e-pvc-model-download-46bff\" (UID: \"2f39d075-5b82-477f-ab02-05583c9e2f1f\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-46bff"
Jul 28 10:52:00.053165 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:00.053124    2557 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-b20895cc-2303-4bdd-89d4-b32b3d2b54f1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09013b836ca500b90\") pod \"e2e-pvc-model-download-46bff\" (UID: \"2f39d075-5b82-477f-ab02-05583c9e2f1f\") device path: \"\"" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-46bff"
Jul 28 10:52:00.148742 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:00.148699    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-b20895cc-2303-4bdd-89d4-b32b3d2b54f1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09013b836ca500b90\") pod \"e2e-pvc-model-download-46bff\" (UID: \"2f39d075-5b82-477f-ab02-05583c9e2f1f\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-46bff"
Jul 28 10:52:00.148917 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:00.148840    2557 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-b20895cc-2303-4bdd-89d4-b32b3d2b54f1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09013b836ca500b90\") pod \"e2e-pvc-model-download-46bff\" (UID: \"2f39d075-5b82-477f-ab02-05583c9e2f1f\") DevicePath \"\"" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-46bff"
Jul 28 10:52:00.151916 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:00.151893    2557 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-b20895cc-2303-4bdd-89d4-b32b3d2b54f1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09013b836ca500b90\") pod \"e2e-pvc-model-download-46bff\" (UID: \"2f39d075-5b82-477f-ab02-05583c9e2f1f\") DevicePath \"csi-7791c75c9d40e8e1a70cbe03afbb5e78c32f7cb3bf7478097bca321c1e68e1f8\"" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-46bff"
Jul 28 10:52:00.373767 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:00.373732    2557 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-b20895cc-2303-4bdd-89d4-b32b3d2b54f1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09013b836ca500b90\") pod \"e2e-pvc-model-download-46bff\" (UID: \"2f39d075-5b82-477f-ab02-05583c9e2f1f\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/71dd163cd5a159e681d684285ccc41b5bb0f8af4ad6482a24eb0975a89adc509/globalmount\"" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-46bff"
Jul 28 10:52:00.386874 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:00.386847    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-b20895cc-2303-4bdd-89d4-b32b3d2b54f1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09013b836ca500b90\") pod \"e2e-pvc-model-download-46bff\" (UID: \"2f39d075-5b82-477f-ab02-05583c9e2f1f\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-46bff"
Jul 28 10:52:00.513888 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:00.513855    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-46bff"
Jul 28 10:52:00.646456 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:00.646427    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-46bff"]
Jul 28 10:52:00.648258 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:52:00.648231    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f39d075_5b82_477f_ab02_05583c9e2f1f.slice/crio-361bf2312e65f0c64310489dea1d53331c7495edd6ecfc6ac239e4ebd98c8248 WatchSource:0}: Error finding container 361bf2312e65f0c64310489dea1d53331c7495edd6ecfc6ac239e4ebd98c8248: Status 404 returned error can't find the container with id 361bf2312e65f0c64310489dea1d53331c7495edd6ecfc6ac239e4ebd98c8248
Jul 28 10:52:01.618666 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:01.618573    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-46bff" event={"ID":"2f39d075-5b82-477f-ab02-05583c9e2f1f","Type":"ContainerStarted","Data":"afbdfeefff6823b2c1acaec33ff8da5a866abcfd83a6b8b77f6e651a19166d6a"}
Jul 28 10:52:01.619188 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:01.618689    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-46bff" event={"ID":"2f39d075-5b82-477f-ab02-05583c9e2f1f","Type":"ContainerStarted","Data":"361bf2312e65f0c64310489dea1d53331c7495edd6ecfc6ac239e4ebd98c8248"}
Jul 28 10:52:01.634344 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:01.634220    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-46bff" podStartSLOduration=10.634195128 podStartE2EDuration="10.634195128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:51:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:52:01.633695924 +0000 UTC m=+2657.592464304" watchObservedRunningTime="2026-07-28 10:52:01.634195128 +0000 UTC m=+2657.592963508"
Jul 28 10:52:07.645084 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:07.645049    2557 generic.go:356] "Generic (PLEG): container finished" podID="2f39d075-5b82-477f-ab02-05583c9e2f1f" containerID="afbdfeefff6823b2c1acaec33ff8da5a866abcfd83a6b8b77f6e651a19166d6a" exitCode=0
Jul 28 10:52:07.645460 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:07.645121    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-46bff" event={"ID":"2f39d075-5b82-477f-ab02-05583c9e2f1f","Type":"ContainerDied","Data":"afbdfeefff6823b2c1acaec33ff8da5a866abcfd83a6b8b77f6e651a19166d6a"}
Jul 28 10:52:08.795128 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:08.795100    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-46bff"
Jul 28 10:52:08.831537 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:08.831500    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09013b836ca500b90\") pod \"2f39d075-5b82-477f-ab02-05583c9e2f1f\" (UID: \"2f39d075-5b82-477f-ab02-05583c9e2f1f\") "
Jul 28 10:52:08.831742 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:08.831630    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b2xgj\" (UniqueName: \"kubernetes.io/projected/2f39d075-5b82-477f-ab02-05583c9e2f1f-kube-api-access-b2xgj\") pod \"2f39d075-5b82-477f-ab02-05583c9e2f1f\" (UID: \"2f39d075-5b82-477f-ab02-05583c9e2f1f\") "
Jul 28 10:52:08.833908 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:08.833876    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f39d075-5b82-477f-ab02-05583c9e2f1f-kube-api-access-b2xgj" (OuterVolumeSpecName: "kube-api-access-b2xgj") pod "2f39d075-5b82-477f-ab02-05583c9e2f1f" (UID: "2f39d075-5b82-477f-ab02-05583c9e2f1f"). InnerVolumeSpecName "kube-api-access-b2xgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:52:08.835796 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:08.835772    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-09013b836ca500b90" (OuterVolumeSpecName: "model-storage") pod "2f39d075-5b82-477f-ab02-05583c9e2f1f" (UID: "2f39d075-5b82-477f-ab02-05583c9e2f1f"). InnerVolumeSpecName "pvc-b20895cc-2303-4bdd-89d4-b32b3d2b54f1". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 28 10:52:08.933385 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:08.933296    2557 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-b20895cc-2303-4bdd-89d4-b32b3d2b54f1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09013b836ca500b90\") on node \"ip-10-0-131-88.ec2.internal\" "
Jul 28 10:52:08.933385 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:08.933329    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2xgj\" (UniqueName: \"kubernetes.io/projected/2f39d075-5b82-477f-ab02-05583c9e2f1f-kube-api-access-b2xgj\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:52:09.052066 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:09.052044    2557 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-b20895cc-2303-4bdd-89d4-b32b3d2b54f1" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-09013b836ca500b90") on node "ip-10-0-131-88.ec2.internal"
Jul 28 10:52:09.134766 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:09.134726    2557 reconciler_common.go:299] "Volume detached for volume \"pvc-b20895cc-2303-4bdd-89d4-b32b3d2b54f1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09013b836ca500b90\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"csi-7791c75c9d40e8e1a70cbe03afbb5e78c32f7cb3bf7478097bca321c1e68e1f8\""
Jul 28 10:52:09.655651 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:09.655579    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-46bff" event={"ID":"2f39d075-5b82-477f-ab02-05583c9e2f1f","Type":"ContainerDied","Data":"361bf2312e65f0c64310489dea1d53331c7495edd6ecfc6ac239e4ebd98c8248"}
Jul 28 10:52:09.655830 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:09.655663    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="361bf2312e65f0c64310489dea1d53331c7495edd6ecfc6ac239e4ebd98c8248"
Jul 28 10:52:09.655830 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:09.655628    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-46bff"
Jul 28 10:52:16.285381 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:16.285340    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws"]
Jul 28 10:52:16.286131 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:16.285821    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f39d075-5b82-477f-ab02-05583c9e2f1f" containerName="storage-initializer"
Jul 28 10:52:16.286131 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:16.285840    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f39d075-5b82-477f-ab02-05583c9e2f1f" containerName="storage-initializer"
Jul 28 10:52:16.286131 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:16.285928    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f39d075-5b82-477f-ab02-05583c9e2f1f" containerName="storage-initializer"
Jul 28 10:52:16.289278 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:16.289256    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws"
Jul 28 10:52:16.291569 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:16.291545    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"openshift-service-ca.crt\""
Jul 28 10:52:16.291701 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:16.291545    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs\""
Jul 28 10:52:16.291701 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:16.291586    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"default-dockercfg-khtqv\""
Jul 28 10:52:16.291701 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:16.291547    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"kube-root-ca.crt\""
Jul 28 10:52:16.300364 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:16.300339    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws"]
Jul 28 10:52:16.399058 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:16.399027    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3451948-a723-44da-8ac3-cfedd2a7eb10-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws\" (UID: \"b3451948-a723-44da-8ac3-cfedd2a7eb10\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws"
Jul 28 10:52:16.399262 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:16.399096    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3451948-a723-44da-8ac3-cfedd2a7eb10-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws\" (UID: \"b3451948-a723-44da-8ac3-cfedd2a7eb10\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws"
Jul 28 10:52:16.399262 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:16.399170    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3451948-a723-44da-8ac3-cfedd2a7eb10-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws\" (UID: \"b3451948-a723-44da-8ac3-cfedd2a7eb10\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws"
Jul 28 10:52:16.399262 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:16.399247    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnqpb\" (UniqueName: \"kubernetes.io/projected/b3451948-a723-44da-8ac3-cfedd2a7eb10-kube-api-access-xnqpb\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws\" (UID: \"b3451948-a723-44da-8ac3-cfedd2a7eb10\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws"
Jul 28 10:52:16.399384 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:16.399315    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3451948-a723-44da-8ac3-cfedd2a7eb10-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws\" (UID: \"b3451948-a723-44da-8ac3-cfedd2a7eb10\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws"
Jul 28 10:52:16.399384 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:16.399356    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3451948-a723-44da-8ac3-cfedd2a7eb10-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws\" (UID: \"b3451948-a723-44da-8ac3-cfedd2a7eb10\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws"
Jul 28 10:52:16.500236 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:16.500200    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3451948-a723-44da-8ac3-cfedd2a7eb10-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws\" (UID: \"b3451948-a723-44da-8ac3-cfedd2a7eb10\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws"
Jul 28 10:52:16.500236 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:16.500238    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3451948-a723-44da-8ac3-cfedd2a7eb10-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws\" (UID: \"b3451948-a723-44da-8ac3-cfedd2a7eb10\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws"
Jul 28 10:52:16.500463 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:16.500269    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3451948-a723-44da-8ac3-cfedd2a7eb10-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws\" (UID: \"b3451948-a723-44da-8ac3-cfedd2a7eb10\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws"
Jul 28 10:52:16.500463 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:16.500317    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3451948-a723-44da-8ac3-cfedd2a7eb10-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws\" (UID: \"b3451948-a723-44da-8ac3-cfedd2a7eb10\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws"
Jul 28 10:52:16.500463 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:16.500338    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3451948-a723-44da-8ac3-cfedd2a7eb10-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws\" (UID: \"b3451948-a723-44da-8ac3-cfedd2a7eb10\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws"
Jul 28 10:52:16.500463 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:16.500362    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnqpb\" (UniqueName: \"kubernetes.io/projected/b3451948-a723-44da-8ac3-cfedd2a7eb10-kube-api-access-xnqpb\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws\" (UID: \"b3451948-a723-44da-8ac3-cfedd2a7eb10\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws"
Jul 28 10:52:16.500753 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:16.500713    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3451948-a723-44da-8ac3-cfedd2a7eb10-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws\" (UID: \"b3451948-a723-44da-8ac3-cfedd2a7eb10\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws"
Jul 28 10:52:16.500820 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:16.500777    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3451948-a723-44da-8ac3-cfedd2a7eb10-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws\" (UID: \"b3451948-a723-44da-8ac3-cfedd2a7eb10\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws"
Jul 28 10:52:16.500820 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:16.500810    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3451948-a723-44da-8ac3-cfedd2a7eb10-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws\" (UID: \"b3451948-a723-44da-8ac3-cfedd2a7eb10\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws"
Jul 28 10:52:16.502688 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:16.502669    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3451948-a723-44da-8ac3-cfedd2a7eb10-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws\" (UID: \"b3451948-a723-44da-8ac3-cfedd2a7eb10\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws"
Jul 28 10:52:16.502929 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:16.502911    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3451948-a723-44da-8ac3-cfedd2a7eb10-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws\" (UID: \"b3451948-a723-44da-8ac3-cfedd2a7eb10\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws"
Jul 28 10:52:16.508707 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:16.508676    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnqpb\" (UniqueName: \"kubernetes.io/projected/b3451948-a723-44da-8ac3-cfedd2a7eb10-kube-api-access-xnqpb\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws\" (UID: \"b3451948-a723-44da-8ac3-cfedd2a7eb10\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws"
Jul 28 10:52:20.435028 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:20.434987    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-b20895cc-2303-4bdd-89d4-b32b3d2b54f1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09013b836ca500b90\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws\" (UID: \"b3451948-a723-44da-8ac3-cfedd2a7eb10\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws"
Jul 28 10:52:20.439874 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:20.439848    2557 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-b20895cc-2303-4bdd-89d4-b32b3d2b54f1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09013b836ca500b90\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws\" (UID: \"b3451948-a723-44da-8ac3-cfedd2a7eb10\") device path: \"\"" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws"
Jul 28 10:52:20.536458 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:20.536414    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-b20895cc-2303-4bdd-89d4-b32b3d2b54f1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09013b836ca500b90\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws\" (UID: \"b3451948-a723-44da-8ac3-cfedd2a7eb10\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws"
Jul 28 10:52:20.536666 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:20.536525    2557 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-b20895cc-2303-4bdd-89d4-b32b3d2b54f1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09013b836ca500b90\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws\" (UID: \"b3451948-a723-44da-8ac3-cfedd2a7eb10\") DevicePath \"\"" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws"
Jul 28 10:52:20.539368 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:20.539348    2557 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-b20895cc-2303-4bdd-89d4-b32b3d2b54f1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09013b836ca500b90\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws\" (UID: \"b3451948-a723-44da-8ac3-cfedd2a7eb10\") DevicePath \"csi-7791c75c9d40e8e1a70cbe03afbb5e78c32f7cb3bf7478097bca321c1e68e1f8\"" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws"
Jul 28 10:52:20.624616 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:20.624564    2557 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-b20895cc-2303-4bdd-89d4-b32b3d2b54f1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09013b836ca500b90\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws\" (UID: \"b3451948-a723-44da-8ac3-cfedd2a7eb10\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/71dd163cd5a159e681d684285ccc41b5bb0f8af4ad6482a24eb0975a89adc509/globalmount\"" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws"
Jul 28 10:52:20.639665 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:20.639631    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-b20895cc-2303-4bdd-89d4-b32b3d2b54f1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09013b836ca500b90\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws\" (UID: \"b3451948-a723-44da-8ac3-cfedd2a7eb10\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws"
Jul 28 10:52:20.799649 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:20.799588    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws"
Jul 28 10:52:20.938593 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:20.938569    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws"]
Jul 28 10:52:20.941229 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:52:20.941202    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3451948_a723_44da_8ac3_cfedd2a7eb10.slice/crio-c12a7a17b876485a31243c88ccb74013a2ae304350c24ecc5043c5dabe1f0896 WatchSource:0}: Error finding container c12a7a17b876485a31243c88ccb74013a2ae304350c24ecc5043c5dabe1f0896: Status 404 returned error can't find the container with id c12a7a17b876485a31243c88ccb74013a2ae304350c24ecc5043c5dabe1f0896
Jul 28 10:52:21.715259 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:21.715224    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws" event={"ID":"b3451948-a723-44da-8ac3-cfedd2a7eb10","Type":"ContainerStarted","Data":"ff2103934d33f0ef37c95812689225a3e3620b9afea3e895425eeeccfe1fe4fc"}
Jul 28 10:52:21.715655 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:21.715301    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws" event={"ID":"b3451948-a723-44da-8ac3-cfedd2a7eb10","Type":"ContainerStarted","Data":"c12a7a17b876485a31243c88ccb74013a2ae304350c24ecc5043c5dabe1f0896"}
Jul 28 10:52:21.733915 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:21.733855    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws" podStartSLOduration=5.733834887 podStartE2EDuration="5.733834887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:52:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:52:21.732425115 +0000 UTC m=+2677.691193494" watchObservedRunningTime="2026-07-28 10:52:21.733834887 +0000 UTC m=+2677.692603267"
Jul 28 10:52:21.800458 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:21.800426    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws"
Jul 28 10:52:30.800379 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:30.800343    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws"
Jul 28 10:52:30.801941 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:30.801910    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws" podUID="b3451948-a723-44da-8ac3-cfedd2a7eb10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Jul 28 10:52:40.800371 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:40.800273    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws" podUID="b3451948-a723-44da-8ac3-cfedd2a7eb10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Jul 28 10:52:50.800834 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:52:50.800789    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws" podUID="b3451948-a723-44da-8ac3-cfedd2a7eb10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Jul 28 10:53:00.800639 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:53:00.800573    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws" podUID="b3451948-a723-44da-8ac3-cfedd2a7eb10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Jul 28 10:53:10.800202 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:53:10.800154    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws" podUID="b3451948-a723-44da-8ac3-cfedd2a7eb10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Jul 28 10:53:20.801027 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:53:20.800980    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws" podUID="b3451948-a723-44da-8ac3-cfedd2a7eb10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Jul 28 10:53:30.800721 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:53:30.800673    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws" podUID="b3451948-a723-44da-8ac3-cfedd2a7eb10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Jul 28 10:53:40.800674 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:53:40.800621    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws" podUID="b3451948-a723-44da-8ac3-cfedd2a7eb10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Jul 28 10:53:50.800211 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:53:50.800162    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws" podUID="b3451948-a723-44da-8ac3-cfedd2a7eb10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Jul 28 10:54:00.800737 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:00.800696    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws" podUID="b3451948-a723-44da-8ac3-cfedd2a7eb10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Jul 28 10:54:10.800992 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:10.800891    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws" podUID="b3451948-a723-44da-8ac3-cfedd2a7eb10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Jul 28 10:54:20.810334 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:20.810299    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws"
Jul 28 10:54:20.818185 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:20.818155    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws"
Jul 28 10:54:25.760167 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:25.760129    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws"]
Jul 28 10:54:40.784265 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:40.784203    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws" podUID="b3451948-a723-44da-8ac3-cfedd2a7eb10" containerName="main" containerID="cri-o://ff2103934d33f0ef37c95812689225a3e3620b9afea3e895425eeeccfe1fe4fc" gracePeriod=45
Jul 28 10:54:41.538450 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:41.538422    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws"
Jul 28 10:54:41.631827 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:41.631737    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09013b836ca500b90\") pod \"b3451948-a723-44da-8ac3-cfedd2a7eb10\" (UID: \"b3451948-a723-44da-8ac3-cfedd2a7eb10\") "
Jul 28 10:54:41.631827 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:41.631793    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3451948-a723-44da-8ac3-cfedd2a7eb10-tls-certs\") pod \"b3451948-a723-44da-8ac3-cfedd2a7eb10\" (UID: \"b3451948-a723-44da-8ac3-cfedd2a7eb10\") "
Jul 28 10:54:41.632067 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:41.631837    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3451948-a723-44da-8ac3-cfedd2a7eb10-tmp-dir\") pod \"b3451948-a723-44da-8ac3-cfedd2a7eb10\" (UID: \"b3451948-a723-44da-8ac3-cfedd2a7eb10\") "
Jul 28 10:54:41.632067 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:41.631869    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3451948-a723-44da-8ac3-cfedd2a7eb10-model-cache\") pod \"b3451948-a723-44da-8ac3-cfedd2a7eb10\" (UID: \"b3451948-a723-44da-8ac3-cfedd2a7eb10\") "
Jul 28 10:54:41.632067 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:41.631908    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3451948-a723-44da-8ac3-cfedd2a7eb10-home\") pod \"b3451948-a723-44da-8ac3-cfedd2a7eb10\" (UID: \"b3451948-a723-44da-8ac3-cfedd2a7eb10\") "
Jul 28 10:54:41.632067 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:41.631947    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3451948-a723-44da-8ac3-cfedd2a7eb10-dshm\") pod \"b3451948-a723-44da-8ac3-cfedd2a7eb10\" (UID: \"b3451948-a723-44da-8ac3-cfedd2a7eb10\") "
Jul 28 10:54:41.632067 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:41.631969    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xnqpb\" (UniqueName: \"kubernetes.io/projected/b3451948-a723-44da-8ac3-cfedd2a7eb10-kube-api-access-xnqpb\") pod \"b3451948-a723-44da-8ac3-cfedd2a7eb10\" (UID: \"b3451948-a723-44da-8ac3-cfedd2a7eb10\") "
Jul 28 10:54:41.632326 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:41.632109    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3451948-a723-44da-8ac3-cfedd2a7eb10-model-cache" (OuterVolumeSpecName: "model-cache") pod "b3451948-a723-44da-8ac3-cfedd2a7eb10" (UID: "b3451948-a723-44da-8ac3-cfedd2a7eb10"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:54:41.632685 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:41.632647    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3451948-a723-44da-8ac3-cfedd2a7eb10-home" (OuterVolumeSpecName: "home") pod "b3451948-a723-44da-8ac3-cfedd2a7eb10" (UID: "b3451948-a723-44da-8ac3-cfedd2a7eb10"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:54:41.634456 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:41.634430    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3451948-a723-44da-8ac3-cfedd2a7eb10-kube-api-access-xnqpb" (OuterVolumeSpecName: "kube-api-access-xnqpb") pod "b3451948-a723-44da-8ac3-cfedd2a7eb10" (UID: "b3451948-a723-44da-8ac3-cfedd2a7eb10"). InnerVolumeSpecName "kube-api-access-xnqpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:54:41.634645 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:41.634590    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3451948-a723-44da-8ac3-cfedd2a7eb10-dshm" (OuterVolumeSpecName: "dshm") pod "b3451948-a723-44da-8ac3-cfedd2a7eb10" (UID: "b3451948-a723-44da-8ac3-cfedd2a7eb10"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:54:41.634765 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:41.634621    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3451948-a723-44da-8ac3-cfedd2a7eb10-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b3451948-a723-44da-8ac3-cfedd2a7eb10" (UID: "b3451948-a723-44da-8ac3-cfedd2a7eb10"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:54:41.634981 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:41.634958    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-09013b836ca500b90" (OuterVolumeSpecName: "kserve-pvc-source") pod "b3451948-a723-44da-8ac3-cfedd2a7eb10" (UID: "b3451948-a723-44da-8ac3-cfedd2a7eb10"). InnerVolumeSpecName "pvc-b20895cc-2303-4bdd-89d4-b32b3d2b54f1". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 28 10:54:41.645551 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:41.645526    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3451948-a723-44da-8ac3-cfedd2a7eb10-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b3451948-a723-44da-8ac3-cfedd2a7eb10" (UID: "b3451948-a723-44da-8ac3-cfedd2a7eb10"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:54:41.733612 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:41.733571    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3451948-a723-44da-8ac3-cfedd2a7eb10-tls-certs\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:54:41.733760 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:41.733625    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3451948-a723-44da-8ac3-cfedd2a7eb10-tmp-dir\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:54:41.733760 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:41.733635    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3451948-a723-44da-8ac3-cfedd2a7eb10-model-cache\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:54:41.733760 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:41.733643    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3451948-a723-44da-8ac3-cfedd2a7eb10-home\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:54:41.733760 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:41.733651    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3451948-a723-44da-8ac3-cfedd2a7eb10-dshm\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:54:41.733760 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:41.733660    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnqpb\" (UniqueName: \"kubernetes.io/projected/b3451948-a723-44da-8ac3-cfedd2a7eb10-kube-api-access-xnqpb\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:54:41.733760 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:41.733688    2557 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-b20895cc-2303-4bdd-89d4-b32b3d2b54f1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09013b836ca500b90\") on node \"ip-10-0-131-88.ec2.internal\" "
Jul 28 10:54:41.763157 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:41.763128    2557 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-b20895cc-2303-4bdd-89d4-b32b3d2b54f1" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-09013b836ca500b90") on node "ip-10-0-131-88.ec2.internal"
Jul 28 10:54:41.834785 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:41.834748    2557 reconciler_common.go:299] "Volume detached for volume \"pvc-b20895cc-2303-4bdd-89d4-b32b3d2b54f1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09013b836ca500b90\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"csi-7791c75c9d40e8e1a70cbe03afbb5e78c32f7cb3bf7478097bca321c1e68e1f8\""
Jul 28 10:54:42.382752 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:42.382720    2557 generic.go:356] "Generic (PLEG): container finished" podID="b3451948-a723-44da-8ac3-cfedd2a7eb10" containerID="ff2103934d33f0ef37c95812689225a3e3620b9afea3e895425eeeccfe1fe4fc" exitCode=0
Jul 28 10:54:42.382947 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:42.382769    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws" event={"ID":"b3451948-a723-44da-8ac3-cfedd2a7eb10","Type":"ContainerDied","Data":"ff2103934d33f0ef37c95812689225a3e3620b9afea3e895425eeeccfe1fe4fc"}
Jul 28 10:54:42.382947 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:42.382773    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws"
Jul 28 10:54:42.382947 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:42.382810    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws" event={"ID":"b3451948-a723-44da-8ac3-cfedd2a7eb10","Type":"ContainerDied","Data":"c12a7a17b876485a31243c88ccb74013a2ae304350c24ecc5043c5dabe1f0896"}
Jul 28 10:54:42.382947 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:42.382832    2557 scope.go:117] "RemoveContainer" containerID="ff2103934d33f0ef37c95812689225a3e3620b9afea3e895425eeeccfe1fe4fc"
Jul 28 10:54:42.391464 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:42.391444    2557 scope.go:117] "RemoveContainer" containerID="ff2103934d33f0ef37c95812689225a3e3620b9afea3e895425eeeccfe1fe4fc"
Jul 28 10:54:42.391736 ip-10-0-131-88 kubenswrapper[2557]: E0728 10:54:42.391714    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff2103934d33f0ef37c95812689225a3e3620b9afea3e895425eeeccfe1fe4fc\": container with ID starting with ff2103934d33f0ef37c95812689225a3e3620b9afea3e895425eeeccfe1fe4fc not found: ID does not exist" containerID="ff2103934d33f0ef37c95812689225a3e3620b9afea3e895425eeeccfe1fe4fc"
Jul 28 10:54:42.391798 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:42.391743    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff2103934d33f0ef37c95812689225a3e3620b9afea3e895425eeeccfe1fe4fc"} err="failed to get container status \"ff2103934d33f0ef37c95812689225a3e3620b9afea3e895425eeeccfe1fe4fc\": rpc error: code = NotFound desc = could not find container \"ff2103934d33f0ef37c95812689225a3e3620b9afea3e895425eeeccfe1fe4fc\": container with ID starting with ff2103934d33f0ef37c95812689225a3e3620b9afea3e895425eeeccfe1fe4fc not found: ID does not exist"
Jul 28 10:54:42.404262 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:42.404237    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws"]
Jul 28 10:54:42.408315 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:42.408296    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcj44ws"]
Jul 28 10:54:42.581051 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:42.580969    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3451948-a723-44da-8ac3-cfedd2a7eb10" path="/var/lib/kubelet/pods/b3451948-a723-44da-8ac3-cfedd2a7eb10/volumes"
Jul 28 10:54:48.912640 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:48.912588    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-2sw7r"]
Jul 28 10:54:48.913034 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:48.912936    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3451948-a723-44da-8ac3-cfedd2a7eb10" containerName="main"
Jul 28 10:54:48.913034 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:48.912949    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3451948-a723-44da-8ac3-cfedd2a7eb10" containerName="main"
Jul 28 10:54:48.913034 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:48.913005    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3451948-a723-44da-8ac3-cfedd2a7eb10" containerName="main"
Jul 28 10:54:48.917376 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:48.917358    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-2sw7r"
Jul 28 10:54:48.919972 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:48.919947    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"default-dockercfg-s8znt\""
Jul 28 10:54:48.920110 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:48.919947    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"openshift-service-ca.crt\""
Jul 28 10:54:48.920110 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:48.919945    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"kube-root-ca.crt\""
Jul 28 10:54:48.926473 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:48.926449    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-2sw7r"]
Jul 28 10:54:48.996867 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:48.996824    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvk62\" (UniqueName: \"kubernetes.io/projected/0990394c-2bfd-4b42-ba55-2e1f87140d6f-kube-api-access-tvk62\") pod \"e2e-pvc-model-download-2sw7r\" (UID: \"0990394c-2bfd-4b42-ba55-2e1f87140d6f\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-2sw7r"
Jul 28 10:54:49.097982 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:49.097939    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvk62\" (UniqueName: \"kubernetes.io/projected/0990394c-2bfd-4b42-ba55-2e1f87140d6f-kube-api-access-tvk62\") pod \"e2e-pvc-model-download-2sw7r\" (UID: \"0990394c-2bfd-4b42-ba55-2e1f87140d6f\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-2sw7r"
Jul 28 10:54:49.107160 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:49.107126    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvk62\" (UniqueName: \"kubernetes.io/projected/0990394c-2bfd-4b42-ba55-2e1f87140d6f-kube-api-access-tvk62\") pod \"e2e-pvc-model-download-2sw7r\" (UID: \"0990394c-2bfd-4b42-ba55-2e1f87140d6f\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-2sw7r"
Jul 28 10:54:50.619240 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:50.619208    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-46bff"]
Jul 28 10:54:50.630210 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:50.626161    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-46bff"]
Jul 28 10:54:52.580318 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:52.580281    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f39d075-5b82-477f-ab02-05583c9e2f1f" path="/var/lib/kubelet/pods/2f39d075-5b82-477f-ab02-05583c9e2f1f/volumes"
Jul 28 10:54:53.534420 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:53.534320    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-f219be45-903a-4603-991c-a0a969f33047\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-050ac456ca5f97bb8\") pod \"e2e-pvc-model-download-2sw7r\" (UID: \"0990394c-2bfd-4b42-ba55-2e1f87140d6f\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-2sw7r"
Jul 28 10:54:53.539110 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:53.539083    2557 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-f219be45-903a-4603-991c-a0a969f33047\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-050ac456ca5f97bb8\") pod \"e2e-pvc-model-download-2sw7r\" (UID: \"0990394c-2bfd-4b42-ba55-2e1f87140d6f\") device path: \"\"" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-2sw7r"
Jul 28 10:54:53.634823 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:53.634792    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-f219be45-903a-4603-991c-a0a969f33047\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-050ac456ca5f97bb8\") pod \"e2e-pvc-model-download-2sw7r\" (UID: \"0990394c-2bfd-4b42-ba55-2e1f87140d6f\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-2sw7r"
Jul 28 10:54:53.635225 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:53.634943    2557 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-f219be45-903a-4603-991c-a0a969f33047\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-050ac456ca5f97bb8\") pod \"e2e-pvc-model-download-2sw7r\" (UID: \"0990394c-2bfd-4b42-ba55-2e1f87140d6f\") DevicePath \"\"" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-2sw7r"
Jul 28 10:54:53.637623 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:53.637585    2557 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-f219be45-903a-4603-991c-a0a969f33047\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-050ac456ca5f97bb8\") pod \"e2e-pvc-model-download-2sw7r\" (UID: \"0990394c-2bfd-4b42-ba55-2e1f87140d6f\") DevicePath \"csi-8f70dca98acf5b82535b96ad7d8628a568697a1059f39f1d7d80b52c57c81302\"" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-2sw7r"
Jul 28 10:54:53.853014 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:53.852976    2557 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-f219be45-903a-4603-991c-a0a969f33047\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-050ac456ca5f97bb8\") pod \"e2e-pvc-model-download-2sw7r\" (UID: \"0990394c-2bfd-4b42-ba55-2e1f87140d6f\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/20ae7e64630c1825a727ff2aaf076c7ccab6d592608f1f4710e2b0623afb974f/globalmount\"" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-2sw7r"
Jul 28 10:54:53.866181 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:53.866151    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-f219be45-903a-4603-991c-a0a969f33047\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-050ac456ca5f97bb8\") pod \"e2e-pvc-model-download-2sw7r\" (UID: \"0990394c-2bfd-4b42-ba55-2e1f87140d6f\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-2sw7r"
Jul 28 10:54:54.027383 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:54.027344    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-2sw7r"
Jul 28 10:54:54.163735 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:54.163707    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-2sw7r"]
Jul 28 10:54:54.166095 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:54:54.166065    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0990394c_2bfd_4b42_ba55_2e1f87140d6f.slice/crio-b37f7b59837f8cb87f2a8b384325e75d38f8bf192c8fc8d6cf08e549e71401e7 WatchSource:0}: Error finding container b37f7b59837f8cb87f2a8b384325e75d38f8bf192c8fc8d6cf08e549e71401e7: Status 404 returned error can't find the container with id b37f7b59837f8cb87f2a8b384325e75d38f8bf192c8fc8d6cf08e549e71401e7
Jul 28 10:54:54.439550 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:54.439464    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-2sw7r" event={"ID":"0990394c-2bfd-4b42-ba55-2e1f87140d6f","Type":"ContainerStarted","Data":"0e8818fe7bccf49228fbe9cf2975c2aef033c102e7053bcf23865910ba96c903"}
Jul 28 10:54:54.439550 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:54.439505    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-2sw7r" event={"ID":"0990394c-2bfd-4b42-ba55-2e1f87140d6f","Type":"ContainerStarted","Data":"b37f7b59837f8cb87f2a8b384325e75d38f8bf192c8fc8d6cf08e549e71401e7"}
Jul 28 10:54:54.455674 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:54:54.455624    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-2sw7r" podStartSLOduration=9.455586392 podStartE2EDuration="9.455586392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:54:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:54:54.454571464 +0000 UTC m=+2830.413339842" watchObservedRunningTime="2026-07-28 10:54:54.455586392 +0000 UTC m=+2830.414354770"
Jul 28 10:55:01.483378 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:01.483341    2557 generic.go:356] "Generic (PLEG): container finished" podID="0990394c-2bfd-4b42-ba55-2e1f87140d6f" containerID="0e8818fe7bccf49228fbe9cf2975c2aef033c102e7053bcf23865910ba96c903" exitCode=0
Jul 28 10:55:01.483878 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:01.483418    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-2sw7r" event={"ID":"0990394c-2bfd-4b42-ba55-2e1f87140d6f","Type":"ContainerDied","Data":"0e8818fe7bccf49228fbe9cf2975c2aef033c102e7053bcf23865910ba96c903"}
Jul 28 10:55:02.641528 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:02.641500    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-2sw7r"
Jul 28 10:55:02.719777 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:02.719745    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-050ac456ca5f97bb8\") pod \"0990394c-2bfd-4b42-ba55-2e1f87140d6f\" (UID: \"0990394c-2bfd-4b42-ba55-2e1f87140d6f\") "
Jul 28 10:55:02.719925 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:02.719791    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvk62\" (UniqueName: \"kubernetes.io/projected/0990394c-2bfd-4b42-ba55-2e1f87140d6f-kube-api-access-tvk62\") pod \"0990394c-2bfd-4b42-ba55-2e1f87140d6f\" (UID: \"0990394c-2bfd-4b42-ba55-2e1f87140d6f\") "
Jul 28 10:55:02.722095 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:02.722065    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0990394c-2bfd-4b42-ba55-2e1f87140d6f-kube-api-access-tvk62" (OuterVolumeSpecName: "kube-api-access-tvk62") pod "0990394c-2bfd-4b42-ba55-2e1f87140d6f" (UID: "0990394c-2bfd-4b42-ba55-2e1f87140d6f"). InnerVolumeSpecName "kube-api-access-tvk62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:55:02.722420 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:02.722398    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-050ac456ca5f97bb8" (OuterVolumeSpecName: "model-storage") pod "0990394c-2bfd-4b42-ba55-2e1f87140d6f" (UID: "0990394c-2bfd-4b42-ba55-2e1f87140d6f"). InnerVolumeSpecName "pvc-f219be45-903a-4603-991c-a0a969f33047". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 28 10:55:02.821002 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:02.820967    2557 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-f219be45-903a-4603-991c-a0a969f33047\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-050ac456ca5f97bb8\") on node \"ip-10-0-131-88.ec2.internal\" "
Jul 28 10:55:02.821002 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:02.821000    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvk62\" (UniqueName: \"kubernetes.io/projected/0990394c-2bfd-4b42-ba55-2e1f87140d6f-kube-api-access-tvk62\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 10:55:02.942877 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:02.942856    2557 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-f219be45-903a-4603-991c-a0a969f33047" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-050ac456ca5f97bb8") on node "ip-10-0-131-88.ec2.internal"
Jul 28 10:55:03.023345 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:03.023310    2557 reconciler_common.go:299] "Volume detached for volume \"pvc-f219be45-903a-4603-991c-a0a969f33047\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-050ac456ca5f97bb8\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"csi-8f70dca98acf5b82535b96ad7d8628a568697a1059f39f1d7d80b52c57c81302\""
Jul 28 10:55:03.495840 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:03.495809    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-2sw7r"
Jul 28 10:55:03.496012 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:03.495846    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-2sw7r" event={"ID":"0990394c-2bfd-4b42-ba55-2e1f87140d6f","Type":"ContainerDied","Data":"b37f7b59837f8cb87f2a8b384325e75d38f8bf192c8fc8d6cf08e549e71401e7"}
Jul 28 10:55:03.496012 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:03.495879    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b37f7b59837f8cb87f2a8b384325e75d38f8bf192c8fc8d6cf08e549e71401e7"
Jul 28 10:55:19.133138 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:19.133106    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv"]
Jul 28 10:55:19.133570 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:19.133434    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0990394c-2bfd-4b42-ba55-2e1f87140d6f" containerName="storage-initializer"
Jul 28 10:55:19.133570 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:19.133447    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0990394c-2bfd-4b42-ba55-2e1f87140d6f" containerName="storage-initializer"
Jul 28 10:55:19.133570 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:19.133512    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="0990394c-2bfd-4b42-ba55-2e1f87140d6f" containerName="storage-initializer"
Jul 28 10:55:19.136919 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:19.136892    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv"
Jul 28 10:55:19.140004 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:19.139978    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"default-dockercfg-s8znt\""
Jul 28 10:55:19.140155 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:19.140021    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs\""
Jul 28 10:55:19.140294 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:19.140275    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-c2c8r\""
Jul 28 10:55:19.140354 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:19.140316    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"kube-root-ca.crt\""
Jul 28 10:55:19.140405 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:19.140356    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"openshift-service-ca.crt\""
Jul 28 10:55:19.153375 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:19.153344    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv"]
Jul 28 10:55:19.267872 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:19.267836    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv\" (UID: \"1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv"
Jul 28 10:55:19.267872 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:19.267876    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv\" (UID: \"1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv"
Jul 28 10:55:19.268102 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:19.267897    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv\" (UID: \"1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv"
Jul 28 10:55:19.268102 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:19.267921    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv\" (UID: \"1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv"
Jul 28 10:55:19.268102 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:19.267944    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sddl4\" (UniqueName: \"kubernetes.io/projected/1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8-kube-api-access-sddl4\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv\" (UID: \"1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv"
Jul 28 10:55:19.268102 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:19.267962    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv\" (UID: \"1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv"
Jul 28 10:55:19.369400 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:19.369356    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv\" (UID: \"1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv"
Jul 28 10:55:19.369400 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:19.369402    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv\" (UID: \"1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv"
Jul 28 10:55:19.369685 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:19.369423    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv\" (UID: \"1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv"
Jul 28 10:55:19.369685 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:19.369440    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv\" (UID: \"1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv"
Jul 28 10:55:19.369685 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:19.369457    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sddl4\" (UniqueName: \"kubernetes.io/projected/1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8-kube-api-access-sddl4\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv\" (UID: \"1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv"
Jul 28 10:55:19.369685 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:19.369487    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv\" (UID: \"1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv"
Jul 28 10:55:19.369894 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:19.369874    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv\" (UID: \"1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv"
Jul 28 10:55:19.369933 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:19.369915    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv\" (UID: \"1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv"
Jul 28 10:55:19.369965 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:19.369934    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv\" (UID: \"1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv"
Jul 28 10:55:19.371837 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:19.371814    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv\" (UID: \"1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv"
Jul 28 10:55:19.372004 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:19.371987    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv\" (UID: \"1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv"
Jul 28 10:55:19.377778 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:19.377755    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sddl4\" (UniqueName: \"kubernetes.io/projected/1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8-kube-api-access-sddl4\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv\" (UID: \"1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv"
Jul 28 10:55:19.413679 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:19.413584    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche5mbz2"]
Jul 28 10:55:19.417052 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:19.417029    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche5mbz2"
Jul 28 10:55:19.419011 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:19.418991    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-cpxm2\""
Jul 28 10:55:19.425134 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:19.425101    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche5mbz2"]
Jul 28 10:55:19.571943 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:19.571910    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b32e5bc0-dc67-488e-9512-9e7bc49415d0-tls-certs\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche5mbz2\" (UID: \"b32e5bc0-dc67-488e-9512-9e7bc49415d0\") " pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche5mbz2"
Jul 28 10:55:19.572132 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:19.571959    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vpmt\" (UniqueName: \"kubernetes.io/projected/b32e5bc0-dc67-488e-9512-9e7bc49415d0-kube-api-access-5vpmt\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche5mbz2\" (UID: \"b32e5bc0-dc67-488e-9512-9e7bc49415d0\") " pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche5mbz2"
Jul 28 10:55:19.673454 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:19.673358    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b32e5bc0-dc67-488e-9512-9e7bc49415d0-tls-certs\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche5mbz2\" (UID: \"b32e5bc0-dc67-488e-9512-9e7bc49415d0\") " pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche5mbz2"
Jul 28 10:55:19.673454 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:19.673414    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vpmt\" (UniqueName: \"kubernetes.io/projected/b32e5bc0-dc67-488e-9512-9e7bc49415d0-kube-api-access-5vpmt\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche5mbz2\" (UID: \"b32e5bc0-dc67-488e-9512-9e7bc49415d0\") " pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche5mbz2"
Jul 28 10:55:19.675915 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:19.675893    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b32e5bc0-dc67-488e-9512-9e7bc49415d0-tls-certs\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche5mbz2\" (UID: \"b32e5bc0-dc67-488e-9512-9e7bc49415d0\") " pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche5mbz2"
Jul 28 10:55:19.681583 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:19.681556    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vpmt\" (UniqueName: \"kubernetes.io/projected/b32e5bc0-dc67-488e-9512-9e7bc49415d0-kube-api-access-5vpmt\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche5mbz2\" (UID: \"b32e5bc0-dc67-488e-9512-9e7bc49415d0\") " pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche5mbz2"
Jul 28 10:55:19.728614 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:19.728576    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche5mbz2"
Jul 28 10:55:19.864950 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:19.864923    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche5mbz2"]
Jul 28 10:55:19.866869 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:55:19.866840    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb32e5bc0_dc67_488e_9512_9e7bc49415d0.slice/crio-010d20e2fac07d36f64605e9968a863993f921a8a872943024d6bdde59476453 WatchSource:0}: Error finding container 010d20e2fac07d36f64605e9968a863993f921a8a872943024d6bdde59476453: Status 404 returned error can't find the container with id 010d20e2fac07d36f64605e9968a863993f921a8a872943024d6bdde59476453
Jul 28 10:55:20.571667 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:20.571625    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche5mbz2" event={"ID":"b32e5bc0-dc67-488e-9512-9e7bc49415d0","Type":"ContainerStarted","Data":"1bce47f7167c065cad94715e9b69de044901dc43db5a21ae522d6658ea3364d9"}
Jul 28 10:55:20.572033 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:20.571691    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche5mbz2"
Jul 28 10:55:20.572033 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:20.571716    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche5mbz2" event={"ID":"b32e5bc0-dc67-488e-9512-9e7bc49415d0","Type":"ContainerStarted","Data":"010d20e2fac07d36f64605e9968a863993f921a8a872943024d6bdde59476453"}
Jul 28 10:55:20.589344 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:20.589283    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche5mbz2" podStartSLOduration=1.5892671950000001 podStartE2EDuration="1.589267195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:55:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:55:20.58822662 +0000 UTC m=+2856.546995010" watchObservedRunningTime="2026-07-28 10:55:20.589267195 +0000 UTC m=+2856.548035574"
Jul 28 10:55:23.810026 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:23.809976    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-f219be45-903a-4603-991c-a0a969f33047\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-050ac456ca5f97bb8\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv\" (UID: \"1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv"
Jul 28 10:55:23.815225 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:23.815191    2557 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-f219be45-903a-4603-991c-a0a969f33047\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-050ac456ca5f97bb8\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv\" (UID: \"1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8\") device path: \"\"" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv"
Jul 28 10:55:23.911462 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:23.911405    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-f219be45-903a-4603-991c-a0a969f33047\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-050ac456ca5f97bb8\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv\" (UID: \"1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv"
Jul 28 10:55:23.911686 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:23.911552    2557 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-f219be45-903a-4603-991c-a0a969f33047\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-050ac456ca5f97bb8\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv\" (UID: \"1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8\") DevicePath \"\"" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv"
Jul 28 10:55:23.929930 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:23.929897    2557 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-f219be45-903a-4603-991c-a0a969f33047\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-050ac456ca5f97bb8\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv\" (UID: \"1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8\") DevicePath \"csi-8f70dca98acf5b82535b96ad7d8628a568697a1059f39f1d7d80b52c57c81302\"" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv"
Jul 28 10:55:24.018519 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:24.018485    2557 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-f219be45-903a-4603-991c-a0a969f33047\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-050ac456ca5f97bb8\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv\" (UID: \"1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/20ae7e64630c1825a727ff2aaf076c7ccab6d592608f1f4710e2b0623afb974f/globalmount\"" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv"
Jul 28 10:55:24.035332 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:24.035303    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-f219be45-903a-4603-991c-a0a969f33047\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-050ac456ca5f97bb8\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv\" (UID: \"1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv"
Jul 28 10:55:24.247486 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:24.247389    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv"
Jul 28 10:55:24.385491 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:24.385463    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv"]
Jul 28 10:55:24.386764 ip-10-0-131-88 kubenswrapper[2557]: W0728 10:55:24.386744    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a81ee1e_7b05_4d81_a30f_b69c71d2d9d8.slice/crio-8f0d8be6c0fee4b814e3b60d42d1f9a2e5407114eecffe0e2aa47197c5c7c707 WatchSource:0}: Error finding container 8f0d8be6c0fee4b814e3b60d42d1f9a2e5407114eecffe0e2aa47197c5c7c707: Status 404 returned error can't find the container with id 8f0d8be6c0fee4b814e3b60d42d1f9a2e5407114eecffe0e2aa47197c5c7c707
Jul 28 10:55:24.590632 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:24.590563    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv" event={"ID":"1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8","Type":"ContainerStarted","Data":"4192bf9d2ed5eae0c6ebd299775d86420492f7fedab99510fcc37def6129a8a4"}
Jul 28 10:55:24.590806 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:24.590766    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv"
Jul 28 10:55:24.590851 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:24.590813    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv" event={"ID":"1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8","Type":"ContainerStarted","Data":"8f0d8be6c0fee4b814e3b60d42d1f9a2e5407114eecffe0e2aa47197c5c7c707"}
Jul 28 10:55:25.598151 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:25.598113    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv" event={"ID":"1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8","Type":"ContainerStarted","Data":"63db77fcb53796bcad40f97f18680a91e18932b0ebb33c4d0ca7a640b473d15a"}
Jul 28 10:55:25.620623 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:25.620489    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv" podStartSLOduration=6.620472823 podStartE2EDuration="6.620472823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:55:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:55:25.618712128 +0000 UTC m=+2861.577480506" watchObservedRunningTime="2026-07-28 10:55:25.620472823 +0000 UTC m=+2861.579241202"
Jul 28 10:55:34.247796 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:34.247750    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv"
Jul 28 10:55:34.248262 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:34.247831    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv"
Jul 28 10:55:34.249182 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:34.249144    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv" podUID="1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8001/health\": dial tcp 10.133.0.62:8001: connect: connection refused"
Jul 28 10:55:34.265938 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:34.265909    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv"
Jul 28 10:55:44.248894 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:44.248781    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv" podUID="1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8001/health\": dial tcp 10.133.0.62:8001: connect: connection refused"
Jul 28 10:55:51.578450 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:51.578416    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche5mbz2"
Jul 28 10:55:54.248724 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:55:54.248678    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv" podUID="1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8001/health\": dial tcp 10.133.0.62:8001: connect: connection refused"
Jul 28 10:56:04.248534 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:56:04.248472    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv" podUID="1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8001/health\": dial tcp 10.133.0.62:8001: connect: connection refused"
Jul 28 10:56:14.248550 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:56:14.248500    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv" podUID="1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8001/health\": dial tcp 10.133.0.62:8001: connect: connection refused"
Jul 28 10:56:24.248834 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:56:24.248787    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv" podUID="1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8001/health\": dial tcp 10.133.0.62:8001: connect: connection refused"
Jul 28 10:56:34.248098 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:56:34.248038    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv" podUID="1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8001/health\": dial tcp 10.133.0.62:8001: connect: connection refused"
Jul 28 10:56:44.247904 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:56:44.247854    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv" podUID="1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8001/health\": dial tcp 10.133.0.62:8001: connect: connection refused"
Jul 28 10:56:54.247980 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:56:54.247933    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv" podUID="1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8001/health\": dial tcp 10.133.0.62:8001: connect: connection refused"
Jul 28 10:57:04.248581 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:57:04.248533    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv" podUID="1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8001/health\": dial tcp 10.133.0.62:8001: connect: connection refused"
Jul 28 10:57:14.248458 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:57:14.248339    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv" podUID="1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8001/health\": dial tcp 10.133.0.62:8001: connect: connection refused"
Jul 28 10:57:24.257946 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:57:24.257916    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv"
Jul 28 10:57:24.269923 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:57:24.269895    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv"
Jul 28 10:58:53.307845 ip-10-0-131-88 kubenswrapper[2557]: I0728 10:58:53.307763    2557 scope.go:117] "RemoveContainer" containerID="afbdfeefff6823b2c1acaec33ff8da5a866abcfd83a6b8b77f6e651a19166d6a"
Jul 28 11:05:47.172937 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:05:47.172904    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-pxkr7"]
Jul 28 11:05:47.176217 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:05:47.176192    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-pxkr7"
Jul 28 11:05:47.179992 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:05:47.179958    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"default-dockercfg-m7rhc\""
Jul 28 11:05:47.180148 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:05:47.179957    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"autoscale-hpa-lws-kserve-self-signed-certs\""
Jul 28 11:05:47.180148 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:05:47.180008    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"autoscale-hpa-lws-epp-sa-dockercfg-dst95\""
Jul 28 11:05:47.180148 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:05:47.179970    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"openshift-service-ca.crt\""
Jul 28 11:05:47.181001 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:05:47.180988    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"kube-root-ca.crt\""
Jul 28 11:05:47.182051 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:05:47.182031    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a86124d-1667-4e53-a120-4d926a3a322c-tls-certs\") pod \"autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-pxkr7\" (UID: \"2a86124d-1667-4e53-a120-4d926a3a322c\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-pxkr7"
Jul 28 11:05:47.182149 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:05:47.182058    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz97x\" (UniqueName: \"kubernetes.io/projected/2a86124d-1667-4e53-a120-4d926a3a322c-kube-api-access-bz97x\") pod \"autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-pxkr7\" (UID: \"2a86124d-1667-4e53-a120-4d926a3a322c\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-pxkr7"
Jul 28 11:05:47.189089 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:05:47.189066    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-pxkr7"]
Jul 28 11:05:47.282670 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:05:47.282631    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a86124d-1667-4e53-a120-4d926a3a322c-tls-certs\") pod \"autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-pxkr7\" (UID: \"2a86124d-1667-4e53-a120-4d926a3a322c\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-pxkr7"
Jul 28 11:05:47.282850 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:05:47.282679    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz97x\" (UniqueName: \"kubernetes.io/projected/2a86124d-1667-4e53-a120-4d926a3a322c-kube-api-access-bz97x\") pod \"autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-pxkr7\" (UID: \"2a86124d-1667-4e53-a120-4d926a3a322c\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-pxkr7"
Jul 28 11:05:47.285136 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:05:47.285109    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a86124d-1667-4e53-a120-4d926a3a322c-tls-certs\") pod \"autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-pxkr7\" (UID: \"2a86124d-1667-4e53-a120-4d926a3a322c\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-pxkr7"
Jul 28 11:05:47.291160 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:05:47.291132    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz97x\" (UniqueName: \"kubernetes.io/projected/2a86124d-1667-4e53-a120-4d926a3a322c-kube-api-access-bz97x\") pod \"autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-pxkr7\" (UID: \"2a86124d-1667-4e53-a120-4d926a3a322c\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-pxkr7"
Jul 28 11:05:47.487050 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:05:47.486967    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-pxkr7"
Jul 28 11:05:47.631486 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:05:47.631371    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-pxkr7"]
Jul 28 11:05:47.634423 ip-10-0-131-88 kubenswrapper[2557]: W0728 11:05:47.634390    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a86124d_1667_4e53_a120_4d926a3a322c.slice/crio-262e34b67541761ae9ac76f35203b4bd0bf6335e9fc73a280170330ca0af6572 WatchSource:0}: Error finding container 262e34b67541761ae9ac76f35203b4bd0bf6335e9fc73a280170330ca0af6572: Status 404 returned error can't find the container with id 262e34b67541761ae9ac76f35203b4bd0bf6335e9fc73a280170330ca0af6572
Jul 28 11:05:48.447988 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:05:48.447949    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-pxkr7" event={"ID":"2a86124d-1667-4e53-a120-4d926a3a322c","Type":"ContainerStarted","Data":"3b63242e6cee5e9db580b097b3f4b20036cb74f84473fdd9e60efebe789ade58"}
Jul 28 11:05:48.448389 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:05:48.448028    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-pxkr7" event={"ID":"2a86124d-1667-4e53-a120-4d926a3a322c","Type":"ContainerStarted","Data":"262e34b67541761ae9ac76f35203b4bd0bf6335e9fc73a280170330ca0af6572"}
Jul 28 11:05:48.448389 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:05:48.448065    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-pxkr7"
Jul 28 11:05:48.466389 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:05:48.466334    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-pxkr7" podStartSLOduration=1.466319377 podStartE2EDuration="1.466319377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:05:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:05:48.46442578 +0000 UTC m=+3484.423194159" watchObservedRunningTime="2026-07-28 11:05:48.466319377 +0000 UTC m=+3484.425087756"
Jul 28 11:06:19.455647 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:06:19.455595    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-pxkr7"
Jul 28 11:10:06.797936 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:10:06.797876    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-48639af5_llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv_1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8/llm-d-routing-sidecar/0.log"
Jul 28 11:10:06.809758 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:10:06.809709    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-48639af5_llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv_1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8/main/0.log"
Jul 28 11:10:06.826131 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:10:06.826098    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-48639af5_llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche5mbz2_b32e5bc0-dc67-488e-9512-9e7bc49415d0/main/0.log"
Jul 28 11:10:48.398418 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:10:48.398339    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 28 11:10:48.401816 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:10:48.401800    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 11:10:48.404559 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:10:48.404535    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"default-dockercfg-lfhh2\""
Jul 28 11:10:48.404726 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:10:48.404620    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"kube-root-ca.crt\""
Jul 28 11:10:48.404726 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:10:48.404627    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs\""
Jul 28 11:10:48.404726 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:10:48.404683    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-zzdrr\""
Jul 28 11:10:48.404951 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:10:48.404826    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"openshift-service-ca.crt\""
Jul 28 11:10:48.411796 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:10:48.411774    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 28 11:10:48.487749 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:10:48.487720    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/306cf4bf-a7df-4f67-ab1b-4e7547006c4a-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"306cf4bf-a7df-4f67-ab1b-4e7547006c4a\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 11:10:48.487918 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:10:48.487766    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/306cf4bf-a7df-4f67-ab1b-4e7547006c4a-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"306cf4bf-a7df-4f67-ab1b-4e7547006c4a\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 11:10:48.487918 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:10:48.487800    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/306cf4bf-a7df-4f67-ab1b-4e7547006c4a-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"306cf4bf-a7df-4f67-ab1b-4e7547006c4a\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 11:10:48.487918 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:10:48.487825    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/306cf4bf-a7df-4f67-ab1b-4e7547006c4a-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"306cf4bf-a7df-4f67-ab1b-4e7547006c4a\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 11:10:48.487918 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:10:48.487862    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/306cf4bf-a7df-4f67-ab1b-4e7547006c4a-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"306cf4bf-a7df-4f67-ab1b-4e7547006c4a\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 11:10:48.487918 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:10:48.487878    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwqtj\" (UniqueName: \"kubernetes.io/projected/306cf4bf-a7df-4f67-ab1b-4e7547006c4a-kube-api-access-vwqtj\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"306cf4bf-a7df-4f67-ab1b-4e7547006c4a\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 11:10:48.588858 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:10:48.588829    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/306cf4bf-a7df-4f67-ab1b-4e7547006c4a-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"306cf4bf-a7df-4f67-ab1b-4e7547006c4a\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 11:10:48.589048 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:10:48.588864    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwqtj\" (UniqueName: \"kubernetes.io/projected/306cf4bf-a7df-4f67-ab1b-4e7547006c4a-kube-api-access-vwqtj\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"306cf4bf-a7df-4f67-ab1b-4e7547006c4a\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 11:10:48.589048 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:10:48.588917    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/306cf4bf-a7df-4f67-ab1b-4e7547006c4a-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"306cf4bf-a7df-4f67-ab1b-4e7547006c4a\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 11:10:48.589048 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:10:48.588948    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/306cf4bf-a7df-4f67-ab1b-4e7547006c4a-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"306cf4bf-a7df-4f67-ab1b-4e7547006c4a\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 11:10:48.589226 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:10:48.589073    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/306cf4bf-a7df-4f67-ab1b-4e7547006c4a-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"306cf4bf-a7df-4f67-ab1b-4e7547006c4a\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 11:10:48.589226 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:10:48.589210    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/306cf4bf-a7df-4f67-ab1b-4e7547006c4a-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"306cf4bf-a7df-4f67-ab1b-4e7547006c4a\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 11:10:48.589381 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:10:48.589361    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/306cf4bf-a7df-4f67-ab1b-4e7547006c4a-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"306cf4bf-a7df-4f67-ab1b-4e7547006c4a\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 11:10:48.589443 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:10:48.589413    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/306cf4bf-a7df-4f67-ab1b-4e7547006c4a-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"306cf4bf-a7df-4f67-ab1b-4e7547006c4a\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 11:10:48.589524 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:10:48.589504    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/306cf4bf-a7df-4f67-ab1b-4e7547006c4a-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"306cf4bf-a7df-4f67-ab1b-4e7547006c4a\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 11:10:48.591139 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:10:48.591117    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/306cf4bf-a7df-4f67-ab1b-4e7547006c4a-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"306cf4bf-a7df-4f67-ab1b-4e7547006c4a\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 11:10:48.591340 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:10:48.591322    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/306cf4bf-a7df-4f67-ab1b-4e7547006c4a-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"306cf4bf-a7df-4f67-ab1b-4e7547006c4a\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 11:10:48.596281 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:10:48.596259    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwqtj\" (UniqueName: \"kubernetes.io/projected/306cf4bf-a7df-4f67-ab1b-4e7547006c4a-kube-api-access-vwqtj\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"306cf4bf-a7df-4f67-ab1b-4e7547006c4a\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 11:10:53.738640 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:10:53.738584    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-22e6f62e-463f-4729-96e2-d0f264811ec3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-021b25aa65f0f0357\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"306cf4bf-a7df-4f67-ab1b-4e7547006c4a\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 11:10:53.743442 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:10:53.743402    2557 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-22e6f62e-463f-4729-96e2-d0f264811ec3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-021b25aa65f0f0357\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"306cf4bf-a7df-4f67-ab1b-4e7547006c4a\") device path: \"\"" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 11:10:53.839541 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:10:53.839507    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-22e6f62e-463f-4729-96e2-d0f264811ec3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-021b25aa65f0f0357\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"306cf4bf-a7df-4f67-ab1b-4e7547006c4a\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 11:10:53.839759 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:10:53.839702    2557 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-22e6f62e-463f-4729-96e2-d0f264811ec3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-021b25aa65f0f0357\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"306cf4bf-a7df-4f67-ab1b-4e7547006c4a\") DevicePath \"\"" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 11:10:53.842244 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:10:53.842222    2557 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-22e6f62e-463f-4729-96e2-d0f264811ec3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-021b25aa65f0f0357\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"306cf4bf-a7df-4f67-ab1b-4e7547006c4a\") DevicePath \"csi-381c9c3d7dd4736bd08fc89624097dc8a411d3eaef78066e938565a7f68d80a4\"" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 11:10:53.932049 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:10:53.932016    2557 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-22e6f62e-463f-4729-96e2-d0f264811ec3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-021b25aa65f0f0357\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"306cf4bf-a7df-4f67-ab1b-4e7547006c4a\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/946f308e63c2899cb5f4f1333e39d10efbf22eb79d52dcb6acf9510bc19079c3/globalmount\"" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 11:10:53.943224 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:10:53.943198    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-22e6f62e-463f-4729-96e2-d0f264811ec3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-021b25aa65f0f0357\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"306cf4bf-a7df-4f67-ab1b-4e7547006c4a\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 11:10:54.113864 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:10:54.113829    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 11:10:54.252163 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:10:54.252135    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 28 11:10:54.254700 ip-10-0-131-88 kubenswrapper[2557]: W0728 11:10:54.254669    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod306cf4bf_a7df_4f67_ab1b_4e7547006c4a.slice/crio-b702073c5a22a3dca1fb91267a956ecda6a83719cb42e62b38757691cc94e933 WatchSource:0}: Error finding container b702073c5a22a3dca1fb91267a956ecda6a83719cb42e62b38757691cc94e933: Status 404 returned error can't find the container with id b702073c5a22a3dca1fb91267a956ecda6a83719cb42e62b38757691cc94e933
Jul 28 11:10:54.852902 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:10:54.852858    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" event={"ID":"306cf4bf-a7df-4f67-ab1b-4e7547006c4a","Type":"ContainerStarted","Data":"e1d6685281b99bbc2aba88fb68908918e6b820d89aa81fe9c64fd8c707d99210"}
Jul 28 11:10:54.853297 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:10:54.852924    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" event={"ID":"306cf4bf-a7df-4f67-ab1b-4e7547006c4a","Type":"ContainerStarted","Data":"b702073c5a22a3dca1fb91267a956ecda6a83719cb42e62b38757691cc94e933"}
Jul 28 11:10:54.869730 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:10:54.869673    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podStartSLOduration=6.869656514 podStartE2EDuration="6.869656514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 11:10:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:10:54.869317368 +0000 UTC m=+3790.828085758" watchObservedRunningTime="2026-07-28 11:10:54.869656514 +0000 UTC m=+3790.828424893"
Jul 28 11:20:42.023374 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:20:42.023332    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-pxkr7"]
Jul 28 11:20:49.455710 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:20:49.455667    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-pxkr7" podUID="2a86124d-1667-4e53-a120-4d926a3a322c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 11:20:57.043079 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:20:57.043027    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-pxkr7" podUID="2a86124d-1667-4e53-a120-4d926a3a322c" containerName="main" containerID="cri-o://3b63242e6cee5e9db580b097b3f4b20036cb74f84473fdd9e60efebe789ade58" gracePeriod=45
Jul 28 11:20:57.297863 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:20:57.297795    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-pxkr7"
Jul 28 11:20:57.354796 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:20:57.354758    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bz97x\" (UniqueName: \"kubernetes.io/projected/2a86124d-1667-4e53-a120-4d926a3a322c-kube-api-access-bz97x\") pod \"2a86124d-1667-4e53-a120-4d926a3a322c\" (UID: \"2a86124d-1667-4e53-a120-4d926a3a322c\") "
Jul 28 11:20:57.354982 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:20:57.354837    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a86124d-1667-4e53-a120-4d926a3a322c-tls-certs\") pod \"2a86124d-1667-4e53-a120-4d926a3a322c\" (UID: \"2a86124d-1667-4e53-a120-4d926a3a322c\") "
Jul 28 11:20:57.356995 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:20:57.356967    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a86124d-1667-4e53-a120-4d926a3a322c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2a86124d-1667-4e53-a120-4d926a3a322c" (UID: "2a86124d-1667-4e53-a120-4d926a3a322c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:20:57.357135 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:20:57.357114    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a86124d-1667-4e53-a120-4d926a3a322c-kube-api-access-bz97x" (OuterVolumeSpecName: "kube-api-access-bz97x") pod "2a86124d-1667-4e53-a120-4d926a3a322c" (UID: "2a86124d-1667-4e53-a120-4d926a3a322c"). InnerVolumeSpecName "kube-api-access-bz97x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:20:57.456307 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:20:57.456270    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz97x\" (UniqueName: \"kubernetes.io/projected/2a86124d-1667-4e53-a120-4d926a3a322c-kube-api-access-bz97x\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 11:20:57.456307 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:20:57.456299    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a86124d-1667-4e53-a120-4d926a3a322c-tls-certs\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 11:20:57.752124 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:20:57.752090    2557 generic.go:356] "Generic (PLEG): container finished" podID="2a86124d-1667-4e53-a120-4d926a3a322c" containerID="3b63242e6cee5e9db580b097b3f4b20036cb74f84473fdd9e60efebe789ade58" exitCode=0
Jul 28 11:20:57.752290 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:20:57.752135    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-pxkr7"
Jul 28 11:20:57.752290 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:20:57.752178    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-pxkr7" event={"ID":"2a86124d-1667-4e53-a120-4d926a3a322c","Type":"ContainerDied","Data":"3b63242e6cee5e9db580b097b3f4b20036cb74f84473fdd9e60efebe789ade58"}
Jul 28 11:20:57.752290 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:20:57.752225    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-pxkr7" event={"ID":"2a86124d-1667-4e53-a120-4d926a3a322c","Type":"ContainerDied","Data":"262e34b67541761ae9ac76f35203b4bd0bf6335e9fc73a280170330ca0af6572"}
Jul 28 11:20:57.752290 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:20:57.752244    2557 scope.go:117] "RemoveContainer" containerID="3b63242e6cee5e9db580b097b3f4b20036cb74f84473fdd9e60efebe789ade58"
Jul 28 11:20:57.762384 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:20:57.762363    2557 scope.go:117] "RemoveContainer" containerID="3b63242e6cee5e9db580b097b3f4b20036cb74f84473fdd9e60efebe789ade58"
Jul 28 11:20:57.762736 ip-10-0-131-88 kubenswrapper[2557]: E0728 11:20:57.762705    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b63242e6cee5e9db580b097b3f4b20036cb74f84473fdd9e60efebe789ade58\": container with ID starting with 3b63242e6cee5e9db580b097b3f4b20036cb74f84473fdd9e60efebe789ade58 not found: ID does not exist" containerID="3b63242e6cee5e9db580b097b3f4b20036cb74f84473fdd9e60efebe789ade58"
Jul 28 11:20:57.762791 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:20:57.762748    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b63242e6cee5e9db580b097b3f4b20036cb74f84473fdd9e60efebe789ade58"} err="failed to get container status \"3b63242e6cee5e9db580b097b3f4b20036cb74f84473fdd9e60efebe789ade58\": rpc error: code = NotFound desc = could not find container \"3b63242e6cee5e9db580b097b3f4b20036cb74f84473fdd9e60efebe789ade58\": container with ID starting with 3b63242e6cee5e9db580b097b3f4b20036cb74f84473fdd9e60efebe789ade58 not found: ID does not exist"
Jul 28 11:20:57.772689 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:20:57.772665    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-pxkr7"]
Jul 28 11:20:57.777258 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:20:57.777235    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-pxkr7"]
Jul 28 11:20:58.580115 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:20:58.580073    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a86124d-1667-4e53-a120-4d926a3a322c" path="/var/lib/kubelet/pods/2a86124d-1667-4e53-a120-4d926a3a322c/volumes"
Jul 28 11:25:31.126316 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:25:31.126282    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7c3633d1_llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1_306cf4bf-a7df-4f67-ab1b-4e7547006c4a/main/0.log"
Jul 28 11:26:34.607469 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:26:34.607432    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-kzspb"]
Jul 28 11:26:34.610227 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:26:34.607952    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a86124d-1667-4e53-a120-4d926a3a322c" containerName="main"
Jul 28 11:26:34.610227 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:26:34.607969    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a86124d-1667-4e53-a120-4d926a3a322c" containerName="main"
Jul 28 11:26:34.610227 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:26:34.608038    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a86124d-1667-4e53-a120-4d926a3a322c" containerName="main"
Jul 28 11:26:34.611584 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:26:34.611562    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-kzspb"
Jul 28 11:26:34.613818 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:26:34.613790    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Jul 28 11:26:34.613987 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:26:34.613842    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-65z7g\""
Jul 28 11:26:34.613987 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:26:34.613880    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 28 11:26:34.613987 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:26:34.613905    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Jul 28 11:26:34.614138 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:26:34.614044    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Jul 28 11:26:34.632979 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:26:34.632942    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-kzspb"]
Jul 28 11:26:34.733831 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:26:34.733790    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/393582ac-4637-44b4-b566-dc5a84575ed6-tls-certs\") pod \"stop-feature-test-kserve-5cff5d4976-kzspb\" (UID: \"393582ac-4637-44b4-b566-dc5a84575ed6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-kzspb"
Jul 28 11:26:34.734040 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:26:34.733843    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/393582ac-4637-44b4-b566-dc5a84575ed6-model-cache\") pod \"stop-feature-test-kserve-5cff5d4976-kzspb\" (UID: \"393582ac-4637-44b4-b566-dc5a84575ed6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-kzspb"
Jul 28 11:26:34.734040 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:26:34.733874    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/393582ac-4637-44b4-b566-dc5a84575ed6-home\") pod \"stop-feature-test-kserve-5cff5d4976-kzspb\" (UID: \"393582ac-4637-44b4-b566-dc5a84575ed6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-kzspb"
Jul 28 11:26:34.734040 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:26:34.733942    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bm4p\" (UniqueName: \"kubernetes.io/projected/393582ac-4637-44b4-b566-dc5a84575ed6-kube-api-access-5bm4p\") pod \"stop-feature-test-kserve-5cff5d4976-kzspb\" (UID: \"393582ac-4637-44b4-b566-dc5a84575ed6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-kzspb"
Jul 28 11:26:34.734040 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:26:34.733998    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/393582ac-4637-44b4-b566-dc5a84575ed6-dshm\") pod \"stop-feature-test-kserve-5cff5d4976-kzspb\" (UID: \"393582ac-4637-44b4-b566-dc5a84575ed6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-kzspb"
Jul 28 11:26:34.734198 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:26:34.734110    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/393582ac-4637-44b4-b566-dc5a84575ed6-kserve-provision-location\") pod \"stop-feature-test-kserve-5cff5d4976-kzspb\" (UID: \"393582ac-4637-44b4-b566-dc5a84575ed6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-kzspb"
Jul 28 11:26:34.734198 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:26:34.734164    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/393582ac-4637-44b4-b566-dc5a84575ed6-tmp-dir\") pod \"stop-feature-test-kserve-5cff5d4976-kzspb\" (UID: \"393582ac-4637-44b4-b566-dc5a84575ed6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-kzspb"
Jul 28 11:26:34.835299 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:26:34.835263    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/393582ac-4637-44b4-b566-dc5a84575ed6-kserve-provision-location\") pod \"stop-feature-test-kserve-5cff5d4976-kzspb\" (UID: \"393582ac-4637-44b4-b566-dc5a84575ed6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-kzspb"
Jul 28 11:26:34.835487 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:26:34.835311    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/393582ac-4637-44b4-b566-dc5a84575ed6-tmp-dir\") pod \"stop-feature-test-kserve-5cff5d4976-kzspb\" (UID: \"393582ac-4637-44b4-b566-dc5a84575ed6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-kzspb"
Jul 28 11:26:34.835487 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:26:34.835342    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/393582ac-4637-44b4-b566-dc5a84575ed6-tls-certs\") pod \"stop-feature-test-kserve-5cff5d4976-kzspb\" (UID: \"393582ac-4637-44b4-b566-dc5a84575ed6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-kzspb"
Jul 28 11:26:34.835487 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:26:34.835370    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/393582ac-4637-44b4-b566-dc5a84575ed6-model-cache\") pod \"stop-feature-test-kserve-5cff5d4976-kzspb\" (UID: \"393582ac-4637-44b4-b566-dc5a84575ed6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-kzspb"
Jul 28 11:26:34.835487 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:26:34.835391    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/393582ac-4637-44b4-b566-dc5a84575ed6-home\") pod \"stop-feature-test-kserve-5cff5d4976-kzspb\" (UID: \"393582ac-4637-44b4-b566-dc5a84575ed6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-kzspb"
Jul 28 11:26:34.835487 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:26:34.835414    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bm4p\" (UniqueName: \"kubernetes.io/projected/393582ac-4637-44b4-b566-dc5a84575ed6-kube-api-access-5bm4p\") pod \"stop-feature-test-kserve-5cff5d4976-kzspb\" (UID: \"393582ac-4637-44b4-b566-dc5a84575ed6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-kzspb"
Jul 28 11:26:34.835487 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:26:34.835451    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/393582ac-4637-44b4-b566-dc5a84575ed6-dshm\") pod \"stop-feature-test-kserve-5cff5d4976-kzspb\" (UID: \"393582ac-4637-44b4-b566-dc5a84575ed6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-kzspb"
Jul 28 11:26:34.835812 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:26:34.835726    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/393582ac-4637-44b4-b566-dc5a84575ed6-kserve-provision-location\") pod \"stop-feature-test-kserve-5cff5d4976-kzspb\" (UID: \"393582ac-4637-44b4-b566-dc5a84575ed6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-kzspb"
Jul 28 11:26:34.835812 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:26:34.835789    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/393582ac-4637-44b4-b566-dc5a84575ed6-home\") pod \"stop-feature-test-kserve-5cff5d4976-kzspb\" (UID: \"393582ac-4637-44b4-b566-dc5a84575ed6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-kzspb"
Jul 28 11:26:34.835928 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:26:34.835833    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/393582ac-4637-44b4-b566-dc5a84575ed6-model-cache\") pod \"stop-feature-test-kserve-5cff5d4976-kzspb\" (UID: \"393582ac-4637-44b4-b566-dc5a84575ed6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-kzspb"
Jul 28 11:26:34.836029 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:26:34.836010    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/393582ac-4637-44b4-b566-dc5a84575ed6-tmp-dir\") pod \"stop-feature-test-kserve-5cff5d4976-kzspb\" (UID: \"393582ac-4637-44b4-b566-dc5a84575ed6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-kzspb"
Jul 28 11:26:34.837727 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:26:34.837706    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/393582ac-4637-44b4-b566-dc5a84575ed6-dshm\") pod \"stop-feature-test-kserve-5cff5d4976-kzspb\" (UID: \"393582ac-4637-44b4-b566-dc5a84575ed6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-kzspb"
Jul 28 11:26:34.837915 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:26:34.837899    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/393582ac-4637-44b4-b566-dc5a84575ed6-tls-certs\") pod \"stop-feature-test-kserve-5cff5d4976-kzspb\" (UID: \"393582ac-4637-44b4-b566-dc5a84575ed6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-kzspb"
Jul 28 11:26:34.843512 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:26:34.843489    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bm4p\" (UniqueName: \"kubernetes.io/projected/393582ac-4637-44b4-b566-dc5a84575ed6-kube-api-access-5bm4p\") pod \"stop-feature-test-kserve-5cff5d4976-kzspb\" (UID: \"393582ac-4637-44b4-b566-dc5a84575ed6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-kzspb"
Jul 28 11:26:34.922329 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:26:34.922224    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-kzspb"
Jul 28 11:26:35.068274 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:26:35.068247    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-kzspb"]
Jul 28 11:26:35.070435 ip-10-0-131-88 kubenswrapper[2557]: W0728 11:26:35.070404    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod393582ac_4637_44b4_b566_dc5a84575ed6.slice/crio-be3592ad61ba4d0b22f35622b859b228c82ac3616fc0a86c7488b5f1e6fc5f34 WatchSource:0}: Error finding container be3592ad61ba4d0b22f35622b859b228c82ac3616fc0a86c7488b5f1e6fc5f34: Status 404 returned error can't find the container with id be3592ad61ba4d0b22f35622b859b228c82ac3616fc0a86c7488b5f1e6fc5f34
Jul 28 11:26:35.325764 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:26:35.325720    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-kzspb" event={"ID":"393582ac-4637-44b4-b566-dc5a84575ed6","Type":"ContainerStarted","Data":"22f871a2055afb908c9c3148b23dfe161fe43f8c1e2caa4d761aeb0df74cfe96"}
Jul 28 11:26:35.325952 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:26:35.325790    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-kzspb" event={"ID":"393582ac-4637-44b4-b566-dc5a84575ed6","Type":"ContainerStarted","Data":"be3592ad61ba4d0b22f35622b859b228c82ac3616fc0a86c7488b5f1e6fc5f34"}
Jul 28 11:26:40.353077 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:26:40.353039    2557 generic.go:356] "Generic (PLEG): container finished" podID="393582ac-4637-44b4-b566-dc5a84575ed6" containerID="22f871a2055afb908c9c3148b23dfe161fe43f8c1e2caa4d761aeb0df74cfe96" exitCode=0
Jul 28 11:26:40.353557 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:26:40.353116    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-kzspb" event={"ID":"393582ac-4637-44b4-b566-dc5a84575ed6","Type":"ContainerDied","Data":"22f871a2055afb908c9c3148b23dfe161fe43f8c1e2caa4d761aeb0df74cfe96"}
Jul 28 11:26:41.361315 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:26:41.361280    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-kzspb" event={"ID":"393582ac-4637-44b4-b566-dc5a84575ed6","Type":"ContainerStarted","Data":"78ee09eab874e754bae2e13ed55df2e0108c5c13706ff32c500c152eb0d79ddb"}
Jul 28 11:26:41.384835 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:26:41.384773    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-kzspb" podStartSLOduration=2.797962181 podStartE2EDuration="7.384755773s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.586793592s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:26:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:26:41.383141773 +0000 UTC m=+4737.341910152" watchObservedRunningTime="2026-07-28 11:26:41.384755773 +0000 UTC m=+4737.343524152"
Jul 28 11:26:41.922677 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:26:41.922639    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-kzspb"
Jul 28 11:26:44.922843 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:26:44.922750    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-kzspb"
Jul 28 11:26:44.924399 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:26:44.924370    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-kzspb" podUID="393582ac-4637-44b4-b566-dc5a84575ed6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Jul 28 11:26:54.923547 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:26:54.923498    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-kzspb" podUID="393582ac-4637-44b4-b566-dc5a84575ed6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Jul 28 11:27:04.922914 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:27:04.922868    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-kzspb" podUID="393582ac-4637-44b4-b566-dc5a84575ed6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Jul 28 11:27:14.922757 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:27:14.922660    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-kzspb" podUID="393582ac-4637-44b4-b566-dc5a84575ed6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Jul 28 11:27:24.923668 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:27:24.923622    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-kzspb" podUID="393582ac-4637-44b4-b566-dc5a84575ed6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Jul 28 11:27:34.922949 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:27:34.922905    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-kzspb" podUID="393582ac-4637-44b4-b566-dc5a84575ed6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Jul 28 11:27:44.923556 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:27:44.923460    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-kzspb" podUID="393582ac-4637-44b4-b566-dc5a84575ed6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Jul 28 11:27:54.923149 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:27:54.923107    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-kzspb" podUID="393582ac-4637-44b4-b566-dc5a84575ed6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Jul 28 11:28:04.923462 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:28:04.923400    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-kzspb" podUID="393582ac-4637-44b4-b566-dc5a84575ed6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Jul 28 11:28:14.922830 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:28:14.922773    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-kzspb" podUID="393582ac-4637-44b4-b566-dc5a84575ed6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Jul 28 11:28:24.922715 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:28:24.922669    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-kzspb" podUID="393582ac-4637-44b4-b566-dc5a84575ed6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Jul 28 11:28:34.922756 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:28:34.922713    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-kzspb" podUID="393582ac-4637-44b4-b566-dc5a84575ed6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Jul 28 11:28:44.933244 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:28:44.933164    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-kzspb"
Jul 28 11:28:44.940803 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:28:44.940777    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-kzspb"
Jul 28 11:29:27.567250 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:27.567213    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-kzspb"]
Jul 28 11:29:42.590323 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:42.590284    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-kzspb" podUID="393582ac-4637-44b4-b566-dc5a84575ed6" containerName="main" containerID="cri-o://78ee09eab874e754bae2e13ed55df2e0108c5c13706ff32c500c152eb0d79ddb" gracePeriod=45
Jul 28 11:29:42.934680 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:42.934562    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-kzspb" podUID="393582ac-4637-44b4-b566-dc5a84575ed6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Jul 28 11:29:43.272062 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:43.272023    2557 generic.go:356] "Generic (PLEG): container finished" podID="393582ac-4637-44b4-b566-dc5a84575ed6" containerID="78ee09eab874e754bae2e13ed55df2e0108c5c13706ff32c500c152eb0d79ddb" exitCode=0
Jul 28 11:29:43.272222 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:43.272089    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-kzspb" event={"ID":"393582ac-4637-44b4-b566-dc5a84575ed6","Type":"ContainerDied","Data":"78ee09eab874e754bae2e13ed55df2e0108c5c13706ff32c500c152eb0d79ddb"}
Jul 28 11:29:43.352957 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:43.352929    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-kzspb"
Jul 28 11:29:43.435240 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:43.435205    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/393582ac-4637-44b4-b566-dc5a84575ed6-dshm\") pod \"393582ac-4637-44b4-b566-dc5a84575ed6\" (UID: \"393582ac-4637-44b4-b566-dc5a84575ed6\") "
Jul 28 11:29:43.435428 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:43.435269    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/393582ac-4637-44b4-b566-dc5a84575ed6-kserve-provision-location\") pod \"393582ac-4637-44b4-b566-dc5a84575ed6\" (UID: \"393582ac-4637-44b4-b566-dc5a84575ed6\") "
Jul 28 11:29:43.435428 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:43.435313    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/393582ac-4637-44b4-b566-dc5a84575ed6-home\") pod \"393582ac-4637-44b4-b566-dc5a84575ed6\" (UID: \"393582ac-4637-44b4-b566-dc5a84575ed6\") "
Jul 28 11:29:43.435428 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:43.435350    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/393582ac-4637-44b4-b566-dc5a84575ed6-tmp-dir\") pod \"393582ac-4637-44b4-b566-dc5a84575ed6\" (UID: \"393582ac-4637-44b4-b566-dc5a84575ed6\") "
Jul 28 11:29:43.435428 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:43.435378    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/393582ac-4637-44b4-b566-dc5a84575ed6-tls-certs\") pod \"393582ac-4637-44b4-b566-dc5a84575ed6\" (UID: \"393582ac-4637-44b4-b566-dc5a84575ed6\") "
Jul 28 11:29:43.435676 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:43.435461    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/393582ac-4637-44b4-b566-dc5a84575ed6-model-cache\") pod \"393582ac-4637-44b4-b566-dc5a84575ed6\" (UID: \"393582ac-4637-44b4-b566-dc5a84575ed6\") "
Jul 28 11:29:43.435676 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:43.435486    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5bm4p\" (UniqueName: \"kubernetes.io/projected/393582ac-4637-44b4-b566-dc5a84575ed6-kube-api-access-5bm4p\") pod \"393582ac-4637-44b4-b566-dc5a84575ed6\" (UID: \"393582ac-4637-44b4-b566-dc5a84575ed6\") "
Jul 28 11:29:43.435861 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:43.435833    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/393582ac-4637-44b4-b566-dc5a84575ed6-model-cache" (OuterVolumeSpecName: "model-cache") pod "393582ac-4637-44b4-b566-dc5a84575ed6" (UID: "393582ac-4637-44b4-b566-dc5a84575ed6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:29:43.436058 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:43.436021    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/393582ac-4637-44b4-b566-dc5a84575ed6-home" (OuterVolumeSpecName: "home") pod "393582ac-4637-44b4-b566-dc5a84575ed6" (UID: "393582ac-4637-44b4-b566-dc5a84575ed6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:29:43.437753 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:43.437724    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/393582ac-4637-44b4-b566-dc5a84575ed6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "393582ac-4637-44b4-b566-dc5a84575ed6" (UID: "393582ac-4637-44b4-b566-dc5a84575ed6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:29:43.437860 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:43.437833    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/393582ac-4637-44b4-b566-dc5a84575ed6-dshm" (OuterVolumeSpecName: "dshm") pod "393582ac-4637-44b4-b566-dc5a84575ed6" (UID: "393582ac-4637-44b4-b566-dc5a84575ed6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:29:43.437943 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:43.437912    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/393582ac-4637-44b4-b566-dc5a84575ed6-kube-api-access-5bm4p" (OuterVolumeSpecName: "kube-api-access-5bm4p") pod "393582ac-4637-44b4-b566-dc5a84575ed6" (UID: "393582ac-4637-44b4-b566-dc5a84575ed6"). InnerVolumeSpecName "kube-api-access-5bm4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:29:43.448162 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:43.448131    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/393582ac-4637-44b4-b566-dc5a84575ed6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "393582ac-4637-44b4-b566-dc5a84575ed6" (UID: "393582ac-4637-44b4-b566-dc5a84575ed6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:29:43.465805 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:43.465744    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/393582ac-4637-44b4-b566-dc5a84575ed6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "393582ac-4637-44b4-b566-dc5a84575ed6" (UID: "393582ac-4637-44b4-b566-dc5a84575ed6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:29:43.536789 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:43.536756    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/393582ac-4637-44b4-b566-dc5a84575ed6-model-cache\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 11:29:43.536789 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:43.536786    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bm4p\" (UniqueName: \"kubernetes.io/projected/393582ac-4637-44b4-b566-dc5a84575ed6-kube-api-access-5bm4p\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 11:29:43.536789 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:43.536796    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/393582ac-4637-44b4-b566-dc5a84575ed6-dshm\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 11:29:43.537026 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:43.536806    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/393582ac-4637-44b4-b566-dc5a84575ed6-kserve-provision-location\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 11:29:43.537026 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:43.536817    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/393582ac-4637-44b4-b566-dc5a84575ed6-home\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 11:29:43.537026 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:43.536826    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/393582ac-4637-44b4-b566-dc5a84575ed6-tmp-dir\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 11:29:43.537026 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:43.536833    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/393582ac-4637-44b4-b566-dc5a84575ed6-tls-certs\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 11:29:44.278240 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:44.278208    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-kzspb" event={"ID":"393582ac-4637-44b4-b566-dc5a84575ed6","Type":"ContainerDied","Data":"be3592ad61ba4d0b22f35622b859b228c82ac3616fc0a86c7488b5f1e6fc5f34"}
Jul 28 11:29:44.278240 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:44.278227    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-kzspb"
Jul 28 11:29:44.278811 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:44.278266    2557 scope.go:117] "RemoveContainer" containerID="78ee09eab874e754bae2e13ed55df2e0108c5c13706ff32c500c152eb0d79ddb"
Jul 28 11:29:44.287251 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:44.287230    2557 scope.go:117] "RemoveContainer" containerID="22f871a2055afb908c9c3148b23dfe161fe43f8c1e2caa4d761aeb0df74cfe96"
Jul 28 11:29:44.299926 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:44.299902    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-kzspb"]
Jul 28 11:29:44.304055 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:44.304033    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-kzspb"]
Jul 28 11:29:44.581082 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:44.581004    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="393582ac-4637-44b4-b566-dc5a84575ed6" path="/var/lib/kubelet/pods/393582ac-4637-44b4-b566-dc5a84575ed6/volumes"
Jul 28 11:29:51.607691 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:51.607648    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-92mnh"]
Jul 28 11:29:51.608196 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:51.608136    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="393582ac-4637-44b4-b566-dc5a84575ed6" containerName="main"
Jul 28 11:29:51.608196 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:51.608152    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="393582ac-4637-44b4-b566-dc5a84575ed6" containerName="main"
Jul 28 11:29:51.608196 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:51.608173    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="393582ac-4637-44b4-b566-dc5a84575ed6" containerName="storage-initializer"
Jul 28 11:29:51.608196 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:51.608183    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="393582ac-4637-44b4-b566-dc5a84575ed6" containerName="storage-initializer"
Jul 28 11:29:51.608341 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:51.608323    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="393582ac-4637-44b4-b566-dc5a84575ed6" containerName="main"
Jul 28 11:29:51.610844 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:51.610823    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-92mnh"
Jul 28 11:29:51.612921 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:51.612894    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Jul 28 11:29:51.613059 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:51.613023    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-65z7g\""
Jul 28 11:29:51.613135 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:51.613065    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Jul 28 11:29:51.613425 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:51.613403    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Jul 28 11:29:51.613529 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:51.613428    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 28 11:29:51.621533 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:51.621506    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-92mnh"]
Jul 28 11:29:51.712722 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:51.712689    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54wl4\" (UniqueName: \"kubernetes.io/projected/a4c659b9-8707-4169-8efc-f56285ea3c52-kube-api-access-54wl4\") pod \"stop-feature-test-kserve-5cff5d4976-92mnh\" (UID: \"a4c659b9-8707-4169-8efc-f56285ea3c52\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-92mnh"
Jul 28 11:29:51.712891 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:51.712785    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4c659b9-8707-4169-8efc-f56285ea3c52-tls-certs\") pod \"stop-feature-test-kserve-5cff5d4976-92mnh\" (UID: \"a4c659b9-8707-4169-8efc-f56285ea3c52\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-92mnh"
Jul 28 11:29:51.712891 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:51.712817    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4c659b9-8707-4169-8efc-f56285ea3c52-home\") pod \"stop-feature-test-kserve-5cff5d4976-92mnh\" (UID: \"a4c659b9-8707-4169-8efc-f56285ea3c52\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-92mnh"
Jul 28 11:29:51.712891 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:51.712836    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4c659b9-8707-4169-8efc-f56285ea3c52-dshm\") pod \"stop-feature-test-kserve-5cff5d4976-92mnh\" (UID: \"a4c659b9-8707-4169-8efc-f56285ea3c52\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-92mnh"
Jul 28 11:29:51.712891 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:51.712863    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4c659b9-8707-4169-8efc-f56285ea3c52-kserve-provision-location\") pod \"stop-feature-test-kserve-5cff5d4976-92mnh\" (UID: \"a4c659b9-8707-4169-8efc-f56285ea3c52\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-92mnh"
Jul 28 11:29:51.713050 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:51.713004    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4c659b9-8707-4169-8efc-f56285ea3c52-model-cache\") pod \"stop-feature-test-kserve-5cff5d4976-92mnh\" (UID: \"a4c659b9-8707-4169-8efc-f56285ea3c52\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-92mnh"
Jul 28 11:29:51.713050 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:51.713032    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4c659b9-8707-4169-8efc-f56285ea3c52-tmp-dir\") pod \"stop-feature-test-kserve-5cff5d4976-92mnh\" (UID: \"a4c659b9-8707-4169-8efc-f56285ea3c52\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-92mnh"
Jul 28 11:29:51.813882 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:51.813844    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4c659b9-8707-4169-8efc-f56285ea3c52-model-cache\") pod \"stop-feature-test-kserve-5cff5d4976-92mnh\" (UID: \"a4c659b9-8707-4169-8efc-f56285ea3c52\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-92mnh"
Jul 28 11:29:51.813882 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:51.813880    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4c659b9-8707-4169-8efc-f56285ea3c52-tmp-dir\") pod \"stop-feature-test-kserve-5cff5d4976-92mnh\" (UID: \"a4c659b9-8707-4169-8efc-f56285ea3c52\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-92mnh"
Jul 28 11:29:51.814102 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:51.813907    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54wl4\" (UniqueName: \"kubernetes.io/projected/a4c659b9-8707-4169-8efc-f56285ea3c52-kube-api-access-54wl4\") pod \"stop-feature-test-kserve-5cff5d4976-92mnh\" (UID: \"a4c659b9-8707-4169-8efc-f56285ea3c52\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-92mnh"
Jul 28 11:29:51.814102 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:51.813978    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4c659b9-8707-4169-8efc-f56285ea3c52-tls-certs\") pod \"stop-feature-test-kserve-5cff5d4976-92mnh\" (UID: \"a4c659b9-8707-4169-8efc-f56285ea3c52\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-92mnh"
Jul 28 11:29:51.814102 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:51.814010    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4c659b9-8707-4169-8efc-f56285ea3c52-home\") pod \"stop-feature-test-kserve-5cff5d4976-92mnh\" (UID: \"a4c659b9-8707-4169-8efc-f56285ea3c52\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-92mnh"
Jul 28 11:29:51.814255 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:51.814132    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4c659b9-8707-4169-8efc-f56285ea3c52-dshm\") pod \"stop-feature-test-kserve-5cff5d4976-92mnh\" (UID: \"a4c659b9-8707-4169-8efc-f56285ea3c52\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-92mnh"
Jul 28 11:29:51.814255 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:51.814190    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4c659b9-8707-4169-8efc-f56285ea3c52-kserve-provision-location\") pod \"stop-feature-test-kserve-5cff5d4976-92mnh\" (UID: \"a4c659b9-8707-4169-8efc-f56285ea3c52\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-92mnh"
Jul 28 11:29:51.814386 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:51.814357    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4c659b9-8707-4169-8efc-f56285ea3c52-home\") pod \"stop-feature-test-kserve-5cff5d4976-92mnh\" (UID: \"a4c659b9-8707-4169-8efc-f56285ea3c52\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-92mnh"
Jul 28 11:29:51.814449 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:51.814379    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4c659b9-8707-4169-8efc-f56285ea3c52-tmp-dir\") pod \"stop-feature-test-kserve-5cff5d4976-92mnh\" (UID: \"a4c659b9-8707-4169-8efc-f56285ea3c52\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-92mnh"
Jul 28 11:29:51.814449 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:51.814390    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4c659b9-8707-4169-8efc-f56285ea3c52-model-cache\") pod \"stop-feature-test-kserve-5cff5d4976-92mnh\" (UID: \"a4c659b9-8707-4169-8efc-f56285ea3c52\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-92mnh"
Jul 28 11:29:51.814557 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:51.814497    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4c659b9-8707-4169-8efc-f56285ea3c52-kserve-provision-location\") pod \"stop-feature-test-kserve-5cff5d4976-92mnh\" (UID: \"a4c659b9-8707-4169-8efc-f56285ea3c52\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-92mnh"
Jul 28 11:29:51.816330 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:51.816299    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4c659b9-8707-4169-8efc-f56285ea3c52-dshm\") pod \"stop-feature-test-kserve-5cff5d4976-92mnh\" (UID: \"a4c659b9-8707-4169-8efc-f56285ea3c52\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-92mnh"
Jul 28 11:29:51.816643 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:51.816623    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4c659b9-8707-4169-8efc-f56285ea3c52-tls-certs\") pod \"stop-feature-test-kserve-5cff5d4976-92mnh\" (UID: \"a4c659b9-8707-4169-8efc-f56285ea3c52\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-92mnh"
Jul 28 11:29:51.822043 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:51.822023    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54wl4\" (UniqueName: \"kubernetes.io/projected/a4c659b9-8707-4169-8efc-f56285ea3c52-kube-api-access-54wl4\") pod \"stop-feature-test-kserve-5cff5d4976-92mnh\" (UID: \"a4c659b9-8707-4169-8efc-f56285ea3c52\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-92mnh"
Jul 28 11:29:51.922639 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:51.922526    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-92mnh"
Jul 28 11:29:52.065752 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:52.065725    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-92mnh"]
Jul 28 11:29:52.067898 ip-10-0-131-88 kubenswrapper[2557]: W0728 11:29:52.067871    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4c659b9_8707_4169_8efc_f56285ea3c52.slice/crio-9eaebf6f211109166345709486a1dab0ed643fa10caacff6f6b0314c1ca5319c WatchSource:0}: Error finding container 9eaebf6f211109166345709486a1dab0ed643fa10caacff6f6b0314c1ca5319c: Status 404 returned error can't find the container with id 9eaebf6f211109166345709486a1dab0ed643fa10caacff6f6b0314c1ca5319c
Jul 28 11:29:52.317275 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:52.317242    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-92mnh" event={"ID":"a4c659b9-8707-4169-8efc-f56285ea3c52","Type":"ContainerStarted","Data":"f21435af2d493c446c8922a7b853c294eced14922974a36fbf8c8a2dbc11a987"}
Jul 28 11:29:52.317451 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:52.317298    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-92mnh" event={"ID":"a4c659b9-8707-4169-8efc-f56285ea3c52","Type":"ContainerStarted","Data":"9eaebf6f211109166345709486a1dab0ed643fa10caacff6f6b0314c1ca5319c"}
Jul 28 11:29:56.339819 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:56.339731    2557 generic.go:356] "Generic (PLEG): container finished" podID="a4c659b9-8707-4169-8efc-f56285ea3c52" containerID="f21435af2d493c446c8922a7b853c294eced14922974a36fbf8c8a2dbc11a987" exitCode=0
Jul 28 11:29:56.339819 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:56.339804    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-92mnh" event={"ID":"a4c659b9-8707-4169-8efc-f56285ea3c52","Type":"ContainerDied","Data":"f21435af2d493c446c8922a7b853c294eced14922974a36fbf8c8a2dbc11a987"}
Jul 28 11:29:57.347259 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:57.347222    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-92mnh" event={"ID":"a4c659b9-8707-4169-8efc-f56285ea3c52","Type":"ContainerStarted","Data":"3717d6b898f8d0be17ccde94d7d57776f8647a25cc2e9df8389842fc704d1c3a"}
Jul 28 11:29:57.371159 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:57.371103    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-92mnh" podStartSLOduration=2.441977155 podStartE2EDuration="6.371085266s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.929108111s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:29:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:29:57.368915028 +0000 UTC m=+4933.327683406" watchObservedRunningTime="2026-07-28 11:29:57.371085266 +0000 UTC m=+4933.329853622"
Jul 28 11:29:57.923615 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:29:57.923572    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-92mnh"
Jul 28 11:30:01.923182 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:30:01.923137    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-92mnh"
Jul 28 11:30:01.924882 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:30:01.924850    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-92mnh" podUID="a4c659b9-8707-4169-8efc-f56285ea3c52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Jul 28 11:30:11.923121 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:30:11.922997    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-92mnh" podUID="a4c659b9-8707-4169-8efc-f56285ea3c52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Jul 28 11:30:21.923014 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:30:21.922964    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-92mnh" podUID="a4c659b9-8707-4169-8efc-f56285ea3c52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Jul 28 11:30:31.923372 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:30:31.923317    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-92mnh" podUID="a4c659b9-8707-4169-8efc-f56285ea3c52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Jul 28 11:30:41.923684 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:30:41.923632    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-92mnh" podUID="a4c659b9-8707-4169-8efc-f56285ea3c52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Jul 28 11:30:51.923524 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:30:51.923474    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-92mnh" podUID="a4c659b9-8707-4169-8efc-f56285ea3c52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Jul 28 11:31:01.923479 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:31:01.923433    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-92mnh" podUID="a4c659b9-8707-4169-8efc-f56285ea3c52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Jul 28 11:31:11.923663 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:31:11.923613    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-92mnh" podUID="a4c659b9-8707-4169-8efc-f56285ea3c52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Jul 28 11:31:21.923558 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:31:21.923512    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-92mnh" podUID="a4c659b9-8707-4169-8efc-f56285ea3c52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Jul 28 11:31:31.923622 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:31:31.923559    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-92mnh" podUID="a4c659b9-8707-4169-8efc-f56285ea3c52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Jul 28 11:31:41.923637 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:31:41.923529    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-92mnh" podUID="a4c659b9-8707-4169-8efc-f56285ea3c52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Jul 28 11:31:51.923278 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:31:51.923230    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-92mnh" podUID="a4c659b9-8707-4169-8efc-f56285ea3c52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Jul 28 11:32:01.933493 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:32:01.933454    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-92mnh"
Jul 28 11:32:01.941081 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:32:01.941054    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-92mnh"
Jul 28 11:32:26.055564 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:32:26.055528    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-92mnh"]
Jul 28 11:32:41.078479 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:32:41.078431    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-92mnh" podUID="a4c659b9-8707-4169-8efc-f56285ea3c52" containerName="main" containerID="cri-o://3717d6b898f8d0be17ccde94d7d57776f8647a25cc2e9df8389842fc704d1c3a" gracePeriod=45
Jul 28 11:32:41.843116 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:32:41.843092    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-92mnh"
Jul 28 11:32:41.917861 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:32:41.917775    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4c659b9-8707-4169-8efc-f56285ea3c52-dshm\") pod \"a4c659b9-8707-4169-8efc-f56285ea3c52\" (UID: \"a4c659b9-8707-4169-8efc-f56285ea3c52\") "
Jul 28 11:32:41.917861 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:32:41.917818    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4c659b9-8707-4169-8efc-f56285ea3c52-tmp-dir\") pod \"a4c659b9-8707-4169-8efc-f56285ea3c52\" (UID: \"a4c659b9-8707-4169-8efc-f56285ea3c52\") "
Jul 28 11:32:41.918052 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:32:41.917873    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54wl4\" (UniqueName: \"kubernetes.io/projected/a4c659b9-8707-4169-8efc-f56285ea3c52-kube-api-access-54wl4\") pod \"a4c659b9-8707-4169-8efc-f56285ea3c52\" (UID: \"a4c659b9-8707-4169-8efc-f56285ea3c52\") "
Jul 28 11:32:41.918052 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:32:41.917914    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4c659b9-8707-4169-8efc-f56285ea3c52-model-cache\") pod \"a4c659b9-8707-4169-8efc-f56285ea3c52\" (UID: \"a4c659b9-8707-4169-8efc-f56285ea3c52\") "
Jul 28 11:32:41.918052 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:32:41.917960    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4c659b9-8707-4169-8efc-f56285ea3c52-kserve-provision-location\") pod \"a4c659b9-8707-4169-8efc-f56285ea3c52\" (UID: \"a4c659b9-8707-4169-8efc-f56285ea3c52\") "
Jul 28 11:32:41.918052 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:32:41.917995    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4c659b9-8707-4169-8efc-f56285ea3c52-tls-certs\") pod \"a4c659b9-8707-4169-8efc-f56285ea3c52\" (UID: \"a4c659b9-8707-4169-8efc-f56285ea3c52\") "
Jul 28 11:32:41.918262 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:32:41.918066    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4c659b9-8707-4169-8efc-f56285ea3c52-home\") pod \"a4c659b9-8707-4169-8efc-f56285ea3c52\" (UID: \"a4c659b9-8707-4169-8efc-f56285ea3c52\") "
Jul 28 11:32:41.918262 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:32:41.918230    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4c659b9-8707-4169-8efc-f56285ea3c52-model-cache" (OuterVolumeSpecName: "model-cache") pod "a4c659b9-8707-4169-8efc-f56285ea3c52" (UID: "a4c659b9-8707-4169-8efc-f56285ea3c52"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:32:41.918449 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:32:41.918383    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4c659b9-8707-4169-8efc-f56285ea3c52-model-cache\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 11:32:41.918911 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:32:41.918884    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4c659b9-8707-4169-8efc-f56285ea3c52-home" (OuterVolumeSpecName: "home") pod "a4c659b9-8707-4169-8efc-f56285ea3c52" (UID: "a4c659b9-8707-4169-8efc-f56285ea3c52"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:32:41.920171 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:32:41.920138    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4c659b9-8707-4169-8efc-f56285ea3c52-dshm" (OuterVolumeSpecName: "dshm") pod "a4c659b9-8707-4169-8efc-f56285ea3c52" (UID: "a4c659b9-8707-4169-8efc-f56285ea3c52"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:32:41.920290 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:32:41.920221    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4c659b9-8707-4169-8efc-f56285ea3c52-kube-api-access-54wl4" (OuterVolumeSpecName: "kube-api-access-54wl4") pod "a4c659b9-8707-4169-8efc-f56285ea3c52" (UID: "a4c659b9-8707-4169-8efc-f56285ea3c52"). InnerVolumeSpecName "kube-api-access-54wl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:32:41.920290 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:32:41.920259    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4c659b9-8707-4169-8efc-f56285ea3c52-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a4c659b9-8707-4169-8efc-f56285ea3c52" (UID: "a4c659b9-8707-4169-8efc-f56285ea3c52"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:32:41.931945 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:32:41.931918    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4c659b9-8707-4169-8efc-f56285ea3c52-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a4c659b9-8707-4169-8efc-f56285ea3c52" (UID: "a4c659b9-8707-4169-8efc-f56285ea3c52"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:32:41.981883 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:32:41.981836    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4c659b9-8707-4169-8efc-f56285ea3c52-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a4c659b9-8707-4169-8efc-f56285ea3c52" (UID: "a4c659b9-8707-4169-8efc-f56285ea3c52"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:32:42.019775 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:32:42.019738    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4c659b9-8707-4169-8efc-f56285ea3c52-kserve-provision-location\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 11:32:42.019775 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:32:42.019769    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4c659b9-8707-4169-8efc-f56285ea3c52-tls-certs\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 11:32:42.019775 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:32:42.019780    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4c659b9-8707-4169-8efc-f56285ea3c52-home\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 11:32:42.019984 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:32:42.019789    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4c659b9-8707-4169-8efc-f56285ea3c52-dshm\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 11:32:42.019984 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:32:42.019797    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4c659b9-8707-4169-8efc-f56285ea3c52-tmp-dir\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 11:32:42.019984 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:32:42.019805    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54wl4\" (UniqueName: \"kubernetes.io/projected/a4c659b9-8707-4169-8efc-f56285ea3c52-kube-api-access-54wl4\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 11:32:42.191100 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:32:42.191002    2557 generic.go:356] "Generic (PLEG): container finished" podID="a4c659b9-8707-4169-8efc-f56285ea3c52" containerID="3717d6b898f8d0be17ccde94d7d57776f8647a25cc2e9df8389842fc704d1c3a" exitCode=0
Jul 28 11:32:42.191100 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:32:42.191052    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-92mnh"
Jul 28 11:32:42.191100 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:32:42.191086    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-92mnh" event={"ID":"a4c659b9-8707-4169-8efc-f56285ea3c52","Type":"ContainerDied","Data":"3717d6b898f8d0be17ccde94d7d57776f8647a25cc2e9df8389842fc704d1c3a"}
Jul 28 11:32:42.191697 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:32:42.191152    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-92mnh" event={"ID":"a4c659b9-8707-4169-8efc-f56285ea3c52","Type":"ContainerDied","Data":"9eaebf6f211109166345709486a1dab0ed643fa10caacff6f6b0314c1ca5319c"}
Jul 28 11:32:42.191697 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:32:42.191184    2557 scope.go:117] "RemoveContainer" containerID="3717d6b898f8d0be17ccde94d7d57776f8647a25cc2e9df8389842fc704d1c3a"
Jul 28 11:32:42.200551 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:32:42.200537    2557 scope.go:117] "RemoveContainer" containerID="f21435af2d493c446c8922a7b853c294eced14922974a36fbf8c8a2dbc11a987"
Jul 28 11:32:42.209373 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:32:42.209354    2557 scope.go:117] "RemoveContainer" containerID="3717d6b898f8d0be17ccde94d7d57776f8647a25cc2e9df8389842fc704d1c3a"
Jul 28 11:32:42.209688 ip-10-0-131-88 kubenswrapper[2557]: E0728 11:32:42.209666    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3717d6b898f8d0be17ccde94d7d57776f8647a25cc2e9df8389842fc704d1c3a\": container with ID starting with 3717d6b898f8d0be17ccde94d7d57776f8647a25cc2e9df8389842fc704d1c3a not found: ID does not exist" containerID="3717d6b898f8d0be17ccde94d7d57776f8647a25cc2e9df8389842fc704d1c3a"
Jul 28 11:32:42.209771 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:32:42.209702    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3717d6b898f8d0be17ccde94d7d57776f8647a25cc2e9df8389842fc704d1c3a"} err="failed to get container status \"3717d6b898f8d0be17ccde94d7d57776f8647a25cc2e9df8389842fc704d1c3a\": rpc error: code = NotFound desc = could not find container \"3717d6b898f8d0be17ccde94d7d57776f8647a25cc2e9df8389842fc704d1c3a\": container with ID starting with 3717d6b898f8d0be17ccde94d7d57776f8647a25cc2e9df8389842fc704d1c3a not found: ID does not exist"
Jul 28 11:32:42.209771 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:32:42.209726    2557 scope.go:117] "RemoveContainer" containerID="f21435af2d493c446c8922a7b853c294eced14922974a36fbf8c8a2dbc11a987"
Jul 28 11:32:42.209994 ip-10-0-131-88 kubenswrapper[2557]: E0728 11:32:42.209974    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f21435af2d493c446c8922a7b853c294eced14922974a36fbf8c8a2dbc11a987\": container with ID starting with f21435af2d493c446c8922a7b853c294eced14922974a36fbf8c8a2dbc11a987 not found: ID does not exist" containerID="f21435af2d493c446c8922a7b853c294eced14922974a36fbf8c8a2dbc11a987"
Jul 28 11:32:42.210038 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:32:42.210001    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f21435af2d493c446c8922a7b853c294eced14922974a36fbf8c8a2dbc11a987"} err="failed to get container status \"f21435af2d493c446c8922a7b853c294eced14922974a36fbf8c8a2dbc11a987\": rpc error: code = NotFound desc = could not find container \"f21435af2d493c446c8922a7b853c294eced14922974a36fbf8c8a2dbc11a987\": container with ID starting with f21435af2d493c446c8922a7b853c294eced14922974a36fbf8c8a2dbc11a987 not found: ID does not exist"
Jul 28 11:32:42.216467 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:32:42.216441    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-92mnh"]
Jul 28 11:32:42.224509 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:32:42.224483    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5cff5d4976-92mnh"]
Jul 28 11:32:42.580700 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:32:42.580663    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4c659b9-8707-4169-8efc-f56285ea3c52" path="/var/lib/kubelet/pods/a4c659b9-8707-4169-8efc-f56285ea3c52/volumes"
Jul 28 11:40:56.951031 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:40:56.950948    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-w4cp8"]
Jul 28 11:40:56.951502 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:40:56.951294    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4c659b9-8707-4169-8efc-f56285ea3c52" containerName="main"
Jul 28 11:40:56.951502 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:40:56.951305    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4c659b9-8707-4169-8efc-f56285ea3c52" containerName="main"
Jul 28 11:40:56.951502 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:40:56.951327    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4c659b9-8707-4169-8efc-f56285ea3c52" containerName="storage-initializer"
Jul 28 11:40:56.951502 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:40:56.951332    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4c659b9-8707-4169-8efc-f56285ea3c52" containerName="storage-initializer"
Jul 28 11:40:56.951502 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:40:56.951388    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4c659b9-8707-4169-8efc-f56285ea3c52" containerName="main"
Jul 28 11:40:56.954196 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:40:56.954177    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-w4cp8"
Jul 28 11:40:56.957082 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:40:56.957048    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-epp-sa-dockercfg-4njs6\""
Jul 28 11:40:56.957082 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:40:56.957059    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-9h6sv\""
Jul 28 11:40:56.957317 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:40:56.957085    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Jul 28 11:40:56.957317 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:40:56.957059    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Jul 28 11:40:56.957432 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:40:56.957394    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Jul 28 11:40:56.968118 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:40:56.968092    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-w4cp8"]
Jul 28 11:40:57.103424 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:40:57.103385    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbzrq\" (UniqueName: \"kubernetes.io/projected/2e297813-5419-409f-bf5a-573c0d0ec9c8-kube-api-access-sbzrq\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-w4cp8\" (UID: \"2e297813-5419-409f-bf5a-573c0d0ec9c8\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-w4cp8"
Jul 28 11:40:57.103638 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:40:57.103434    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e297813-5419-409f-bf5a-573c0d0ec9c8-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-w4cp8\" (UID: \"2e297813-5419-409f-bf5a-573c0d0ec9c8\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-w4cp8"
Jul 28 11:40:57.204479 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:40:57.204377    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e297813-5419-409f-bf5a-573c0d0ec9c8-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-w4cp8\" (UID: \"2e297813-5419-409f-bf5a-573c0d0ec9c8\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-w4cp8"
Jul 28 11:40:57.204670 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:40:57.204508    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbzrq\" (UniqueName: \"kubernetes.io/projected/2e297813-5419-409f-bf5a-573c0d0ec9c8-kube-api-access-sbzrq\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-w4cp8\" (UID: \"2e297813-5419-409f-bf5a-573c0d0ec9c8\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-w4cp8"
Jul 28 11:40:57.207105 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:40:57.207081    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e297813-5419-409f-bf5a-573c0d0ec9c8-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-w4cp8\" (UID: \"2e297813-5419-409f-bf5a-573c0d0ec9c8\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-w4cp8"
Jul 28 11:40:57.212770 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:40:57.212741    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbzrq\" (UniqueName: \"kubernetes.io/projected/2e297813-5419-409f-bf5a-573c0d0ec9c8-kube-api-access-sbzrq\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-w4cp8\" (UID: \"2e297813-5419-409f-bf5a-573c0d0ec9c8\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-w4cp8"
Jul 28 11:40:57.267160 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:40:57.267125    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-w4cp8"
Jul 28 11:40:57.405553 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:40:57.405524    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-w4cp8"]
Jul 28 11:40:57.408181 ip-10-0-131-88 kubenswrapper[2557]: W0728 11:40:57.408153    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e297813_5419_409f_bf5a_573c0d0ec9c8.slice/crio-4065bc00523338a13fb7990c6497286698901235679df9e4b4ba31fda8976e7c WatchSource:0}: Error finding container 4065bc00523338a13fb7990c6497286698901235679df9e4b4ba31fda8976e7c: Status 404 returned error can't find the container with id 4065bc00523338a13fb7990c6497286698901235679df9e4b4ba31fda8976e7c
Jul 28 11:40:57.518878 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:40:57.518841    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-w4cp8" event={"ID":"2e297813-5419-409f-bf5a-573c0d0ec9c8","Type":"ContainerStarted","Data":"aeee99d37dca789bbcb74c5e191c42b7132a3d5db3aecf0f4e66b318da5f0cae"}
Jul 28 11:40:57.519026 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:40:57.518892    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-w4cp8" event={"ID":"2e297813-5419-409f-bf5a-573c0d0ec9c8","Type":"ContainerStarted","Data":"4065bc00523338a13fb7990c6497286698901235679df9e4b4ba31fda8976e7c"}
Jul 28 11:40:57.519026 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:40:57.518988    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-w4cp8"
Jul 28 11:40:57.535586 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:40:57.535533    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-w4cp8" podStartSLOduration=1.5355150229999999 podStartE2EDuration="1.535515023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:40:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:40:57.533792375 +0000 UTC m=+5593.492560755" watchObservedRunningTime="2026-07-28 11:40:57.535515023 +0000 UTC m=+5593.494283403"
Jul 28 11:41:28.525149 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:41:28.525119    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-w4cp8"
Jul 28 11:44:07.510807 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:44:07.510713    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-w4cp8"]
Jul 28 11:44:08.525161 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:44:08.525118    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-w4cp8" podUID="2e297813-5419-409f-bf5a-573c0d0ec9c8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 11:44:18.524966 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:44:18.524926    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-w4cp8" podUID="2e297813-5419-409f-bf5a-573c0d0ec9c8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 11:44:22.531166 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:44:22.531099    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-w4cp8" podUID="2e297813-5419-409f-bf5a-573c0d0ec9c8" containerName="main" containerID="cri-o://aeee99d37dca789bbcb74c5e191c42b7132a3d5db3aecf0f4e66b318da5f0cae" gracePeriod=45
Jul 28 11:44:22.790460 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:44:22.790389    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-w4cp8"
Jul 28 11:44:22.875424 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:44:22.875388    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sbzrq\" (UniqueName: \"kubernetes.io/projected/2e297813-5419-409f-bf5a-573c0d0ec9c8-kube-api-access-sbzrq\") pod \"2e297813-5419-409f-bf5a-573c0d0ec9c8\" (UID: \"2e297813-5419-409f-bf5a-573c0d0ec9c8\") "
Jul 28 11:44:22.875622 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:44:22.875444    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e297813-5419-409f-bf5a-573c0d0ec9c8-tls-certs\") pod \"2e297813-5419-409f-bf5a-573c0d0ec9c8\" (UID: \"2e297813-5419-409f-bf5a-573c0d0ec9c8\") "
Jul 28 11:44:22.877664 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:44:22.877622    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e297813-5419-409f-bf5a-573c0d0ec9c8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2e297813-5419-409f-bf5a-573c0d0ec9c8" (UID: "2e297813-5419-409f-bf5a-573c0d0ec9c8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:44:22.877823 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:44:22.877732    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e297813-5419-409f-bf5a-573c0d0ec9c8-kube-api-access-sbzrq" (OuterVolumeSpecName: "kube-api-access-sbzrq") pod "2e297813-5419-409f-bf5a-573c0d0ec9c8" (UID: "2e297813-5419-409f-bf5a-573c0d0ec9c8"). InnerVolumeSpecName "kube-api-access-sbzrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:44:22.976576 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:44:22.976532    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbzrq\" (UniqueName: \"kubernetes.io/projected/2e297813-5419-409f-bf5a-573c0d0ec9c8-kube-api-access-sbzrq\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 11:44:22.976576 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:44:22.976568    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e297813-5419-409f-bf5a-573c0d0ec9c8-tls-certs\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 11:44:23.503350 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:44:23.503312    2557 generic.go:356] "Generic (PLEG): container finished" podID="2e297813-5419-409f-bf5a-573c0d0ec9c8" containerID="aeee99d37dca789bbcb74c5e191c42b7132a3d5db3aecf0f4e66b318da5f0cae" exitCode=0
Jul 28 11:44:23.503350 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:44:23.503359    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-w4cp8" event={"ID":"2e297813-5419-409f-bf5a-573c0d0ec9c8","Type":"ContainerDied","Data":"aeee99d37dca789bbcb74c5e191c42b7132a3d5db3aecf0f4e66b318da5f0cae"}
Jul 28 11:44:23.503571 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:44:23.503385    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-w4cp8"
Jul 28 11:44:23.503571 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:44:23.503446    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-w4cp8" event={"ID":"2e297813-5419-409f-bf5a-573c0d0ec9c8","Type":"ContainerDied","Data":"4065bc00523338a13fb7990c6497286698901235679df9e4b4ba31fda8976e7c"}
Jul 28 11:44:23.503571 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:44:23.503478    2557 scope.go:117] "RemoveContainer" containerID="aeee99d37dca789bbcb74c5e191c42b7132a3d5db3aecf0f4e66b318da5f0cae"
Jul 28 11:44:23.513970 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:44:23.513945    2557 scope.go:117] "RemoveContainer" containerID="aeee99d37dca789bbcb74c5e191c42b7132a3d5db3aecf0f4e66b318da5f0cae"
Jul 28 11:44:23.514293 ip-10-0-131-88 kubenswrapper[2557]: E0728 11:44:23.514274    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aeee99d37dca789bbcb74c5e191c42b7132a3d5db3aecf0f4e66b318da5f0cae\": container with ID starting with aeee99d37dca789bbcb74c5e191c42b7132a3d5db3aecf0f4e66b318da5f0cae not found: ID does not exist" containerID="aeee99d37dca789bbcb74c5e191c42b7132a3d5db3aecf0f4e66b318da5f0cae"
Jul 28 11:44:23.514406 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:44:23.514301    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aeee99d37dca789bbcb74c5e191c42b7132a3d5db3aecf0f4e66b318da5f0cae"} err="failed to get container status \"aeee99d37dca789bbcb74c5e191c42b7132a3d5db3aecf0f4e66b318da5f0cae\": rpc error: code = NotFound desc = could not find container \"aeee99d37dca789bbcb74c5e191c42b7132a3d5db3aecf0f4e66b318da5f0cae\": container with ID starting with aeee99d37dca789bbcb74c5e191c42b7132a3d5db3aecf0f4e66b318da5f0cae not found: ID does not exist"
Jul 28 11:44:23.527168 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:44:23.527139    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-w4cp8"]
Jul 28 11:44:23.529462 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:44:23.529441    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-w4cp8"]
Jul 28 11:44:24.580416 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:44:24.580385    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e297813-5419-409f-bf5a-573c0d0ec9c8" path="/var/lib/kubelet/pods/2e297813-5419-409f-bf5a-573c0d0ec9c8/volumes"
Jul 28 11:44:40.601181 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:44:40.601140    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-698b7"]
Jul 28 11:44:40.601755 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:44:40.601496    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e297813-5419-409f-bf5a-573c0d0ec9c8" containerName="main"
Jul 28 11:44:40.601755 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:44:40.601510    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e297813-5419-409f-bf5a-573c0d0ec9c8" containerName="main"
Jul 28 11:44:40.601755 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:44:40.601577    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e297813-5419-409f-bf5a-573c0d0ec9c8" containerName="main"
Jul 28 11:44:40.606185 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:44:40.606160    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-698b7"
Jul 28 11:44:40.608230 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:44:40.608207    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Jul 28 11:44:40.608359 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:44:40.608218    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-7jch4\""
Jul 28 11:44:40.608738 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:44:40.608719    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Jul 28 11:44:40.608820 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:44:40.608793    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-epp-sa-dockercfg-wnd4b\""
Jul 28 11:44:40.608929 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:44:40.608907    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Jul 28 11:44:40.613472 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:44:40.613448    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-698b7"]
Jul 28 11:44:40.620743 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:44:40.620709    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsh4t\" (UniqueName: \"kubernetes.io/projected/851dfec9-4179-470f-a411-0f12be26413b-kube-api-access-vsh4t\") pod \"prestop-hook-test-kserve-router-scheduler-6dc6dff85-698b7\" (UID: \"851dfec9-4179-470f-a411-0f12be26413b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-698b7"
Jul 28 11:44:40.620866 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:44:40.620763    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/851dfec9-4179-470f-a411-0f12be26413b-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-6dc6dff85-698b7\" (UID: \"851dfec9-4179-470f-a411-0f12be26413b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-698b7"
Jul 28 11:44:40.721959 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:44:40.721901    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsh4t\" (UniqueName: \"kubernetes.io/projected/851dfec9-4179-470f-a411-0f12be26413b-kube-api-access-vsh4t\") pod \"prestop-hook-test-kserve-router-scheduler-6dc6dff85-698b7\" (UID: \"851dfec9-4179-470f-a411-0f12be26413b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-698b7"
Jul 28 11:44:40.722168 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:44:40.721966    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/851dfec9-4179-470f-a411-0f12be26413b-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-6dc6dff85-698b7\" (UID: \"851dfec9-4179-470f-a411-0f12be26413b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-698b7"
Jul 28 11:44:40.725086 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:44:40.725054    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/851dfec9-4179-470f-a411-0f12be26413b-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-6dc6dff85-698b7\" (UID: \"851dfec9-4179-470f-a411-0f12be26413b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-698b7"
Jul 28 11:44:40.732326 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:44:40.732300    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsh4t\" (UniqueName: \"kubernetes.io/projected/851dfec9-4179-470f-a411-0f12be26413b-kube-api-access-vsh4t\") pod \"prestop-hook-test-kserve-router-scheduler-6dc6dff85-698b7\" (UID: \"851dfec9-4179-470f-a411-0f12be26413b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-698b7"
Jul 28 11:44:40.919084 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:44:40.918988    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-698b7"
Jul 28 11:44:41.072213 ip-10-0-131-88 kubenswrapper[2557]: W0728 11:44:41.072176    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod851dfec9_4179_470f_a411_0f12be26413b.slice/crio-1502aa107ddfe966633c85c84fe40521cf02a5ed9f5244c6a876afbba0f5fca8 WatchSource:0}: Error finding container 1502aa107ddfe966633c85c84fe40521cf02a5ed9f5244c6a876afbba0f5fca8: Status 404 returned error can't find the container with id 1502aa107ddfe966633c85c84fe40521cf02a5ed9f5244c6a876afbba0f5fca8
Jul 28 11:44:41.072435 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:44:41.072411    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-698b7"]
Jul 28 11:44:41.595655 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:44:41.595611    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-698b7" event={"ID":"851dfec9-4179-470f-a411-0f12be26413b","Type":"ContainerStarted","Data":"51589912aa5d55ec8153c6d2e8b7e22abaf24b916e5b7a7c68e8c336525bced1"}
Jul 28 11:44:41.595855 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:44:41.595675    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-698b7" event={"ID":"851dfec9-4179-470f-a411-0f12be26413b","Type":"ContainerStarted","Data":"1502aa107ddfe966633c85c84fe40521cf02a5ed9f5244c6a876afbba0f5fca8"}
Jul 28 11:44:41.595855 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:44:41.595732    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-698b7"
Jul 28 11:44:41.626866 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:44:41.626813    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-698b7" podStartSLOduration=1.626797743 podStartE2EDuration="1.626797743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:44:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:44:41.626065191 +0000 UTC m=+5817.584833570" watchObservedRunningTime="2026-07-28 11:44:41.626797743 +0000 UTC m=+5817.585566121"
Jul 28 11:45:12.601909 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:45:12.601836    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-698b7"
Jul 28 11:47:17.298129 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:47:17.298086    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-698b7"]
Jul 28 11:47:22.602755 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:47:22.602716    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-698b7" podUID="851dfec9-4179-470f-a411-0f12be26413b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 11:47:32.318117 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:47:32.318065    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-698b7" podUID="851dfec9-4179-470f-a411-0f12be26413b" containerName="main" containerID="cri-o://51589912aa5d55ec8153c6d2e8b7e22abaf24b916e5b7a7c68e8c336525bced1" gracePeriod=45
Jul 28 11:47:32.572270 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:47:32.572208    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-698b7"
Jul 28 11:47:32.687165 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:47:32.687130    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/851dfec9-4179-470f-a411-0f12be26413b-tls-certs\") pod \"851dfec9-4179-470f-a411-0f12be26413b\" (UID: \"851dfec9-4179-470f-a411-0f12be26413b\") "
Jul 28 11:47:32.687165 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:47:32.687180    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vsh4t\" (UniqueName: \"kubernetes.io/projected/851dfec9-4179-470f-a411-0f12be26413b-kube-api-access-vsh4t\") pod \"851dfec9-4179-470f-a411-0f12be26413b\" (UID: \"851dfec9-4179-470f-a411-0f12be26413b\") "
Jul 28 11:47:32.689510 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:47:32.689477    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/851dfec9-4179-470f-a411-0f12be26413b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "851dfec9-4179-470f-a411-0f12be26413b" (UID: "851dfec9-4179-470f-a411-0f12be26413b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:47:32.689656 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:47:32.689522    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/851dfec9-4179-470f-a411-0f12be26413b-kube-api-access-vsh4t" (OuterVolumeSpecName: "kube-api-access-vsh4t") pod "851dfec9-4179-470f-a411-0f12be26413b" (UID: "851dfec9-4179-470f-a411-0f12be26413b"). InnerVolumeSpecName "kube-api-access-vsh4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:47:32.788874 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:47:32.788838    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/851dfec9-4179-470f-a411-0f12be26413b-tls-certs\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 11:47:32.788874 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:47:32.788870    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsh4t\" (UniqueName: \"kubernetes.io/projected/851dfec9-4179-470f-a411-0f12be26413b-kube-api-access-vsh4t\") on node \"ip-10-0-131-88.ec2.internal\" DevicePath \"\""
Jul 28 11:47:33.399004 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:47:33.398965    2557 generic.go:356] "Generic (PLEG): container finished" podID="851dfec9-4179-470f-a411-0f12be26413b" containerID="51589912aa5d55ec8153c6d2e8b7e22abaf24b916e5b7a7c68e8c336525bced1" exitCode=0
Jul 28 11:47:33.399431 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:47:33.399009    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-698b7"
Jul 28 11:47:33.399431 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:47:33.399056    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-698b7" event={"ID":"851dfec9-4179-470f-a411-0f12be26413b","Type":"ContainerDied","Data":"51589912aa5d55ec8153c6d2e8b7e22abaf24b916e5b7a7c68e8c336525bced1"}
Jul 28 11:47:33.399431 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:47:33.399116    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-698b7" event={"ID":"851dfec9-4179-470f-a411-0f12be26413b","Type":"ContainerDied","Data":"1502aa107ddfe966633c85c84fe40521cf02a5ed9f5244c6a876afbba0f5fca8"}
Jul 28 11:47:33.399431 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:47:33.399142    2557 scope.go:117] "RemoveContainer" containerID="51589912aa5d55ec8153c6d2e8b7e22abaf24b916e5b7a7c68e8c336525bced1"
Jul 28 11:47:33.408646 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:47:33.408622    2557 scope.go:117] "RemoveContainer" containerID="51589912aa5d55ec8153c6d2e8b7e22abaf24b916e5b7a7c68e8c336525bced1"
Jul 28 11:47:33.408999 ip-10-0-131-88 kubenswrapper[2557]: E0728 11:47:33.408974    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51589912aa5d55ec8153c6d2e8b7e22abaf24b916e5b7a7c68e8c336525bced1\": container with ID starting with 51589912aa5d55ec8153c6d2e8b7e22abaf24b916e5b7a7c68e8c336525bced1 not found: ID does not exist" containerID="51589912aa5d55ec8153c6d2e8b7e22abaf24b916e5b7a7c68e8c336525bced1"
Jul 28 11:47:33.409099 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:47:33.409010    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51589912aa5d55ec8153c6d2e8b7e22abaf24b916e5b7a7c68e8c336525bced1"} err="failed to get container status \"51589912aa5d55ec8153c6d2e8b7e22abaf24b916e5b7a7c68e8c336525bced1\": rpc error: code = NotFound desc = could not find container \"51589912aa5d55ec8153c6d2e8b7e22abaf24b916e5b7a7c68e8c336525bced1\": container with ID starting with 51589912aa5d55ec8153c6d2e8b7e22abaf24b916e5b7a7c68e8c336525bced1 not found: ID does not exist"
Jul 28 11:47:33.422275 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:47:33.422247    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-698b7"]
Jul 28 11:47:33.426526 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:47:33.426502    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-698b7"]
Jul 28 11:47:34.583015 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:47:34.582982    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="851dfec9-4179-470f-a411-0f12be26413b" path="/var/lib/kubelet/pods/851dfec9-4179-470f-a411-0f12be26413b/volumes"
Jul 28 11:52:53.096256 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:52:53.096177    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5xmkm_676a3b2e-543d-47a4-8805-fea3a7ae14a7/authorino/0.log"
Jul 28 11:52:53.331180 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:52:53.331140    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-5grlb_59142a1a-3fd6-4148-bad7-92e0b43f979f/manager/0.log"
Jul 28 11:52:54.404772 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:52:54.404730    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5xmkm_676a3b2e-543d-47a4-8805-fea3a7ae14a7/authorino/0.log"
Jul 28 11:52:54.775023 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:52:54.774988    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-5grlb_59142a1a-3fd6-4148-bad7-92e0b43f979f/manager/0.log"
Jul 28 11:52:55.830131 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:52:55.830093    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5xmkm_676a3b2e-543d-47a4-8805-fea3a7ae14a7/authorino/0.log"
Jul 28 11:52:56.105616 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:52:56.105517    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-5grlb_59142a1a-3fd6-4148-bad7-92e0b43f979f/manager/0.log"
Jul 28 11:52:57.137842 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:52:57.137800    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5xmkm_676a3b2e-543d-47a4-8805-fea3a7ae14a7/authorino/0.log"
Jul 28 11:52:57.375732 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:52:57.375704    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-5grlb_59142a1a-3fd6-4148-bad7-92e0b43f979f/manager/0.log"
Jul 28 11:52:58.400719 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:52:58.400683    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5xmkm_676a3b2e-543d-47a4-8805-fea3a7ae14a7/authorino/0.log"
Jul 28 11:52:58.649303 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:52:58.649269    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-5grlb_59142a1a-3fd6-4148-bad7-92e0b43f979f/manager/0.log"
Jul 28 11:52:59.679234 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:52:59.679185    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5xmkm_676a3b2e-543d-47a4-8805-fea3a7ae14a7/authorino/0.log"
Jul 28 11:52:59.916835 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:52:59.916798    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-5grlb_59142a1a-3fd6-4148-bad7-92e0b43f979f/manager/0.log"
Jul 28 11:53:00.932478 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:00.932448    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5xmkm_676a3b2e-543d-47a4-8805-fea3a7ae14a7/authorino/0.log"
Jul 28 11:53:01.168814 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:01.168761    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-5grlb_59142a1a-3fd6-4148-bad7-92e0b43f979f/manager/0.log"
Jul 28 11:53:02.182795 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:02.182753    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5xmkm_676a3b2e-543d-47a4-8805-fea3a7ae14a7/authorino/0.log"
Jul 28 11:53:02.429298 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:02.429262    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-5grlb_59142a1a-3fd6-4148-bad7-92e0b43f979f/manager/0.log"
Jul 28 11:53:03.432036 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:03.431998    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5xmkm_676a3b2e-543d-47a4-8805-fea3a7ae14a7/authorino/0.log"
Jul 28 11:53:03.702374 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:03.702281    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-5grlb_59142a1a-3fd6-4148-bad7-92e0b43f979f/manager/0.log"
Jul 28 11:53:04.747860 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:04.747820    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5xmkm_676a3b2e-543d-47a4-8805-fea3a7ae14a7/authorino/0.log"
Jul 28 11:53:04.991819 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:04.991784    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-5grlb_59142a1a-3fd6-4148-bad7-92e0b43f979f/manager/0.log"
Jul 28 11:53:06.030186 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:06.030141    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5xmkm_676a3b2e-543d-47a4-8805-fea3a7ae14a7/authorino/0.log"
Jul 28 11:53:06.267369 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:06.267334    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-5grlb_59142a1a-3fd6-4148-bad7-92e0b43f979f/manager/0.log"
Jul 28 11:53:07.280329 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:07.280292    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5xmkm_676a3b2e-543d-47a4-8805-fea3a7ae14a7/authorino/0.log"
Jul 28 11:53:07.517774 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:07.517737    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-5grlb_59142a1a-3fd6-4148-bad7-92e0b43f979f/manager/0.log"
Jul 28 11:53:08.562434 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:08.562390    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5xmkm_676a3b2e-543d-47a4-8805-fea3a7ae14a7/authorino/0.log"
Jul 28 11:53:08.806894 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:08.806862    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-5grlb_59142a1a-3fd6-4148-bad7-92e0b43f979f/manager/0.log"
Jul 28 11:53:09.825144 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:09.825114    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5xmkm_676a3b2e-543d-47a4-8805-fea3a7ae14a7/authorino/0.log"
Jul 28 11:53:10.066244 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:10.066211    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-5grlb_59142a1a-3fd6-4148-bad7-92e0b43f979f/manager/0.log"
Jul 28 11:53:11.069799 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:11.069758    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5xmkm_676a3b2e-543d-47a4-8805-fea3a7ae14a7/authorino/0.log"
Jul 28 11:53:11.314696 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:11.314650    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-5grlb_59142a1a-3fd6-4148-bad7-92e0b43f979f/manager/0.log"
Jul 28 11:53:12.332879 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:12.332841    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5xmkm_676a3b2e-543d-47a4-8805-fea3a7ae14a7/authorino/0.log"
Jul 28 11:53:12.578954 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:12.578923    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-5grlb_59142a1a-3fd6-4148-bad7-92e0b43f979f/manager/0.log"
Jul 28 11:53:13.580955 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:13.580923    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5xmkm_676a3b2e-543d-47a4-8805-fea3a7ae14a7/authorino/0.log"
Jul 28 11:53:13.830933 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:13.830898    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-5grlb_59142a1a-3fd6-4148-bad7-92e0b43f979f/manager/0.log"
Jul 28 11:53:14.914483 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:14.914442    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5xmkm_676a3b2e-543d-47a4-8805-fea3a7ae14a7/authorino/0.log"
Jul 28 11:53:15.150103 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:15.150068    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-5grlb_59142a1a-3fd6-4148-bad7-92e0b43f979f/manager/0.log"
Jul 28 11:53:16.169340 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:16.169303    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5xmkm_676a3b2e-543d-47a4-8805-fea3a7ae14a7/authorino/0.log"
Jul 28 11:53:16.410120 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:16.409114    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-5grlb_59142a1a-3fd6-4148-bad7-92e0b43f979f/manager/0.log"
Jul 28 11:53:17.434758 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:17.434723    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5xmkm_676a3b2e-543d-47a4-8805-fea3a7ae14a7/authorino/0.log"
Jul 28 11:53:17.673384 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:17.673351    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-5grlb_59142a1a-3fd6-4148-bad7-92e0b43f979f/manager/0.log"
Jul 28 11:53:18.727035 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:18.726997    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5xmkm_676a3b2e-543d-47a4-8805-fea3a7ae14a7/authorino/0.log"
Jul 28 11:53:18.966049 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:18.966015    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-5grlb_59142a1a-3fd6-4148-bad7-92e0b43f979f/manager/0.log"
Jul 28 11:53:20.000431 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:20.000392    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5xmkm_676a3b2e-543d-47a4-8805-fea3a7ae14a7/authorino/0.log"
Jul 28 11:53:20.232387 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:20.232354    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-5grlb_59142a1a-3fd6-4148-bad7-92e0b43f979f/manager/0.log"
Jul 28 11:53:21.234386 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:21.234348    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5xmkm_676a3b2e-543d-47a4-8805-fea3a7ae14a7/authorino/0.log"
Jul 28 11:53:21.511680 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:21.511645    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-5grlb_59142a1a-3fd6-4148-bad7-92e0b43f979f/manager/0.log"
Jul 28 11:53:22.539291 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:22.539256    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5xmkm_676a3b2e-543d-47a4-8805-fea3a7ae14a7/authorino/0.log"
Jul 28 11:53:22.777863 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:22.777819    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-5grlb_59142a1a-3fd6-4148-bad7-92e0b43f979f/manager/0.log"
Jul 28 11:53:23.813312 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:23.813271    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5xmkm_676a3b2e-543d-47a4-8805-fea3a7ae14a7/authorino/0.log"
Jul 28 11:53:24.088214 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:24.088125    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-5grlb_59142a1a-3fd6-4148-bad7-92e0b43f979f/manager/0.log"
Jul 28 11:53:29.360929 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:29.360894    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jhkxw_ed2b30c5-aea1-46f5-a28a-1aff6f3d80bb/global-pull-secret-syncer/0.log"
Jul 28 11:53:29.573108 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:29.573071    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-v7gsk_6d704fd4-c194-4c37-a80e-dbde31ebd184/konnectivity-agent/0.log"
Jul 28 11:53:29.631499 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:29.631407    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-88.ec2.internal_6ff9d18249fb2db5835b1f8658874702/haproxy/0.log"
Jul 28 11:53:34.428444 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:34.428406    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5xmkm_676a3b2e-543d-47a4-8805-fea3a7ae14a7/authorino/0.log"
Jul 28 11:53:34.757070 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:34.757039    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-5grlb_59142a1a-3fd6-4148-bad7-92e0b43f979f/manager/0.log"
Jul 28 11:53:35.947084 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:35.947044    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-27zms_dde5d137-fa6b-4c1e-ad94-07f174179c6c/kube-state-metrics/0.log"
Jul 28 11:53:35.974156 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:35.974116    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-27zms_dde5d137-fa6b-4c1e-ad94-07f174179c6c/kube-rbac-proxy-main/0.log"
Jul 28 11:53:35.998383 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:35.998356    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-27zms_dde5d137-fa6b-4c1e-ad94-07f174179c6c/kube-rbac-proxy-self/0.log"
Jul 28 11:53:36.026733 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:36.026699    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5b7c89b76c-w2zrh_8c1a3068-bb8a-4a90-97ca-f29ebebd3d87/metrics-server/0.log"
Jul 28 11:53:36.090340 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:36.090298    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6ps9j_03488ce4-1c3e-4258-854e-913b25aac684/node-exporter/0.log"
Jul 28 11:53:36.112274 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:36.112246    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6ps9j_03488ce4-1c3e-4258-854e-913b25aac684/kube-rbac-proxy/0.log"
Jul 28 11:53:36.136043 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:36.136016    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6ps9j_03488ce4-1c3e-4258-854e-913b25aac684/init-textfile/0.log"
Jul 28 11:53:36.432378 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:36.432343    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bd83a23b-232a-4ed4-a5a8-562ff2704131/prometheus/0.log"
Jul 28 11:53:36.450768 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:36.450746    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bd83a23b-232a-4ed4-a5a8-562ff2704131/config-reloader/0.log"
Jul 28 11:53:36.478328 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:36.478293    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bd83a23b-232a-4ed4-a5a8-562ff2704131/thanos-sidecar/0.log"
Jul 28 11:53:36.501947 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:36.501921    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bd83a23b-232a-4ed4-a5a8-562ff2704131/kube-rbac-proxy-web/0.log"
Jul 28 11:53:36.525786 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:36.525754    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bd83a23b-232a-4ed4-a5a8-562ff2704131/kube-rbac-proxy/0.log"
Jul 28 11:53:36.548207 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:36.548182    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bd83a23b-232a-4ed4-a5a8-562ff2704131/kube-rbac-proxy-thanos/0.log"
Jul 28 11:53:36.571115 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:36.571082    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bd83a23b-232a-4ed4-a5a8-562ff2704131/init-config-reloader/0.log"
Jul 28 11:53:36.602144 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:36.602113    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-ngzcw_949b3403-ad3f-4057-8470-4aa86a2b97c8/prometheus-operator/0.log"
Jul 28 11:53:36.630187 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:36.630155    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-ngzcw_949b3403-ad3f-4057-8470-4aa86a2b97c8/kube-rbac-proxy/0.log"
Jul 28 11:53:37.988196 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:37.988167    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-m4n6h_30e72787-1717-4a7a-b2e2-43e955527f77/networking-console-plugin/0.log"
Jul 28 11:53:38.857446 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:38.857418    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-598bbfdd7c-tb6nc_a9aa4162-c8f4-410b-b596-cbeb8f904aa6/console/0.log"
Jul 28 11:53:38.914135 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:38.914081    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-xktrq_a49e6876-498a-40b4-9ccf-94cf1d02bd97/download-server/0.log"
Jul 28 11:53:40.416199 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:40.416169    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nsq8v_c3fe785d-4bae-4988-85f0-2a02259913b1/dns/0.log"
Jul 28 11:53:40.440375 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:40.440344    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nsq8v_c3fe785d-4bae-4988-85f0-2a02259913b1/kube-rbac-proxy/0.log"
Jul 28 11:53:40.587955 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:40.587916    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5z8xh_f8ad89c4-5ef2-477b-ab4c-e2c97b9a68fa/dns-node-resolver/0.log"
Jul 28 11:53:41.181651 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:41.181620    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-64cd7c8686-n6ktg_dee5fe51-f82d-411e-8f64-9b36bd2623a3/registry/0.log"
Jul 28 11:53:41.238135 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:41.238102    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-h8krb_a1a543f9-5413-4260-b48d-f767a66f23c7/node-ca/0.log"
Jul 28 11:53:43.306058 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:43.306004    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ljlnk_74cc1682-9675-406b-8fe7-78713ec81657/serve-healthcheck-canary/0.log"
Jul 28 11:53:44.177924 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:44.177893    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mpcmc_a3e0525c-3c2b-4d2b-ba59-e815dfa07d51/kube-rbac-proxy/0.log"
Jul 28 11:53:44.212839 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:44.212797    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mpcmc_a3e0525c-3c2b-4d2b-ba59-e815dfa07d51/exporter/0.log"
Jul 28 11:53:44.249588 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:44.249497    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mpcmc_a3e0525c-3c2b-4d2b-ba59-e815dfa07d51/extractor/0.log"
Jul 28 11:53:45.310718 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:45.310681    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wxwvv/perf-node-gather-daemonset-79vwr"]
Jul 28 11:53:45.311128 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:45.311046    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="851dfec9-4179-470f-a411-0f12be26413b" containerName="main"
Jul 28 11:53:45.311128 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:45.311058    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="851dfec9-4179-470f-a411-0f12be26413b" containerName="main"
Jul 28 11:53:45.311128 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:45.311130    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="851dfec9-4179-470f-a411-0f12be26413b" containerName="main"
Jul 28 11:53:45.314656 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:45.314633    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wxwvv/perf-node-gather-daemonset-79vwr"
Jul 28 11:53:45.316871 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:45.316847    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wxwvv\"/\"kube-root-ca.crt\""
Jul 28 11:53:45.317203 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:45.317188    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wxwvv\"/\"default-dockercfg-crk46\""
Jul 28 11:53:45.317260 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:45.317239    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wxwvv\"/\"openshift-service-ca.crt\""
Jul 28 11:53:45.321168 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:45.321143    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wxwvv/perf-node-gather-daemonset-79vwr"]
Jul 28 11:53:45.448134 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:45.448088    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6871496a-2fc5-4597-a1b1-1c24425a51cf-podres\") pod \"perf-node-gather-daemonset-79vwr\" (UID: \"6871496a-2fc5-4597-a1b1-1c24425a51cf\") " pod="openshift-must-gather-wxwvv/perf-node-gather-daemonset-79vwr"
Jul 28 11:53:45.448134 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:45.448134    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6871496a-2fc5-4597-a1b1-1c24425a51cf-proc\") pod \"perf-node-gather-daemonset-79vwr\" (UID: \"6871496a-2fc5-4597-a1b1-1c24425a51cf\") " pod="openshift-must-gather-wxwvv/perf-node-gather-daemonset-79vwr"
Jul 28 11:53:45.448363 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:45.448154    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6871496a-2fc5-4597-a1b1-1c24425a51cf-lib-modules\") pod \"perf-node-gather-daemonset-79vwr\" (UID: \"6871496a-2fc5-4597-a1b1-1c24425a51cf\") " pod="openshift-must-gather-wxwvv/perf-node-gather-daemonset-79vwr"
Jul 28 11:53:45.448363 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:45.448229    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6871496a-2fc5-4597-a1b1-1c24425a51cf-sys\") pod \"perf-node-gather-daemonset-79vwr\" (UID: \"6871496a-2fc5-4597-a1b1-1c24425a51cf\") " pod="openshift-must-gather-wxwvv/perf-node-gather-daemonset-79vwr"
Jul 28 11:53:45.448363 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:45.448285    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th45f\" (UniqueName: \"kubernetes.io/projected/6871496a-2fc5-4597-a1b1-1c24425a51cf-kube-api-access-th45f\") pod \"perf-node-gather-daemonset-79vwr\" (UID: \"6871496a-2fc5-4597-a1b1-1c24425a51cf\") " pod="openshift-must-gather-wxwvv/perf-node-gather-daemonset-79vwr"
Jul 28 11:53:45.549224 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:45.549183    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6871496a-2fc5-4597-a1b1-1c24425a51cf-podres\") pod \"perf-node-gather-daemonset-79vwr\" (UID: \"6871496a-2fc5-4597-a1b1-1c24425a51cf\") " pod="openshift-must-gather-wxwvv/perf-node-gather-daemonset-79vwr"
Jul 28 11:53:45.549224 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:45.549231    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6871496a-2fc5-4597-a1b1-1c24425a51cf-proc\") pod \"perf-node-gather-daemonset-79vwr\" (UID: \"6871496a-2fc5-4597-a1b1-1c24425a51cf\") " pod="openshift-must-gather-wxwvv/perf-node-gather-daemonset-79vwr"
Jul 28 11:53:45.549471 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:45.549248    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6871496a-2fc5-4597-a1b1-1c24425a51cf-lib-modules\") pod \"perf-node-gather-daemonset-79vwr\" (UID: \"6871496a-2fc5-4597-a1b1-1c24425a51cf\") " pod="openshift-must-gather-wxwvv/perf-node-gather-daemonset-79vwr"
Jul 28 11:53:45.549471 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:45.549271    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6871496a-2fc5-4597-a1b1-1c24425a51cf-sys\") pod \"perf-node-gather-daemonset-79vwr\" (UID: \"6871496a-2fc5-4597-a1b1-1c24425a51cf\") " pod="openshift-must-gather-wxwvv/perf-node-gather-daemonset-79vwr"
Jul 28 11:53:45.549471 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:45.549310    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th45f\" (UniqueName: \"kubernetes.io/projected/6871496a-2fc5-4597-a1b1-1c24425a51cf-kube-api-access-th45f\") pod \"perf-node-gather-daemonset-79vwr\" (UID: \"6871496a-2fc5-4597-a1b1-1c24425a51cf\") " pod="openshift-must-gather-wxwvv/perf-node-gather-daemonset-79vwr"
Jul 28 11:53:45.549471 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:45.549323    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6871496a-2fc5-4597-a1b1-1c24425a51cf-proc\") pod \"perf-node-gather-daemonset-79vwr\" (UID: \"6871496a-2fc5-4597-a1b1-1c24425a51cf\") " pod="openshift-must-gather-wxwvv/perf-node-gather-daemonset-79vwr"
Jul 28 11:53:45.549471 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:45.549361    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6871496a-2fc5-4597-a1b1-1c24425a51cf-podres\") pod \"perf-node-gather-daemonset-79vwr\" (UID: \"6871496a-2fc5-4597-a1b1-1c24425a51cf\") " pod="openshift-must-gather-wxwvv/perf-node-gather-daemonset-79vwr"
Jul 28 11:53:45.549471 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:45.549399    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6871496a-2fc5-4597-a1b1-1c24425a51cf-sys\") pod \"perf-node-gather-daemonset-79vwr\" (UID: \"6871496a-2fc5-4597-a1b1-1c24425a51cf\") " pod="openshift-must-gather-wxwvv/perf-node-gather-daemonset-79vwr"
Jul 28 11:53:45.549471 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:45.549398    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6871496a-2fc5-4597-a1b1-1c24425a51cf-lib-modules\") pod \"perf-node-gather-daemonset-79vwr\" (UID: \"6871496a-2fc5-4597-a1b1-1c24425a51cf\") " pod="openshift-must-gather-wxwvv/perf-node-gather-daemonset-79vwr"
Jul 28 11:53:45.557112 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:45.557082    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-th45f\" (UniqueName: \"kubernetes.io/projected/6871496a-2fc5-4597-a1b1-1c24425a51cf-kube-api-access-th45f\") pod \"perf-node-gather-daemonset-79vwr\" (UID: \"6871496a-2fc5-4597-a1b1-1c24425a51cf\") " pod="openshift-must-gather-wxwvv/perf-node-gather-daemonset-79vwr"
Jul 28 11:53:45.626253 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:45.626159    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wxwvv/perf-node-gather-daemonset-79vwr"
Jul 28 11:53:45.771437 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:45.771412    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wxwvv/perf-node-gather-daemonset-79vwr"]
Jul 28 11:53:45.773891 ip-10-0-131-88 kubenswrapper[2557]: W0728 11:53:45.773862    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6871496a_2fc5_4597_a1b1_1c24425a51cf.slice/crio-136d4f9b27947661e83f5fcc3af2949c081012a6bfbbc31f3fbd0c0d9fed5a69 WatchSource:0}: Error finding container 136d4f9b27947661e83f5fcc3af2949c081012a6bfbbc31f3fbd0c0d9fed5a69: Status 404 returned error can't find the container with id 136d4f9b27947661e83f5fcc3af2949c081012a6bfbbc31f3fbd0c0d9fed5a69
Jul 28 11:53:46.147030 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:46.146991    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wxwvv/perf-node-gather-daemonset-79vwr" event={"ID":"6871496a-2fc5-4597-a1b1-1c24425a51cf","Type":"ContainerStarted","Data":"137abbaa5836231bf6034df304d5795c639cc76c5bc033be7e992e9f93502d1a"}
Jul 28 11:53:46.147231 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:46.147047    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wxwvv/perf-node-gather-daemonset-79vwr" event={"ID":"6871496a-2fc5-4597-a1b1-1c24425a51cf","Type":"ContainerStarted","Data":"136d4f9b27947661e83f5fcc3af2949c081012a6bfbbc31f3fbd0c0d9fed5a69"}
Jul 28 11:53:46.147231 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:46.147080    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wxwvv/perf-node-gather-daemonset-79vwr"
Jul 28 11:53:46.163937 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:46.163878    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wxwvv/perf-node-gather-daemonset-79vwr" podStartSLOduration=1.163858679 podStartE2EDuration="1.163858679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 11:53:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:53:46.161657029 +0000 UTC m=+6362.120425399" watchObservedRunningTime="2026-07-28 11:53:46.163858679 +0000 UTC m=+6362.122627063"
Jul 28 11:53:47.420240 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:47.420207    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-68cc75d5fc-8886b_96f79162-8c39-4e0e-8c22-0513f8b708c7/manager/0.log"
Jul 28 11:53:52.161825 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:52.161791    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wxwvv/perf-node-gather-daemonset-79vwr"
Jul 28 11:53:55.970640 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:55.970534    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gzgcl_f9fe54ae-5731-4fdd-b668-c70aca78b5bf/kube-multus-additional-cni-plugins/0.log"
Jul 28 11:53:55.999503 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:55.999471    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gzgcl_f9fe54ae-5731-4fdd-b668-c70aca78b5bf/egress-router-binary-copy/0.log"
Jul 28 11:53:56.022995 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:56.022971    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gzgcl_f9fe54ae-5731-4fdd-b668-c70aca78b5bf/cni-plugins/0.log"
Jul 28 11:53:56.050067 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:56.050031    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gzgcl_f9fe54ae-5731-4fdd-b668-c70aca78b5bf/bond-cni-plugin/0.log"
Jul 28 11:53:56.078025 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:56.077997    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gzgcl_f9fe54ae-5731-4fdd-b668-c70aca78b5bf/routeoverride-cni/0.log"
Jul 28 11:53:56.101422 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:56.101396    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gzgcl_f9fe54ae-5731-4fdd-b668-c70aca78b5bf/whereabouts-cni-bincopy/0.log"
Jul 28 11:53:56.126824 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:56.126792    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gzgcl_f9fe54ae-5731-4fdd-b668-c70aca78b5bf/whereabouts-cni/0.log"
Jul 28 11:53:56.228399 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:56.228310    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-p2rv8_4e80f436-0cf4-45a3-8fdf-53bdae6a7fad/kube-multus/0.log"
Jul 28 11:53:56.316594 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:56.316564    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-8fltb_d23b40b0-9299-4a0b-a87c-d2f2cb921fd9/network-metrics-daemon/0.log"
Jul 28 11:53:56.340594 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:56.340566    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-8fltb_d23b40b0-9299-4a0b-a87c-d2f2cb921fd9/kube-rbac-proxy/0.log"
Jul 28 11:53:58.012837 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:58.012806    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hdqls_ceccf442-0ab2-4993-bf49-d0375abe1e62/ovn-controller/0.log"
Jul 28 11:53:58.035913 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:58.035869    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hdqls_ceccf442-0ab2-4993-bf49-d0375abe1e62/ovn-acl-logging/0.log"
Jul 28 11:53:58.069643 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:58.069592    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hdqls_ceccf442-0ab2-4993-bf49-d0375abe1e62/kube-rbac-proxy-node/0.log"
Jul 28 11:53:58.101909 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:58.101869    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hdqls_ceccf442-0ab2-4993-bf49-d0375abe1e62/kube-rbac-proxy-ovn-metrics/0.log"
Jul 28 11:53:58.130042 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:58.130016    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hdqls_ceccf442-0ab2-4993-bf49-d0375abe1e62/northd/0.log"
Jul 28 11:53:58.163577 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:58.163542    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hdqls_ceccf442-0ab2-4993-bf49-d0375abe1e62/nbdb/0.log"
Jul 28 11:53:58.192094 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:58.192062    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hdqls_ceccf442-0ab2-4993-bf49-d0375abe1e62/sbdb/0.log"
Jul 28 11:53:58.289334 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:58.289246    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hdqls_ceccf442-0ab2-4993-bf49-d0375abe1e62/ovnkube-controller/0.log"
Jul 28 11:53:59.800962 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:53:59.800934    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-ddrhj_0bc74d3d-3abf-47f5-8149-7d07a9a34e7c/network-check-target-container/0.log"
Jul 28 11:54:00.962665 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:54:00.962629    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-vgktv_58c000f3-06db-4572-8abb-89ca11078c3c/iptables-alerter/0.log"
Jul 28 11:54:01.823368 ip-10-0-131-88 kubenswrapper[2557]: I0728 11:54:01.823334    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-msf7n_2c5a2bd4-8e86-493f-9479-1a183cc866e6/tuned/0.log"