Jul 31 23:43:54.937681 ip-10-0-130-148 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 31 23:43:54.937696 ip-10-0-130-148 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 31 23:43:54.937705 ip-10-0-130-148 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 31 23:43:54.938018 ip-10-0-130-148 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 31 23:44:06.164019 ip-10-0-130-148 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 31 23:44:06.164035 ip-10-0-130-148 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot c765851366bb4fa8adbe0bc1c51bb96f --
Jul 31 23:45:37.154017 ip-10-0-130-148 systemd[1]: Starting Kubernetes Kubelet...
Jul 31 23:45:37.599544 ip-10-0-130-148 kubenswrapper[2559]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 31 23:45:37.599544 ip-10-0-130-148 kubenswrapper[2559]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 31 23:45:37.599544 ip-10-0-130-148 kubenswrapper[2559]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 31 23:45:37.599544 ip-10-0-130-148 kubenswrapper[2559]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 31 23:45:37.599544 ip-10-0-130-148 kubenswrapper[2559]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 31 23:45:37.600249 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.599597    2559 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 31 23:45:37.604386 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604371    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 23:45:37.604386 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604385    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 23:45:37.604456 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604390    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 23:45:37.604456 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604393    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 23:45:37.604456 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604396    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 23:45:37.604456 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604399    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 23:45:37.604456 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604403    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 23:45:37.604456 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604407    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 23:45:37.604456 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604411    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 23:45:37.604456 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604414    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 23:45:37.604456 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604417    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 23:45:37.604456 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604420    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 23:45:37.604456 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604423    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 23:45:37.604456 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604425    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 23:45:37.604456 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604431    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 23:45:37.604456 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604435    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 23:45:37.604456 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604437    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 23:45:37.604456 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604440    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 23:45:37.604456 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604442    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 23:45:37.604456 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604446    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 23:45:37.604456 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604450    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 23:45:37.604930 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604453    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 23:45:37.604930 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604456    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 23:45:37.604930 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604459    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 23:45:37.604930 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604462    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 23:45:37.604930 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604465    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 23:45:37.604930 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604468    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 23:45:37.604930 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604471    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 23:45:37.604930 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604474    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 23:45:37.604930 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604476    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 23:45:37.604930 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604479    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 23:45:37.604930 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604481    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 23:45:37.604930 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604485    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 23:45:37.604930 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604489    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 23:45:37.604930 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604493    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 23:45:37.604930 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604496    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 23:45:37.604930 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604498    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 23:45:37.604930 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604501    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 23:45:37.604930 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604503    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 23:45:37.604930 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604506    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 23:45:37.605419 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604508    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 23:45:37.605419 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604511    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 23:45:37.605419 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604514    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 23:45:37.605419 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604517    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 23:45:37.605419 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604519    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 23:45:37.605419 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604522    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 23:45:37.605419 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604524    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 23:45:37.605419 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604527    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 23:45:37.605419 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604529    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 23:45:37.605419 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604532    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 23:45:37.605419 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604534    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 23:45:37.605419 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604537    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 23:45:37.605419 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604539    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 23:45:37.605419 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604542    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 23:45:37.605419 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604546    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 23:45:37.605419 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604549    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 23:45:37.605419 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604551    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 23:45:37.605419 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604554    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 23:45:37.605419 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604556    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 23:45:37.605925 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604559    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 23:45:37.605925 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604561    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 23:45:37.605925 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604564    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 23:45:37.605925 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604566    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 23:45:37.605925 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604569    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 23:45:37.605925 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604571    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 23:45:37.605925 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604574    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 23:45:37.605925 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604576    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 23:45:37.605925 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604579    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 23:45:37.605925 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604581    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 23:45:37.605925 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604584    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 23:45:37.605925 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604586    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 23:45:37.605925 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604589    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 23:45:37.605925 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604592    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 23:45:37.605925 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604594    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 23:45:37.605925 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604597    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 23:45:37.605925 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604600    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 23:45:37.605925 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604602    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 23:45:37.605925 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604605    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 23:45:37.605925 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604607    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 23:45:37.606436 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604610    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 23:45:37.606436 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604612    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 23:45:37.606436 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604615    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 23:45:37.606436 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604617    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 23:45:37.606436 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604620    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 23:45:37.606436 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604622    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 23:45:37.606436 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604625    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 23:45:37.606436 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604628    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 23:45:37.606436 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604630    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 23:45:37.606436 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604633    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 23:45:37.606436 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604635    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 23:45:37.606436 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604638    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 23:45:37.606436 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604640    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 23:45:37.606436 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604643    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 23:45:37.606436 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604646    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 23:45:37.606436 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604648    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 23:45:37.606436 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604651    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 23:45:37.606436 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604653    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 23:45:37.606436 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604656    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 23:45:37.606436 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.604658    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 23:45:37.607139 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.606813    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 23:45:37.607139 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.606827    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 23:45:37.607139 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.606833    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 23:45:37.607139 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.606844    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 23:45:37.607139 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.606849    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 23:45:37.607139 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.606854    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 23:45:37.607139 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.606858    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 23:45:37.607139 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.606863    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 23:45:37.607139 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.606867    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 23:45:37.607139 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.606883    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 23:45:37.607139 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.606887    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 23:45:37.607139 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.606892    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 23:45:37.607139 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.606896    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 23:45:37.607139 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.606900    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 23:45:37.607139 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.606904    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 23:45:37.607139 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.606909    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 23:45:37.607139 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.606918    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 23:45:37.607139 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.606923    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 23:45:37.607139 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.606927    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 23:45:37.607844 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.606931    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 23:45:37.607844 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.606936    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 23:45:37.607844 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.606940    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 23:45:37.607844 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.606944    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 23:45:37.607844 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.606948    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 23:45:37.607844 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.606953    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 23:45:37.607844 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.606957    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 23:45:37.607844 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.606961    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 23:45:37.607844 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.606965    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 23:45:37.607844 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.606974    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 23:45:37.607844 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.606979    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 23:45:37.607844 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.606983    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 23:45:37.607844 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.606987    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 23:45:37.607844 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.606993    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 23:45:37.607844 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.606998    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 23:45:37.607844 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607003    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 23:45:37.607844 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607007    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 23:45:37.607844 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607011    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 23:45:37.607844 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607015    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 23:45:37.607844 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607019    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 23:45:37.608381 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607023    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 23:45:37.608381 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607027    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 23:45:37.608381 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607036    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 23:45:37.608381 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607044    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 23:45:37.608381 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607050    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 23:45:37.608381 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607054    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 23:45:37.608381 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607059    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 23:45:37.608381 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607064    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 23:45:37.608381 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607068    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 23:45:37.608381 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607072    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 23:45:37.608381 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607078    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 23:45:37.608381 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607084    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 23:45:37.608381 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607088    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 23:45:37.608381 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607092    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 23:45:37.608381 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607102    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 23:45:37.608381 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607106    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 23:45:37.608381 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607111    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 23:45:37.608381 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607115    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 23:45:37.608381 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607120    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 23:45:37.608884 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607125    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 23:45:37.608884 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607132    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 23:45:37.608884 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607136    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 23:45:37.608884 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607143    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 23:45:37.608884 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607148    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 23:45:37.608884 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607154    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 23:45:37.608884 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607184    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 23:45:37.608884 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607192    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 23:45:37.608884 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607198    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 23:45:37.608884 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607204    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 23:45:37.608884 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607209    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 23:45:37.608884 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607214    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 23:45:37.608884 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607218    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 23:45:37.608884 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607223    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 23:45:37.608884 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607228    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 23:45:37.608884 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607232    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 23:45:37.608884 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607238    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 23:45:37.608884 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607242    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 23:45:37.608884 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607254    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 23:45:37.609360 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607258    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 23:45:37.609360 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607263    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 23:45:37.609360 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607268    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 23:45:37.609360 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607273    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 23:45:37.609360 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607287    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 23:45:37.609360 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607294    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 23:45:37.609360 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607298    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 23:45:37.609360 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607302    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 23:45:37.609360 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607307    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 23:45:37.609360 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607311    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 23:45:37.609360 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607321    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 23:45:37.609360 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607325    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 23:45:37.609360 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607329    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 23:45:37.609360 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607334    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 23:45:37.609360 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607338    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 23:45:37.609360 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607343    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 23:45:37.609360 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607348    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 23:45:37.609360 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607352    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 23:45:37.609360 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607357    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 23:45:37.609360 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607362    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 23:45:37.609878 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607367    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 23:45:37.609878 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.607371    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 23:45:37.609878 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608022    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 23:45:37.609878 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608029    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 23:45:37.609878 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608032    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 23:45:37.609878 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608035    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 23:45:37.609878 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608038    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 23:45:37.609878 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608041    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 23:45:37.609878 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608043    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 23:45:37.609878 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608046    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 23:45:37.609878 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608048    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 23:45:37.609878 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608051    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 23:45:37.609878 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608055    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 23:45:37.609878 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608060    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 23:45:37.609878 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608064    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 23:45:37.609878 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608067    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 23:45:37.609878 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608070    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 23:45:37.609878 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608073    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 23:45:37.609878 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608076    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 23:45:37.610344 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608079    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 23:45:37.610344 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608081    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 23:45:37.610344 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608084    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 23:45:37.610344 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608086    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 23:45:37.610344 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608089    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 23:45:37.610344 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608092    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 23:45:37.610344 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608094    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 23:45:37.610344 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608097    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 23:45:37.610344 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608100    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 23:45:37.610344 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608103    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 23:45:37.610344 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608105    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 23:45:37.610344 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608108    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 23:45:37.610344 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608111    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 23:45:37.610344 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608114    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 23:45:37.610344 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608117    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 23:45:37.610344 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608119    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 23:45:37.610344 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608122    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 23:45:37.610344 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608124    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 23:45:37.610344 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608127    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 23:45:37.610344 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608130    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 23:45:37.610858 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608132    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 23:45:37.610858 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608134    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 23:45:37.610858 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608137    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 23:45:37.610858 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608140    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 23:45:37.610858 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608142    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 23:45:37.610858 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608145    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 23:45:37.610858 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608148    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 23:45:37.610858 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608151    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 23:45:37.610858 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608153    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 23:45:37.610858 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608156    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 23:45:37.610858 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608158    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 23:45:37.610858 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608160    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 23:45:37.610858 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608163    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 23:45:37.610858 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608165    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 23:45:37.610858 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608168    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 23:45:37.610858 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608171    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 23:45:37.610858 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608173    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 23:45:37.610858 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608176    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 23:45:37.610858 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608179    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 23:45:37.610858 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608181    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 23:45:37.611353 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608184    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 23:45:37.611353 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608186    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 23:45:37.611353 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608190    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 23:45:37.611353 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608196    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 23:45:37.611353 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608199    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 23:45:37.611353 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608202    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 23:45:37.611353 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608204    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 23:45:37.611353 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608207    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 23:45:37.611353 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608209    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 23:45:37.611353 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608212    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 23:45:37.611353 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608215    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 23:45:37.611353 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608217    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 23:45:37.611353 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608220    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 23:45:37.611353 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608222    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 23:45:37.611353 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608224    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 23:45:37.611353 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608227    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 23:45:37.611353 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608230    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 23:45:37.611353 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608233    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 23:45:37.611353 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608235    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 23:45:37.611849 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608238    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 23:45:37.611849 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608240    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 23:45:37.611849 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608243    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 23:45:37.611849 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608245    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 23:45:37.611849 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608248    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 23:45:37.611849 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608251    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 23:45:37.611849 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608254    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 23:45:37.611849 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608256    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 23:45:37.611849 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608259    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 23:45:37.611849 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608261    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 23:45:37.611849 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608265    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 23:45:37.611849 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608267    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 23:45:37.611849 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608270    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 23:45:37.611849 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608272    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 23:45:37.611849 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608275    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 23:45:37.611849 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608278    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 23:45:37.611849 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608282    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 23:45:37.611849 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608285    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 23:45:37.611849 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608287    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 23:45:37.611849 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608290    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 23:45:37.612328 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608292    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 23:45:37.612328 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608294    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 23:45:37.612328 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.608297    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 23:45:37.612328 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608366    2559 flags.go:64] FLAG: --address="0.0.0.0"
Jul 31 23:45:37.612328 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608373    2559 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 31 23:45:37.612328 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608379    2559 flags.go:64] FLAG: --anonymous-auth="true"
Jul 31 23:45:37.612328 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608383    2559 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 31 23:45:37.612328 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608387    2559 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 31 23:45:37.612328 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608391    2559 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 31 23:45:37.612328 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608395    2559 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 31 23:45:37.612328 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608400    2559 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 31 23:45:37.612328 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608403    2559 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 31 23:45:37.612328 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608406    2559 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 31 23:45:37.612328 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608409    2559 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 31 23:45:37.612328 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608412    2559 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 31 23:45:37.612328 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608415    2559 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 31 23:45:37.612328 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608418    2559 flags.go:64] FLAG: --cgroup-root=""
Jul 31 23:45:37.612328 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608421    2559 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 31 23:45:37.612328 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608425    2559 flags.go:64] FLAG: --client-ca-file=""
Jul 31 23:45:37.612328 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608434    2559 flags.go:64] FLAG: --cloud-provider="external"
Jul 31 23:45:37.612328 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608437    2559 flags.go:64] FLAG: --cluster-dns="[]"
Jul 31 23:45:37.612328 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608441    2559 flags.go:64] FLAG: --cluster-domain=""
Jul 31 23:45:37.612328 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608444    2559 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 31 23:45:37.612328 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608447    2559 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 31 23:45:37.612931 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608452    2559 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 31 23:45:37.612931 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608455    2559 flags.go:64] FLAG: --container-log-max-files="5"
Jul 31 23:45:37.612931 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608459    2559 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 31 23:45:37.612931 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608462    2559 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 31 23:45:37.612931 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608466    2559 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 31 23:45:37.612931 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608469    2559 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 31 23:45:37.612931 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608472    2559 flags.go:64] FLAG: --contention-profiling="false"
Jul 31 23:45:37.612931 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608475    2559 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 31 23:45:37.612931 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608478    2559 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 31 23:45:37.612931 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608481    2559 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 31 23:45:37.612931 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608484    2559 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 31 23:45:37.612931 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608488    2559 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 31 23:45:37.612931 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608491    2559 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 31 23:45:37.612931 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608494    2559 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 31 23:45:37.612931 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608496    2559 flags.go:64] FLAG: --enable-load-reader="false"
Jul 31 23:45:37.612931 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608499    2559 flags.go:64] FLAG: --enable-server="true"
Jul 31 23:45:37.612931 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608502    2559 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 31 23:45:37.612931 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608509    2559 flags.go:64] FLAG: --event-burst="100"
Jul 31 23:45:37.612931 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608512    2559 flags.go:64] FLAG: --event-qps="50"
Jul 31 23:45:37.612931 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608515    2559 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 31 23:45:37.612931 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608518    2559 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 31 23:45:37.612931 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608521    2559 flags.go:64] FLAG: --eviction-hard=""
Jul 31 23:45:37.612931 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608525    2559 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 31 23:45:37.612931 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608527    2559 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 31 23:45:37.613542 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608530    2559 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 31 23:45:37.613542 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608533    2559 flags.go:64] FLAG: --eviction-soft=""
Jul 31 23:45:37.613542 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608536    2559 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 31 23:45:37.613542 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608539    2559 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 31 23:45:37.613542 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608547    2559 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 31 23:45:37.613542 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608550    2559 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 31 23:45:37.613542 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608554    2559 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 31 23:45:37.613542 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608557    2559 flags.go:64] FLAG: --fail-swap-on="true"
Jul 31 23:45:37.613542 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608561    2559 flags.go:64] FLAG: --feature-gates=""
Jul 31 23:45:37.613542 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608569    2559 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 31 23:45:37.613542 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608572    2559 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 31 23:45:37.613542 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608575    2559 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 31 23:45:37.613542 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608579    2559 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 31 23:45:37.613542 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608582    2559 flags.go:64] FLAG: --healthz-port="10248"
Jul 31 23:45:37.613542 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608585    2559 flags.go:64] FLAG: --help="false"
Jul 31 23:45:37.613542 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608588    2559 flags.go:64] FLAG: --hostname-override="ip-10-0-130-148.ec2.internal"
Jul 31 23:45:37.613542 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608591    2559 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 31 23:45:37.613542 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608594    2559 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 31 23:45:37.613542 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608597    2559 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 31 23:45:37.613542 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608600    2559 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 31 23:45:37.613542 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608604    2559 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 31 23:45:37.613542 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608607    2559 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 31 23:45:37.613542 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608609    2559 flags.go:64] FLAG: --image-service-endpoint=""
Jul 31 23:45:37.614206 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608612    2559 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 31 23:45:37.614206 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608615    2559 flags.go:64] FLAG: --kube-api-burst="100"
Jul 31 23:45:37.614206 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608617    2559 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 31 23:45:37.614206 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608620    2559 flags.go:64] FLAG: --kube-api-qps="50"
Jul 31 23:45:37.614206 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608623    2559 flags.go:64] FLAG: --kube-reserved=""
Jul 31 23:45:37.614206 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608626    2559 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 31 23:45:37.614206 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608629    2559 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 31 23:45:37.614206 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608632    2559 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 31 23:45:37.614206 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608635    2559 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 31 23:45:37.614206 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608638    2559 flags.go:64] FLAG: --lock-file=""
Jul 31 23:45:37.614206 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608640    2559 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 31 23:45:37.614206 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608643    2559 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 31 23:45:37.614206 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608646    2559 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 31 23:45:37.614206 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608651    2559 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 31 23:45:37.614206 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608661    2559 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 31 23:45:37.614206 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608665    2559 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 31 23:45:37.614206 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608667    2559 flags.go:64] FLAG: --logging-format="text"
Jul 31 23:45:37.614206 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608671    2559 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 31 23:45:37.614206 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608674    2559 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 31 23:45:37.614206 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608677    2559 flags.go:64] FLAG: --manifest-url=""
Jul 31 23:45:37.614206 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608680    2559 flags.go:64] FLAG: --manifest-url-header=""
Jul 31 23:45:37.614206 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608685    2559 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 31 23:45:37.614206 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608688    2559 flags.go:64] FLAG: --max-open-files="1000000"
Jul 31 23:45:37.614206 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608692    2559 flags.go:64] FLAG: --max-pods="110"
Jul 31 23:45:37.614206 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608695    2559 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 31 23:45:37.614848 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608698    2559 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 31 23:45:37.614848 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608701    2559 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 31 23:45:37.614848 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608704    2559 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 31 23:45:37.614848 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608706    2559 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 31 23:45:37.614848 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608709    2559 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 31 23:45:37.614848 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608712    2559 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 31 23:45:37.614848 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608719    2559 flags.go:64] FLAG: --node-status-max-images="50"
Jul 31 23:45:37.614848 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608722    2559 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 31 23:45:37.614848 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608724    2559 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 31 23:45:37.614848 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608727    2559 flags.go:64] FLAG: --pod-cidr=""
Jul 31 23:45:37.614848 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608730    2559 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8"
Jul 31 23:45:37.614848 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608735    2559 flags.go:64] FLAG: --pod-manifest-path=""
Jul 31 23:45:37.614848 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608738    2559 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 31 23:45:37.614848 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608757    2559 flags.go:64] FLAG: --pods-per-core="0"
Jul 31 23:45:37.614848 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608760    2559 flags.go:64] FLAG: --port="10250"
Jul 31 23:45:37.614848 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608763    2559 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 31 23:45:37.614848 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608766    2559 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c8645677f86c52c0"
Jul 31 23:45:37.614848 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608769    2559 flags.go:64] FLAG: --qos-reserved=""
Jul 31 23:45:37.614848 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608772    2559 flags.go:64] FLAG: --read-only-port="10255"
Jul 31 23:45:37.614848 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608775    2559 flags.go:64] FLAG: --register-node="true"
Jul 31 23:45:37.614848 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608778    2559 flags.go:64] FLAG: --register-with-taints=""
Jul 31 23:45:37.614848 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608782    2559 flags.go:64] FLAG: --registry-burst="10"
Jul 31 23:45:37.614848 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608785    2559 flags.go:64] FLAG: --registry-qps="5"
Jul 31 23:45:37.614848 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608788    2559 flags.go:64] FLAG: --reserved-cpus=""
Jul 31 23:45:37.615568 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608791    2559 flags.go:64] FLAG: --reserved-memory=""
Jul 31 23:45:37.615568 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608796    2559 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 31 23:45:37.615568 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608799    2559 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 31 23:45:37.615568 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608802    2559 flags.go:64] FLAG: --rotate-certificates="false"
Jul 31 23:45:37.615568 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608806    2559 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 31 23:45:37.615568 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608809    2559 flags.go:64] FLAG: --runonce="false"
Jul 31 23:45:37.615568 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608812    2559 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 31 23:45:37.615568 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608815    2559 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 31 23:45:37.615568 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608818    2559 flags.go:64] FLAG: --seccomp-default="false"
Jul 31 23:45:37.615568 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608820    2559 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 31 23:45:37.615568 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608823    2559 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 31 23:45:37.615568 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608826    2559 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 31 23:45:37.615568 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608829    2559 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 31 23:45:37.615568 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608832    2559 flags.go:64] FLAG: --storage-driver-password="root"
Jul 31 23:45:37.615568 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608835    2559 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 31 23:45:37.615568 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608838    2559 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 31 23:45:37.615568 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608841    2559 flags.go:64] FLAG: --storage-driver-user="root"
Jul 31 23:45:37.615568 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608844    2559 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 31 23:45:37.615568 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608847    2559 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 31 23:45:37.615568 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608850    2559 flags.go:64] FLAG: --system-cgroups=""
Jul 31 23:45:37.615568 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608853    2559 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 31 23:45:37.615568 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608858    2559 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 31 23:45:37.615568 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608861    2559 flags.go:64] FLAG: --tls-cert-file=""
Jul 31 23:45:37.615568 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608864    2559 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 31 23:45:37.615568 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608870    2559 flags.go:64] FLAG: --tls-min-version=""
Jul 31 23:45:37.616190 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608873    2559 flags.go:64] FLAG: --tls-private-key-file=""
Jul 31 23:45:37.616190 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608875    2559 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 31 23:45:37.616190 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608878    2559 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 31 23:45:37.616190 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608881    2559 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 31 23:45:37.616190 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608884    2559 flags.go:64] FLAG: --v="2"
Jul 31 23:45:37.616190 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608888    2559 flags.go:64] FLAG: --version="false"
Jul 31 23:45:37.616190 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608893    2559 flags.go:64] FLAG: --vmodule=""
Jul 31 23:45:37.616190 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608897    2559 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 31 23:45:37.616190 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.608902    2559 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 31 23:45:37.616190 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609023    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 23:45:37.616190 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609027    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 23:45:37.616190 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609031    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 23:45:37.616190 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609034    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 23:45:37.616190 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609037    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 23:45:37.616190 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609040    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 23:45:37.616190 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609042    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 23:45:37.616190 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609045    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 23:45:37.616190 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609047    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 23:45:37.616190 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609050    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 23:45:37.616190 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609053    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 23:45:37.616190 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609055    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 23:45:37.616190 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609058    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 23:45:37.616734 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609060    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 23:45:37.616734 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609063    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 23:45:37.616734 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609066    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 23:45:37.616734 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609068    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 23:45:37.616734 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609071    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 23:45:37.616734 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609073    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 23:45:37.616734 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609076    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 23:45:37.616734 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609078    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 23:45:37.616734 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609081    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 23:45:37.616734 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609083    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 23:45:37.616734 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609085    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 23:45:37.616734 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609088    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 23:45:37.616734 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609090    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 23:45:37.616734 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609093    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 23:45:37.616734 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609096    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 23:45:37.616734 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609098    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 23:45:37.616734 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609101    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 23:45:37.616734 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609106    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 23:45:37.616734 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609110    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 23:45:37.616734 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609113    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 23:45:37.617262 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609116    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 23:45:37.617262 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609120    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 23:45:37.617262 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609122    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 23:45:37.617262 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609125    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 23:45:37.617262 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609127    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 23:45:37.617262 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609130    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 23:45:37.617262 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609132    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 23:45:37.617262 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609135    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 23:45:37.617262 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609138    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 23:45:37.617262 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609140    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 23:45:37.617262 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609144    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 23:45:37.617262 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609147    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 23:45:37.617262 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609151    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 23:45:37.617262 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609154    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 23:45:37.617262 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609156    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 23:45:37.617262 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609159    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 23:45:37.617262 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609162    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 23:45:37.617262 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609165    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 23:45:37.617262 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609167    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 23:45:37.617262 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609170    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 23:45:37.617836 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609172    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 23:45:37.617836 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609175    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 23:45:37.617836 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609177    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 23:45:37.617836 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609180    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 23:45:37.617836 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609182    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 23:45:37.617836 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609185    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 23:45:37.617836 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609187    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 23:45:37.617836 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609190    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 23:45:37.617836 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609192    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 23:45:37.617836 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609195    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 23:45:37.617836 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609199    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 23:45:37.617836 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609202    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 23:45:37.617836 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609204    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 23:45:37.617836 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609208    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 23:45:37.617836 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609211    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 23:45:37.617836 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609213    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 23:45:37.617836 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609216    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 23:45:37.617836 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609218    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 23:45:37.617836 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609221    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 23:45:37.618370 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609223    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 23:45:37.618370 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609226    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 23:45:37.618370 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609228    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 23:45:37.618370 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609231    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 23:45:37.618370 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609233    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 23:45:37.618370 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609236    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 23:45:37.618370 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609240    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 23:45:37.618370 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609244    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 23:45:37.618370 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609248    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 23:45:37.618370 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609251    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 23:45:37.618370 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609254    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 23:45:37.618370 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609256    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 23:45:37.618370 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609259    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 23:45:37.618370 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609261    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 23:45:37.618370 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609264    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 23:45:37.618370 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609266    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 23:45:37.618370 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609269    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 23:45:37.618370 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609271    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 23:45:37.618370 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609274    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 23:45:37.618855 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609277    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 23:45:37.618855 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609279    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 23:45:37.618855 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609282    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 23:45:37.618855 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609284    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 23:45:37.618855 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609288    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 23:45:37.618855 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609291    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 23:45:37.618855 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609294    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 23:45:37.618855 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.609298    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 23:45:37.618855 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.609968    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 31 23:45:37.618855 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.616790    2559 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 31 23:45:37.618855 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.616805    2559 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 31 23:45:37.618855 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.616819    2559 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 31 23:45:37.618855 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.616825    2559 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 31 23:45:37.618855 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.616879    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 23:45:37.618855 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.616883    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 23:45:37.619239 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.616887    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 23:45:37.619239 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.616891    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 23:45:37.619239 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.616896    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 23:45:37.619239 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.616899    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 23:45:37.619239 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.616902    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 23:45:37.619239 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.616905    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 23:45:37.619239 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.616908    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 23:45:37.619239 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.616911    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 23:45:37.619239 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.616913    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 23:45:37.619239 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.616916    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 23:45:37.619239 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.616919    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 23:45:37.619239 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.616921    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 23:45:37.619239 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.616925    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 23:45:37.619239 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.616927    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 23:45:37.619239 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.616930    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 23:45:37.619239 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.616932    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 23:45:37.619239 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.616934    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 23:45:37.619239 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.616937    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 23:45:37.619239 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.616940    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 23:45:37.619693 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.616942    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 23:45:37.619693 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.616946    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 23:45:37.619693 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.616950    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 23:45:37.619693 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.616952    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 23:45:37.619693 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.616956    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 23:45:37.619693 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.616958    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 23:45:37.619693 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.616961    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 23:45:37.619693 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.616964    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 23:45:37.619693 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.616967    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 23:45:37.619693 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.616970    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 23:45:37.619693 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.616973    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 23:45:37.619693 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.616976    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 23:45:37.619693 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.616978    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 23:45:37.619693 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.616981    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 23:45:37.619693 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.616983    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 23:45:37.619693 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.616986    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 23:45:37.619693 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.616989    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 23:45:37.619693 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.616991    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 23:45:37.619693 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.616993    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 23:45:37.620217 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.616996    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 23:45:37.620217 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.616998    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 23:45:37.620217 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617001    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 23:45:37.620217 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617003    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 23:45:37.620217 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617006    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 23:45:37.620217 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617008    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 23:45:37.620217 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617011    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 23:45:37.620217 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617013    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 23:45:37.620217 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617016    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 23:45:37.620217 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617018    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 23:45:37.620217 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617021    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 23:45:37.620217 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617024    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 23:45:37.620217 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617026    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 23:45:37.620217 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617029    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 23:45:37.620217 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617031    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 23:45:37.620217 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617034    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 23:45:37.620217 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617036    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 23:45:37.620217 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617039    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 23:45:37.620217 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617041    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 23:45:37.620217 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617044    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 23:45:37.620709 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617047    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 23:45:37.620709 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617050    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 23:45:37.620709 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617053    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 23:45:37.620709 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617055    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 23:45:37.620709 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617059    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 23:45:37.620709 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617063    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 23:45:37.620709 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617066    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 23:45:37.620709 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617068    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 23:45:37.620709 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617071    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 23:45:37.620709 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617074    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 23:45:37.620709 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617077    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 23:45:37.620709 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617079    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 23:45:37.620709 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617082    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 23:45:37.620709 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617084    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 23:45:37.620709 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617087    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 23:45:37.620709 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617089    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 23:45:37.620709 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617092    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 23:45:37.620709 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617094    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 23:45:37.620709 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617097    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 23:45:37.621240 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617099    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 23:45:37.621240 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617102    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 23:45:37.621240 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617104    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 23:45:37.621240 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617107    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 23:45:37.621240 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617109    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 23:45:37.621240 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617112    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 23:45:37.621240 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617114    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 23:45:37.621240 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617117    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 23:45:37.621240 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617119    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 23:45:37.621240 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617122    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 23:45:37.621240 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617125    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 23:45:37.621240 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617128    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 23:45:37.621240 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617130    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 23:45:37.621240 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617133    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 23:45:37.621240 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617136    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 23:45:37.621240 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617139    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 23:45:37.621240 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617142    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 23:45:37.621240 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617145    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 23:45:37.621240 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617148    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 23:45:37.621240 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617150    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 23:45:37.621894 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.617155    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 31 23:45:37.621894 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617264    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 23:45:37.621894 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617268    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 23:45:37.621894 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617271    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 23:45:37.621894 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617274    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 23:45:37.621894 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617277    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 23:45:37.621894 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617281    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 23:45:37.621894 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617285    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 23:45:37.621894 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617288    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 23:45:37.621894 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617291    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 23:45:37.621894 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617293    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 23:45:37.621894 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617296    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 23:45:37.621894 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617299    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 23:45:37.621894 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617302    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 23:45:37.621894 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617304    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 23:45:37.622262 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617307    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 23:45:37.622262 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617309    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 23:45:37.622262 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617312    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 23:45:37.622262 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617314    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 23:45:37.622262 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617317    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 23:45:37.622262 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617319    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 23:45:37.622262 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617322    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 23:45:37.622262 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617324    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 23:45:37.622262 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617327    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 23:45:37.622262 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617329    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 23:45:37.622262 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617332    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 23:45:37.622262 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617335    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 23:45:37.622262 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617337    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 23:45:37.622262 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617340    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 23:45:37.622262 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617343    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 23:45:37.622262 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617346    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 23:45:37.622262 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617348    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 23:45:37.622262 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617351    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 23:45:37.622262 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617353    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 23:45:37.622262 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617355    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 23:45:37.622752 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617358    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 23:45:37.622752 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617361    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 23:45:37.622752 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617363    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 23:45:37.622752 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617365    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 23:45:37.622752 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617368    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 23:45:37.622752 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617370    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 23:45:37.622752 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617373    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 23:45:37.622752 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617377    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 23:45:37.622752 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617380    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 23:45:37.622752 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617382    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 23:45:37.622752 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617385    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 23:45:37.622752 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617387    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 23:45:37.622752 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617390    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 23:45:37.622752 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617392    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 23:45:37.622752 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617395    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 23:45:37.622752 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617397    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 23:45:37.622752 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617400    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 23:45:37.622752 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617402    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 23:45:37.622752 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617405    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 23:45:37.623519 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617407    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 23:45:37.623519 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617410    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 23:45:37.623519 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617412    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 23:45:37.623519 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617415    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 23:45:37.623519 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617418    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 23:45:37.623519 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617420    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 23:45:37.623519 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617423    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 23:45:37.623519 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617426    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 23:45:37.623519 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617429    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 23:45:37.623519 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617432    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 23:45:37.623519 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617434    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 23:45:37.623519 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617437    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 23:45:37.623519 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617439    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 23:45:37.623519 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617442    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 23:45:37.623519 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617444    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 23:45:37.623519 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617447    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 23:45:37.623519 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617449    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 23:45:37.623519 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617452    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 23:45:37.623519 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617454    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 23:45:37.623519 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617457    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 23:45:37.624239 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617459    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 23:45:37.624239 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617462    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 23:45:37.624239 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617464    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 23:45:37.624239 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617466    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 23:45:37.624239 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617469    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 23:45:37.624239 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617471    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 23:45:37.624239 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617474    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 23:45:37.624239 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617476    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 23:45:37.624239 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617479    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 23:45:37.624239 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617482    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 23:45:37.624239 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617484    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 23:45:37.624239 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617488    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 23:45:37.624239 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617491    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 23:45:37.624239 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617494    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 23:45:37.624239 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617497    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 23:45:37.624239 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617500    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 23:45:37.624239 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617503    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 23:45:37.624239 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617506    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 23:45:37.624239 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617508    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 23:45:37.624707 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617511    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 23:45:37.624707 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617513    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 23:45:37.624707 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617516    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 23:45:37.624707 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617519    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 23:45:37.624707 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617521    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 23:45:37.624707 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617524    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 23:45:37.624707 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:37.617527    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 23:45:37.624707 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.617531    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 31 23:45:37.624707 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.617641    2559 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 31 23:45:37.624707 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.619977    2559 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 31 23:45:37.624707 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.621480    2559 server.go:1019] "Starting client certificate rotation"
Jul 31 23:45:37.624707 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.621591    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 31 23:45:37.624707 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.621633    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 31 23:45:37.624707 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.624343    2559 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 31 23:45:37.625902 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.625881    2559 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 31 23:45:37.635860 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.635845    2559 log.go:25] "Validated CRI v1 runtime API"
Jul 31 23:45:37.638915 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.638899    2559 log.go:25] "Validated CRI v1 image API"
Jul 31 23:45:37.640541 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.640526    2559 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 31 23:45:37.643556 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.643494    2559 fs.go:135] Filesystem UUIDs: map[5373f0f4-6f70-49bb-87de-f7b73e85de09:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 aec3fcb1-ccaf-48e5-82e0-e5f2b800dfab:/dev/nvme0n1p4]
Jul 31 23:45:37.643644 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.643547    2559 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 31 23:45:37.649538 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.649432    2559 manager.go:216] Machine: {Timestamp:2026-07-31 23:45:37.648967238 +0000 UTC m=+0.368767075 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3121127 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20910bbe787abbe310bd5aba058ff9 SystemUUID:ec20910b-be78-7abb-e310-bd5aba058ff9 BootID:c7658513-66bb-4fa8-adbe-0bc1c51bb96f Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:52:54:25:a9:5f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:52:54:25:a9:5f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7e:0b:e3:d6:3d:6f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 31 23:45:37.649538 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.649529    2559 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 31 23:45:37.649669 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.649612    2559 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 31 23:45:37.652245 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.652226    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 31 23:45:37.653093 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.653067    2559 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 31 23:45:37.653230 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.653097    2559 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-148.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 31 23:45:37.653967 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.653957    2559 topology_manager.go:138] "Creating topology manager with none policy"
Jul 31 23:45:37.654001 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.653969    2559 container_manager_linux.go:309] "Creating device plugin manager"
Jul 31 23:45:37.654001 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.653977    2559 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 31 23:45:37.654001 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.653991    2559 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 31 23:45:37.655354 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.655344    2559 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 31 23:45:37.660166 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.660155    2559 state_mem.go:36] "Initialized new in-memory state store"
Jul 31 23:45:37.660281 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.660272    2559 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 31 23:45:37.660312 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.660302    2559 kubelet.go:486] "Attempting to sync node with API server"
Jul 31 23:45:37.660312 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.660311    2559 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 31 23:45:37.660362 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.660327    2559 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 31 23:45:37.660362 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.660336    2559 kubelet.go:398] "Adding apiserver pod source"
Jul 31 23:45:37.660362 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.660347    2559 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 31 23:45:37.660700 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.660689    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 31 23:45:37.660765 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.660707    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 31 23:45:37.663172 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.663155    2559 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 31 23:45:37.664998 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.664984    2559 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 31 23:45:37.665058 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.665014    2559 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 31 23:45:37.666506 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.666494    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 31 23:45:37.666544 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.666511    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 31 23:45:37.666544 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.666517    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 31 23:45:37.666544 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.666523    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 31 23:45:37.666544 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.666532    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 31 23:45:37.666544 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.666537    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 31 23:45:37.666544 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.666546    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 31 23:45:37.666697 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.666559    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 31 23:45:37.666697 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.666566    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 31 23:45:37.666697 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.666571    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 31 23:45:37.666697 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.666584    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 31 23:45:37.666697 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.666590    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 31 23:45:37.667545 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.667535    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 31 23:45:37.667598 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.667545    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 31 23:45:37.670806 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.670786    2559 server.go:1268] "Started kubelet"
Jul 31 23:45:37.670884 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.670867    2559 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 31 23:45:37.671602 ip-10-0-130-148 systemd[1]: Started Kubernetes Kubelet.
Jul 31 23:45:37.672704 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.671678    2559 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 31 23:45:37.672704 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.671778    2559 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 31 23:45:37.672841 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.672805    2559 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-148.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 31 23:45:37.672987 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:37.672964    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-148.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 31 23:45:37.674163 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.673543    2559 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 31 23:45:37.674310 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:37.674265    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 31 23:45:37.674849 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.674836    2559 server.go:310] "Adding debug handlers to kubelet server"
Jul 31 23:45:37.676149 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:37.676117    2559 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 31 23:45:37.677610 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:37.676403    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-148.ec2.internal.18c784917c91faf8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-148.ec2.internal,UID:ip-10-0-130-148.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-148.ec2.internal,},FirstTimestamp:2026-07-31 23:45:37.670757112 +0000 UTC m=+0.390556952,LastTimestamp:2026-07-31 23:45:37.670757112 +0000 UTC m=+0.390556952,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-148.ec2.internal,}"
Jul 31 23:45:37.681082 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.681064    2559 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 31 23:45:37.681082 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.681077    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 31 23:45:37.681206 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:37.681142    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-148.ec2.internal\" not found"
Jul 31 23:45:37.682243 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.682112    2559 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 31 23:45:37.682243 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.682109    2559 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 31 23:45:37.682243 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.682247    2559 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 31 23:45:37.682580 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.682562    2559 reconstruct.go:97] "Volume reconstruction finished"
Jul 31 23:45:37.682580 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.682580    2559 reconciler.go:29] "Reconciler: start to sync state"
Jul 31 23:45:37.683962 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.683939    2559 factory.go:55] Registering systemd factory
Jul 31 23:45:37.684039 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.683967    2559 factory.go:223] Registration of the systemd container factory successfully
Jul 31 23:45:37.684272 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.684256    2559 factory.go:153] Registering CRI-O factory
Jul 31 23:45:37.684272 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.684274    2559 factory.go:223] Registration of the crio container factory successfully
Jul 31 23:45:37.684403 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.684323    2559 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 31 23:45:37.684403 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.684347    2559 factory.go:103] Registering Raw factory
Jul 31 23:45:37.684403 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.684374    2559 manager.go:1196] Started watching for new ooms in manager
Jul 31 23:45:37.685090 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.685072    2559 manager.go:349] Starting recovery of all containers
Jul 31 23:45:37.685909 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:37.685883    2559 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-148.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 31 23:45:37.685993 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:37.685969    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 31 23:45:37.695942 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.695925    2559 manager.go:354] Recovery completed
Jul 31 23:45:37.697400 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.697384    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pdx8t"
Jul 31 23:45:37.702143 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.702049    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 23:45:37.703549 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.703534    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-148.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 23:45:37.703636 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.703566    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-148.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 23:45:37.703636 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.703580    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-148.ec2.internal" event="NodeHasSufficientPID"
Jul 31 23:45:37.704130 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.704115    2559 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 31 23:45:37.704130 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.704129    2559 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 31 23:45:37.704264 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.704147    2559 state_mem.go:36] "Initialized new in-memory state store"
Jul 31 23:45:37.704264 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.704120    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 23:45:37.704582 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.704566    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pdx8t"
Jul 31 23:45:37.704870 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.704856    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-148.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 23:45:37.704955 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.704884    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-148.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 23:45:37.704955 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.704910    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-148.ec2.internal" event="NodeHasSufficientPID"
Jul 31 23:45:37.705607 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:37.705536    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-148.ec2.internal.18c784917e864fa2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-148.ec2.internal,UID:ip-10-0-130-148.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-148.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-148.ec2.internal,},FirstTimestamp:2026-07-31 23:45:37.703546786 +0000 UTC m=+0.423346629,LastTimestamp:2026-07-31 23:45:37.703546786 +0000 UTC m=+0.423346629,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-148.ec2.internal,}"
Jul 31 23:45:37.705867 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.705850    2559 policy_none.go:49] "None policy: Start"
Jul 31 23:45:37.705947 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.705874    2559 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 31 23:45:37.705947 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.705890    2559 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 31 23:45:37.707054 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.707042    2559 policy_none.go:47] "Start"
Jul 31 23:45:37.746587 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.745315    2559 manager.go:337] "Starting Device Plugin manager"
Jul 31 23:45:37.746587 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:37.745339    2559 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 31 23:45:37.746587 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.745349    2559 server.go:87] "Starting device plugin registration server"
Jul 31 23:45:37.746587 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.745576    2559 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 31 23:45:37.746587 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.745587    2559 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 31 23:45:37.746587 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.745689    2559 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 31 23:45:37.746587 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.745850    2559 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 31 23:45:37.746587 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.745860    2559 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 31 23:45:37.746587 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:37.746286    2559 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 31 23:45:37.746587 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:37.746333    2559 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-148.ec2.internal\" not found"
Jul 31 23:45:37.778753 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.778717    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 31 23:45:37.779826 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.779807    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 31 23:45:37.779916 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.779842    2559 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 31 23:45:37.779916 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.779862    2559 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 31 23:45:37.779916 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:37.779891    2559 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 31 23:45:37.781844 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.781825    2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 23:45:37.845817 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.845789    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 23:45:37.846577 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.846557    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-148.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 23:45:37.846651 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.846587    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-148.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 23:45:37.846651 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.846598    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-148.ec2.internal" event="NodeHasSufficientPID"
Jul 31 23:45:37.846651 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.846618    2559 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-148.ec2.internal"
Jul 31 23:45:37.855457 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.855418    2559 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-148.ec2.internal"
Jul 31 23:45:37.855457 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:37.855439    2559 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-148.ec2.internal\": node \"ip-10-0-130-148.ec2.internal\" not found"
Jul 31 23:45:37.879674 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:37.879653    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-148.ec2.internal\" not found"
Jul 31 23:45:37.880666 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.880653    2559 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-130-148.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-148.ec2.internal"]
Jul 31 23:45:37.880715 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.880700    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 23:45:37.882053 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.882039    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-148.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 23:45:37.882146 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.882068    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-148.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 23:45:37.882146 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.882083    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-148.ec2.internal" event="NodeHasSufficientPID"
Jul 31 23:45:37.883169 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.883152    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4016d03779e0d094ad64607388fac106-config\") pod \"kube-apiserver-proxy-ip-10-0-130-148.ec2.internal\" (UID: \"4016d03779e0d094ad64607388fac106\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-148.ec2.internal"
Jul 31 23:45:37.883254 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.883240    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 23:45:37.883364 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.883348    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-148.ec2.internal"
Jul 31 23:45:37.883409 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.883378    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 23:45:37.883964 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.883949    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-148.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 23:45:37.884025 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.883978    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-148.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 23:45:37.884025 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.883989    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-148.ec2.internal" event="NodeHasSufficientPID"
Jul 31 23:45:37.885033 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.885021    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-148.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 23:45:37.885092 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.885043    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-148.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 23:45:37.885092 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.885056    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-148.ec2.internal" event="NodeHasSufficientPID"
Jul 31 23:45:37.886387 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.886374    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-148.ec2.internal"
Jul 31 23:45:37.886429 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.886400    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 23:45:37.887021 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.887005    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-148.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 23:45:37.887105 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.887028    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-148.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 23:45:37.887105 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.887037    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-148.ec2.internal" event="NodeHasSufficientPID"
Jul 31 23:45:37.912354 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:37.912336    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-148.ec2.internal\" not found" node="ip-10-0-130-148.ec2.internal"
Jul 31 23:45:37.916842 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:37.916824    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-148.ec2.internal\" not found" node="ip-10-0-130-148.ec2.internal"
Jul 31 23:45:37.980718 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:37.980696    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-148.ec2.internal\" not found"
Jul 31 23:45:37.983971 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.983956    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4016d03779e0d094ad64607388fac106-config\") pod \"kube-apiserver-proxy-ip-10-0-130-148.ec2.internal\" (UID: \"4016d03779e0d094ad64607388fac106\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-148.ec2.internal"
Jul 31 23:45:37.984043 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.983988    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/484eb4178e0de6efea5b73965227443b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-148.ec2.internal\" (UID: \"484eb4178e0de6efea5b73965227443b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-148.ec2.internal"
Jul 31 23:45:37.984043 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.984015    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/484eb4178e0de6efea5b73965227443b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-148.ec2.internal\" (UID: \"484eb4178e0de6efea5b73965227443b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-148.ec2.internal"
Jul 31 23:45:37.984119 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:37.984061    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4016d03779e0d094ad64607388fac106-config\") pod \"kube-apiserver-proxy-ip-10-0-130-148.ec2.internal\" (UID: \"4016d03779e0d094ad64607388fac106\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-148.ec2.internal"
Jul 31 23:45:38.081154 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:38.081121    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-148.ec2.internal\" not found"
Jul 31 23:45:38.084392 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.084374    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/484eb4178e0de6efea5b73965227443b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-148.ec2.internal\" (UID: \"484eb4178e0de6efea5b73965227443b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-148.ec2.internal"
Jul 31 23:45:38.084440 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.084403    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/484eb4178e0de6efea5b73965227443b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-148.ec2.internal\" (UID: \"484eb4178e0de6efea5b73965227443b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-148.ec2.internal"
Jul 31 23:45:38.084514 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.084442    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/484eb4178e0de6efea5b73965227443b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-148.ec2.internal\" (UID: \"484eb4178e0de6efea5b73965227443b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-148.ec2.internal"
Jul 31 23:45:38.084514 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.084469    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/484eb4178e0de6efea5b73965227443b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-148.ec2.internal\" (UID: \"484eb4178e0de6efea5b73965227443b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-148.ec2.internal"
Jul 31 23:45:38.181824 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:38.181764    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-148.ec2.internal\" not found"
Jul 31 23:45:38.215170 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.215151    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-148.ec2.internal"
Jul 31 23:45:38.219695 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.219406    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-148.ec2.internal"
Jul 31 23:45:38.282558 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:38.282531    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-148.ec2.internal\" not found"
Jul 31 23:45:38.383051 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:38.383016    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-148.ec2.internal\" not found"
Jul 31 23:45:38.483440 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:38.483413    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-148.ec2.internal\" not found"
Jul 31 23:45:38.572513 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.572491    2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 23:45:38.581747 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.581707    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-148.ec2.internal"
Jul 31 23:45:38.589897 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.589871    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 31 23:45:38.591921 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.591905    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-148.ec2.internal"
Jul 31 23:45:38.603066 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.603045    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 31 23:45:38.621572 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.621552    2559 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 31 23:45:38.621674 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.621652    2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 31 23:45:38.621674 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.621660    2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 31 23:45:38.662957 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.662932    2559 apiserver.go:52] "Watching apiserver"
Jul 31 23:45:38.681212 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.681185    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 31 23:45:38.684493 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.684473    2559 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 31 23:45:38.684868 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.684847    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-jqggp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw4n9","openshift-image-registry/node-ca-7mgjz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-148.ec2.internal","openshift-multus/multus-c8d8d","openshift-multus/network-metrics-daemon-bpv44","openshift-network-diagnostics/network-check-target-s9j95","kube-system/konnectivity-agent-92v2f","kube-system/kube-apiserver-proxy-ip-10-0-130-148.ec2.internal","openshift-cluster-node-tuning-operator/tuned-wggwt","openshift-dns/node-resolver-ld78r","openshift-multus/multus-additional-cni-plugins-r9jcj","openshift-network-operator/iptables-alerter-ck2fx","openshift-ovn-kubernetes/ovnkube-node-nmsz6"]
Jul 31 23:45:38.686823 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.686787    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ld78r"
Jul 31 23:45:38.688209 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.688190    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.689604 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.689561    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 31 23:45:38.689720 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.689672    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 31 23:45:38.689720 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.689679    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-sktv6\""
Jul 31 23:45:38.690517 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.690492    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7mgjz"
Jul 31 23:45:38.690723 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.690703    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2pkbp\""
Jul 31 23:45:38.691126 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.691104    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 31 23:45:38.692285 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.692094    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 31 23:45:38.692285 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.692102    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 31 23:45:38.692285 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.692111    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 31 23:45:38.692285 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.692191    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 31 23:45:38.692483 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.692425    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ck2fx"
Jul 31 23:45:38.692483 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.692448    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 31 23:45:38.693432 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.693418    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 31 23:45:38.693512 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.693487    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-84w28\""
Jul 31 23:45:38.693641 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.693622    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 31 23:45:38.693705 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.693691    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 31 23:45:38.694622 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.694604    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 31 23:45:38.694712 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.694694    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-c8d8d"
Jul 31 23:45:38.694920 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.694876    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 31 23:45:38.695071 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.695056    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hwbp8\""
Jul 31 23:45:38.695071 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.695063    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 31 23:45:38.695187 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.695071    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 31 23:45:38.696151 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.696135    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpv44"
Jul 31 23:45:38.696233 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:38.696195    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpv44" podUID="450206d3-1b10-45ae-b6e9-6575c0d0be94"
Jul 31 23:45:38.697090 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.697074    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 31 23:45:38.697173 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.697074    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-9q92m\""
Jul 31 23:45:38.697233 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.697081    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 31 23:45:38.697233 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.697189    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 31 23:45:38.697499 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.697476    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 31 23:45:38.697596 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.697567    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s9j95"
Jul 31 23:45:38.697656 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:38.697632    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s9j95" podUID="64e3d3a1-0422-4faa-a938-a523a0a7b827"
Jul 31 23:45:38.699139 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.699118    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-r9jcj"
Jul 31 23:45:38.700794 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.700650    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wggwt"
Jul 31 23:45:38.701470 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.701451    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vfmzb\""
Jul 31 23:45:38.701589 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.701496    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 31 23:45:38.701589 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.701529    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 31 23:45:38.702276 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.702262    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-92v2f"
Jul 31 23:45:38.703305 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.703291    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 31 23:45:38.703489 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.703478    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jqggp"
Jul 31 23:45:38.703525 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.703493    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5cdj6\""
Jul 31 23:45:38.703555 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:38.703524    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jqggp" podUID="e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06"
Jul 31 23:45:38.704157 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.704144    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 31 23:45:38.704701 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.704685    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 31 23:45:38.704761 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.704749    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 31 23:45:38.704895 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.704880    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-x97fg\""
Jul 31 23:45:38.705270 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.705247    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw4n9"
Jul 31 23:45:38.707516 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.707491    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 31 23:45:38.707614 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.707591    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 31 23:45:38.707823 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.707806    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-299z5\""
Jul 31 23:45:38.707901 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.707807    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 31 23:45:38.709501 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.709444    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-30 23:40:38 +0000 UTC" deadline="2028-04-05 06:22:37.993159126 +0000 UTC"
Jul 31 23:45:38.709501 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.709494    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14718h36m59.28366893s"
Jul 31 23:45:38.768501 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.768474    2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 23:45:38.783832 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.783793    2559 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 31 23:45:38.788484 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.788464    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpxtk\" (UniqueName: \"kubernetes.io/projected/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-kube-api-access-mpxtk\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.788602 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.788491    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-host-kubelet\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.788602 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.788509    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-etc-openvswitch\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.788602 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.788540    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/60b82286-9dac-458a-8b62-6bb90df47e04-etc-modprobe-d\") pod \"tuned-wggwt\" (UID: \"60b82286-9dac-458a-8b62-6bb90df47e04\") " pod="openshift-cluster-node-tuning-operator/tuned-wggwt"
Jul 31 23:45:38.788602 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.788567    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b490be26-0249-4036-8841-93b022ae3b1f-host-run-netns\") pod \"multus-c8d8d\" (UID: \"b490be26-0249-4036-8841-93b022ae3b1f\") " pod="openshift-multus/multus-c8d8d"
Jul 31 23:45:38.788602 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.788591    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06-kubelet-config\") pod \"global-pull-secret-syncer-jqggp\" (UID: \"e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06\") " pod="kube-system/global-pull-secret-syncer-jqggp"
Jul 31 23:45:38.788792 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.788613    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e5d655df-5dc6-4796-b7e7-b7e6596261bf-device-dir\") pod \"aws-ebs-csi-driver-node-dw4n9\" (UID: \"e5d655df-5dc6-4796-b7e7-b7e6596261bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw4n9"
Jul 31 23:45:38.788792 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.788669    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee8bfdcd-3661-46f6-93a0-93bb3facacd4-cnibin\") pod \"multus-additional-cni-plugins-r9jcj\" (UID: \"ee8bfdcd-3661-46f6-93a0-93bb3facacd4\") " pod="openshift-multus/multus-additional-cni-plugins-r9jcj"
Jul 31 23:45:38.788792 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.788704    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r987\" (UniqueName: \"kubernetes.io/projected/482f5dcb-a14d-426f-93e6-b1ce4c9b89e6-kube-api-access-8r987\") pod \"node-ca-7mgjz\" (UID: \"482f5dcb-a14d-426f-93e6-b1ce4c9b89e6\") " pod="openshift-image-registry/node-ca-7mgjz"
Jul 31 23:45:38.788792 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.788727    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b490be26-0249-4036-8841-93b022ae3b1f-cnibin\") pod \"multus-c8d8d\" (UID: \"b490be26-0249-4036-8841-93b022ae3b1f\") " pod="openshift-multus/multus-c8d8d"
Jul 31 23:45:38.788792 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.788762    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-ovnkube-script-lib\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.788792 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.788780    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/482f5dcb-a14d-426f-93e6-b1ce4c9b89e6-serviceca\") pod \"node-ca-7mgjz\" (UID: \"482f5dcb-a14d-426f-93e6-b1ce4c9b89e6\") " pod="openshift-image-registry/node-ca-7mgjz"
Jul 31 23:45:38.789075 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.788795    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b490be26-0249-4036-8841-93b022ae3b1f-system-cni-dir\") pod \"multus-c8d8d\" (UID: \"b490be26-0249-4036-8841-93b022ae3b1f\") " pod="openshift-multus/multus-c8d8d"
Jul 31 23:45:38.789075 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.788810    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/539b6c48-1833-4461-8ab1-580974198fb3-konnectivity-ca\") pod \"konnectivity-agent-92v2f\" (UID: \"539b6c48-1833-4461-8ab1-580974198fb3\") " pod="kube-system/konnectivity-agent-92v2f"
Jul 31 23:45:38.789075 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.788827    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96d86114-06a1-4c32-9d63-c7f4262fa871-tmp-dir\") pod \"node-resolver-ld78r\" (UID: \"96d86114-06a1-4c32-9d63-c7f4262fa871\") " pod="openshift-dns/node-resolver-ld78r"
Jul 31 23:45:38.789075 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.788842    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b490be26-0249-4036-8841-93b022ae3b1f-multus-cni-dir\") pod \"multus-c8d8d\" (UID: \"b490be26-0249-4036-8841-93b022ae3b1f\") " pod="openshift-multus/multus-c8d8d"
Jul 31 23:45:38.789075 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.788891    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e5d655df-5dc6-4796-b7e7-b7e6596261bf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dw4n9\" (UID: \"e5d655df-5dc6-4796-b7e7-b7e6596261bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw4n9"
Jul 31 23:45:38.789075 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.788927    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-log-socket\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.789075 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.788958    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee8bfdcd-3661-46f6-93a0-93bb3facacd4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r9jcj\" (UID: \"ee8bfdcd-3661-46f6-93a0-93bb3facacd4\") " pod="openshift-multus/multus-additional-cni-plugins-r9jcj"
Jul 31 23:45:38.789075 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.788987    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60b82286-9dac-458a-8b62-6bb90df47e04-etc-kubernetes\") pod \"tuned-wggwt\" (UID: \"60b82286-9dac-458a-8b62-6bb90df47e04\") " pod="openshift-cluster-node-tuning-operator/tuned-wggwt"
Jul 31 23:45:38.789075 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.789015    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0046f547-2701-496c-b596-578ce8949694-iptables-alerter-script\") pod \"iptables-alerter-ck2fx\" (UID: \"0046f547-2701-496c-b596-578ce8949694\") " pod="openshift-network-operator/iptables-alerter-ck2fx"
Jul 31 23:45:38.789075 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.789040    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s9tv\" (UniqueName: \"kubernetes.io/projected/b490be26-0249-4036-8841-93b022ae3b1f-kube-api-access-9s9tv\") pod \"multus-c8d8d\" (UID: \"b490be26-0249-4036-8841-93b022ae3b1f\") " pod="openshift-multus/multus-c8d8d"
Jul 31 23:45:38.789075 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.789070    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e5d655df-5dc6-4796-b7e7-b7e6596261bf-etc-selinux\") pod \"aws-ebs-csi-driver-node-dw4n9\" (UID: \"e5d655df-5dc6-4796-b7e7-b7e6596261bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw4n9"
Jul 31 23:45:38.789613 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.789095    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-var-lib-openvswitch\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.789613 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.789119    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-ovnkube-config\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.789613 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.789142    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee8bfdcd-3661-46f6-93a0-93bb3facacd4-system-cni-dir\") pod \"multus-additional-cni-plugins-r9jcj\" (UID: \"ee8bfdcd-3661-46f6-93a0-93bb3facacd4\") " pod="openshift-multus/multus-additional-cni-plugins-r9jcj"
Jul 31 23:45:38.789613 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.789166    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/60b82286-9dac-458a-8b62-6bb90df47e04-etc-tuned\") pod \"tuned-wggwt\" (UID: \"60b82286-9dac-458a-8b62-6bb90df47e04\") " pod="openshift-cluster-node-tuning-operator/tuned-wggwt"
Jul 31 23:45:38.789613 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.789192    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06-original-pull-secret\") pod \"global-pull-secret-syncer-jqggp\" (UID: \"e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06\") " pod="kube-system/global-pull-secret-syncer-jqggp"
Jul 31 23:45:38.789613 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.789217    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e5d655df-5dc6-4796-b7e7-b7e6596261bf-sys-fs\") pod \"aws-ebs-csi-driver-node-dw4n9\" (UID: \"e5d655df-5dc6-4796-b7e7-b7e6596261bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw4n9"
Jul 31 23:45:38.789613 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.789243    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxphm\" (UniqueName: \"kubernetes.io/projected/e5d655df-5dc6-4796-b7e7-b7e6596261bf-kube-api-access-mxphm\") pod \"aws-ebs-csi-driver-node-dw4n9\" (UID: \"e5d655df-5dc6-4796-b7e7-b7e6596261bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw4n9"
Jul 31 23:45:38.789613 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.789276    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-systemd-units\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.789613 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.789326    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ee8bfdcd-3661-46f6-93a0-93bb3facacd4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r9jcj\" (UID: \"ee8bfdcd-3661-46f6-93a0-93bb3facacd4\") " pod="openshift-multus/multus-additional-cni-plugins-r9jcj"
Jul 31 23:45:38.789613 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.789353    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/60b82286-9dac-458a-8b62-6bb90df47e04-etc-systemd\") pod \"tuned-wggwt\" (UID: \"60b82286-9dac-458a-8b62-6bb90df47e04\") " pod="openshift-cluster-node-tuning-operator/tuned-wggwt"
Jul 31 23:45:38.789613 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.789379    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzdnz\" (UniqueName: \"kubernetes.io/projected/60b82286-9dac-458a-8b62-6bb90df47e04-kube-api-access-fzdnz\") pod \"tuned-wggwt\" (UID: \"60b82286-9dac-458a-8b62-6bb90df47e04\") " pod="openshift-cluster-node-tuning-operator/tuned-wggwt"
Jul 31 23:45:38.789613 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.789417    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/539b6c48-1833-4461-8ab1-580974198fb3-agent-certs\") pod \"konnectivity-agent-92v2f\" (UID: \"539b6c48-1833-4461-8ab1-580974198fb3\") " pod="kube-system/konnectivity-agent-92v2f"
Jul 31 23:45:38.789613 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.789465    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-run-ovn\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.789613 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.789491    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pgnt\" (UniqueName: \"kubernetes.io/projected/64e3d3a1-0422-4faa-a938-a523a0a7b827-kube-api-access-7pgnt\") pod \"network-check-target-s9j95\" (UID: \"64e3d3a1-0422-4faa-a938-a523a0a7b827\") " pod="openshift-network-diagnostics/network-check-target-s9j95"
Jul 31 23:45:38.789613 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.789526    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl5nb\" (UniqueName: \"kubernetes.io/projected/ee8bfdcd-3661-46f6-93a0-93bb3facacd4-kube-api-access-hl5nb\") pod \"multus-additional-cni-plugins-r9jcj\" (UID: \"ee8bfdcd-3661-46f6-93a0-93bb3facacd4\") " pod="openshift-multus/multus-additional-cni-plugins-r9jcj"
Jul 31 23:45:38.789613 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.789552    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60b82286-9dac-458a-8b62-6bb90df47e04-var-lib-kubelet\") pod \"tuned-wggwt\" (UID: \"60b82286-9dac-458a-8b62-6bb90df47e04\") " pod="openshift-cluster-node-tuning-operator/tuned-wggwt"
Jul 31 23:45:38.790172 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.789570    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b490be26-0249-4036-8841-93b022ae3b1f-os-release\") pod \"multus-c8d8d\" (UID: \"b490be26-0249-4036-8841-93b022ae3b1f\") " pod="openshift-multus/multus-c8d8d"
Jul 31 23:45:38.790172 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.789585    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e5d655df-5dc6-4796-b7e7-b7e6596261bf-socket-dir\") pod \"aws-ebs-csi-driver-node-dw4n9\" (UID: \"e5d655df-5dc6-4796-b7e7-b7e6596261bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw4n9"
Jul 31 23:45:38.790172 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.789613    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-node-log\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.790172 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.789663    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-host-cni-bin\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.790172 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.789682    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/60b82286-9dac-458a-8b62-6bb90df47e04-run\") pod \"tuned-wggwt\" (UID: \"60b82286-9dac-458a-8b62-6bb90df47e04\") " pod="openshift-cluster-node-tuning-operator/tuned-wggwt"
Jul 31 23:45:38.790172 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.789699    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b490be26-0249-4036-8841-93b022ae3b1f-multus-socket-dir-parent\") pod \"multus-c8d8d\" (UID: \"b490be26-0249-4036-8841-93b022ae3b1f\") " pod="openshift-multus/multus-c8d8d"
Jul 31 23:45:38.790172 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.789715    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06-dbus\") pod \"global-pull-secret-syncer-jqggp\" (UID: \"e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06\") " pod="kube-system/global-pull-secret-syncer-jqggp"
Jul 31 23:45:38.790172 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.789731    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-host-run-netns\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.790172 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.789752    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-run-systemd\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.790172 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.789783    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-host-cni-netd\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.790172 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.789797    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-env-overrides\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.790172 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.789811    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60b82286-9dac-458a-8b62-6bb90df47e04-tmp\") pod \"tuned-wggwt\" (UID: \"60b82286-9dac-458a-8b62-6bb90df47e04\") " pod="openshift-cluster-node-tuning-operator/tuned-wggwt"
Jul 31 23:45:38.790172 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.789844    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b490be26-0249-4036-8841-93b022ae3b1f-cni-binary-copy\") pod \"multus-c8d8d\" (UID: \"b490be26-0249-4036-8841-93b022ae3b1f\") " pod="openshift-multus/multus-c8d8d"
Jul 31 23:45:38.790172 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.789868    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-host-run-ovn-kubernetes\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.790172 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.789892    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/60b82286-9dac-458a-8b62-6bb90df47e04-etc-sysctl-d\") pod \"tuned-wggwt\" (UID: \"60b82286-9dac-458a-8b62-6bb90df47e04\") " pod="openshift-cluster-node-tuning-operator/tuned-wggwt"
Jul 31 23:45:38.790172 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.789912    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/60b82286-9dac-458a-8b62-6bb90df47e04-etc-sysctl-conf\") pod \"tuned-wggwt\" (UID: \"60b82286-9dac-458a-8b62-6bb90df47e04\") " pod="openshift-cluster-node-tuning-operator/tuned-wggwt"
Jul 31 23:45:38.790172 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.789950    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/60b82286-9dac-458a-8b62-6bb90df47e04-lib-modules\") pod \"tuned-wggwt\" (UID: \"60b82286-9dac-458a-8b62-6bb90df47e04\") " pod="openshift-cluster-node-tuning-operator/tuned-wggwt"
Jul 31 23:45:38.790701 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.789972    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/60b82286-9dac-458a-8b62-6bb90df47e04-host\") pod \"tuned-wggwt\" (UID: \"60b82286-9dac-458a-8b62-6bb90df47e04\") " pod="openshift-cluster-node-tuning-operator/tuned-wggwt"
Jul 31 23:45:38.790701 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.789991    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b490be26-0249-4036-8841-93b022ae3b1f-host-run-k8s-cni-cncf-io\") pod \"multus-c8d8d\" (UID: \"b490be26-0249-4036-8841-93b022ae3b1f\") " pod="openshift-multus/multus-c8d8d"
Jul 31 23:45:38.790701 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.790015    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/450206d3-1b10-45ae-b6e9-6575c0d0be94-metrics-certs\") pod \"network-metrics-daemon-bpv44\" (UID: \"450206d3-1b10-45ae-b6e9-6575c0d0be94\") " pod="openshift-multus/network-metrics-daemon-bpv44"
Jul 31 23:45:38.790701 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.790035    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s94lb\" (UniqueName: \"kubernetes.io/projected/450206d3-1b10-45ae-b6e9-6575c0d0be94-kube-api-access-s94lb\") pod \"network-metrics-daemon-bpv44\" (UID: \"450206d3-1b10-45ae-b6e9-6575c0d0be94\") " pod="openshift-multus/network-metrics-daemon-bpv44"
Jul 31 23:45:38.790701 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.790051    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b490be26-0249-4036-8841-93b022ae3b1f-host-var-lib-cni-bin\") pod \"multus-c8d8d\" (UID: \"b490be26-0249-4036-8841-93b022ae3b1f\") " pod="openshift-multus/multus-c8d8d"
Jul 31 23:45:38.790701 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.790067    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b490be26-0249-4036-8841-93b022ae3b1f-hostroot\") pod \"multus-c8d8d\" (UID: \"b490be26-0249-4036-8841-93b022ae3b1f\") " pod="openshift-multus/multus-c8d8d"
Jul 31 23:45:38.790701 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.790087    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/96d86114-06a1-4c32-9d63-c7f4262fa871-hosts-file\") pod \"node-resolver-ld78r\" (UID: \"96d86114-06a1-4c32-9d63-c7f4262fa871\") " pod="openshift-dns/node-resolver-ld78r"
Jul 31 23:45:38.790701 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.790118    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-host-slash\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.790701 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.790140    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-run-openvswitch\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.790701 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.790156    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee8bfdcd-3661-46f6-93a0-93bb3facacd4-cni-binary-copy\") pod \"multus-additional-cni-plugins-r9jcj\" (UID: \"ee8bfdcd-3661-46f6-93a0-93bb3facacd4\") " pod="openshift-multus/multus-additional-cni-plugins-r9jcj"
Jul 31 23:45:38.790701 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.790191    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/482f5dcb-a14d-426f-93e6-b1ce4c9b89e6-host\") pod \"node-ca-7mgjz\" (UID: \"482f5dcb-a14d-426f-93e6-b1ce4c9b89e6\") " pod="openshift-image-registry/node-ca-7mgjz"
Jul 31 23:45:38.790701 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.790210    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0046f547-2701-496c-b596-578ce8949694-host-slash\") pod \"iptables-alerter-ck2fx\" (UID: \"0046f547-2701-496c-b596-578ce8949694\") " pod="openshift-network-operator/iptables-alerter-ck2fx"
Jul 31 23:45:38.790701 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.790225    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b490be26-0249-4036-8841-93b022ae3b1f-host-var-lib-kubelet\") pod \"multus-c8d8d\" (UID: \"b490be26-0249-4036-8841-93b022ae3b1f\") " pod="openshift-multus/multus-c8d8d"
Jul 31 23:45:38.790701 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.790240    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b490be26-0249-4036-8841-93b022ae3b1f-multus-daemon-config\") pod \"multus-c8d8d\" (UID: \"b490be26-0249-4036-8841-93b022ae3b1f\") " pod="openshift-multus/multus-c8d8d"
Jul 31 23:45:38.790701 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.790266    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-ovn-node-metrics-cert\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.790701 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.790295    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.791382 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.790323    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ee8bfdcd-3661-46f6-93a0-93bb3facacd4-os-release\") pod \"multus-additional-cni-plugins-r9jcj\" (UID: \"ee8bfdcd-3661-46f6-93a0-93bb3facacd4\") " pod="openshift-multus/multus-additional-cni-plugins-r9jcj"
Jul 31 23:45:38.791382 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.790365    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ee8bfdcd-3661-46f6-93a0-93bb3facacd4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r9jcj\" (UID: \"ee8bfdcd-3661-46f6-93a0-93bb3facacd4\") " pod="openshift-multus/multus-additional-cni-plugins-r9jcj"
Jul 31 23:45:38.791382 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.790390    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/60b82286-9dac-458a-8b62-6bb90df47e04-sys\") pod \"tuned-wggwt\" (UID: \"60b82286-9dac-458a-8b62-6bb90df47e04\") " pod="openshift-cluster-node-tuning-operator/tuned-wggwt"
Jul 31 23:45:38.791382 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.790404    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b490be26-0249-4036-8841-93b022ae3b1f-host-var-lib-cni-multus\") pod \"multus-c8d8d\" (UID: \"b490be26-0249-4036-8841-93b022ae3b1f\") " pod="openshift-multus/multus-c8d8d"
Jul 31 23:45:38.791382 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.790423    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b490be26-0249-4036-8841-93b022ae3b1f-etc-kubernetes\") pod \"multus-c8d8d\" (UID: \"b490be26-0249-4036-8841-93b022ae3b1f\") " pod="openshift-multus/multus-c8d8d"
Jul 31 23:45:38.791382 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.790447    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/60b82286-9dac-458a-8b62-6bb90df47e04-etc-sysconfig\") pod \"tuned-wggwt\" (UID: \"60b82286-9dac-458a-8b62-6bb90df47e04\") " pod="openshift-cluster-node-tuning-operator/tuned-wggwt"
Jul 31 23:45:38.791382 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.790462    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9vqm\" (UniqueName: \"kubernetes.io/projected/0046f547-2701-496c-b596-578ce8949694-kube-api-access-q9vqm\") pod \"iptables-alerter-ck2fx\" (UID: \"0046f547-2701-496c-b596-578ce8949694\") " pod="openshift-network-operator/iptables-alerter-ck2fx"
Jul 31 23:45:38.791382 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.790509    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b490be26-0249-4036-8841-93b022ae3b1f-host-run-multus-certs\") pod \"multus-c8d8d\" (UID: \"b490be26-0249-4036-8841-93b022ae3b1f\") " pod="openshift-multus/multus-c8d8d"
Jul 31 23:45:38.791382 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.790532    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w48d\" (UniqueName: \"kubernetes.io/projected/96d86114-06a1-4c32-9d63-c7f4262fa871-kube-api-access-7w48d\") pod \"node-resolver-ld78r\" (UID: \"96d86114-06a1-4c32-9d63-c7f4262fa871\") " pod="openshift-dns/node-resolver-ld78r"
Jul 31 23:45:38.791382 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.790554    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b490be26-0249-4036-8841-93b022ae3b1f-multus-conf-dir\") pod \"multus-c8d8d\" (UID: \"b490be26-0249-4036-8841-93b022ae3b1f\") " pod="openshift-multus/multus-c8d8d"
Jul 31 23:45:38.791382 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.790584    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e5d655df-5dc6-4796-b7e7-b7e6596261bf-registration-dir\") pod \"aws-ebs-csi-driver-node-dw4n9\" (UID: \"e5d655df-5dc6-4796-b7e7-b7e6596261bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw4n9"
Jul 31 23:45:38.828392 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.828367    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wwln4"
Jul 31 23:45:38.838049 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.838021    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wwln4"
Jul 31 23:45:38.890900 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.890870    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-log-socket\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.890900 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.890901    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee8bfdcd-3661-46f6-93a0-93bb3facacd4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r9jcj\" (UID: \"ee8bfdcd-3661-46f6-93a0-93bb3facacd4\") " pod="openshift-multus/multus-additional-cni-plugins-r9jcj"
Jul 31 23:45:38.891105 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.890919    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60b82286-9dac-458a-8b62-6bb90df47e04-etc-kubernetes\") pod \"tuned-wggwt\" (UID: \"60b82286-9dac-458a-8b62-6bb90df47e04\") " pod="openshift-cluster-node-tuning-operator/tuned-wggwt"
Jul 31 23:45:38.891105 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.890944    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0046f547-2701-496c-b596-578ce8949694-iptables-alerter-script\") pod \"iptables-alerter-ck2fx\" (UID: \"0046f547-2701-496c-b596-578ce8949694\") " pod="openshift-network-operator/iptables-alerter-ck2fx"
Jul 31 23:45:38.891105 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.890987    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-log-socket\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.891105 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.891016    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s9tv\" (UniqueName: \"kubernetes.io/projected/b490be26-0249-4036-8841-93b022ae3b1f-kube-api-access-9s9tv\") pod \"multus-c8d8d\" (UID: \"b490be26-0249-4036-8841-93b022ae3b1f\") " pod="openshift-multus/multus-c8d8d"
Jul 31 23:45:38.891105 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.891022    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60b82286-9dac-458a-8b62-6bb90df47e04-etc-kubernetes\") pod \"tuned-wggwt\" (UID: \"60b82286-9dac-458a-8b62-6bb90df47e04\") " pod="openshift-cluster-node-tuning-operator/tuned-wggwt"
Jul 31 23:45:38.891105 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.891059    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e5d655df-5dc6-4796-b7e7-b7e6596261bf-etc-selinux\") pod \"aws-ebs-csi-driver-node-dw4n9\" (UID: \"e5d655df-5dc6-4796-b7e7-b7e6596261bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw4n9"
Jul 31 23:45:38.891105 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.891092    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-var-lib-openvswitch\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.891105 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.891097    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee8bfdcd-3661-46f6-93a0-93bb3facacd4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r9jcj\" (UID: \"ee8bfdcd-3661-46f6-93a0-93bb3facacd4\") " pod="openshift-multus/multus-additional-cni-plugins-r9jcj"
Jul 31 23:45:38.891546 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.891116    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-ovnkube-config\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.891546 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.891147    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee8bfdcd-3661-46f6-93a0-93bb3facacd4-system-cni-dir\") pod \"multus-additional-cni-plugins-r9jcj\" (UID: \"ee8bfdcd-3661-46f6-93a0-93bb3facacd4\") " pod="openshift-multus/multus-additional-cni-plugins-r9jcj"
Jul 31 23:45:38.891546 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.891156    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-var-lib-openvswitch\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.891546 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.891173    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/60b82286-9dac-458a-8b62-6bb90df47e04-etc-tuned\") pod \"tuned-wggwt\" (UID: \"60b82286-9dac-458a-8b62-6bb90df47e04\") " pod="openshift-cluster-node-tuning-operator/tuned-wggwt"
Jul 31 23:45:38.891546 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.891194    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e5d655df-5dc6-4796-b7e7-b7e6596261bf-etc-selinux\") pod \"aws-ebs-csi-driver-node-dw4n9\" (UID: \"e5d655df-5dc6-4796-b7e7-b7e6596261bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw4n9"
Jul 31 23:45:38.891546 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.891201    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06-original-pull-secret\") pod \"global-pull-secret-syncer-jqggp\" (UID: \"e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06\") " pod="kube-system/global-pull-secret-syncer-jqggp"
Jul 31 23:45:38.891546 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.891226    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e5d655df-5dc6-4796-b7e7-b7e6596261bf-sys-fs\") pod \"aws-ebs-csi-driver-node-dw4n9\" (UID: \"e5d655df-5dc6-4796-b7e7-b7e6596261bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw4n9"
Jul 31 23:45:38.891546 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.891250    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxphm\" (UniqueName: \"kubernetes.io/projected/e5d655df-5dc6-4796-b7e7-b7e6596261bf-kube-api-access-mxphm\") pod \"aws-ebs-csi-driver-node-dw4n9\" (UID: \"e5d655df-5dc6-4796-b7e7-b7e6596261bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw4n9"
Jul 31 23:45:38.891546 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.891290    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-systemd-units\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.891546 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.891315    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ee8bfdcd-3661-46f6-93a0-93bb3facacd4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r9jcj\" (UID: \"ee8bfdcd-3661-46f6-93a0-93bb3facacd4\") " pod="openshift-multus/multus-additional-cni-plugins-r9jcj"
Jul 31 23:45:38.891546 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.891341    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/60b82286-9dac-458a-8b62-6bb90df47e04-etc-systemd\") pod \"tuned-wggwt\" (UID: \"60b82286-9dac-458a-8b62-6bb90df47e04\") " pod="openshift-cluster-node-tuning-operator/tuned-wggwt"
Jul 31 23:45:38.891546 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.891330    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e5d655df-5dc6-4796-b7e7-b7e6596261bf-sys-fs\") pod \"aws-ebs-csi-driver-node-dw4n9\" (UID: \"e5d655df-5dc6-4796-b7e7-b7e6596261bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw4n9"
Jul 31 23:45:38.891546 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.891394    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzdnz\" (UniqueName: \"kubernetes.io/projected/60b82286-9dac-458a-8b62-6bb90df47e04-kube-api-access-fzdnz\") pod \"tuned-wggwt\" (UID: \"60b82286-9dac-458a-8b62-6bb90df47e04\") " pod="openshift-cluster-node-tuning-operator/tuned-wggwt"
Jul 31 23:45:38.891546 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:38.891402    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 23:45:38.891546 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.891426    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/539b6c48-1833-4461-8ab1-580974198fb3-agent-certs\") pod \"konnectivity-agent-92v2f\" (UID: \"539b6c48-1833-4461-8ab1-580974198fb3\") " pod="kube-system/konnectivity-agent-92v2f"
Jul 31 23:45:38.891546 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.891455    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-run-ovn\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.891546 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.891487    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pgnt\" (UniqueName: \"kubernetes.io/projected/64e3d3a1-0422-4faa-a938-a523a0a7b827-kube-api-access-7pgnt\") pod \"network-check-target-s9j95\" (UID: \"64e3d3a1-0422-4faa-a938-a523a0a7b827\") " pod="openshift-network-diagnostics/network-check-target-s9j95"
Jul 31 23:45:38.892393 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.891557    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl5nb\" (UniqueName: \"kubernetes.io/projected/ee8bfdcd-3661-46f6-93a0-93bb3facacd4-kube-api-access-hl5nb\") pod \"multus-additional-cni-plugins-r9jcj\" (UID: \"ee8bfdcd-3661-46f6-93a0-93bb3facacd4\") " pod="openshift-multus/multus-additional-cni-plugins-r9jcj"
Jul 31 23:45:38.892393 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.891538    2559 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 31 23:45:38.892393 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.891593    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0046f547-2701-496c-b596-578ce8949694-iptables-alerter-script\") pod \"iptables-alerter-ck2fx\" (UID: \"0046f547-2701-496c-b596-578ce8949694\") " pod="openshift-network-operator/iptables-alerter-ck2fx"
Jul 31 23:45:38.892393 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.891246    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee8bfdcd-3661-46f6-93a0-93bb3facacd4-system-cni-dir\") pod \"multus-additional-cni-plugins-r9jcj\" (UID: \"ee8bfdcd-3661-46f6-93a0-93bb3facacd4\") " pod="openshift-multus/multus-additional-cni-plugins-r9jcj"
Jul 31 23:45:38.892393 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.891674    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-systemd-units\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.892393 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.891740    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/60b82286-9dac-458a-8b62-6bb90df47e04-etc-systemd\") pod \"tuned-wggwt\" (UID: \"60b82286-9dac-458a-8b62-6bb90df47e04\") " pod="openshift-cluster-node-tuning-operator/tuned-wggwt"
Jul 31 23:45:38.892393 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.891766    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-ovnkube-config\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.892393 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.891850    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-run-ovn\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.892393 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.892201    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ee8bfdcd-3661-46f6-93a0-93bb3facacd4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r9jcj\" (UID: \"ee8bfdcd-3661-46f6-93a0-93bb3facacd4\") " pod="openshift-multus/multus-additional-cni-plugins-r9jcj"
Jul 31 23:45:38.892852 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:38.892690    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06-original-pull-secret podName:e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06 nodeName:}" failed. No retries permitted until 2026-07-31 23:45:39.391579257 +0000 UTC m=+2.111379086 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06-original-pull-secret") pod "global-pull-secret-syncer-jqggp" (UID: "e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06") : object "kube-system"/"original-pull-secret" not registered
Jul 31 23:45:38.892852 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.892720    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60b82286-9dac-458a-8b62-6bb90df47e04-var-lib-kubelet\") pod \"tuned-wggwt\" (UID: \"60b82286-9dac-458a-8b62-6bb90df47e04\") " pod="openshift-cluster-node-tuning-operator/tuned-wggwt"
Jul 31 23:45:38.892852 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.892769    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b490be26-0249-4036-8841-93b022ae3b1f-os-release\") pod \"multus-c8d8d\" (UID: \"b490be26-0249-4036-8841-93b022ae3b1f\") " pod="openshift-multus/multus-c8d8d"
Jul 31 23:45:38.892852 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.892780    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60b82286-9dac-458a-8b62-6bb90df47e04-var-lib-kubelet\") pod \"tuned-wggwt\" (UID: \"60b82286-9dac-458a-8b62-6bb90df47e04\") " pod="openshift-cluster-node-tuning-operator/tuned-wggwt"
Jul 31 23:45:38.892852 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.892808    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e5d655df-5dc6-4796-b7e7-b7e6596261bf-socket-dir\") pod \"aws-ebs-csi-driver-node-dw4n9\" (UID: \"e5d655df-5dc6-4796-b7e7-b7e6596261bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw4n9"
Jul 31 23:45:38.892852 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.892835    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-node-log\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.893236 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.892856    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-host-cni-bin\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.893236 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.892877    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b490be26-0249-4036-8841-93b022ae3b1f-os-release\") pod \"multus-c8d8d\" (UID: \"b490be26-0249-4036-8841-93b022ae3b1f\") " pod="openshift-multus/multus-c8d8d"
Jul 31 23:45:38.893236 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.892881    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/60b82286-9dac-458a-8b62-6bb90df47e04-run\") pod \"tuned-wggwt\" (UID: \"60b82286-9dac-458a-8b62-6bb90df47e04\") " pod="openshift-cluster-node-tuning-operator/tuned-wggwt"
Jul 31 23:45:38.893236 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.892929    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b490be26-0249-4036-8841-93b022ae3b1f-multus-socket-dir-parent\") pod \"multus-c8d8d\" (UID: \"b490be26-0249-4036-8841-93b022ae3b1f\") " pod="openshift-multus/multus-c8d8d"
Jul 31 23:45:38.893236 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.892952    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e5d655df-5dc6-4796-b7e7-b7e6596261bf-socket-dir\") pod \"aws-ebs-csi-driver-node-dw4n9\" (UID: \"e5d655df-5dc6-4796-b7e7-b7e6596261bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw4n9"
Jul 31 23:45:38.893236 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.892973    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06-dbus\") pod \"global-pull-secret-syncer-jqggp\" (UID: \"e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06\") " pod="kube-system/global-pull-secret-syncer-jqggp"
Jul 31 23:45:38.893236 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.893002    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-host-run-netns\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.893236 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.893004    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-node-log\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.893236 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.892984    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/60b82286-9dac-458a-8b62-6bb90df47e04-run\") pod \"tuned-wggwt\" (UID: \"60b82286-9dac-458a-8b62-6bb90df47e04\") " pod="openshift-cluster-node-tuning-operator/tuned-wggwt"
Jul 31 23:45:38.893236 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.893050    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-run-systemd\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.893236 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.893082    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-host-cni-netd\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.893236 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.893106    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-env-overrides\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.893236 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.893131    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60b82286-9dac-458a-8b62-6bb90df47e04-tmp\") pod \"tuned-wggwt\" (UID: \"60b82286-9dac-458a-8b62-6bb90df47e04\") " pod="openshift-cluster-node-tuning-operator/tuned-wggwt"
Jul 31 23:45:38.893236 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.893152    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-run-systemd\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.893236 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.893152    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06-dbus\") pod \"global-pull-secret-syncer-jqggp\" (UID: \"e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06\") " pod="kube-system/global-pull-secret-syncer-jqggp"
Jul 31 23:45:38.893236 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.893212    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b490be26-0249-4036-8841-93b022ae3b1f-multus-socket-dir-parent\") pod \"multus-c8d8d\" (UID: \"b490be26-0249-4036-8841-93b022ae3b1f\") " pod="openshift-multus/multus-c8d8d"
Jul 31 23:45:38.893236 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.893224    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b490be26-0249-4036-8841-93b022ae3b1f-cni-binary-copy\") pod \"multus-c8d8d\" (UID: \"b490be26-0249-4036-8841-93b022ae3b1f\") " pod="openshift-multus/multus-c8d8d"
Jul 31 23:45:38.893236 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.893223    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-host-run-netns\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.894209 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.893107    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-host-cni-bin\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.894209 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.893258    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-host-cni-netd\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.894209 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.893267    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-host-run-ovn-kubernetes\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.894209 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.893301    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-host-run-ovn-kubernetes\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.894209 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.893314    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/60b82286-9dac-458a-8b62-6bb90df47e04-etc-sysctl-d\") pod \"tuned-wggwt\" (UID: \"60b82286-9dac-458a-8b62-6bb90df47e04\") " pod="openshift-cluster-node-tuning-operator/tuned-wggwt"
Jul 31 23:45:38.894209 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.893339    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/60b82286-9dac-458a-8b62-6bb90df47e04-etc-sysctl-conf\") pod \"tuned-wggwt\" (UID: \"60b82286-9dac-458a-8b62-6bb90df47e04\") " pod="openshift-cluster-node-tuning-operator/tuned-wggwt"
Jul 31 23:45:38.894209 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.893956    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/60b82286-9dac-458a-8b62-6bb90df47e04-lib-modules\") pod \"tuned-wggwt\" (UID: \"60b82286-9dac-458a-8b62-6bb90df47e04\") " pod="openshift-cluster-node-tuning-operator/tuned-wggwt"
Jul 31 23:45:38.894209 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.894088    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-env-overrides\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.894209 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.894127    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/60b82286-9dac-458a-8b62-6bb90df47e04-lib-modules\") pod \"tuned-wggwt\" (UID: \"60b82286-9dac-458a-8b62-6bb90df47e04\") " pod="openshift-cluster-node-tuning-operator/tuned-wggwt"
Jul 31 23:45:38.894209 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.894171    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/60b82286-9dac-458a-8b62-6bb90df47e04-host\") pod \"tuned-wggwt\" (UID: \"60b82286-9dac-458a-8b62-6bb90df47e04\") " pod="openshift-cluster-node-tuning-operator/tuned-wggwt"
Jul 31 23:45:38.894209 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.894204    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b490be26-0249-4036-8841-93b022ae3b1f-host-run-k8s-cni-cncf-io\") pod \"multus-c8d8d\" (UID: \"b490be26-0249-4036-8841-93b022ae3b1f\") " pod="openshift-multus/multus-c8d8d"
Jul 31 23:45:38.894760 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.894241    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/450206d3-1b10-45ae-b6e9-6575c0d0be94-metrics-certs\") pod \"network-metrics-daemon-bpv44\" (UID: \"450206d3-1b10-45ae-b6e9-6575c0d0be94\") " pod="openshift-multus/network-metrics-daemon-bpv44"
Jul 31 23:45:38.894760 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.894283    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s94lb\" (UniqueName: \"kubernetes.io/projected/450206d3-1b10-45ae-b6e9-6575c0d0be94-kube-api-access-s94lb\") pod \"network-metrics-daemon-bpv44\" (UID: \"450206d3-1b10-45ae-b6e9-6575c0d0be94\") " pod="openshift-multus/network-metrics-daemon-bpv44"
Jul 31 23:45:38.894760 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.894321    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b490be26-0249-4036-8841-93b022ae3b1f-host-var-lib-cni-bin\") pod \"multus-c8d8d\" (UID: \"b490be26-0249-4036-8841-93b022ae3b1f\") " pod="openshift-multus/multus-c8d8d"
Jul 31 23:45:38.894760 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.894354    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b490be26-0249-4036-8841-93b022ae3b1f-hostroot\") pod \"multus-c8d8d\" (UID: \"b490be26-0249-4036-8841-93b022ae3b1f\") " pod="openshift-multus/multus-c8d8d"
Jul 31 23:45:38.894760 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.894382    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/96d86114-06a1-4c32-9d63-c7f4262fa871-hosts-file\") pod \"node-resolver-ld78r\" (UID: \"96d86114-06a1-4c32-9d63-c7f4262fa871\") " pod="openshift-dns/node-resolver-ld78r"
Jul 31 23:45:38.894760 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.894419    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-host-slash\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.894760 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.894455    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-run-openvswitch\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.894760 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.894493    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee8bfdcd-3661-46f6-93a0-93bb3facacd4-cni-binary-copy\") pod \"multus-additional-cni-plugins-r9jcj\" (UID: \"ee8bfdcd-3661-46f6-93a0-93bb3facacd4\") " pod="openshift-multus/multus-additional-cni-plugins-r9jcj"
Jul 31 23:45:38.894760 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.894526    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/482f5dcb-a14d-426f-93e6-b1ce4c9b89e6-host\") pod \"node-ca-7mgjz\" (UID: \"482f5dcb-a14d-426f-93e6-b1ce4c9b89e6\") " pod="openshift-image-registry/node-ca-7mgjz"
Jul 31 23:45:38.894760 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.894557    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0046f547-2701-496c-b596-578ce8949694-host-slash\") pod \"iptables-alerter-ck2fx\" (UID: \"0046f547-2701-496c-b596-578ce8949694\") " pod="openshift-network-operator/iptables-alerter-ck2fx"
Jul 31 23:45:38.894760 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.894592    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b490be26-0249-4036-8841-93b022ae3b1f-host-var-lib-kubelet\") pod \"multus-c8d8d\" (UID: \"b490be26-0249-4036-8841-93b022ae3b1f\") " pod="openshift-multus/multus-c8d8d"
Jul 31 23:45:38.894760 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.894626    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b490be26-0249-4036-8841-93b022ae3b1f-multus-daemon-config\") pod \"multus-c8d8d\" (UID: \"b490be26-0249-4036-8841-93b022ae3b1f\") " pod="openshift-multus/multus-c8d8d"
Jul 31 23:45:38.894760 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.894663    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-ovn-node-metrics-cert\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.894760 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.894699    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.894760 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.894732    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ee8bfdcd-3661-46f6-93a0-93bb3facacd4-os-release\") pod \"multus-additional-cni-plugins-r9jcj\" (UID: \"ee8bfdcd-3661-46f6-93a0-93bb3facacd4\") " pod="openshift-multus/multus-additional-cni-plugins-r9jcj"
Jul 31 23:45:38.894760 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.894743    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b490be26-0249-4036-8841-93b022ae3b1f-cni-binary-copy\") pod \"multus-c8d8d\" (UID: \"b490be26-0249-4036-8841-93b022ae3b1f\") " pod="openshift-multus/multus-c8d8d"
Jul 31 23:45:38.895612 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.894768    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ee8bfdcd-3661-46f6-93a0-93bb3facacd4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r9jcj\" (UID: \"ee8bfdcd-3661-46f6-93a0-93bb3facacd4\") " pod="openshift-multus/multus-additional-cni-plugins-r9jcj"
Jul 31 23:45:38.895612 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.894094    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/60b82286-9dac-458a-8b62-6bb90df47e04-etc-sysctl-d\") pod \"tuned-wggwt\" (UID: \"60b82286-9dac-458a-8b62-6bb90df47e04\") " pod="openshift-cluster-node-tuning-operator/tuned-wggwt"
Jul 31 23:45:38.895612 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.894803    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/60b82286-9dac-458a-8b62-6bb90df47e04-sys\") pod \"tuned-wggwt\" (UID: \"60b82286-9dac-458a-8b62-6bb90df47e04\") " pod="openshift-cluster-node-tuning-operator/tuned-wggwt"
Jul 31 23:45:38.895612 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.894886    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/60b82286-9dac-458a-8b62-6bb90df47e04-sys\") pod \"tuned-wggwt\" (UID: \"60b82286-9dac-458a-8b62-6bb90df47e04\") " pod="openshift-cluster-node-tuning-operator/tuned-wggwt"
Jul 31 23:45:38.895612 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.894957    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/60b82286-9dac-458a-8b62-6bb90df47e04-etc-sysctl-conf\") pod \"tuned-wggwt\" (UID: \"60b82286-9dac-458a-8b62-6bb90df47e04\") " pod="openshift-cluster-node-tuning-operator/tuned-wggwt"
Jul 31 23:45:38.895884 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.895638    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b490be26-0249-4036-8841-93b022ae3b1f-host-var-lib-cni-multus\") pod \"multus-c8d8d\" (UID: \"b490be26-0249-4036-8841-93b022ae3b1f\") " pod="openshift-multus/multus-c8d8d"
Jul 31 23:45:38.895884 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.895842    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b490be26-0249-4036-8841-93b022ae3b1f-host-var-lib-cni-multus\") pod \"multus-c8d8d\" (UID: \"b490be26-0249-4036-8841-93b022ae3b1f\") " pod="openshift-multus/multus-c8d8d"
Jul 31 23:45:38.895988 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.895882    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b490be26-0249-4036-8841-93b022ae3b1f-host-run-k8s-cni-cncf-io\") pod \"multus-c8d8d\" (UID: \"b490be26-0249-4036-8841-93b022ae3b1f\") " pod="openshift-multus/multus-c8d8d"
Jul 31 23:45:38.895988 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.895842    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/60b82286-9dac-458a-8b62-6bb90df47e04-host\") pod \"tuned-wggwt\" (UID: \"60b82286-9dac-458a-8b62-6bb90df47e04\") " pod="openshift-cluster-node-tuning-operator/tuned-wggwt"
Jul 31 23:45:38.895988 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:38.895965    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:45:38.896162 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.896132    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/60b82286-9dac-458a-8b62-6bb90df47e04-etc-tuned\") pod \"tuned-wggwt\" (UID: \"60b82286-9dac-458a-8b62-6bb90df47e04\") " pod="openshift-cluster-node-tuning-operator/tuned-wggwt"
Jul 31 23:45:38.896444 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:38.896424    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/450206d3-1b10-45ae-b6e9-6575c0d0be94-metrics-certs podName:450206d3-1b10-45ae-b6e9-6575c0d0be94 nodeName:}" failed. No retries permitted until 2026-07-31 23:45:39.396108421 +0000 UTC m=+2.115908256 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/450206d3-1b10-45ae-b6e9-6575c0d0be94-metrics-certs") pod "network-metrics-daemon-bpv44" (UID: "450206d3-1b10-45ae-b6e9-6575c0d0be94") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:45:38.896716 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.896696    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b490be26-0249-4036-8841-93b022ae3b1f-host-var-lib-cni-bin\") pod \"multus-c8d8d\" (UID: \"b490be26-0249-4036-8841-93b022ae3b1f\") " pod="openshift-multus/multus-c8d8d"
Jul 31 23:45:38.896792 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.896775    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b490be26-0249-4036-8841-93b022ae3b1f-hostroot\") pod \"multus-c8d8d\" (UID: \"b490be26-0249-4036-8841-93b022ae3b1f\") " pod="openshift-multus/multus-c8d8d"
Jul 31 23:45:38.896906 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.896833    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/96d86114-06a1-4c32-9d63-c7f4262fa871-hosts-file\") pod \"node-resolver-ld78r\" (UID: \"96d86114-06a1-4c32-9d63-c7f4262fa871\") " pod="openshift-dns/node-resolver-ld78r"
Jul 31 23:45:38.896906 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.896879    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b490be26-0249-4036-8841-93b022ae3b1f-host-var-lib-kubelet\") pod \"multus-c8d8d\" (UID: \"b490be26-0249-4036-8841-93b022ae3b1f\") " pod="openshift-multus/multus-c8d8d"
Jul 31 23:45:38.897022 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.896924    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-host-slash\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.897113 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.897091    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-run-openvswitch\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.897173 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.897125    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/539b6c48-1833-4461-8ab1-580974198fb3-agent-certs\") pod \"konnectivity-agent-92v2f\" (UID: \"539b6c48-1833-4461-8ab1-580974198fb3\") " pod="kube-system/konnectivity-agent-92v2f"
Jul 31 23:45:38.897871 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.897434    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b490be26-0249-4036-8841-93b022ae3b1f-multus-daemon-config\") pod \"multus-c8d8d\" (UID: \"b490be26-0249-4036-8841-93b022ae3b1f\") " pod="openshift-multus/multus-c8d8d"
Jul 31 23:45:38.897871 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.897607    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.897871 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.897681    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ee8bfdcd-3661-46f6-93a0-93bb3facacd4-os-release\") pod \"multus-additional-cni-plugins-r9jcj\" (UID: \"ee8bfdcd-3661-46f6-93a0-93bb3facacd4\") " pod="openshift-multus/multus-additional-cni-plugins-r9jcj"
Jul 31 23:45:38.897871 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.897768    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/482f5dcb-a14d-426f-93e6-b1ce4c9b89e6-host\") pod \"node-ca-7mgjz\" (UID: \"482f5dcb-a14d-426f-93e6-b1ce4c9b89e6\") " pod="openshift-image-registry/node-ca-7mgjz"
Jul 31 23:45:38.898158 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.897941    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee8bfdcd-3661-46f6-93a0-93bb3facacd4-cni-binary-copy\") pod \"multus-additional-cni-plugins-r9jcj\" (UID: \"ee8bfdcd-3661-46f6-93a0-93bb3facacd4\") " pod="openshift-multus/multus-additional-cni-plugins-r9jcj"
Jul 31 23:45:38.898279 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.898258    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ee8bfdcd-3661-46f6-93a0-93bb3facacd4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r9jcj\" (UID: \"ee8bfdcd-3661-46f6-93a0-93bb3facacd4\") " pod="openshift-multus/multus-additional-cni-plugins-r9jcj"
Jul 31 23:45:38.898363 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.898325    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0046f547-2701-496c-b596-578ce8949694-host-slash\") pod \"iptables-alerter-ck2fx\" (UID: \"0046f547-2701-496c-b596-578ce8949694\") " pod="openshift-network-operator/iptables-alerter-ck2fx"
Jul 31 23:45:38.898421 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.898360    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b490be26-0249-4036-8841-93b022ae3b1f-etc-kubernetes\") pod \"multus-c8d8d\" (UID: \"b490be26-0249-4036-8841-93b022ae3b1f\") " pod="openshift-multus/multus-c8d8d"
Jul 31 23:45:38.898421 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.898410    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/60b82286-9dac-458a-8b62-6bb90df47e04-etc-sysconfig\") pod \"tuned-wggwt\" (UID: \"60b82286-9dac-458a-8b62-6bb90df47e04\") " pod="openshift-cluster-node-tuning-operator/tuned-wggwt"
Jul 31 23:45:38.898532 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.898443    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9vqm\" (UniqueName: \"kubernetes.io/projected/0046f547-2701-496c-b596-578ce8949694-kube-api-access-q9vqm\") pod \"iptables-alerter-ck2fx\" (UID: \"0046f547-2701-496c-b596-578ce8949694\") " pod="openshift-network-operator/iptables-alerter-ck2fx"
Jul 31 23:45:38.898532 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.898469    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b490be26-0249-4036-8841-93b022ae3b1f-host-run-multus-certs\") pod \"multus-c8d8d\" (UID: \"b490be26-0249-4036-8841-93b022ae3b1f\") " pod="openshift-multus/multus-c8d8d"
Jul 31 23:45:38.898532 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.898516    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w48d\" (UniqueName: \"kubernetes.io/projected/96d86114-06a1-4c32-9d63-c7f4262fa871-kube-api-access-7w48d\") pod \"node-resolver-ld78r\" (UID: \"96d86114-06a1-4c32-9d63-c7f4262fa871\") " pod="openshift-dns/node-resolver-ld78r"
Jul 31 23:45:38.898689 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.898554    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b490be26-0249-4036-8841-93b022ae3b1f-multus-conf-dir\") pod \"multus-c8d8d\" (UID: \"b490be26-0249-4036-8841-93b022ae3b1f\") " pod="openshift-multus/multus-c8d8d"
Jul 31 23:45:38.898689 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.898584    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e5d655df-5dc6-4796-b7e7-b7e6596261bf-registration-dir\") pod \"aws-ebs-csi-driver-node-dw4n9\" (UID: \"e5d655df-5dc6-4796-b7e7-b7e6596261bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw4n9"
Jul 31 23:45:38.898689 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.898613    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpxtk\" (UniqueName: \"kubernetes.io/projected/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-kube-api-access-mpxtk\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.898689 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.898645    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-host-kubelet\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.901430 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.898683    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-etc-openvswitch\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.901430 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.898722    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/60b82286-9dac-458a-8b62-6bb90df47e04-etc-modprobe-d\") pod \"tuned-wggwt\" (UID: \"60b82286-9dac-458a-8b62-6bb90df47e04\") " pod="openshift-cluster-node-tuning-operator/tuned-wggwt"
Jul 31 23:45:38.901430 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.898752    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b490be26-0249-4036-8841-93b022ae3b1f-host-run-netns\") pod \"multus-c8d8d\" (UID: \"b490be26-0249-4036-8841-93b022ae3b1f\") " pod="openshift-multus/multus-c8d8d"
Jul 31 23:45:38.901430 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.898780    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06-kubelet-config\") pod \"global-pull-secret-syncer-jqggp\" (UID: \"e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06\") " pod="kube-system/global-pull-secret-syncer-jqggp"
Jul 31 23:45:38.901430 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.898814    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e5d655df-5dc6-4796-b7e7-b7e6596261bf-device-dir\") pod \"aws-ebs-csi-driver-node-dw4n9\" (UID: \"e5d655df-5dc6-4796-b7e7-b7e6596261bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw4n9"
Jul 31 23:45:38.901430 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.899545    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee8bfdcd-3661-46f6-93a0-93bb3facacd4-cnibin\") pod \"multus-additional-cni-plugins-r9jcj\" (UID: \"ee8bfdcd-3661-46f6-93a0-93bb3facacd4\") " pod="openshift-multus/multus-additional-cni-plugins-r9jcj"
Jul 31 23:45:38.901430 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.899586    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r987\" (UniqueName: \"kubernetes.io/projected/482f5dcb-a14d-426f-93e6-b1ce4c9b89e6-kube-api-access-8r987\") pod \"node-ca-7mgjz\" (UID: \"482f5dcb-a14d-426f-93e6-b1ce4c9b89e6\") " pod="openshift-image-registry/node-ca-7mgjz"
Jul 31 23:45:38.901430 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.899618    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b490be26-0249-4036-8841-93b022ae3b1f-cnibin\") pod \"multus-c8d8d\" (UID: \"b490be26-0249-4036-8841-93b022ae3b1f\") " pod="openshift-multus/multus-c8d8d"
Jul 31 23:45:38.901430 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.899141    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b490be26-0249-4036-8841-93b022ae3b1f-host-run-netns\") pod \"multus-c8d8d\" (UID: \"b490be26-0249-4036-8841-93b022ae3b1f\") " pod="openshift-multus/multus-c8d8d"
Jul 31 23:45:38.901430 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.899188    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06-kubelet-config\") pod \"global-pull-secret-syncer-jqggp\" (UID: \"e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06\") " pod="kube-system/global-pull-secret-syncer-jqggp"
Jul 31 23:45:38.901430 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.899303    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e5d655df-5dc6-4796-b7e7-b7e6596261bf-device-dir\") pod \"aws-ebs-csi-driver-node-dw4n9\" (UID: \"e5d655df-5dc6-4796-b7e7-b7e6596261bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw4n9"
Jul 31 23:45:38.901430 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.899664    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-host-kubelet\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.901430 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.899372    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/60b82286-9dac-458a-8b62-6bb90df47e04-etc-sysconfig\") pod \"tuned-wggwt\" (UID: \"60b82286-9dac-458a-8b62-6bb90df47e04\") " pod="openshift-cluster-node-tuning-operator/tuned-wggwt"
Jul 31 23:45:38.901430 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.899424    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e5d655df-5dc6-4796-b7e7-b7e6596261bf-registration-dir\") pod \"aws-ebs-csi-driver-node-dw4n9\" (UID: \"e5d655df-5dc6-4796-b7e7-b7e6596261bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw4n9"
Jul 31 23:45:38.901430 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.899732    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b490be26-0249-4036-8841-93b022ae3b1f-host-run-multus-certs\") pod \"multus-c8d8d\" (UID: \"b490be26-0249-4036-8841-93b022ae3b1f\") " pod="openshift-multus/multus-c8d8d"
Jul 31 23:45:38.901430 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.898927    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b490be26-0249-4036-8841-93b022ae3b1f-multus-conf-dir\") pod \"multus-c8d8d\" (UID: \"b490be26-0249-4036-8841-93b022ae3b1f\") " pod="openshift-multus/multus-c8d8d"
Jul 31 23:45:38.901430 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.899750    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-etc-openvswitch\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.901430 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.898969    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b490be26-0249-4036-8841-93b022ae3b1f-etc-kubernetes\") pod \"multus-c8d8d\" (UID: \"b490be26-0249-4036-8841-93b022ae3b1f\") " pod="openshift-multus/multus-c8d8d"
Jul 31 23:45:38.902426 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.899087    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/60b82286-9dac-458a-8b62-6bb90df47e04-etc-modprobe-d\") pod \"tuned-wggwt\" (UID: \"60b82286-9dac-458a-8b62-6bb90df47e04\") " pod="openshift-cluster-node-tuning-operator/tuned-wggwt"
Jul 31 23:45:38.902426 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.899803    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee8bfdcd-3661-46f6-93a0-93bb3facacd4-cnibin\") pod \"multus-additional-cni-plugins-r9jcj\" (UID: \"ee8bfdcd-3661-46f6-93a0-93bb3facacd4\") " pod="openshift-multus/multus-additional-cni-plugins-r9jcj"
Jul 31 23:45:38.902426 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.899868    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b490be26-0249-4036-8841-93b022ae3b1f-cnibin\") pod \"multus-c8d8d\" (UID: \"b490be26-0249-4036-8841-93b022ae3b1f\") " pod="openshift-multus/multus-c8d8d"
Jul 31 23:45:38.902426 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.900021    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-ovnkube-script-lib\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.902426 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.900067    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/482f5dcb-a14d-426f-93e6-b1ce4c9b89e6-serviceca\") pod \"node-ca-7mgjz\" (UID: \"482f5dcb-a14d-426f-93e6-b1ce4c9b89e6\") " pod="openshift-image-registry/node-ca-7mgjz"
Jul 31 23:45:38.902426 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.900104    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b490be26-0249-4036-8841-93b022ae3b1f-system-cni-dir\") pod \"multus-c8d8d\" (UID: \"b490be26-0249-4036-8841-93b022ae3b1f\") " pod="openshift-multus/multus-c8d8d"
Jul 31 23:45:38.902426 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.900124    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-ovn-node-metrics-cert\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.902426 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.900148    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/539b6c48-1833-4461-8ab1-580974198fb3-konnectivity-ca\") pod \"konnectivity-agent-92v2f\" (UID: \"539b6c48-1833-4461-8ab1-580974198fb3\") " pod="kube-system/konnectivity-agent-92v2f"
Jul 31 23:45:38.902426 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.900195    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96d86114-06a1-4c32-9d63-c7f4262fa871-tmp-dir\") pod \"node-resolver-ld78r\" (UID: \"96d86114-06a1-4c32-9d63-c7f4262fa871\") " pod="openshift-dns/node-resolver-ld78r"
Jul 31 23:45:38.902426 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.900210    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b490be26-0249-4036-8841-93b022ae3b1f-system-cni-dir\") pod \"multus-c8d8d\" (UID: \"b490be26-0249-4036-8841-93b022ae3b1f\") " pod="openshift-multus/multus-c8d8d"
Jul 31 23:45:38.902426 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.900232    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b490be26-0249-4036-8841-93b022ae3b1f-multus-cni-dir\") pod \"multus-c8d8d\" (UID: \"b490be26-0249-4036-8841-93b022ae3b1f\") " pod="openshift-multus/multus-c8d8d"
Jul 31 23:45:38.902426 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.900260    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e5d655df-5dc6-4796-b7e7-b7e6596261bf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dw4n9\" (UID: \"e5d655df-5dc6-4796-b7e7-b7e6596261bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw4n9"
Jul 31 23:45:38.902426 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.900424    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e5d655df-5dc6-4796-b7e7-b7e6596261bf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dw4n9\" (UID: \"e5d655df-5dc6-4796-b7e7-b7e6596261bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw4n9"
Jul 31 23:45:38.902426 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.900498    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b490be26-0249-4036-8841-93b022ae3b1f-multus-cni-dir\") pod \"multus-c8d8d\" (UID: \"b490be26-0249-4036-8841-93b022ae3b1f\") " pod="openshift-multus/multus-c8d8d"
Jul 31 23:45:38.902426 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.900560    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/482f5dcb-a14d-426f-93e6-b1ce4c9b89e6-serviceca\") pod \"node-ca-7mgjz\" (UID: \"482f5dcb-a14d-426f-93e6-b1ce4c9b89e6\") " pod="openshift-image-registry/node-ca-7mgjz"
Jul 31 23:45:38.902426 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.900576    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96d86114-06a1-4c32-9d63-c7f4262fa871-tmp-dir\") pod \"node-resolver-ld78r\" (UID: \"96d86114-06a1-4c32-9d63-c7f4262fa871\") " pod="openshift-dns/node-resolver-ld78r"
Jul 31 23:45:38.902426 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.900646    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-ovnkube-script-lib\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.902426 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.901016    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/539b6c48-1833-4461-8ab1-580974198fb3-konnectivity-ca\") pod \"konnectivity-agent-92v2f\" (UID: \"539b6c48-1833-4461-8ab1-580974198fb3\") " pod="kube-system/konnectivity-agent-92v2f"
Jul 31 23:45:38.903343 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:38.901433    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 23:45:38.903343 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:38.901460    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 23:45:38.903343 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:38.901478    2559 projected.go:196] Error preparing data for projected volume kube-api-access-7pgnt for pod openshift-network-diagnostics/network-check-target-s9j95: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:45:38.903343 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:38.901838    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64e3d3a1-0422-4faa-a938-a523a0a7b827-kube-api-access-7pgnt podName:64e3d3a1-0422-4faa-a938-a523a0a7b827 nodeName:}" failed. No retries permitted until 2026-07-31 23:45:39.401785694 +0000 UTC m=+2.121585541 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7pgnt" (UniqueName: "kubernetes.io/projected/64e3d3a1-0422-4faa-a938-a523a0a7b827-kube-api-access-7pgnt") pod "network-check-target-s9j95" (UID: "64e3d3a1-0422-4faa-a938-a523a0a7b827") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:45:38.903343 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.902001    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60b82286-9dac-458a-8b62-6bb90df47e04-tmp\") pod \"tuned-wggwt\" (UID: \"60b82286-9dac-458a-8b62-6bb90df47e04\") " pod="openshift-cluster-node-tuning-operator/tuned-wggwt"
Jul 31 23:45:38.903343 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:38.902562    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4016d03779e0d094ad64607388fac106.slice/crio-e5c8c9789dde5af8913b9ce712f889ebc3815ead83364dd78c370c4cde0aad4c WatchSource:0}: Error finding container e5c8c9789dde5af8913b9ce712f889ebc3815ead83364dd78c370c4cde0aad4c: Status 404 returned error can't find the container with id e5c8c9789dde5af8913b9ce712f889ebc3815ead83364dd78c370c4cde0aad4c
Jul 31 23:45:38.904602 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.904530    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s9tv\" (UniqueName: \"kubernetes.io/projected/b490be26-0249-4036-8841-93b022ae3b1f-kube-api-access-9s9tv\") pod \"multus-c8d8d\" (UID: \"b490be26-0249-4036-8841-93b022ae3b1f\") " pod="openshift-multus/multus-c8d8d"
Jul 31 23:45:38.904700 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.904641    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl5nb\" (UniqueName: \"kubernetes.io/projected/ee8bfdcd-3661-46f6-93a0-93bb3facacd4-kube-api-access-hl5nb\") pod \"multus-additional-cni-plugins-r9jcj\" (UID: \"ee8bfdcd-3661-46f6-93a0-93bb3facacd4\") " pod="openshift-multus/multus-additional-cni-plugins-r9jcj"
Jul 31 23:45:38.904868 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:38.904844    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod484eb4178e0de6efea5b73965227443b.slice/crio-cc53f57331f12ac6b33864cb59fa4cac1cde1dcb1d845462aba87744a1200554 WatchSource:0}: Error finding container cc53f57331f12ac6b33864cb59fa4cac1cde1dcb1d845462aba87744a1200554: Status 404 returned error can't find the container with id cc53f57331f12ac6b33864cb59fa4cac1cde1dcb1d845462aba87744a1200554
Jul 31 23:45:38.905410 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.905361    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzdnz\" (UniqueName: \"kubernetes.io/projected/60b82286-9dac-458a-8b62-6bb90df47e04-kube-api-access-fzdnz\") pod \"tuned-wggwt\" (UID: \"60b82286-9dac-458a-8b62-6bb90df47e04\") " pod="openshift-cluster-node-tuning-operator/tuned-wggwt"
Jul 31 23:45:38.905516 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.905479    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxphm\" (UniqueName: \"kubernetes.io/projected/e5d655df-5dc6-4796-b7e7-b7e6596261bf-kube-api-access-mxphm\") pod \"aws-ebs-csi-driver-node-dw4n9\" (UID: \"e5d655df-5dc6-4796-b7e7-b7e6596261bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw4n9"
Jul 31 23:45:38.908223 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.907840    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 23:45:38.908223 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.908053    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w48d\" (UniqueName: \"kubernetes.io/projected/96d86114-06a1-4c32-9d63-c7f4262fa871-kube-api-access-7w48d\") pod \"node-resolver-ld78r\" (UID: \"96d86114-06a1-4c32-9d63-c7f4262fa871\") " pod="openshift-dns/node-resolver-ld78r"
Jul 31 23:45:38.908223 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.908089    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s94lb\" (UniqueName: \"kubernetes.io/projected/450206d3-1b10-45ae-b6e9-6575c0d0be94-kube-api-access-s94lb\") pod \"network-metrics-daemon-bpv44\" (UID: \"450206d3-1b10-45ae-b6e9-6575c0d0be94\") " pod="openshift-multus/network-metrics-daemon-bpv44"
Jul 31 23:45:38.908223 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.908173    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9vqm\" (UniqueName: \"kubernetes.io/projected/0046f547-2701-496c-b596-578ce8949694-kube-api-access-q9vqm\") pod \"iptables-alerter-ck2fx\" (UID: \"0046f547-2701-496c-b596-578ce8949694\") " pod="openshift-network-operator/iptables-alerter-ck2fx"
Jul 31 23:45:38.908681 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.908663    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpxtk\" (UniqueName: \"kubernetes.io/projected/71cbd9c3-1692-4ded-a7fc-bf48635b0c29-kube-api-access-mpxtk\") pod \"ovnkube-node-nmsz6\" (UID: \"71cbd9c3-1692-4ded-a7fc-bf48635b0c29\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:38.908939 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.908923    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r987\" (UniqueName: \"kubernetes.io/projected/482f5dcb-a14d-426f-93e6-b1ce4c9b89e6-kube-api-access-8r987\") pod \"node-ca-7mgjz\" (UID: \"482f5dcb-a14d-426f-93e6-b1ce4c9b89e6\") " pod="openshift-image-registry/node-ca-7mgjz"
Jul 31 23:45:38.970531 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:38.970501    2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 23:45:39.011593 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:39.011528    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ld78r"
Jul 31 23:45:39.020794 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:39.020764    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96d86114_06a1_4c32_9d63_c7f4262fa871.slice/crio-ac58634665385abaef30b48d148863419559fc5fb15b37d74d09098dba661cbb WatchSource:0}: Error finding container ac58634665385abaef30b48d148863419559fc5fb15b37d74d09098dba661cbb: Status 404 returned error can't find the container with id ac58634665385abaef30b48d148863419559fc5fb15b37d74d09098dba661cbb
Jul 31 23:45:39.028186 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:39.028167    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:45:39.035202 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:39.035181    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71cbd9c3_1692_4ded_a7fc_bf48635b0c29.slice/crio-4212d7387c7457e125fb36ce3ac16401249164c164538d684b5f6b1aac63afdd WatchSource:0}: Error finding container 4212d7387c7457e125fb36ce3ac16401249164c164538d684b5f6b1aac63afdd: Status 404 returned error can't find the container with id 4212d7387c7457e125fb36ce3ac16401249164c164538d684b5f6b1aac63afdd
Jul 31 23:45:39.045943 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:39.045925    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7mgjz"
Jul 31 23:45:39.053351 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:39.053330    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod482f5dcb_a14d_426f_93e6_b1ce4c9b89e6.slice/crio-6c1da0b4cc57c43702acdb91d473daec95a70cd2143531528d7690d2a0019829 WatchSource:0}: Error finding container 6c1da0b4cc57c43702acdb91d473daec95a70cd2143531528d7690d2a0019829: Status 404 returned error can't find the container with id 6c1da0b4cc57c43702acdb91d473daec95a70cd2143531528d7690d2a0019829
Jul 31 23:45:39.058899 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:39.058878    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ck2fx"
Jul 31 23:45:39.065316 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:39.065299    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0046f547_2701_496c_b596_578ce8949694.slice/crio-cf6f034cfd793069cac74f1c0b3f51bcf96612489c8cfac6f64d6d2b6202aff9 WatchSource:0}: Error finding container cf6f034cfd793069cac74f1c0b3f51bcf96612489c8cfac6f64d6d2b6202aff9: Status 404 returned error can't find the container with id cf6f034cfd793069cac74f1c0b3f51bcf96612489c8cfac6f64d6d2b6202aff9
Jul 31 23:45:39.078065 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:39.078042    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-c8d8d"
Jul 31 23:45:39.085296 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:39.085273    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb490be26_0249_4036_8841_93b022ae3b1f.slice/crio-ef6c7593a640749c09ec24d58ae54044364439b0818b233ac859252cac808000 WatchSource:0}: Error finding container ef6c7593a640749c09ec24d58ae54044364439b0818b233ac859252cac808000: Status 404 returned error can't find the container with id ef6c7593a640749c09ec24d58ae54044364439b0818b233ac859252cac808000
Jul 31 23:45:39.093293 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:39.093275    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-r9jcj"
Jul 31 23:45:39.100547 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:39.100526    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee8bfdcd_3661_46f6_93a0_93bb3facacd4.slice/crio-0db838e16726f99b0a386a6823bd330cc8c7e7a4b8dda5e492d6357a36340f51 WatchSource:0}: Error finding container 0db838e16726f99b0a386a6823bd330cc8c7e7a4b8dda5e492d6357a36340f51: Status 404 returned error can't find the container with id 0db838e16726f99b0a386a6823bd330cc8c7e7a4b8dda5e492d6357a36340f51
Jul 31 23:45:39.108591 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:39.108575    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wggwt"
Jul 31 23:45:39.115143 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:39.115124    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60b82286_9dac_458a_8b62_6bb90df47e04.slice/crio-954195e2bccacb087214977ee77f22cf88edca6be417a5a0d924dd64b3b5b167 WatchSource:0}: Error finding container 954195e2bccacb087214977ee77f22cf88edca6be417a5a0d924dd64b3b5b167: Status 404 returned error can't find the container with id 954195e2bccacb087214977ee77f22cf88edca6be417a5a0d924dd64b3b5b167
Jul 31 23:45:39.123286 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:39.123265    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-92v2f"
Jul 31 23:45:39.128583 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:39.128569    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw4n9"
Jul 31 23:45:39.130549 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:39.130532    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod539b6c48_1833_4461_8ab1_580974198fb3.slice/crio-54c7c9668db80663a8a9eda5c6e89b467d0a380d601fece2106b7bf89c034185 WatchSource:0}: Error finding container 54c7c9668db80663a8a9eda5c6e89b467d0a380d601fece2106b7bf89c034185: Status 404 returned error can't find the container with id 54c7c9668db80663a8a9eda5c6e89b467d0a380d601fece2106b7bf89c034185
Jul 31 23:45:39.136155 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:45:39.136132    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5d655df_5dc6_4796_b7e7_b7e6596261bf.slice/crio-1c0941add72076b185819807c7eae0dcb91763ac37efbd871f9b1c74cfcaf84d WatchSource:0}: Error finding container 1c0941add72076b185819807c7eae0dcb91763ac37efbd871f9b1c74cfcaf84d: Status 404 returned error can't find the container with id 1c0941add72076b185819807c7eae0dcb91763ac37efbd871f9b1c74cfcaf84d
Jul 31 23:45:39.403309 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:39.403234    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06-original-pull-secret\") pod \"global-pull-secret-syncer-jqggp\" (UID: \"e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06\") " pod="kube-system/global-pull-secret-syncer-jqggp"
Jul 31 23:45:39.403309 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:39.403270    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pgnt\" (UniqueName: \"kubernetes.io/projected/64e3d3a1-0422-4faa-a938-a523a0a7b827-kube-api-access-7pgnt\") pod \"network-check-target-s9j95\" (UID: \"64e3d3a1-0422-4faa-a938-a523a0a7b827\") " pod="openshift-network-diagnostics/network-check-target-s9j95"
Jul 31 23:45:39.403537 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:39.403393    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 23:45:39.403537 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:39.403411    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 23:45:39.403537 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:39.403426    2559 projected.go:196] Error preparing data for projected volume kube-api-access-7pgnt for pod openshift-network-diagnostics/network-check-target-s9j95: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:45:39.403537 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:39.403437    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 23:45:39.403537 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:39.403432    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/450206d3-1b10-45ae-b6e9-6575c0d0be94-metrics-certs\") pod \"network-metrics-daemon-bpv44\" (UID: \"450206d3-1b10-45ae-b6e9-6575c0d0be94\") " pod="openshift-multus/network-metrics-daemon-bpv44"
Jul 31 23:45:39.403537 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:39.403509    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06-original-pull-secret podName:e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06 nodeName:}" failed. No retries permitted until 2026-07-31 23:45:40.403488994 +0000 UTC m=+3.123288824 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06-original-pull-secret") pod "global-pull-secret-syncer-jqggp" (UID: "e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06") : object "kube-system"/"original-pull-secret" not registered
Jul 31 23:45:39.403537 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:39.403521    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:45:39.403867 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:39.403562    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64e3d3a1-0422-4faa-a938-a523a0a7b827-kube-api-access-7pgnt podName:64e3d3a1-0422-4faa-a938-a523a0a7b827 nodeName:}" failed. No retries permitted until 2026-07-31 23:45:40.403537273 +0000 UTC m=+3.123337104 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7pgnt" (UniqueName: "kubernetes.io/projected/64e3d3a1-0422-4faa-a938-a523a0a7b827-kube-api-access-7pgnt") pod "network-check-target-s9j95" (UID: "64e3d3a1-0422-4faa-a938-a523a0a7b827") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:45:39.403867 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:39.403637    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/450206d3-1b10-45ae-b6e9-6575c0d0be94-metrics-certs podName:450206d3-1b10-45ae-b6e9-6575c0d0be94 nodeName:}" failed. No retries permitted until 2026-07-31 23:45:40.403625126 +0000 UTC m=+3.123424955 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/450206d3-1b10-45ae-b6e9-6575c0d0be94-metrics-certs") pod "network-metrics-daemon-bpv44" (UID: "450206d3-1b10-45ae-b6e9-6575c0d0be94") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:45:39.586991 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:39.586955    2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 23:45:39.793438 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:39.793308    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-92v2f" event={"ID":"539b6c48-1833-4461-8ab1-580974198fb3","Type":"ContainerStarted","Data":"54c7c9668db80663a8a9eda5c6e89b467d0a380d601fece2106b7bf89c034185"}
Jul 31 23:45:39.808674 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:39.808598    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r9jcj" event={"ID":"ee8bfdcd-3661-46f6-93a0-93bb3facacd4","Type":"ContainerStarted","Data":"0db838e16726f99b0a386a6823bd330cc8c7e7a4b8dda5e492d6357a36340f51"}
Jul 31 23:45:39.816821 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:39.816697    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7mgjz" event={"ID":"482f5dcb-a14d-426f-93e6-b1ce4c9b89e6","Type":"ContainerStarted","Data":"6c1da0b4cc57c43702acdb91d473daec95a70cd2143531528d7690d2a0019829"}
Jul 31 23:45:39.840896 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:39.840844    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-30 23:40:39 +0000 UTC" deadline="2027-12-30 11:10:02.860485549 +0000 UTC"
Jul 31 23:45:39.840896 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:39.840880    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12395h24m23.019610353s"
Jul 31 23:45:39.841865 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:39.841746    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-148.ec2.internal" event={"ID":"4016d03779e0d094ad64607388fac106","Type":"ContainerStarted","Data":"e5c8c9789dde5af8913b9ce712f889ebc3815ead83364dd78c370c4cde0aad4c"}
Jul 31 23:45:39.850189 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:39.850088    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw4n9" event={"ID":"e5d655df-5dc6-4796-b7e7-b7e6596261bf","Type":"ContainerStarted","Data":"1c0941add72076b185819807c7eae0dcb91763ac37efbd871f9b1c74cfcaf84d"}
Jul 31 23:45:39.869426 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:39.869392    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wggwt" event={"ID":"60b82286-9dac-458a-8b62-6bb90df47e04","Type":"ContainerStarted","Data":"954195e2bccacb087214977ee77f22cf88edca6be417a5a0d924dd64b3b5b167"}
Jul 31 23:45:39.876257 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:39.876002    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-c8d8d" event={"ID":"b490be26-0249-4036-8841-93b022ae3b1f","Type":"ContainerStarted","Data":"ef6c7593a640749c09ec24d58ae54044364439b0818b233ac859252cac808000"}
Jul 31 23:45:39.880886 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:39.880858    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ck2fx" event={"ID":"0046f547-2701-496c-b596-578ce8949694","Type":"ContainerStarted","Data":"cf6f034cfd793069cac74f1c0b3f51bcf96612489c8cfac6f64d6d2b6202aff9"}
Jul 31 23:45:39.888780 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:39.888653    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6" event={"ID":"71cbd9c3-1692-4ded-a7fc-bf48635b0c29","Type":"ContainerStarted","Data":"4212d7387c7457e125fb36ce3ac16401249164c164538d684b5f6b1aac63afdd"}
Jul 31 23:45:39.902982 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:39.902956    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ld78r" event={"ID":"96d86114-06a1-4c32-9d63-c7f4262fa871","Type":"ContainerStarted","Data":"ac58634665385abaef30b48d148863419559fc5fb15b37d74d09098dba661cbb"}
Jul 31 23:45:39.913547 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:39.913515    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-148.ec2.internal" event={"ID":"484eb4178e0de6efea5b73965227443b","Type":"ContainerStarted","Data":"cc53f57331f12ac6b33864cb59fa4cac1cde1dcb1d845462aba87744a1200554"}
Jul 31 23:45:40.412633 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:40.412595    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06-original-pull-secret\") pod \"global-pull-secret-syncer-jqggp\" (UID: \"e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06\") " pod="kube-system/global-pull-secret-syncer-jqggp"
Jul 31 23:45:40.412853 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:40.412655    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pgnt\" (UniqueName: \"kubernetes.io/projected/64e3d3a1-0422-4faa-a938-a523a0a7b827-kube-api-access-7pgnt\") pod \"network-check-target-s9j95\" (UID: \"64e3d3a1-0422-4faa-a938-a523a0a7b827\") " pod="openshift-network-diagnostics/network-check-target-s9j95"
Jul 31 23:45:40.412853 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:40.412710    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/450206d3-1b10-45ae-b6e9-6575c0d0be94-metrics-certs\") pod \"network-metrics-daemon-bpv44\" (UID: \"450206d3-1b10-45ae-b6e9-6575c0d0be94\") " pod="openshift-multus/network-metrics-daemon-bpv44"
Jul 31 23:45:40.413002 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:40.412865    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:45:40.413002 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:40.412928    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/450206d3-1b10-45ae-b6e9-6575c0d0be94-metrics-certs podName:450206d3-1b10-45ae-b6e9-6575c0d0be94 nodeName:}" failed. No retries permitted until 2026-07-31 23:45:42.412909481 +0000 UTC m=+5.132709321 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/450206d3-1b10-45ae-b6e9-6575c0d0be94-metrics-certs") pod "network-metrics-daemon-bpv44" (UID: "450206d3-1b10-45ae-b6e9-6575c0d0be94") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:45:40.413372 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:40.413350    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 23:45:40.413467 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:40.413413    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06-original-pull-secret podName:e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06 nodeName:}" failed. No retries permitted until 2026-07-31 23:45:42.413397015 +0000 UTC m=+5.133196857 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06-original-pull-secret") pod "global-pull-secret-syncer-jqggp" (UID: "e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06") : object "kube-system"/"original-pull-secret" not registered
Jul 31 23:45:40.413544 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:40.413496    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 23:45:40.413544 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:40.413516    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 23:45:40.413544 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:40.413529    2559 projected.go:196] Error preparing data for projected volume kube-api-access-7pgnt for pod openshift-network-diagnostics/network-check-target-s9j95: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:45:40.413697 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:40.413565    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64e3d3a1-0422-4faa-a938-a523a0a7b827-kube-api-access-7pgnt podName:64e3d3a1-0422-4faa-a938-a523a0a7b827 nodeName:}" failed. No retries permitted until 2026-07-31 23:45:42.413553597 +0000 UTC m=+5.133353437 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7pgnt" (UniqueName: "kubernetes.io/projected/64e3d3a1-0422-4faa-a938-a523a0a7b827-kube-api-access-7pgnt") pod "network-check-target-s9j95" (UID: "64e3d3a1-0422-4faa-a938-a523a0a7b827") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:45:40.664023 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:40.663938    2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 23:45:40.780866 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:40.780830    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jqggp"
Jul 31 23:45:40.781064 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:40.780975    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jqggp" podUID="e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06"
Jul 31 23:45:40.781064 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:40.781042    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpv44"
Jul 31 23:45:40.781186 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:40.781169    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpv44" podUID="450206d3-1b10-45ae-b6e9-6575c0d0be94"
Jul 31 23:45:40.781404 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:40.781246    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s9j95"
Jul 31 23:45:40.781404 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:40.781360    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s9j95" podUID="64e3d3a1-0422-4faa-a938-a523a0a7b827"
Jul 31 23:45:40.841357 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:40.841314    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-30 23:40:39 +0000 UTC" deadline="2028-03-23 01:28:27.954798499 +0000 UTC"
Jul 31 23:45:40.841357 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:40.841355    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14401h42m47.113448387s"
Jul 31 23:45:42.429691 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:42.428614    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06-original-pull-secret\") pod \"global-pull-secret-syncer-jqggp\" (UID: \"e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06\") " pod="kube-system/global-pull-secret-syncer-jqggp"
Jul 31 23:45:42.429691 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:42.428675    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pgnt\" (UniqueName: \"kubernetes.io/projected/64e3d3a1-0422-4faa-a938-a523a0a7b827-kube-api-access-7pgnt\") pod \"network-check-target-s9j95\" (UID: \"64e3d3a1-0422-4faa-a938-a523a0a7b827\") " pod="openshift-network-diagnostics/network-check-target-s9j95"
Jul 31 23:45:42.429691 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:42.428720    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/450206d3-1b10-45ae-b6e9-6575c0d0be94-metrics-certs\") pod \"network-metrics-daemon-bpv44\" (UID: \"450206d3-1b10-45ae-b6e9-6575c0d0be94\") " pod="openshift-multus/network-metrics-daemon-bpv44"
Jul 31 23:45:42.429691 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:42.428887    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:45:42.429691 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:42.428955    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/450206d3-1b10-45ae-b6e9-6575c0d0be94-metrics-certs podName:450206d3-1b10-45ae-b6e9-6575c0d0be94 nodeName:}" failed. No retries permitted until 2026-07-31 23:45:46.428934188 +0000 UTC m=+9.148734043 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/450206d3-1b10-45ae-b6e9-6575c0d0be94-metrics-certs") pod "network-metrics-daemon-bpv44" (UID: "450206d3-1b10-45ae-b6e9-6575c0d0be94") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:45:42.429691 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:42.429414    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 23:45:42.429691 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:42.429472    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06-original-pull-secret podName:e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06 nodeName:}" failed. No retries permitted until 2026-07-31 23:45:46.429450187 +0000 UTC m=+9.149250038 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06-original-pull-secret") pod "global-pull-secret-syncer-jqggp" (UID: "e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06") : object "kube-system"/"original-pull-secret" not registered
Jul 31 23:45:42.429691 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:42.429560    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 23:45:42.429691 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:42.429578    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 23:45:42.429691 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:42.429592    2559 projected.go:196] Error preparing data for projected volume kube-api-access-7pgnt for pod openshift-network-diagnostics/network-check-target-s9j95: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:45:42.429691 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:42.429631    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64e3d3a1-0422-4faa-a938-a523a0a7b827-kube-api-access-7pgnt podName:64e3d3a1-0422-4faa-a938-a523a0a7b827 nodeName:}" failed. No retries permitted until 2026-07-31 23:45:46.429619231 +0000 UTC m=+9.149419076 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7pgnt" (UniqueName: "kubernetes.io/projected/64e3d3a1-0422-4faa-a938-a523a0a7b827-kube-api-access-7pgnt") pod "network-check-target-s9j95" (UID: "64e3d3a1-0422-4faa-a938-a523a0a7b827") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:45:42.781148 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:42.781110    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jqggp"
Jul 31 23:45:42.781341 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:42.781262    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jqggp" podUID="e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06"
Jul 31 23:45:42.781677 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:42.781658    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpv44"
Jul 31 23:45:42.781786 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:42.781766    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpv44" podUID="450206d3-1b10-45ae-b6e9-6575c0d0be94"
Jul 31 23:45:42.781861 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:42.781846    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s9j95"
Jul 31 23:45:42.781930 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:42.781915    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s9j95" podUID="64e3d3a1-0422-4faa-a938-a523a0a7b827"
Jul 31 23:45:44.781373 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:44.780728    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s9j95"
Jul 31 23:45:44.781373 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:44.780856    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s9j95" podUID="64e3d3a1-0422-4faa-a938-a523a0a7b827"
Jul 31 23:45:44.781373 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:44.781175    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpv44"
Jul 31 23:45:44.781373 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:44.781259    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpv44" podUID="450206d3-1b10-45ae-b6e9-6575c0d0be94"
Jul 31 23:45:44.781373 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:44.781294    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jqggp"
Jul 31 23:45:44.781373 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:44.781343    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jqggp" podUID="e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06"
Jul 31 23:45:46.464340 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:46.463968    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06-original-pull-secret\") pod \"global-pull-secret-syncer-jqggp\" (UID: \"e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06\") " pod="kube-system/global-pull-secret-syncer-jqggp"
Jul 31 23:45:46.464340 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:46.464025    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pgnt\" (UniqueName: \"kubernetes.io/projected/64e3d3a1-0422-4faa-a938-a523a0a7b827-kube-api-access-7pgnt\") pod \"network-check-target-s9j95\" (UID: \"64e3d3a1-0422-4faa-a938-a523a0a7b827\") " pod="openshift-network-diagnostics/network-check-target-s9j95"
Jul 31 23:45:46.464340 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:46.464066    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/450206d3-1b10-45ae-b6e9-6575c0d0be94-metrics-certs\") pod \"network-metrics-daemon-bpv44\" (UID: \"450206d3-1b10-45ae-b6e9-6575c0d0be94\") " pod="openshift-multus/network-metrics-daemon-bpv44"
Jul 31 23:45:46.464340 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:46.464121    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 23:45:46.464340 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:46.464157    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:45:46.464340 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:46.464197    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06-original-pull-secret podName:e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06 nodeName:}" failed. No retries permitted until 2026-07-31 23:45:54.464167702 +0000 UTC m=+17.183967537 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06-original-pull-secret") pod "global-pull-secret-syncer-jqggp" (UID: "e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06") : object "kube-system"/"original-pull-secret" not registered
Jul 31 23:45:46.464340 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:46.464217    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/450206d3-1b10-45ae-b6e9-6575c0d0be94-metrics-certs podName:450206d3-1b10-45ae-b6e9-6575c0d0be94 nodeName:}" failed. No retries permitted until 2026-07-31 23:45:54.464206716 +0000 UTC m=+17.184006559 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/450206d3-1b10-45ae-b6e9-6575c0d0be94-metrics-certs") pod "network-metrics-daemon-bpv44" (UID: "450206d3-1b10-45ae-b6e9-6575c0d0be94") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:45:46.464340 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:46.464249    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 23:45:46.464340 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:46.464263    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 23:45:46.464340 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:46.464276    2559 projected.go:196] Error preparing data for projected volume kube-api-access-7pgnt for pod openshift-network-diagnostics/network-check-target-s9j95: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:45:46.464340 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:46.464311    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64e3d3a1-0422-4faa-a938-a523a0a7b827-kube-api-access-7pgnt podName:64e3d3a1-0422-4faa-a938-a523a0a7b827 nodeName:}" failed. No retries permitted until 2026-07-31 23:45:54.464301391 +0000 UTC m=+17.184101225 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7pgnt" (UniqueName: "kubernetes.io/projected/64e3d3a1-0422-4faa-a938-a523a0a7b827-kube-api-access-7pgnt") pod "network-check-target-s9j95" (UID: "64e3d3a1-0422-4faa-a938-a523a0a7b827") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:45:46.780548 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:46.780507    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s9j95"
Jul 31 23:45:46.780749 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:46.780663    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpv44"
Jul 31 23:45:46.780861 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:46.780780    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s9j95" podUID="64e3d3a1-0422-4faa-a938-a523a0a7b827"
Jul 31 23:45:46.780861 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:46.780803    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpv44" podUID="450206d3-1b10-45ae-b6e9-6575c0d0be94"
Jul 31 23:45:46.780861 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:46.780858    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jqggp"
Jul 31 23:45:46.781029 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:46.780936    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jqggp" podUID="e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06"
Jul 31 23:45:48.781132 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:48.780591    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s9j95"
Jul 31 23:45:48.781132 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:48.780726    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s9j95" podUID="64e3d3a1-0422-4faa-a938-a523a0a7b827"
Jul 31 23:45:48.781132 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:48.780921    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jqggp"
Jul 31 23:45:48.781132 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:48.780932    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpv44"
Jul 31 23:45:48.781132 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:48.781026    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jqggp" podUID="e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06"
Jul 31 23:45:48.781132 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:48.781123    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpv44" podUID="450206d3-1b10-45ae-b6e9-6575c0d0be94"
Jul 31 23:45:50.780815 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:50.780768    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jqggp"
Jul 31 23:45:50.780815 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:50.780807    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpv44"
Jul 31 23:45:50.781325 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:50.780776    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s9j95"
Jul 31 23:45:50.781325 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:50.780891    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jqggp" podUID="e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06"
Jul 31 23:45:50.781325 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:50.780941    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s9j95" podUID="64e3d3a1-0422-4faa-a938-a523a0a7b827"
Jul 31 23:45:50.781325 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:50.781023    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpv44" podUID="450206d3-1b10-45ae-b6e9-6575c0d0be94"
Jul 31 23:45:52.781087 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:52.781054    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jqggp"
Jul 31 23:45:52.781542 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:52.781172    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpv44"
Jul 31 23:45:52.781542 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:52.781178    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jqggp" podUID="e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06"
Jul 31 23:45:52.781542 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:52.781179    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s9j95"
Jul 31 23:45:52.781542 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:52.781282    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpv44" podUID="450206d3-1b10-45ae-b6e9-6575c0d0be94"
Jul 31 23:45:52.781542 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:52.781332    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s9j95" podUID="64e3d3a1-0422-4faa-a938-a523a0a7b827"
Jul 31 23:45:54.529926 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:54.529892    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pgnt\" (UniqueName: \"kubernetes.io/projected/64e3d3a1-0422-4faa-a938-a523a0a7b827-kube-api-access-7pgnt\") pod \"network-check-target-s9j95\" (UID: \"64e3d3a1-0422-4faa-a938-a523a0a7b827\") " pod="openshift-network-diagnostics/network-check-target-s9j95"
Jul 31 23:45:54.530333 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:54.529945    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/450206d3-1b10-45ae-b6e9-6575c0d0be94-metrics-certs\") pod \"network-metrics-daemon-bpv44\" (UID: \"450206d3-1b10-45ae-b6e9-6575c0d0be94\") " pod="openshift-multus/network-metrics-daemon-bpv44"
Jul 31 23:45:54.530333 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:54.529984    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06-original-pull-secret\") pod \"global-pull-secret-syncer-jqggp\" (UID: \"e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06\") " pod="kube-system/global-pull-secret-syncer-jqggp"
Jul 31 23:45:54.530333 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:54.530058    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 23:45:54.530333 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:54.530098    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 23:45:54.530333 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:54.530111    2559 projected.go:196] Error preparing data for projected volume kube-api-access-7pgnt for pod openshift-network-diagnostics/network-check-target-s9j95: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:45:54.530333 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:54.530062    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:45:54.530333 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:54.530161    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64e3d3a1-0422-4faa-a938-a523a0a7b827-kube-api-access-7pgnt podName:64e3d3a1-0422-4faa-a938-a523a0a7b827 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:10.530149056 +0000 UTC m=+33.249948880 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7pgnt" (UniqueName: "kubernetes.io/projected/64e3d3a1-0422-4faa-a938-a523a0a7b827-kube-api-access-7pgnt") pod "network-check-target-s9j95" (UID: "64e3d3a1-0422-4faa-a938-a523a0a7b827") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:45:54.530333 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:54.530188    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/450206d3-1b10-45ae-b6e9-6575c0d0be94-metrics-certs podName:450206d3-1b10-45ae-b6e9-6575c0d0be94 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:10.530176181 +0000 UTC m=+33.249976005 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/450206d3-1b10-45ae-b6e9-6575c0d0be94-metrics-certs") pod "network-metrics-daemon-bpv44" (UID: "450206d3-1b10-45ae-b6e9-6575c0d0be94") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:45:54.530333 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:54.530062    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 23:45:54.530333 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:54.530211    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06-original-pull-secret podName:e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:10.530205033 +0000 UTC m=+33.250004856 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06-original-pull-secret") pod "global-pull-secret-syncer-jqggp" (UID: "e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06") : object "kube-system"/"original-pull-secret" not registered
Jul 31 23:45:54.780388 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:54.780287    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s9j95"
Jul 31 23:45:54.780568 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:54.780292    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpv44"
Jul 31 23:45:54.780568 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:54.780414    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s9j95" podUID="64e3d3a1-0422-4faa-a938-a523a0a7b827"
Jul 31 23:45:54.780568 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:54.780292    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jqggp"
Jul 31 23:45:54.780568 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:54.780513    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpv44" podUID="450206d3-1b10-45ae-b6e9-6575c0d0be94"
Jul 31 23:45:54.780568 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:54.780558    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jqggp" podUID="e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06"
Jul 31 23:45:56.780798 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:56.780694    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jqggp"
Jul 31 23:45:56.780798 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:56.780758    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpv44"
Jul 31 23:45:56.781385 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:56.780845    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s9j95"
Jul 31 23:45:56.781385 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:56.780852    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jqggp" podUID="e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06"
Jul 31 23:45:56.781385 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:56.780949    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s9j95" podUID="64e3d3a1-0422-4faa-a938-a523a0a7b827"
Jul 31 23:45:56.781385 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:56.781018    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpv44" podUID="450206d3-1b10-45ae-b6e9-6575c0d0be94"
Jul 31 23:45:58.781115 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:58.781078    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jqggp"
Jul 31 23:45:58.781522 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:58.781192    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s9j95"
Jul 31 23:45:58.781522 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:58.781202    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpv44"
Jul 31 23:45:58.781522 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:58.781198    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jqggp" podUID="e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06"
Jul 31 23:45:58.781522 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:58.781295    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s9j95" podUID="64e3d3a1-0422-4faa-a938-a523a0a7b827"
Jul 31 23:45:58.781522 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:45:58.781378    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpv44" podUID="450206d3-1b10-45ae-b6e9-6575c0d0be94"
Jul 31 23:45:59.977200 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:59.976948    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-c8d8d" event={"ID":"b490be26-0249-4036-8841-93b022ae3b1f","Type":"ContainerStarted","Data":"05062c913b17bb5558886e926b1b104269950ee595565002fb8f3f40d8f55c4d"}
Jul 31 23:45:59.979550 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:59.979522    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6" event={"ID":"71cbd9c3-1692-4ded-a7fc-bf48635b0c29","Type":"ContainerStarted","Data":"02d202c485731481500a510157fc60286aed2b36598b528421f901c46c7ab751"}
Jul 31 23:45:59.979665 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:59.979608    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6" event={"ID":"71cbd9c3-1692-4ded-a7fc-bf48635b0c29","Type":"ContainerStarted","Data":"8eb3684612b2f33dbc3a6ba0f7b14bd382d8a946285f68ff8bf2fdc4c2b7c8d7"}
Jul 31 23:45:59.979665 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:59.979660    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6" event={"ID":"71cbd9c3-1692-4ded-a7fc-bf48635b0c29","Type":"ContainerStarted","Data":"a55b0c11337d604586f0b42d27e1978e7a202fe7c5ade3f695ed22b8db00ab3a"}
Jul 31 23:45:59.982020 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:59.981994    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-148.ec2.internal" event={"ID":"4016d03779e0d094ad64607388fac106","Type":"ContainerStarted","Data":"03bfeaf140117cd790aa62faf72feeae82124dd03639028102de7fabfdcdc606"}
Jul 31 23:45:59.983415 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:59.983394    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wggwt" event={"ID":"60b82286-9dac-458a-8b62-6bb90df47e04","Type":"ContainerStarted","Data":"a86dc3631bfb3b735a711e082dd634624c134615ab93224390ac4ca17e9a7b6b"}
Jul 31 23:45:59.999077 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:45:59.999027    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-c8d8d" podStartSLOduration=1.410867837 podStartE2EDuration="21.999008615s" totalImagesPullingTime="20.588140778s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:45:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:45:59.998425352 +0000 UTC m=+22.718225199" watchObservedRunningTime="2026-07-31 23:45:59.999008615 +0000 UTC m=+22.718808462"
Jul 31 23:46:00.013450 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:00.013411    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-148.ec2.internal" podStartSLOduration=21.013399645 podStartE2EDuration="21.013399645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:45:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:00.012996613 +0000 UTC m=+22.732796460" watchObservedRunningTime="2026-07-31 23:46:00.013399645 +0000 UTC m=+22.733199491"
Jul 31 23:46:00.029543 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:00.029496    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wggwt" podStartSLOduration=0.514037405 podStartE2EDuration="21.029479372s" totalImagesPullingTime="20.515441967s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:45:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:00.029140537 +0000 UTC m=+22.748940383" watchObservedRunningTime="2026-07-31 23:46:00.029479372 +0000 UTC m=+22.749279219"
Jul 31 23:46:00.781087 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:00.781051    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s9j95"
Jul 31 23:46:00.781248 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:00.781063    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jqggp"
Jul 31 23:46:00.781248 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:00.781192    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpv44"
Jul 31 23:46:00.781248 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:00.781188    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s9j95" podUID="64e3d3a1-0422-4faa-a938-a523a0a7b827"
Jul 31 23:46:00.781418 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:00.781262    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jqggp" podUID="e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06"
Jul 31 23:46:00.781418 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:00.781341    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpv44" podUID="450206d3-1b10-45ae-b6e9-6575c0d0be94"
Jul 31 23:46:00.986972 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:00.986937    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ck2fx" event={"ID":"0046f547-2701-496c-b596-578ce8949694","Type":"ContainerStarted","Data":"7addade008a3276c1b152617ed4a8eebaf5ee8bb46f223bc58c75da08c65beb9"}
Jul 31 23:46:00.989623 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:00.989599    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6" event={"ID":"71cbd9c3-1692-4ded-a7fc-bf48635b0c29","Type":"ContainerStarted","Data":"96b1841fb557672b1e269829c1269dfa24ffabff56472dec02230a03c23d8d76"}
Jul 31 23:46:00.989767 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:00.989670    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6" event={"ID":"71cbd9c3-1692-4ded-a7fc-bf48635b0c29","Type":"ContainerStarted","Data":"24042fa157a7d404e5c53f444cddc0600b7eb87e61adb7a396f192cb1e68dbf1"}
Jul 31 23:46:00.989767 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:00.989718    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6" event={"ID":"71cbd9c3-1692-4ded-a7fc-bf48635b0c29","Type":"ContainerStarted","Data":"d208c1e01988eb2dab86c6e2e258d068a9a0c2e62777617e667869d8de812a44"}
Jul 31 23:46:00.991048 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:00.991026    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ld78r" event={"ID":"96d86114-06a1-4c32-9d63-c7f4262fa871","Type":"ContainerStarted","Data":"a6e0e4a65e6f91e6a0e18903a7e4f9bf1e9f4643d66fc49d4d0b753cb3288ee7"}
Jul 31 23:46:00.992387 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:00.992368    2559 generic.go:356] "Generic (PLEG): container finished" podID="484eb4178e0de6efea5b73965227443b" containerID="1d2185c3689cd444f6541a6182961fef50fa987118b90306d5fb9e1698688c33" exitCode=0
Jul 31 23:46:00.992464 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:00.992428    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-148.ec2.internal" event={"ID":"484eb4178e0de6efea5b73965227443b","Type":"ContainerDied","Data":"1d2185c3689cd444f6541a6182961fef50fa987118b90306d5fb9e1698688c33"}
Jul 31 23:46:00.993701 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:00.993683    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-92v2f" event={"ID":"539b6c48-1833-4461-8ab1-580974198fb3","Type":"ContainerStarted","Data":"0e31778f87b795a4f58254fef7d1f85b7eed53b3151e11cd351e5d7d566229d3"}
Jul 31 23:46:00.995080 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:00.995058    2559 generic.go:356] "Generic (PLEG): container finished" podID="ee8bfdcd-3661-46f6-93a0-93bb3facacd4" containerID="2474226e0361f2bddf4b9d170af673684dbc8590ed35d8d0a560318a5932d81a" exitCode=0
Jul 31 23:46:00.995152 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:00.995132    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r9jcj" event={"ID":"ee8bfdcd-3661-46f6-93a0-93bb3facacd4","Type":"ContainerDied","Data":"2474226e0361f2bddf4b9d170af673684dbc8590ed35d8d0a560318a5932d81a"}
Jul 31 23:46:00.996534 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:00.996513    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7mgjz" event={"ID":"482f5dcb-a14d-426f-93e6-b1ce4c9b89e6","Type":"ContainerStarted","Data":"6ab802cda9808126b3f86697edb8aa901d021a3805137e594c0180693bd49328"}
Jul 31 23:46:00.997818 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:00.997798    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw4n9" event={"ID":"e5d655df-5dc6-4796-b7e7-b7e6596261bf","Type":"ContainerStarted","Data":"0f2389b440025258c53816ef23474c8a6c0027918a90fc69926b55041eaf767b"}
Jul 31 23:46:01.002773 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:01.002717    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ck2fx" podStartSLOduration=2.452990325 podStartE2EDuration="23.002704624s" totalImagesPullingTime="20.549714299s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:45:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:01.002468169 +0000 UTC m=+23.722268014" watchObservedRunningTime="2026-07-31 23:46:01.002704624 +0000 UTC m=+23.722504470"
Jul 31 23:46:01.036527 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:01.036350    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-ld78r" podStartSLOduration=2.440607448 podStartE2EDuration="23.036336707s" totalImagesPullingTime="20.595729259s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:45:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:01.036088809 +0000 UTC m=+23.755888654" watchObservedRunningTime="2026-07-31 23:46:01.036336707 +0000 UTC m=+23.756136552"
Jul 31 23:46:01.053998 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:01.053945    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7mgjz" podStartSLOduration=2.488846859 podStartE2EDuration="23.053928952s" totalImagesPullingTime="20.565082093s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:45:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:01.053814056 +0000 UTC m=+23.773613903" watchObservedRunningTime="2026-07-31 23:46:01.053928952 +0000 UTC m=+23.773728799"
Jul 31 23:46:01.067811 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:01.067768    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-92v2f" podStartSLOduration=1.5801032849999999 podStartE2EDuration="22.067755363s" totalImagesPullingTime="20.487652078s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:45:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:01.067350131 +0000 UTC m=+23.787149995" watchObservedRunningTime="2026-07-31 23:46:01.067755363 +0000 UTC m=+23.787555201"
Jul 31 23:46:01.162083 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:01.162049    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-92v2f"
Jul 31 23:46:01.552315 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:01.552289    2559 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 31 23:46:01.761503 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:01.761310    2559 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-31T23:46:01.552310042Z","UUID":"4b606b9a-693d-4082-b2f3-3dc9262e858a","Handler":null,"Name":"","Endpoint":""}
Jul 31 23:46:01.763005 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:01.762983    2559 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 31 23:46:01.763145 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:01.763016    2559 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 31 23:46:02.003996 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:02.003963    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw4n9" event={"ID":"e5d655df-5dc6-4796-b7e7-b7e6596261bf","Type":"ContainerStarted","Data":"6436426a84a13e7781e68ee3e975e9835e187dd1ee526922a27d1ad1c18a0195"}
Jul 31 23:46:02.005878 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:02.005847    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-148.ec2.internal" event={"ID":"484eb4178e0de6efea5b73965227443b","Type":"ContainerStarted","Data":"8b77adfb0901b124550891f1c138a59ce15049e7db7edf6b8882e1fba5cb9de7"}
Jul 31 23:46:02.021111 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:02.021012    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-148.ec2.internal" podStartSLOduration=23.020996058 podStartE2EDuration="23.020996058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:45:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:02.020836086 +0000 UTC m=+24.740635932" watchObservedRunningTime="2026-07-31 23:46:02.020996058 +0000 UTC m=+24.740795906"
Jul 31 23:46:02.780953 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:02.780915    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpv44"
Jul 31 23:46:02.781170 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:02.781063    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpv44" podUID="450206d3-1b10-45ae-b6e9-6575c0d0be94"
Jul 31 23:46:02.781170 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:02.780918    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jqggp"
Jul 31 23:46:02.781170 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:02.781158    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jqggp" podUID="e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06"
Jul 31 23:46:02.781170 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:02.780915    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s9j95"
Jul 31 23:46:02.781330 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:02.781228    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s9j95" podUID="64e3d3a1-0422-4faa-a938-a523a0a7b827"
Jul 31 23:46:03.011757 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:03.011715    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6" event={"ID":"71cbd9c3-1692-4ded-a7fc-bf48635b0c29","Type":"ContainerStarted","Data":"cd7e5d49c161502928f6aeed4569633489a294d227e476cb45448d880ad83416"}
Jul 31 23:46:04.016101 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:04.016056    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw4n9" event={"ID":"e5d655df-5dc6-4796-b7e7-b7e6596261bf","Type":"ContainerStarted","Data":"e7fd72b97bb0eba8feb424c6c6fc77bed8a574ae9ba4578bd53ace0bee2547d8"}
Jul 31 23:46:04.780916 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:04.780795    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jqggp"
Jul 31 23:46:04.781092 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:04.780796    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s9j95"
Jul 31 23:46:04.781092 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:04.780992    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jqggp" podUID="e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06"
Jul 31 23:46:04.781210 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:04.781086    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s9j95" podUID="64e3d3a1-0422-4faa-a938-a523a0a7b827"
Jul 31 23:46:04.781210 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:04.780804    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpv44"
Jul 31 23:46:04.781210 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:04.781180    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpv44" podUID="450206d3-1b10-45ae-b6e9-6575c0d0be94"
Jul 31 23:46:05.022382 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:05.022348    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6" event={"ID":"71cbd9c3-1692-4ded-a7fc-bf48635b0c29","Type":"ContainerStarted","Data":"599004df4b44ed1861bac468537ce57138abaefc6c9bce9af2660af79949ffd3"}
Jul 31 23:46:05.052081 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:05.052032    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6" podStartSLOduration=6.395870088 podStartE2EDuration="27.052015998s" totalImagesPullingTime="20.65614591s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:45:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:05.05025512 +0000 UTC m=+27.770054966" watchObservedRunningTime="2026-07-31 23:46:05.052015998 +0000 UTC m=+27.771815842"
Jul 31 23:46:05.052259 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:05.052238    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw4n9" podStartSLOduration=11.013148955 podStartE2EDuration="26.052225818s" totalImagesPullingTime="15.039076863s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:45:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:04.033542585 +0000 UTC m=+26.753342431" watchObservedRunningTime="2026-07-31 23:46:05.052225818 +0000 UTC m=+27.772025669"
Jul 31 23:46:05.759564 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:05.759533    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-92v2f"
Jul 31 23:46:05.760095 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:05.760074    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-92v2f"
Jul 31 23:46:06.029732 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:06.028483    2559 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 31 23:46:06.029732 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:06.029504    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:46:06.029732 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:06.029636    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:46:06.030439 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:06.030237    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-92v2f"
Jul 31 23:46:06.044489 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:06.044465    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:46:06.044726 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:06.044702    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:46:06.137491 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:06.137462    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:46:06.768963 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:06.768785    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jqggp"]
Jul 31 23:46:06.769097 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:06.769073    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jqggp"
Jul 31 23:46:06.769206 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:06.769166    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jqggp" podUID="e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06"
Jul 31 23:46:06.772387 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:06.772343    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bpv44"]
Jul 31 23:46:06.772493 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:06.772472    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpv44"
Jul 31 23:46:06.772589 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:06.772568    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpv44" podUID="450206d3-1b10-45ae-b6e9-6575c0d0be94"
Jul 31 23:46:06.773038 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:06.773021    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-s9j95"]
Jul 31 23:46:06.773107 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:06.773094    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s9j95"
Jul 31 23:46:06.773200 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:06.773180    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s9j95" podUID="64e3d3a1-0422-4faa-a938-a523a0a7b827"
Jul 31 23:46:07.031718 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:07.031686    2559 generic.go:356] "Generic (PLEG): container finished" podID="ee8bfdcd-3661-46f6-93a0-93bb3facacd4" containerID="db176191306507604e3600fceee15c9b30ed7118ef20caf37ef2a4406aa24f86" exitCode=0
Jul 31 23:46:07.032409 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:07.031771    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r9jcj" event={"ID":"ee8bfdcd-3661-46f6-93a0-93bb3facacd4","Type":"ContainerDied","Data":"db176191306507604e3600fceee15c9b30ed7118ef20caf37ef2a4406aa24f86"}
Jul 31 23:46:08.035323 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:08.035296    2559 generic.go:356] "Generic (PLEG): container finished" podID="ee8bfdcd-3661-46f6-93a0-93bb3facacd4" containerID="342e9f298066ef12a9c53d84e7420b3e781c3dafebb9b677eaadfde1a3662bba" exitCode=0
Jul 31 23:46:08.035609 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:08.035383    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r9jcj" event={"ID":"ee8bfdcd-3661-46f6-93a0-93bb3facacd4","Type":"ContainerDied","Data":"342e9f298066ef12a9c53d84e7420b3e781c3dafebb9b677eaadfde1a3662bba"}
Jul 31 23:46:08.780907 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:08.780882    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpv44"
Jul 31 23:46:08.781039 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:08.780882    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jqggp"
Jul 31 23:46:08.781039 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:08.780985    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpv44" podUID="450206d3-1b10-45ae-b6e9-6575c0d0be94"
Jul 31 23:46:08.781129 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:08.781089    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jqggp" podUID="e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06"
Jul 31 23:46:08.781129 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:08.780886    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s9j95"
Jul 31 23:46:08.781196 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:08.781183    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s9j95" podUID="64e3d3a1-0422-4faa-a938-a523a0a7b827"
Jul 31 23:46:09.039871 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:09.039838    2559 generic.go:356] "Generic (PLEG): container finished" podID="ee8bfdcd-3661-46f6-93a0-93bb3facacd4" containerID="80cf3be302d4a719daff237ebff935d316c4cef7aaa5c9ea6063bb43006f1fb8" exitCode=0
Jul 31 23:46:09.040232 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:09.039903    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r9jcj" event={"ID":"ee8bfdcd-3661-46f6-93a0-93bb3facacd4","Type":"ContainerDied","Data":"80cf3be302d4a719daff237ebff935d316c4cef7aaa5c9ea6063bb43006f1fb8"}
Jul 31 23:46:10.560911 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:10.560832    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06-original-pull-secret\") pod \"global-pull-secret-syncer-jqggp\" (UID: \"e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06\") " pod="kube-system/global-pull-secret-syncer-jqggp"
Jul 31 23:46:10.560911 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:10.560878    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pgnt\" (UniqueName: \"kubernetes.io/projected/64e3d3a1-0422-4faa-a938-a523a0a7b827-kube-api-access-7pgnt\") pod \"network-check-target-s9j95\" (UID: \"64e3d3a1-0422-4faa-a938-a523a0a7b827\") " pod="openshift-network-diagnostics/network-check-target-s9j95"
Jul 31 23:46:10.561332 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:10.560915    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/450206d3-1b10-45ae-b6e9-6575c0d0be94-metrics-certs\") pod \"network-metrics-daemon-bpv44\" (UID: \"450206d3-1b10-45ae-b6e9-6575c0d0be94\") " pod="openshift-multus/network-metrics-daemon-bpv44"
Jul 31 23:46:10.561332 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:10.560983    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:10.561332 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:10.561014    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:10.561332 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:10.561050    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 23:46:10.561332 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:10.561069    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 23:46:10.561332 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:10.561078    2559 projected.go:196] Error preparing data for projected volume kube-api-access-7pgnt for pod openshift-network-diagnostics/network-check-target-s9j95: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:10.561332 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:10.561057    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06-original-pull-secret podName:e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:42.561038323 +0000 UTC m=+65.280838168 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06-original-pull-secret") pod "global-pull-secret-syncer-jqggp" (UID: "e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06") : object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:10.561332 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:10.561121    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/450206d3-1b10-45ae-b6e9-6575c0d0be94-metrics-certs podName:450206d3-1b10-45ae-b6e9-6575c0d0be94 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:42.561105735 +0000 UTC m=+65.280905569 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/450206d3-1b10-45ae-b6e9-6575c0d0be94-metrics-certs") pod "network-metrics-daemon-bpv44" (UID: "450206d3-1b10-45ae-b6e9-6575c0d0be94") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:10.561332 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:10.561143    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64e3d3a1-0422-4faa-a938-a523a0a7b827-kube-api-access-7pgnt podName:64e3d3a1-0422-4faa-a938-a523a0a7b827 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:42.561133514 +0000 UTC m=+65.280933339 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-7pgnt" (UniqueName: "kubernetes.io/projected/64e3d3a1-0422-4faa-a938-a523a0a7b827-kube-api-access-7pgnt") pod "network-check-target-s9j95" (UID: "64e3d3a1-0422-4faa-a938-a523a0a7b827") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:10.780682 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:10.780648    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpv44"
Jul 31 23:46:10.780888 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:10.780648    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jqggp"
Jul 31 23:46:10.780888 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:10.780656    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s9j95"
Jul 31 23:46:10.780888 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:10.780813    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpv44" podUID="450206d3-1b10-45ae-b6e9-6575c0d0be94"
Jul 31 23:46:10.780888 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:10.780879    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s9j95" podUID="64e3d3a1-0422-4faa-a938-a523a0a7b827"
Jul 31 23:46:10.781191 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:10.780984    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jqggp" podUID="e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06"
Jul 31 23:46:12.553954 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.553927    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-148.ec2.internal" event="NodeReady"
Jul 31 23:46:12.554341 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.554099    2559 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 31 23:46:12.591308 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.591170    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5b46bd6c8c-m7dfb"]
Jul 31 23:46:12.629593 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.626598    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b46bd6c8c-m7dfb"]
Jul 31 23:46:12.629593 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.626670    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9zcz2"]
Jul 31 23:46:12.629593 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.628042    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b46bd6c8c-m7dfb"
Jul 31 23:46:12.631843 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.631817    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 31 23:46:12.631843 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.631831    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 31 23:46:12.632013 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.631995    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mwlw9\""
Jul 31 23:46:12.632088 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.632063    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 31 23:46:12.637498 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.637475    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 31 23:46:12.642615 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.642595    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-fxfvr"]
Jul 31 23:46:12.642728 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.642712    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9zcz2"
Jul 31 23:46:12.646862 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.646839    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 31 23:46:12.646967 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.646869    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 31 23:46:12.647224 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.647208    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 31 23:46:12.647352 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.647335    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qx4sg\""
Jul 31 23:46:12.660605 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.660585    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9zcz2"]
Jul 31 23:46:12.660693 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.660621    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fxfvr"]
Jul 31 23:46:12.660693 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.660675    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fxfvr"
Jul 31 23:46:12.662982 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.662962    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 31 23:46:12.663271 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.663256    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 31 23:46:12.663360 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.663270    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tss5j\""
Jul 31 23:46:12.760042 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.759999    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92v2f_539b6c48-1833-4461-8ab1-580974198fb3/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:12.778089 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.778056    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7fb6400-a564-4c03-8381-b76eb7aac58b-registry-tls\") pod \"image-registry-5b46bd6c8c-m7dfb\" (UID: \"c7fb6400-a564-4c03-8381-b76eb7aac58b\") " pod="openshift-image-registry/image-registry-5b46bd6c8c-m7dfb"
Jul 31 23:46:12.778089 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.778091    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c7fb6400-a564-4c03-8381-b76eb7aac58b-registry-certificates\") pod \"image-registry-5b46bd6c8c-m7dfb\" (UID: \"c7fb6400-a564-4c03-8381-b76eb7aac58b\") " pod="openshift-image-registry/image-registry-5b46bd6c8c-m7dfb"
Jul 31 23:46:12.778276 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.778108    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c7fb6400-a564-4c03-8381-b76eb7aac58b-trusted-ca\") pod \"image-registry-5b46bd6c8c-m7dfb\" (UID: \"c7fb6400-a564-4c03-8381-b76eb7aac58b\") " pod="openshift-image-registry/image-registry-5b46bd6c8c-m7dfb"
Jul 31 23:46:12.778276 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.778129    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hc9r\" (UniqueName: \"kubernetes.io/projected/c7fb6400-a564-4c03-8381-b76eb7aac58b-kube-api-access-7hc9r\") pod \"image-registry-5b46bd6c8c-m7dfb\" (UID: \"c7fb6400-a564-4c03-8381-b76eb7aac58b\") " pod="openshift-image-registry/image-registry-5b46bd6c8c-m7dfb"
Jul 31 23:46:12.778276 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.778171    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ae80080-0f96-4a21-b13c-87738f73a578-metrics-tls\") pod \"dns-default-fxfvr\" (UID: \"0ae80080-0f96-4a21-b13c-87738f73a578\") " pod="openshift-dns/dns-default-fxfvr"
Jul 31 23:46:12.778276 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.778205    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tfc4\" (UniqueName: \"kubernetes.io/projected/1d81c21b-5141-40fb-bb77-b676e9461c82-kube-api-access-4tfc4\") pod \"ingress-canary-9zcz2\" (UID: \"1d81c21b-5141-40fb-bb77-b676e9461c82\") " pod="openshift-ingress-canary/ingress-canary-9zcz2"
Jul 31 23:46:12.778276 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.778231    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c7fb6400-a564-4c03-8381-b76eb7aac58b-image-registry-private-configuration\") pod \"image-registry-5b46bd6c8c-m7dfb\" (UID: \"c7fb6400-a564-4c03-8381-b76eb7aac58b\") " pod="openshift-image-registry/image-registry-5b46bd6c8c-m7dfb"
Jul 31 23:46:12.778276 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.778270    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c7fb6400-a564-4c03-8381-b76eb7aac58b-ca-trust-extracted\") pod \"image-registry-5b46bd6c8c-m7dfb\" (UID: \"c7fb6400-a564-4c03-8381-b76eb7aac58b\") " pod="openshift-image-registry/image-registry-5b46bd6c8c-m7dfb"
Jul 31 23:46:12.778471 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.778291    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0ae80080-0f96-4a21-b13c-87738f73a578-config-volume\") pod \"dns-default-fxfvr\" (UID: \"0ae80080-0f96-4a21-b13c-87738f73a578\") " pod="openshift-dns/dns-default-fxfvr"
Jul 31 23:46:12.778471 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.778332    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d81c21b-5141-40fb-bb77-b676e9461c82-cert\") pod \"ingress-canary-9zcz2\" (UID: \"1d81c21b-5141-40fb-bb77-b676e9461c82\") " pod="openshift-ingress-canary/ingress-canary-9zcz2"
Jul 31 23:46:12.778471 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.778388    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c7fb6400-a564-4c03-8381-b76eb7aac58b-installation-pull-secrets\") pod \"image-registry-5b46bd6c8c-m7dfb\" (UID: \"c7fb6400-a564-4c03-8381-b76eb7aac58b\") " pod="openshift-image-registry/image-registry-5b46bd6c8c-m7dfb"
Jul 31 23:46:12.778592 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.778470    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c7fb6400-a564-4c03-8381-b76eb7aac58b-bound-sa-token\") pod \"image-registry-5b46bd6c8c-m7dfb\" (UID: \"c7fb6400-a564-4c03-8381-b76eb7aac58b\") " pod="openshift-image-registry/image-registry-5b46bd6c8c-m7dfb"
Jul 31 23:46:12.778592 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.778509    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4t4b\" (UniqueName: \"kubernetes.io/projected/0ae80080-0f96-4a21-b13c-87738f73a578-kube-api-access-c4t4b\") pod \"dns-default-fxfvr\" (UID: \"0ae80080-0f96-4a21-b13c-87738f73a578\") " pod="openshift-dns/dns-default-fxfvr"
Jul 31 23:46:12.778592 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.778546    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ae80080-0f96-4a21-b13c-87738f73a578-tmp-dir\") pod \"dns-default-fxfvr\" (UID: \"0ae80080-0f96-4a21-b13c-87738f73a578\") " pod="openshift-dns/dns-default-fxfvr"
Jul 31 23:46:12.780069 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.780048    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s9j95"
Jul 31 23:46:12.780153 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.780056    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jqggp"
Jul 31 23:46:12.780192 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.780164    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpv44"
Jul 31 23:46:12.783019 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.782999    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 31 23:46:12.783124 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.783032    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 31 23:46:12.783124 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.783046    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 31 23:46:12.783237 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.783160    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-45dqd\""
Jul 31 23:46:12.783285 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.783253    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 31 23:46:12.783332 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.783284    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-c2rbb\""
Jul 31 23:46:12.836337 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.836264    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92v2f_539b6c48-1833-4461-8ab1-580974198fb3/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:12.879316 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.879281    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c7fb6400-a564-4c03-8381-b76eb7aac58b-ca-trust-extracted\") pod \"image-registry-5b46bd6c8c-m7dfb\" (UID: \"c7fb6400-a564-4c03-8381-b76eb7aac58b\") " pod="openshift-image-registry/image-registry-5b46bd6c8c-m7dfb"
Jul 31 23:46:12.879468 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.879325    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0ae80080-0f96-4a21-b13c-87738f73a578-config-volume\") pod \"dns-default-fxfvr\" (UID: \"0ae80080-0f96-4a21-b13c-87738f73a578\") " pod="openshift-dns/dns-default-fxfvr"
Jul 31 23:46:12.879468 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.879372    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d81c21b-5141-40fb-bb77-b676e9461c82-cert\") pod \"ingress-canary-9zcz2\" (UID: \"1d81c21b-5141-40fb-bb77-b676e9461c82\") " pod="openshift-ingress-canary/ingress-canary-9zcz2"
Jul 31 23:46:12.879468 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.879399    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c7fb6400-a564-4c03-8381-b76eb7aac58b-installation-pull-secrets\") pod \"image-registry-5b46bd6c8c-m7dfb\" (UID: \"c7fb6400-a564-4c03-8381-b76eb7aac58b\") " pod="openshift-image-registry/image-registry-5b46bd6c8c-m7dfb"
Jul 31 23:46:12.879468 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.879446    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c7fb6400-a564-4c03-8381-b76eb7aac58b-bound-sa-token\") pod \"image-registry-5b46bd6c8c-m7dfb\" (UID: \"c7fb6400-a564-4c03-8381-b76eb7aac58b\") " pod="openshift-image-registry/image-registry-5b46bd6c8c-m7dfb"
Jul 31 23:46:12.879659 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.879474    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4t4b\" (UniqueName: \"kubernetes.io/projected/0ae80080-0f96-4a21-b13c-87738f73a578-kube-api-access-c4t4b\") pod \"dns-default-fxfvr\" (UID: \"0ae80080-0f96-4a21-b13c-87738f73a578\") " pod="openshift-dns/dns-default-fxfvr"
Jul 31 23:46:12.879659 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.879501    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ae80080-0f96-4a21-b13c-87738f73a578-tmp-dir\") pod \"dns-default-fxfvr\" (UID: \"0ae80080-0f96-4a21-b13c-87738f73a578\") " pod="openshift-dns/dns-default-fxfvr"
Jul 31 23:46:12.879659 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.879530    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7fb6400-a564-4c03-8381-b76eb7aac58b-registry-tls\") pod \"image-registry-5b46bd6c8c-m7dfb\" (UID: \"c7fb6400-a564-4c03-8381-b76eb7aac58b\") " pod="openshift-image-registry/image-registry-5b46bd6c8c-m7dfb"
Jul 31 23:46:12.879659 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.879558    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c7fb6400-a564-4c03-8381-b76eb7aac58b-registry-certificates\") pod \"image-registry-5b46bd6c8c-m7dfb\" (UID: \"c7fb6400-a564-4c03-8381-b76eb7aac58b\") " pod="openshift-image-registry/image-registry-5b46bd6c8c-m7dfb"
Jul 31 23:46:12.879659 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.879585    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c7fb6400-a564-4c03-8381-b76eb7aac58b-trusted-ca\") pod \"image-registry-5b46bd6c8c-m7dfb\" (UID: \"c7fb6400-a564-4c03-8381-b76eb7aac58b\") " pod="openshift-image-registry/image-registry-5b46bd6c8c-m7dfb"
Jul 31 23:46:12.879659 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.879610    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hc9r\" (UniqueName: \"kubernetes.io/projected/c7fb6400-a564-4c03-8381-b76eb7aac58b-kube-api-access-7hc9r\") pod \"image-registry-5b46bd6c8c-m7dfb\" (UID: \"c7fb6400-a564-4c03-8381-b76eb7aac58b\") " pod="openshift-image-registry/image-registry-5b46bd6c8c-m7dfb"
Jul 31 23:46:12.879659 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.879635    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ae80080-0f96-4a21-b13c-87738f73a578-metrics-tls\") pod \"dns-default-fxfvr\" (UID: \"0ae80080-0f96-4a21-b13c-87738f73a578\") " pod="openshift-dns/dns-default-fxfvr"
Jul 31 23:46:12.879996 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.879662    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tfc4\" (UniqueName: \"kubernetes.io/projected/1d81c21b-5141-40fb-bb77-b676e9461c82-kube-api-access-4tfc4\") pod \"ingress-canary-9zcz2\" (UID: \"1d81c21b-5141-40fb-bb77-b676e9461c82\") " pod="openshift-ingress-canary/ingress-canary-9zcz2"
Jul 31 23:46:12.879996 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.879686    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c7fb6400-a564-4c03-8381-b76eb7aac58b-image-registry-private-configuration\") pod \"image-registry-5b46bd6c8c-m7dfb\" (UID: \"c7fb6400-a564-4c03-8381-b76eb7aac58b\") " pod="openshift-image-registry/image-registry-5b46bd6c8c-m7dfb"
Jul 31 23:46:12.879996 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.879790    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c7fb6400-a564-4c03-8381-b76eb7aac58b-ca-trust-extracted\") pod \"image-registry-5b46bd6c8c-m7dfb\" (UID: \"c7fb6400-a564-4c03-8381-b76eb7aac58b\") " pod="openshift-image-registry/image-registry-5b46bd6c8c-m7dfb"
Jul 31 23:46:12.879996 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:12.879906    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 31 23:46:12.879996 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:12.879923    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b46bd6c8c-m7dfb: secret "image-registry-tls" not found
Jul 31 23:46:12.879996 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.879975    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0ae80080-0f96-4a21-b13c-87738f73a578-config-volume\") pod \"dns-default-fxfvr\" (UID: \"0ae80080-0f96-4a21-b13c-87738f73a578\") " pod="openshift-dns/dns-default-fxfvr"
Jul 31 23:46:12.879996 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:12.879987    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7fb6400-a564-4c03-8381-b76eb7aac58b-registry-tls podName:c7fb6400-a564-4c03-8381-b76eb7aac58b nodeName:}" failed. No retries permitted until 2026-07-31 23:46:13.379968896 +0000 UTC m=+36.099768749 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c7fb6400-a564-4c03-8381-b76eb7aac58b-registry-tls") pod "image-registry-5b46bd6c8c-m7dfb" (UID: "c7fb6400-a564-4c03-8381-b76eb7aac58b") : secret "image-registry-tls" not found
Jul 31 23:46:12.880305 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:12.880208    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 23:46:12.880305 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:12.880259    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ae80080-0f96-4a21-b13c-87738f73a578-metrics-tls podName:0ae80080-0f96-4a21-b13c-87738f73a578 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:13.380242998 +0000 UTC m=+36.100042830 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0ae80080-0f96-4a21-b13c-87738f73a578-metrics-tls") pod "dns-default-fxfvr" (UID: "0ae80080-0f96-4a21-b13c-87738f73a578") : secret "dns-default-metrics-tls" not found
Jul 31 23:46:12.880735 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.880631    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c7fb6400-a564-4c03-8381-b76eb7aac58b-registry-certificates\") pod \"image-registry-5b46bd6c8c-m7dfb\" (UID: \"c7fb6400-a564-4c03-8381-b76eb7aac58b\") " pod="openshift-image-registry/image-registry-5b46bd6c8c-m7dfb"
Jul 31 23:46:12.880914 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.880887    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ae80080-0f96-4a21-b13c-87738f73a578-tmp-dir\") pod \"dns-default-fxfvr\" (UID: \"0ae80080-0f96-4a21-b13c-87738f73a578\") " pod="openshift-dns/dns-default-fxfvr"
Jul 31 23:46:12.881282 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.881246    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c7fb6400-a564-4c03-8381-b76eb7aac58b-trusted-ca\") pod \"image-registry-5b46bd6c8c-m7dfb\" (UID: \"c7fb6400-a564-4c03-8381-b76eb7aac58b\") " pod="openshift-image-registry/image-registry-5b46bd6c8c-m7dfb"
Jul 31 23:46:12.885038 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.885021    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c7fb6400-a564-4c03-8381-b76eb7aac58b-installation-pull-secrets\") pod \"image-registry-5b46bd6c8c-m7dfb\" (UID: \"c7fb6400-a564-4c03-8381-b76eb7aac58b\") " pod="openshift-image-registry/image-registry-5b46bd6c8c-m7dfb"
Jul 31 23:46:12.885114 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.885027    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c7fb6400-a564-4c03-8381-b76eb7aac58b-image-registry-private-configuration\") pod \"image-registry-5b46bd6c8c-m7dfb\" (UID: \"c7fb6400-a564-4c03-8381-b76eb7aac58b\") " pod="openshift-image-registry/image-registry-5b46bd6c8c-m7dfb"
Jul 31 23:46:12.885206 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.885137    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d81c21b-5141-40fb-bb77-b676e9461c82-cert\") pod \"ingress-canary-9zcz2\" (UID: \"1d81c21b-5141-40fb-bb77-b676e9461c82\") " pod="openshift-ingress-canary/ingress-canary-9zcz2"
Jul 31 23:46:12.889619 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.889595    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4t4b\" (UniqueName: \"kubernetes.io/projected/0ae80080-0f96-4a21-b13c-87738f73a578-kube-api-access-c4t4b\") pod \"dns-default-fxfvr\" (UID: \"0ae80080-0f96-4a21-b13c-87738f73a578\") " pod="openshift-dns/dns-default-fxfvr"
Jul 31 23:46:12.889810 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.889793    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c7fb6400-a564-4c03-8381-b76eb7aac58b-bound-sa-token\") pod \"image-registry-5b46bd6c8c-m7dfb\" (UID: \"c7fb6400-a564-4c03-8381-b76eb7aac58b\") " pod="openshift-image-registry/image-registry-5b46bd6c8c-m7dfb"
Jul 31 23:46:12.889894 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.889856    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tfc4\" (UniqueName: \"kubernetes.io/projected/1d81c21b-5141-40fb-bb77-b676e9461c82-kube-api-access-4tfc4\") pod \"ingress-canary-9zcz2\" (UID: \"1d81c21b-5141-40fb-bb77-b676e9461c82\") " pod="openshift-ingress-canary/ingress-canary-9zcz2"
Jul 31 23:46:12.889995 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.889933    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hc9r\" (UniqueName: \"kubernetes.io/projected/c7fb6400-a564-4c03-8381-b76eb7aac58b-kube-api-access-7hc9r\") pod \"image-registry-5b46bd6c8c-m7dfb\" (UID: \"c7fb6400-a564-4c03-8381-b76eb7aac58b\") " pod="openshift-image-registry/image-registry-5b46bd6c8c-m7dfb"
Jul 31 23:46:12.952360 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:12.952334    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9zcz2"
Jul 31 23:46:13.384686 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:13.384481    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7fb6400-a564-4c03-8381-b76eb7aac58b-registry-tls\") pod \"image-registry-5b46bd6c8c-m7dfb\" (UID: \"c7fb6400-a564-4c03-8381-b76eb7aac58b\") " pod="openshift-image-registry/image-registry-5b46bd6c8c-m7dfb"
Jul 31 23:46:13.384897 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:13.384716    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ae80080-0f96-4a21-b13c-87738f73a578-metrics-tls\") pod \"dns-default-fxfvr\" (UID: \"0ae80080-0f96-4a21-b13c-87738f73a578\") " pod="openshift-dns/dns-default-fxfvr"
Jul 31 23:46:13.384897 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:13.384640    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 31 23:46:13.384897 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:13.384785    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b46bd6c8c-m7dfb: secret "image-registry-tls" not found
Jul 31 23:46:13.384897 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:13.384843    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 23:46:13.384897 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:13.384855    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7fb6400-a564-4c03-8381-b76eb7aac58b-registry-tls podName:c7fb6400-a564-4c03-8381-b76eb7aac58b nodeName:}" failed. No retries permitted until 2026-07-31 23:46:14.384837188 +0000 UTC m=+37.104637033 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c7fb6400-a564-4c03-8381-b76eb7aac58b-registry-tls") pod "image-registry-5b46bd6c8c-m7dfb" (UID: "c7fb6400-a564-4c03-8381-b76eb7aac58b") : secret "image-registry-tls" not found
Jul 31 23:46:13.384897 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:13.384901    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ae80080-0f96-4a21-b13c-87738f73a578-metrics-tls podName:0ae80080-0f96-4a21-b13c-87738f73a578 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:14.384887681 +0000 UTC m=+37.104687504 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0ae80080-0f96-4a21-b13c-87738f73a578-metrics-tls") pod "dns-default-fxfvr" (UID: "0ae80080-0f96-4a21-b13c-87738f73a578") : secret "dns-default-metrics-tls" not found
Jul 31 23:46:13.920588 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:13.920552    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9zcz2"]
Jul 31 23:46:13.990482 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:46:13.990452    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d81c21b_5141_40fb_bb77_b676e9461c82.slice/crio-3988b5f0966b7c8b6758359dd2d6f9b1712bdd1f7ad14dd5b69b172907ae9fd7 WatchSource:0}: Error finding container 3988b5f0966b7c8b6758359dd2d6f9b1712bdd1f7ad14dd5b69b172907ae9fd7: Status 404 returned error can't find the container with id 3988b5f0966b7c8b6758359dd2d6f9b1712bdd1f7ad14dd5b69b172907ae9fd7
Jul 31 23:46:14.052710 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:14.052679    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9zcz2" event={"ID":"1d81c21b-5141-40fb-bb77-b676e9461c82","Type":"ContainerStarted","Data":"3988b5f0966b7c8b6758359dd2d6f9b1712bdd1f7ad14dd5b69b172907ae9fd7"}
Jul 31 23:46:14.393436 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:14.393363    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7fb6400-a564-4c03-8381-b76eb7aac58b-registry-tls\") pod \"image-registry-5b46bd6c8c-m7dfb\" (UID: \"c7fb6400-a564-4c03-8381-b76eb7aac58b\") " pod="openshift-image-registry/image-registry-5b46bd6c8c-m7dfb"
Jul 31 23:46:14.393436 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:14.393398    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ae80080-0f96-4a21-b13c-87738f73a578-metrics-tls\") pod \"dns-default-fxfvr\" (UID: \"0ae80080-0f96-4a21-b13c-87738f73a578\") " pod="openshift-dns/dns-default-fxfvr"
Jul 31 23:46:14.393590 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:14.393511    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 23:46:14.393590 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:14.393519    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 31 23:46:14.393590 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:14.393536    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b46bd6c8c-m7dfb: secret "image-registry-tls" not found
Jul 31 23:46:14.393590 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:14.393560    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ae80080-0f96-4a21-b13c-87738f73a578-metrics-tls podName:0ae80080-0f96-4a21-b13c-87738f73a578 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:16.393547395 +0000 UTC m=+39.113347219 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0ae80080-0f96-4a21-b13c-87738f73a578-metrics-tls") pod "dns-default-fxfvr" (UID: "0ae80080-0f96-4a21-b13c-87738f73a578") : secret "dns-default-metrics-tls" not found
Jul 31 23:46:14.393590 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:14.393584    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7fb6400-a564-4c03-8381-b76eb7aac58b-registry-tls podName:c7fb6400-a564-4c03-8381-b76eb7aac58b nodeName:}" failed. No retries permitted until 2026-07-31 23:46:16.393568062 +0000 UTC m=+39.113367885 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c7fb6400-a564-4c03-8381-b76eb7aac58b-registry-tls") pod "image-registry-5b46bd6c8c-m7dfb" (UID: "c7fb6400-a564-4c03-8381-b76eb7aac58b") : secret "image-registry-tls" not found
Jul 31 23:46:14.828322 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:14.828285    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92v2f_539b6c48-1833-4461-8ab1-580974198fb3/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:14.879631 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:14.879596    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92v2f_539b6c48-1833-4461-8ab1-580974198fb3/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:15.058552 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:15.058516    2559 generic.go:356] "Generic (PLEG): container finished" podID="ee8bfdcd-3661-46f6-93a0-93bb3facacd4" containerID="249c0ab7d3da0b8d5cd23bb4ff5b20ce708236a3e8e76290fb5e80f60224b8d2" exitCode=0
Jul 31 23:46:15.059049 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:15.058567    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r9jcj" event={"ID":"ee8bfdcd-3661-46f6-93a0-93bb3facacd4","Type":"ContainerDied","Data":"249c0ab7d3da0b8d5cd23bb4ff5b20ce708236a3e8e76290fb5e80f60224b8d2"}
Jul 31 23:46:16.064598 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:16.064566    2559 generic.go:356] "Generic (PLEG): container finished" podID="ee8bfdcd-3661-46f6-93a0-93bb3facacd4" containerID="78155d2580439c41ff762cc00e10c6aef5bd3264e18c519b3e1fdc7c49158ec2" exitCode=0
Jul 31 23:46:16.065181 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:16.064620    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r9jcj" event={"ID":"ee8bfdcd-3661-46f6-93a0-93bb3facacd4","Type":"ContainerDied","Data":"78155d2580439c41ff762cc00e10c6aef5bd3264e18c519b3e1fdc7c49158ec2"}
Jul 31 23:46:16.408654 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:16.408620    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7fb6400-a564-4c03-8381-b76eb7aac58b-registry-tls\") pod \"image-registry-5b46bd6c8c-m7dfb\" (UID: \"c7fb6400-a564-4c03-8381-b76eb7aac58b\") " pod="openshift-image-registry/image-registry-5b46bd6c8c-m7dfb"
Jul 31 23:46:16.408654 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:16.408658    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ae80080-0f96-4a21-b13c-87738f73a578-metrics-tls\") pod \"dns-default-fxfvr\" (UID: \"0ae80080-0f96-4a21-b13c-87738f73a578\") " pod="openshift-dns/dns-default-fxfvr"
Jul 31 23:46:16.408891 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:16.408791    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 23:46:16.408891 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:16.408800    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 31 23:46:16.408891 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:16.408821    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b46bd6c8c-m7dfb: secret "image-registry-tls" not found
Jul 31 23:46:16.408891 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:16.408843    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ae80080-0f96-4a21-b13c-87738f73a578-metrics-tls podName:0ae80080-0f96-4a21-b13c-87738f73a578 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:20.40883044 +0000 UTC m=+43.128630265 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0ae80080-0f96-4a21-b13c-87738f73a578-metrics-tls") pod "dns-default-fxfvr" (UID: "0ae80080-0f96-4a21-b13c-87738f73a578") : secret "dns-default-metrics-tls" not found
Jul 31 23:46:16.408891 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:16.408870    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7fb6400-a564-4c03-8381-b76eb7aac58b-registry-tls podName:c7fb6400-a564-4c03-8381-b76eb7aac58b nodeName:}" failed. No retries permitted until 2026-07-31 23:46:20.408854966 +0000 UTC m=+43.128654792 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c7fb6400-a564-4c03-8381-b76eb7aac58b-registry-tls") pod "image-registry-5b46bd6c8c-m7dfb" (UID: "c7fb6400-a564-4c03-8381-b76eb7aac58b") : secret "image-registry-tls" not found
Jul 31 23:46:17.070496 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:17.070296    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r9jcj" event={"ID":"ee8bfdcd-3661-46f6-93a0-93bb3facacd4","Type":"ContainerStarted","Data":"b3cf65be7e53ebba4c1459ae9031b6c1287ceddda2df5f539ee565955841fb25"}
Jul 31 23:46:17.071669 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:17.071646    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9zcz2" event={"ID":"1d81c21b-5141-40fb-bb77-b676e9461c82","Type":"ContainerStarted","Data":"bcb3df17557e7059a5565e5eccf9a31104c2e16ccb69e8ece89529428867de19"}
Jul 31 23:46:17.095498 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:17.095452    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-r9jcj" podStartSLOduration=6.424019923 podStartE2EDuration="39.095441034s" totalImagesPullingTime="32.671421111s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:45:38 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:17.094125596 +0000 UTC m=+39.813925463" watchObservedRunningTime="2026-07-31 23:46:17.095441034 +0000 UTC m=+39.815240880"
Jul 31 23:46:17.110691 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:17.110650    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9zcz2" podStartSLOduration=2.507153895 podStartE2EDuration="5.110638591s" totalImagesPullingTime="2.603484696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:17.110285276 +0000 UTC m=+39.830085123" watchObservedRunningTime="2026-07-31 23:46:17.110638591 +0000 UTC m=+39.830438437"
Jul 31 23:46:17.226150 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:17.226118    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92v2f_539b6c48-1833-4461-8ab1-580974198fb3/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:17.326301 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:17.326231    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92v2f_539b6c48-1833-4461-8ab1-580974198fb3/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:17.897048 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:17.897019    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92v2f_539b6c48-1833-4461-8ab1-580974198fb3/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:17.950728 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:17.950675    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92v2f_539b6c48-1833-4461-8ab1-580974198fb3/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:18.453864 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.453831    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vknnx"]
Jul 31 23:46:18.481580 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.481553    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vknnx"]
Jul 31 23:46:18.481708 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.481644    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vknnx"
Jul 31 23:46:18.484375 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.484343    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 31 23:46:18.486912 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.486890    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 31 23:46:18.486912 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.486906    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 31 23:46:18.487086 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.486919    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-j8lfq\""
Jul 31 23:46:18.487086 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.487031    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 31 23:46:18.561995 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.561967    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-slttz"]
Jul 31 23:46:18.577312 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.577287    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5579c88784-ntwhl"]
Jul 31 23:46:18.577449 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.577416    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-slttz"
Jul 31 23:46:18.579927 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.579906    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-wnt4l\""
Jul 31 23:46:18.580037 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.579927    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 31 23:46:18.580037 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.579971    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 31 23:46:18.580037 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.580028    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 31 23:46:18.580275 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.580261    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 31 23:46:18.603243 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.603220    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-spblg"]
Jul 31 23:46:18.603350 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.603336    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5579c88784-ntwhl"
Jul 31 23:46:18.605880 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.605863    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 31 23:46:18.606005 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.605863    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 31 23:46:18.606005 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.605972    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 31 23:46:18.606005 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.605993    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 31 23:46:18.606189 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.606175    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 31 23:46:18.606271 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.606255    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-gqffv\""
Jul 31 23:46:18.606609 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.606595    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 31 23:46:18.626947 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.626921    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/13eeae3e-dd1f-4829-8fd8-9a867df2630c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-vknnx\" (UID: \"13eeae3e-dd1f-4829-8fd8-9a867df2630c\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vknnx"
Jul 31 23:46:18.626947 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.626946    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdfcn\" (UniqueName: \"kubernetes.io/projected/13eeae3e-dd1f-4829-8fd8-9a867df2630c-kube-api-access-tdfcn\") pod \"cluster-monitoring-operator-6f55c4f58d-vknnx\" (UID: \"13eeae3e-dd1f-4829-8fd8-9a867df2630c\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vknnx"
Jul 31 23:46:18.627093 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.626965    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/13eeae3e-dd1f-4829-8fd8-9a867df2630c-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-vknnx\" (UID: \"13eeae3e-dd1f-4829-8fd8-9a867df2630c\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vknnx"
Jul 31 23:46:18.627168 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.627155    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-wvqkv"]
Jul 31 23:46:18.627283 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.627267    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-5ccd6b445f-spblg"
Jul 31 23:46:18.629445 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.629429    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 31 23:46:18.629640 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.629623    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-9qpp2\""
Jul 31 23:46:18.629712 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.629652    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 31 23:46:18.629776 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.629653    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 31 23:46:18.629776 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.629630    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 31 23:46:18.637967 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.637948    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 31 23:46:18.644161 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.644135    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-4zdc5"]
Jul 31 23:46:18.644255 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.644240    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-wvqkv"
Jul 31 23:46:18.646979 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.646962    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 31 23:46:18.647099 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.647085    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 31 23:46:18.647430 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.647406    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 31 23:46:18.647532 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.647428    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 31 23:46:18.647532 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.647489    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-z2t6l\""
Jul 31 23:46:18.666308 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.666284    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-slttz"]
Jul 31 23:46:18.666431 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.666322    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-spblg"]
Jul 31 23:46:18.666431 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.666414    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-4zdc5"
Jul 31 23:46:18.666685 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.666669    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-wvqkv"]
Jul 31 23:46:18.666773 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.666698    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5579c88784-ntwhl"]
Jul 31 23:46:18.666773 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.666724    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-4zdc5"]
Jul 31 23:46:18.667382 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.667365    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-2lwvf"]
Jul 31 23:46:18.670140 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.670121    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 31 23:46:18.670245 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.670148    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 31 23:46:18.670245 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.670223    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-hzp6k\""
Jul 31 23:46:18.690419 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.690397    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-nspll"]
Jul 31 23:46:18.690535 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.690516    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-2lwvf"
Jul 31 23:46:18.692927 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.692912    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-9zjdg\""
Jul 31 23:46:18.714515 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.714452    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-77696685b-xjqzb"]
Jul 31 23:46:18.714621 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.714565    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-nspll"
Jul 31 23:46:18.718335 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.718314    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-qg5b6\""
Jul 31 23:46:18.718719 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.718707    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 31 23:46:18.718910 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.718891    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 31 23:46:18.719005 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.718970    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 31 23:46:18.720781 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.720757    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92v2f_539b6c48-1833-4461-8ab1-580974198fb3/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:18.728337 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.728321    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/13eeae3e-dd1f-4829-8fd8-9a867df2630c-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-vknnx\" (UID: \"13eeae3e-dd1f-4829-8fd8-9a867df2630c\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vknnx"
Jul 31 23:46:18.728422 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.728381    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c33c7c85-f678-49ff-bcc0-5b1de21c563b-stats-auth\") pod \"router-default-5579c88784-ntwhl\" (UID: \"c33c7c85-f678-49ff-bcc0-5b1de21c563b\") " pod="openshift-ingress/router-default-5579c88784-ntwhl"
Jul 31 23:46:18.728527 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.728508    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q76wb\" (UniqueName: \"kubernetes.io/projected/e709e147-60bb-46dc-be7d-3d5269dbcd8a-kube-api-access-q76wb\") pod \"kube-storage-version-migrator-operator-7878f4b88c-slttz\" (UID: \"e709e147-60bb-46dc-be7d-3d5269dbcd8a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-slttz"
Jul 31 23:46:18.728593 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.728536    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdfcn\" (UniqueName: \"kubernetes.io/projected/13eeae3e-dd1f-4829-8fd8-9a867df2630c-kube-api-access-tdfcn\") pod \"cluster-monitoring-operator-6f55c4f58d-vknnx\" (UID: \"13eeae3e-dd1f-4829-8fd8-9a867df2630c\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vknnx"
Jul 31 23:46:18.728593 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.728554    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vczc\" (UniqueName: \"kubernetes.io/projected/c2e50946-aea4-4886-82ec-0bd5a31196cf-kube-api-access-9vczc\") pod \"console-operator-5ccd6b445f-spblg\" (UID: \"c2e50946-aea4-4886-82ec-0bd5a31196cf\") " pod="openshift-console-operator/console-operator-5ccd6b445f-spblg"
Jul 31 23:46:18.728593 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.728574    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/13eeae3e-dd1f-4829-8fd8-9a867df2630c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-vknnx\" (UID: \"13eeae3e-dd1f-4829-8fd8-9a867df2630c\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vknnx"
Jul 31 23:46:18.728761 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.728599    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c33c7c85-f678-49ff-bcc0-5b1de21c563b-default-certificate\") pod \"router-default-5579c88784-ntwhl\" (UID: \"c33c7c85-f678-49ff-bcc0-5b1de21c563b\") " pod="openshift-ingress/router-default-5579c88784-ntwhl"
Jul 31 23:46:18.728761 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.728618    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c2e50946-aea4-4886-82ec-0bd5a31196cf-serving-cert\") pod \"console-operator-5ccd6b445f-spblg\" (UID: \"c2e50946-aea4-4886-82ec-0bd5a31196cf\") " pod="openshift-console-operator/console-operator-5ccd6b445f-spblg"
Jul 31 23:46:18.728761 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:18.728643    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 31 23:46:18.728761 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.728643    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e709e147-60bb-46dc-be7d-3d5269dbcd8a-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-slttz\" (UID: \"e709e147-60bb-46dc-be7d-3d5269dbcd8a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-slttz"
Jul 31 23:46:18.728761 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:18.728703    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13eeae3e-dd1f-4829-8fd8-9a867df2630c-cluster-monitoring-operator-tls podName:13eeae3e-dd1f-4829-8fd8-9a867df2630c nodeName:}" failed. No retries permitted until 2026-07-31 23:46:19.228686699 +0000 UTC m=+41.948486527 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/13eeae3e-dd1f-4829-8fd8-9a867df2630c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-vknnx" (UID: "13eeae3e-dd1f-4829-8fd8-9a867df2630c") : secret "cluster-monitoring-operator-tls" not found
Jul 31 23:46:18.728761 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.728737    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c2e50946-aea4-4886-82ec-0bd5a31196cf-config\") pod \"console-operator-5ccd6b445f-spblg\" (UID: \"c2e50946-aea4-4886-82ec-0bd5a31196cf\") " pod="openshift-console-operator/console-operator-5ccd6b445f-spblg"
Jul 31 23:46:18.729115 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.728784    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c33c7c85-f678-49ff-bcc0-5b1de21c563b-service-ca-bundle\") pod \"router-default-5579c88784-ntwhl\" (UID: \"c33c7c85-f678-49ff-bcc0-5b1de21c563b\") " pod="openshift-ingress/router-default-5579c88784-ntwhl"
Jul 31 23:46:18.729115 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.728821    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e709e147-60bb-46dc-be7d-3d5269dbcd8a-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-slttz\" (UID: \"e709e147-60bb-46dc-be7d-3d5269dbcd8a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-slttz"
Jul 31 23:46:18.729115 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.728840    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c33c7c85-f678-49ff-bcc0-5b1de21c563b-metrics-certs\") pod \"router-default-5579c88784-ntwhl\" (UID: \"c33c7c85-f678-49ff-bcc0-5b1de21c563b\") " pod="openshift-ingress/router-default-5579c88784-ntwhl"
Jul 31 23:46:18.729115 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.728856    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c2e50946-aea4-4886-82ec-0bd5a31196cf-trusted-ca\") pod \"console-operator-5ccd6b445f-spblg\" (UID: \"c2e50946-aea4-4886-82ec-0bd5a31196cf\") " pod="openshift-console-operator/console-operator-5ccd6b445f-spblg"
Jul 31 23:46:18.729115 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.728884    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwwjr\" (UniqueName: \"kubernetes.io/projected/c33c7c85-f678-49ff-bcc0-5b1de21c563b-kube-api-access-pwwjr\") pod \"router-default-5579c88784-ntwhl\" (UID: \"c33c7c85-f678-49ff-bcc0-5b1de21c563b\") " pod="openshift-ingress/router-default-5579c88784-ntwhl"
Jul 31 23:46:18.729297 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.729176    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/13eeae3e-dd1f-4829-8fd8-9a867df2630c-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-vknnx\" (UID: \"13eeae3e-dd1f-4829-8fd8-9a867df2630c\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vknnx"
Jul 31 23:46:18.734537 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.734523    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-2lwvf"]
Jul 31 23:46:18.734598 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.734546    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-nspll"]
Jul 31 23:46:18.734598 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.734567    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-77696685b-xjqzb"]
Jul 31 23:46:18.734664 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.734625    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-77696685b-xjqzb"
Jul 31 23:46:18.737366 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.737334    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 31 23:46:18.737486 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.737472    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 31 23:46:18.741244 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.741229    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 31 23:46:18.741244 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.741238    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 31 23:46:18.741582 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.741568    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-j5fxf\""
Jul 31 23:46:18.745828 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.745813    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 31 23:46:18.757065 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.757041    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdfcn\" (UniqueName: \"kubernetes.io/projected/13eeae3e-dd1f-4829-8fd8-9a867df2630c-kube-api-access-tdfcn\") pod \"cluster-monitoring-operator-6f55c4f58d-vknnx\" (UID: \"13eeae3e-dd1f-4829-8fd8-9a867df2630c\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vknnx"
Jul 31 23:46:18.807971 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.807948    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92v2f_539b6c48-1833-4461-8ab1-580974198fb3/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:18.829495 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.829472    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ee2a1542-5293-4a53-a469-b2dc3317b711-serving-cert\") pod \"service-ca-operator-784f97bcf6-wvqkv\" (UID: \"ee2a1542-5293-4a53-a469-b2dc3317b711\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-wvqkv"
Jul 31 23:46:18.829580 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.829501    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ee2a1542-5293-4a53-a469-b2dc3317b711-config\") pod \"service-ca-operator-784f97bcf6-wvqkv\" (UID: \"ee2a1542-5293-4a53-a469-b2dc3317b711\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-wvqkv"
Jul 31 23:46:18.829580 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.829519    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njskj\" (UniqueName: \"kubernetes.io/projected/7d4b598c-cec0-4607-95bb-63f05f1eab65-kube-api-access-njskj\") pod \"volume-data-source-validator-5b8776669-4zdc5\" (UID: \"7d4b598c-cec0-4607-95bb-63f05f1eab65\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-4zdc5"
Jul 31 23:46:18.829580 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.829545    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/87ffcd92-302a-4e74-bca9-b64311b2a0b1-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-nspll\" (UID: \"87ffcd92-302a-4e74-bca9-b64311b2a0b1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-nspll"
Jul 31 23:46:18.829674 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.829612    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd7jz\" (UniqueName: \"kubernetes.io/projected/506a81ea-f212-4403-a5da-c92961c86683-kube-api-access-gd7jz\") pod \"network-check-source-65bdbbff45-2lwvf\" (UID: \"506a81ea-f212-4403-a5da-c92961c86683\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-2lwvf"
Jul 31 23:46:18.829674 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.829632    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt8j6\" (UniqueName: \"kubernetes.io/projected/87ffcd92-302a-4e74-bca9-b64311b2a0b1-kube-api-access-rt8j6\") pod \"cluster-samples-operator-559cbf5745-nspll\" (UID: \"87ffcd92-302a-4e74-bca9-b64311b2a0b1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-nspll"
Jul 31 23:46:18.829674 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.829653    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c33c7c85-f678-49ff-bcc0-5b1de21c563b-stats-auth\") pod \"router-default-5579c88784-ntwhl\" (UID: \"c33c7c85-f678-49ff-bcc0-5b1de21c563b\") " pod="openshift-ingress/router-default-5579c88784-ntwhl"
Jul 31 23:46:18.829800 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.829680    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q76wb\" (UniqueName: \"kubernetes.io/projected/e709e147-60bb-46dc-be7d-3d5269dbcd8a-kube-api-access-q76wb\") pod \"kube-storage-version-migrator-operator-7878f4b88c-slttz\" (UID: \"e709e147-60bb-46dc-be7d-3d5269dbcd8a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-slttz"
Jul 31 23:46:18.829800 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.829707    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vczc\" (UniqueName: \"kubernetes.io/projected/c2e50946-aea4-4886-82ec-0bd5a31196cf-kube-api-access-9vczc\") pod \"console-operator-5ccd6b445f-spblg\" (UID: \"c2e50946-aea4-4886-82ec-0bd5a31196cf\") " pod="openshift-console-operator/console-operator-5ccd6b445f-spblg"
Jul 31 23:46:18.829800 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.829733    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c33c7c85-f678-49ff-bcc0-5b1de21c563b-default-certificate\") pod \"router-default-5579c88784-ntwhl\" (UID: \"c33c7c85-f678-49ff-bcc0-5b1de21c563b\") " pod="openshift-ingress/router-default-5579c88784-ntwhl"
Jul 31 23:46:18.829800 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.829770    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c2e50946-aea4-4886-82ec-0bd5a31196cf-serving-cert\") pod \"console-operator-5ccd6b445f-spblg\" (UID: \"c2e50946-aea4-4886-82ec-0bd5a31196cf\") " pod="openshift-console-operator/console-operator-5ccd6b445f-spblg"
Jul 31 23:46:18.829923 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.829894    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e709e147-60bb-46dc-be7d-3d5269dbcd8a-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-slttz\" (UID: \"e709e147-60bb-46dc-be7d-3d5269dbcd8a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-slttz"
Jul 31 23:46:18.829971 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.829920    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c2e50946-aea4-4886-82ec-0bd5a31196cf-config\") pod \"console-operator-5ccd6b445f-spblg\" (UID: \"c2e50946-aea4-4886-82ec-0bd5a31196cf\") " pod="openshift-console-operator/console-operator-5ccd6b445f-spblg"
Jul 31 23:46:18.829971 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.829960    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c33c7c85-f678-49ff-bcc0-5b1de21c563b-service-ca-bundle\") pod \"router-default-5579c88784-ntwhl\" (UID: \"c33c7c85-f678-49ff-bcc0-5b1de21c563b\") " pod="openshift-ingress/router-default-5579c88784-ntwhl"
Jul 31 23:46:18.830070 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.829997    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e709e147-60bb-46dc-be7d-3d5269dbcd8a-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-slttz\" (UID: \"e709e147-60bb-46dc-be7d-3d5269dbcd8a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-slttz"
Jul 31 23:46:18.830070 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.830032    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c33c7c85-f678-49ff-bcc0-5b1de21c563b-metrics-certs\") pod \"router-default-5579c88784-ntwhl\" (UID: \"c33c7c85-f678-49ff-bcc0-5b1de21c563b\") " pod="openshift-ingress/router-default-5579c88784-ntwhl"
Jul 31 23:46:18.830070 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.830060    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c2e50946-aea4-4886-82ec-0bd5a31196cf-trusted-ca\") pod \"console-operator-5ccd6b445f-spblg\" (UID: \"c2e50946-aea4-4886-82ec-0bd5a31196cf\") " pod="openshift-console-operator/console-operator-5ccd6b445f-spblg"
Jul 31 23:46:18.830213 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.830088    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwwjr\" (UniqueName: \"kubernetes.io/projected/c33c7c85-f678-49ff-bcc0-5b1de21c563b-kube-api-access-pwwjr\") pod \"router-default-5579c88784-ntwhl\" (UID: \"c33c7c85-f678-49ff-bcc0-5b1de21c563b\") " pod="openshift-ingress/router-default-5579c88784-ntwhl"
Jul 31 23:46:18.830213 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.830136    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms9fw\" (UniqueName: \"kubernetes.io/projected/ee2a1542-5293-4a53-a469-b2dc3317b711-kube-api-access-ms9fw\") pod \"service-ca-operator-784f97bcf6-wvqkv\" (UID: \"ee2a1542-5293-4a53-a469-b2dc3317b711\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-wvqkv"
Jul 31 23:46:18.830213 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:18.830148    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c33c7c85-f678-49ff-bcc0-5b1de21c563b-service-ca-bundle podName:c33c7c85-f678-49ff-bcc0-5b1de21c563b nodeName:}" failed. No retries permitted until 2026-07-31 23:46:19.330126624 +0000 UTC m=+42.049926473 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c33c7c85-f678-49ff-bcc0-5b1de21c563b-service-ca-bundle") pod "router-default-5579c88784-ntwhl" (UID: "c33c7c85-f678-49ff-bcc0-5b1de21c563b") : configmap references non-existent config key: service-ca.crt
Jul 31 23:46:18.830359 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:18.830223    2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 31 23:46:18.832154 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:18.831079    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c33c7c85-f678-49ff-bcc0-5b1de21c563b-metrics-certs podName:c33c7c85-f678-49ff-bcc0-5b1de21c563b nodeName:}" failed. No retries permitted until 2026-07-31 23:46:19.331059237 +0000 UTC m=+42.050859063 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c33c7c85-f678-49ff-bcc0-5b1de21c563b-metrics-certs") pod "router-default-5579c88784-ntwhl" (UID: "c33c7c85-f678-49ff-bcc0-5b1de21c563b") : secret "router-metrics-certs-default" not found
Jul 31 23:46:18.832154 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.831174    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c2e50946-aea4-4886-82ec-0bd5a31196cf-config\") pod \"console-operator-5ccd6b445f-spblg\" (UID: \"c2e50946-aea4-4886-82ec-0bd5a31196cf\") " pod="openshift-console-operator/console-operator-5ccd6b445f-spblg"
Jul 31 23:46:18.832546 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.832524    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e709e147-60bb-46dc-be7d-3d5269dbcd8a-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-slttz\" (UID: \"e709e147-60bb-46dc-be7d-3d5269dbcd8a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-slttz"
Jul 31 23:46:18.832629 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.832610    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c2e50946-aea4-4886-82ec-0bd5a31196cf-trusted-ca\") pod \"console-operator-5ccd6b445f-spblg\" (UID: \"c2e50946-aea4-4886-82ec-0bd5a31196cf\") " pod="openshift-console-operator/console-operator-5ccd6b445f-spblg"
Jul 31 23:46:18.832687 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.832613    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c33c7c85-f678-49ff-bcc0-5b1de21c563b-default-certificate\") pod \"router-default-5579c88784-ntwhl\" (UID: \"c33c7c85-f678-49ff-bcc0-5b1de21c563b\") " pod="openshift-ingress/router-default-5579c88784-ntwhl"
Jul 31 23:46:18.832825 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.832801    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e709e147-60bb-46dc-be7d-3d5269dbcd8a-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-slttz\" (UID: \"e709e147-60bb-46dc-be7d-3d5269dbcd8a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-slttz"
Jul 31 23:46:18.833047 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.833026    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c33c7c85-f678-49ff-bcc0-5b1de21c563b-stats-auth\") pod \"router-default-5579c88784-ntwhl\" (UID: \"c33c7c85-f678-49ff-bcc0-5b1de21c563b\") " pod="openshift-ingress/router-default-5579c88784-ntwhl"
Jul 31 23:46:18.835981 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.835963    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c2e50946-aea4-4886-82ec-0bd5a31196cf-serving-cert\") pod \"console-operator-5ccd6b445f-spblg\" (UID: \"c2e50946-aea4-4886-82ec-0bd5a31196cf\") " pod="openshift-console-operator/console-operator-5ccd6b445f-spblg"
Jul 31 23:46:18.837907 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.837888    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q76wb\" (UniqueName: \"kubernetes.io/projected/e709e147-60bb-46dc-be7d-3d5269dbcd8a-kube-api-access-q76wb\") pod \"kube-storage-version-migrator-operator-7878f4b88c-slttz\" (UID: \"e709e147-60bb-46dc-be7d-3d5269dbcd8a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-slttz"
Jul 31 23:46:18.838283 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.838264    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vczc\" (UniqueName: \"kubernetes.io/projected/c2e50946-aea4-4886-82ec-0bd5a31196cf-kube-api-access-9vczc\") pod \"console-operator-5ccd6b445f-spblg\" (UID: \"c2e50946-aea4-4886-82ec-0bd5a31196cf\") " pod="openshift-console-operator/console-operator-5ccd6b445f-spblg"
Jul 31 23:46:18.838636 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.838621    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwwjr\" (UniqueName: \"kubernetes.io/projected/c33c7c85-f678-49ff-bcc0-5b1de21c563b-kube-api-access-pwwjr\") pod \"router-default-5579c88784-ntwhl\" (UID: \"c33c7c85-f678-49ff-bcc0-5b1de21c563b\") " pod="openshift-ingress/router-default-5579c88784-ntwhl"
Jul 31 23:46:18.886909 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.886886    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-slttz"
Jul 31 23:46:18.931495 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.931392    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/967d9bc9-7bc0-496f-b18b-3b7b95cab4f2-snapshots\") pod \"insights-operator-77696685b-xjqzb\" (UID: \"967d9bc9-7bc0-496f-b18b-3b7b95cab4f2\") " pod="openshift-insights/insights-operator-77696685b-xjqzb"
Jul 31 23:46:18.931606 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.931486    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/967d9bc9-7bc0-496f-b18b-3b7b95cab4f2-service-ca-bundle\") pod \"insights-operator-77696685b-xjqzb\" (UID: \"967d9bc9-7bc0-496f-b18b-3b7b95cab4f2\") " pod="openshift-insights/insights-operator-77696685b-xjqzb"
Jul 31 23:46:18.931606 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.931544    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlhq8\" (UniqueName: \"kubernetes.io/projected/967d9bc9-7bc0-496f-b18b-3b7b95cab4f2-kube-api-access-dlhq8\") pod \"insights-operator-77696685b-xjqzb\" (UID: \"967d9bc9-7bc0-496f-b18b-3b7b95cab4f2\") " pod="openshift-insights/insights-operator-77696685b-xjqzb"
Jul 31 23:46:18.931697 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.931616    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/967d9bc9-7bc0-496f-b18b-3b7b95cab4f2-serving-cert\") pod \"insights-operator-77696685b-xjqzb\" (UID: \"967d9bc9-7bc0-496f-b18b-3b7b95cab4f2\") " pod="openshift-insights/insights-operator-77696685b-xjqzb"
Jul 31 23:46:18.931697 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.931671    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms9fw\" (UniqueName: \"kubernetes.io/projected/ee2a1542-5293-4a53-a469-b2dc3317b711-kube-api-access-ms9fw\") pod \"service-ca-operator-784f97bcf6-wvqkv\" (UID: \"ee2a1542-5293-4a53-a469-b2dc3317b711\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-wvqkv"
Jul 31 23:46:18.931821 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.931715    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ee2a1542-5293-4a53-a469-b2dc3317b711-serving-cert\") pod \"service-ca-operator-784f97bcf6-wvqkv\" (UID: \"ee2a1542-5293-4a53-a469-b2dc3317b711\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-wvqkv"
Jul 31 23:46:18.931821 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.931761    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ee2a1542-5293-4a53-a469-b2dc3317b711-config\") pod \"service-ca-operator-784f97bcf6-wvqkv\" (UID: \"ee2a1542-5293-4a53-a469-b2dc3317b711\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-wvqkv"
Jul 31 23:46:18.931922 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.931895    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/967d9bc9-7bc0-496f-b18b-3b7b95cab4f2-tmp\") pod \"insights-operator-77696685b-xjqzb\" (UID: \"967d9bc9-7bc0-496f-b18b-3b7b95cab4f2\") " pod="openshift-insights/insights-operator-77696685b-xjqzb"
Jul 31 23:46:18.931974 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.931928    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/967d9bc9-7bc0-496f-b18b-3b7b95cab4f2-trusted-ca-bundle\") pod \"insights-operator-77696685b-xjqzb\" (UID: \"967d9bc9-7bc0-496f-b18b-3b7b95cab4f2\") " pod="openshift-insights/insights-operator-77696685b-xjqzb"
Jul 31 23:46:18.931974 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.931968    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njskj\" (UniqueName: \"kubernetes.io/projected/7d4b598c-cec0-4607-95bb-63f05f1eab65-kube-api-access-njskj\") pod \"volume-data-source-validator-5b8776669-4zdc5\" (UID: \"7d4b598c-cec0-4607-95bb-63f05f1eab65\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-4zdc5"
Jul 31 23:46:18.932066 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.931993    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/87ffcd92-302a-4e74-bca9-b64311b2a0b1-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-nspll\" (UID: \"87ffcd92-302a-4e74-bca9-b64311b2a0b1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-nspll"
Jul 31 23:46:18.932066 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.932055    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd7jz\" (UniqueName: \"kubernetes.io/projected/506a81ea-f212-4403-a5da-c92961c86683-kube-api-access-gd7jz\") pod \"network-check-source-65bdbbff45-2lwvf\" (UID: \"506a81ea-f212-4403-a5da-c92961c86683\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-2lwvf"
Jul 31 23:46:18.932163 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.932081    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt8j6\" (UniqueName: \"kubernetes.io/projected/87ffcd92-302a-4e74-bca9-b64311b2a0b1-kube-api-access-rt8j6\") pod \"cluster-samples-operator-559cbf5745-nspll\" (UID: \"87ffcd92-302a-4e74-bca9-b64311b2a0b1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-nspll"
Jul 31 23:46:18.932231 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:18.932197    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 31 23:46:18.932309 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:18.932291    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87ffcd92-302a-4e74-bca9-b64311b2a0b1-samples-operator-tls podName:87ffcd92-302a-4e74-bca9-b64311b2a0b1 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:19.432270058 +0000 UTC m=+42.152070114 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/87ffcd92-302a-4e74-bca9-b64311b2a0b1-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-nspll" (UID: "87ffcd92-302a-4e74-bca9-b64311b2a0b1") : secret "samples-operator-tls" not found
Jul 31 23:46:18.932309 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.932292    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ee2a1542-5293-4a53-a469-b2dc3317b711-config\") pod \"service-ca-operator-784f97bcf6-wvqkv\" (UID: \"ee2a1542-5293-4a53-a469-b2dc3317b711\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-wvqkv"
Jul 31 23:46:18.933996 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.933978    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ee2a1542-5293-4a53-a469-b2dc3317b711-serving-cert\") pod \"service-ca-operator-784f97bcf6-wvqkv\" (UID: \"ee2a1542-5293-4a53-a469-b2dc3317b711\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-wvqkv"
Jul 31 23:46:18.939084 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.939066    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-5ccd6b445f-spblg"
Jul 31 23:46:18.942601 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.942580    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd7jz\" (UniqueName: \"kubernetes.io/projected/506a81ea-f212-4403-a5da-c92961c86683-kube-api-access-gd7jz\") pod \"network-check-source-65bdbbff45-2lwvf\" (UID: \"506a81ea-f212-4403-a5da-c92961c86683\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-2lwvf"
Jul 31 23:46:18.943024 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.942994    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms9fw\" (UniqueName: \"kubernetes.io/projected/ee2a1542-5293-4a53-a469-b2dc3317b711-kube-api-access-ms9fw\") pod \"service-ca-operator-784f97bcf6-wvqkv\" (UID: \"ee2a1542-5293-4a53-a469-b2dc3317b711\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-wvqkv"
Jul 31 23:46:18.943275 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.943256    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njskj\" (UniqueName: \"kubernetes.io/projected/7d4b598c-cec0-4607-95bb-63f05f1eab65-kube-api-access-njskj\") pod \"volume-data-source-validator-5b8776669-4zdc5\" (UID: \"7d4b598c-cec0-4607-95bb-63f05f1eab65\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-4zdc5"
Jul 31 23:46:18.943401 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.943386    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt8j6\" (UniqueName: \"kubernetes.io/projected/87ffcd92-302a-4e74-bca9-b64311b2a0b1-kube-api-access-rt8j6\") pod \"cluster-samples-operator-559cbf5745-nspll\" (UID: \"87ffcd92-302a-4e74-bca9-b64311b2a0b1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-nspll"
Jul 31 23:46:18.952182 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.952162    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-wvqkv"
Jul 31 23:46:18.975120 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.975056    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-4zdc5"
Jul 31 23:46:18.998186 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:18.998115    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-2lwvf"
Jul 31 23:46:19.032826 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:19.032800    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/967d9bc9-7bc0-496f-b18b-3b7b95cab4f2-tmp\") pod \"insights-operator-77696685b-xjqzb\" (UID: \"967d9bc9-7bc0-496f-b18b-3b7b95cab4f2\") " pod="openshift-insights/insights-operator-77696685b-xjqzb"
Jul 31 23:46:19.032968 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:19.032831    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/967d9bc9-7bc0-496f-b18b-3b7b95cab4f2-trusted-ca-bundle\") pod \"insights-operator-77696685b-xjqzb\" (UID: \"967d9bc9-7bc0-496f-b18b-3b7b95cab4f2\") " pod="openshift-insights/insights-operator-77696685b-xjqzb"
Jul 31 23:46:19.033143 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:19.033118    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/967d9bc9-7bc0-496f-b18b-3b7b95cab4f2-snapshots\") pod \"insights-operator-77696685b-xjqzb\" (UID: \"967d9bc9-7bc0-496f-b18b-3b7b95cab4f2\") " pod="openshift-insights/insights-operator-77696685b-xjqzb"
Jul 31 23:46:19.033213 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:19.033167    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/967d9bc9-7bc0-496f-b18b-3b7b95cab4f2-service-ca-bundle\") pod \"insights-operator-77696685b-xjqzb\" (UID: \"967d9bc9-7bc0-496f-b18b-3b7b95cab4f2\") " pod="openshift-insights/insights-operator-77696685b-xjqzb"
Jul 31 23:46:19.033272 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:19.033217    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlhq8\" (UniqueName: \"kubernetes.io/projected/967d9bc9-7bc0-496f-b18b-3b7b95cab4f2-kube-api-access-dlhq8\") pod \"insights-operator-77696685b-xjqzb\" (UID: \"967d9bc9-7bc0-496f-b18b-3b7b95cab4f2\") " pod="openshift-insights/insights-operator-77696685b-xjqzb"
Jul 31 23:46:19.033272 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:19.033253    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/967d9bc9-7bc0-496f-b18b-3b7b95cab4f2-serving-cert\") pod \"insights-operator-77696685b-xjqzb\" (UID: \"967d9bc9-7bc0-496f-b18b-3b7b95cab4f2\") " pod="openshift-insights/insights-operator-77696685b-xjqzb"
Jul 31 23:46:19.034276 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:19.034131    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/967d9bc9-7bc0-496f-b18b-3b7b95cab4f2-service-ca-bundle\") pod \"insights-operator-77696685b-xjqzb\" (UID: \"967d9bc9-7bc0-496f-b18b-3b7b95cab4f2\") " pod="openshift-insights/insights-operator-77696685b-xjqzb"
Jul 31 23:46:19.034276 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:19.034241    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/967d9bc9-7bc0-496f-b18b-3b7b95cab4f2-tmp\") pod \"insights-operator-77696685b-xjqzb\" (UID: \"967d9bc9-7bc0-496f-b18b-3b7b95cab4f2\") " pod="openshift-insights/insights-operator-77696685b-xjqzb"
Jul 31 23:46:19.034276 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:19.034252    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/967d9bc9-7bc0-496f-b18b-3b7b95cab4f2-snapshots\") pod \"insights-operator-77696685b-xjqzb\" (UID: \"967d9bc9-7bc0-496f-b18b-3b7b95cab4f2\") " pod="openshift-insights/insights-operator-77696685b-xjqzb"
Jul 31 23:46:19.036136 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:19.035386    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/967d9bc9-7bc0-496f-b18b-3b7b95cab4f2-trusted-ca-bundle\") pod \"insights-operator-77696685b-xjqzb\" (UID: \"967d9bc9-7bc0-496f-b18b-3b7b95cab4f2\") " pod="openshift-insights/insights-operator-77696685b-xjqzb"
Jul 31 23:46:19.036830 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:19.036398    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-slttz"]
Jul 31 23:46:19.036830 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:19.036825    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/967d9bc9-7bc0-496f-b18b-3b7b95cab4f2-serving-cert\") pod \"insights-operator-77696685b-xjqzb\" (UID: \"967d9bc9-7bc0-496f-b18b-3b7b95cab4f2\") " pod="openshift-insights/insights-operator-77696685b-xjqzb"
Jul 31 23:46:19.040591 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:46:19.040567    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode709e147_60bb_46dc_be7d_3d5269dbcd8a.slice/crio-c33dc973ae45d6f4631dc9bfb9894207494b1eea007e6c385445277ca7532112 WatchSource:0}: Error finding container c33dc973ae45d6f4631dc9bfb9894207494b1eea007e6c385445277ca7532112: Status 404 returned error can't find the container with id c33dc973ae45d6f4631dc9bfb9894207494b1eea007e6c385445277ca7532112
Jul 31 23:46:19.045854 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:19.045833    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlhq8\" (UniqueName: \"kubernetes.io/projected/967d9bc9-7bc0-496f-b18b-3b7b95cab4f2-kube-api-access-dlhq8\") pod \"insights-operator-77696685b-xjqzb\" (UID: \"967d9bc9-7bc0-496f-b18b-3b7b95cab4f2\") " pod="openshift-insights/insights-operator-77696685b-xjqzb"
Jul 31 23:46:19.080334 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:19.079609    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-slttz" event={"ID":"e709e147-60bb-46dc-be7d-3d5269dbcd8a","Type":"ContainerStarted","Data":"c33dc973ae45d6f4631dc9bfb9894207494b1eea007e6c385445277ca7532112"}
Jul 31 23:46:19.110767 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:19.105379    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-spblg"]
Jul 31 23:46:19.186216 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:19.186192    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-wvqkv"]
Jul 31 23:46:19.189703 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:46:19.189670    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee2a1542_5293_4a53_a469_b2dc3317b711.slice/crio-fdda0de23721aea4b63d0e2cf565ca4298c26ac1ad17acb432528e84ba0e3e01 WatchSource:0}: Error finding container fdda0de23721aea4b63d0e2cf565ca4298c26ac1ad17acb432528e84ba0e3e01: Status 404 returned error can't find the container with id fdda0de23721aea4b63d0e2cf565ca4298c26ac1ad17acb432528e84ba0e3e01
Jul 31 23:46:19.210472 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:19.210419    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-4zdc5"]
Jul 31 23:46:19.212832 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:46:19.212810    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d4b598c_cec0_4607_95bb_63f05f1eab65.slice/crio-c63c8e88f4f40fa9287a3a3ec5074593e3c4a9c978b003db398304b0884dae9c WatchSource:0}: Error finding container c63c8e88f4f40fa9287a3a3ec5074593e3c4a9c978b003db398304b0884dae9c: Status 404 returned error can't find the container with id c63c8e88f4f40fa9287a3a3ec5074593e3c4a9c978b003db398304b0884dae9c
Jul 31 23:46:19.225513 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:19.225462    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-2lwvf"]
Jul 31 23:46:19.228261 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:46:19.228239    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod506a81ea_f212_4403_a5da_c92961c86683.slice/crio-bbd944af7016e12b91e06d11d64573c0e74d1f54fdcea3c6e460c84772c29b49 WatchSource:0}: Error finding container bbd944af7016e12b91e06d11d64573c0e74d1f54fdcea3c6e460c84772c29b49: Status 404 returned error can't find the container with id bbd944af7016e12b91e06d11d64573c0e74d1f54fdcea3c6e460c84772c29b49
Jul 31 23:46:19.237633 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:19.237614    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/13eeae3e-dd1f-4829-8fd8-9a867df2630c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-vknnx\" (UID: \"13eeae3e-dd1f-4829-8fd8-9a867df2630c\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vknnx"
Jul 31 23:46:19.237785 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:19.237769    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 31 23:46:19.237835 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:19.237825    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13eeae3e-dd1f-4829-8fd8-9a867df2630c-cluster-monitoring-operator-tls podName:13eeae3e-dd1f-4829-8fd8-9a867df2630c nodeName:}" failed. No retries permitted until 2026-07-31 23:46:20.237810046 +0000 UTC m=+42.957609870 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/13eeae3e-dd1f-4829-8fd8-9a867df2630c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-vknnx" (UID: "13eeae3e-dd1f-4829-8fd8-9a867df2630c") : secret "cluster-monitoring-operator-tls" not found
Jul 31 23:46:19.338215 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:19.338189    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c33c7c85-f678-49ff-bcc0-5b1de21c563b-service-ca-bundle\") pod \"router-default-5579c88784-ntwhl\" (UID: \"c33c7c85-f678-49ff-bcc0-5b1de21c563b\") " pod="openshift-ingress/router-default-5579c88784-ntwhl"
Jul 31 23:46:19.338337 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:19.338227    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c33c7c85-f678-49ff-bcc0-5b1de21c563b-metrics-certs\") pod \"router-default-5579c88784-ntwhl\" (UID: \"c33c7c85-f678-49ff-bcc0-5b1de21c563b\") " pod="openshift-ingress/router-default-5579c88784-ntwhl"
Jul 31 23:46:19.338378 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:19.338360    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c33c7c85-f678-49ff-bcc0-5b1de21c563b-service-ca-bundle podName:c33c7c85-f678-49ff-bcc0-5b1de21c563b nodeName:}" failed. No retries permitted until 2026-07-31 23:46:20.338342663 +0000 UTC m=+43.058142504 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c33c7c85-f678-49ff-bcc0-5b1de21c563b-service-ca-bundle") pod "router-default-5579c88784-ntwhl" (UID: "c33c7c85-f678-49ff-bcc0-5b1de21c563b") : configmap references non-existent config key: service-ca.crt
Jul 31 23:46:19.338424 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:19.338380    2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 31 23:46:19.338424 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:19.338420    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c33c7c85-f678-49ff-bcc0-5b1de21c563b-metrics-certs podName:c33c7c85-f678-49ff-bcc0-5b1de21c563b nodeName:}" failed. No retries permitted until 2026-07-31 23:46:20.338408911 +0000 UTC m=+43.058208735 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c33c7c85-f678-49ff-bcc0-5b1de21c563b-metrics-certs") pod "router-default-5579c88784-ntwhl" (UID: "c33c7c85-f678-49ff-bcc0-5b1de21c563b") : secret "router-metrics-certs-default" not found
Jul 31 23:46:19.343166 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:19.343149    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-77696685b-xjqzb"
Jul 31 23:46:19.439618 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:19.439592    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/87ffcd92-302a-4e74-bca9-b64311b2a0b1-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-nspll\" (UID: \"87ffcd92-302a-4e74-bca9-b64311b2a0b1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-nspll"
Jul 31 23:46:19.439816 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:19.439794    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 31 23:46:19.439888 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:19.439878    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87ffcd92-302a-4e74-bca9-b64311b2a0b1-samples-operator-tls podName:87ffcd92-302a-4e74-bca9-b64311b2a0b1 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:20.439856281 +0000 UTC m=+43.159656117 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/87ffcd92-302a-4e74-bca9-b64311b2a0b1-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-nspll" (UID: "87ffcd92-302a-4e74-bca9-b64311b2a0b1") : secret "samples-operator-tls" not found
Jul 31 23:46:19.488856 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:19.488778    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-77696685b-xjqzb"]
Jul 31 23:46:19.491561 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:46:19.491540    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod967d9bc9_7bc0_496f_b18b_3b7b95cab4f2.slice/crio-21877d3104c86bcaa3d1faadc61f8ba336d8f4b0720da21c65d2ff2db8646669 WatchSource:0}: Error finding container 21877d3104c86bcaa3d1faadc61f8ba336d8f4b0720da21c65d2ff2db8646669: Status 404 returned error can't find the container with id 21877d3104c86bcaa3d1faadc61f8ba336d8f4b0720da21c65d2ff2db8646669
Jul 31 23:46:20.086144 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:20.086051    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-xjqzb" event={"ID":"967d9bc9-7bc0-496f-b18b-3b7b95cab4f2","Type":"ContainerStarted","Data":"21877d3104c86bcaa3d1faadc61f8ba336d8f4b0720da21c65d2ff2db8646669"}
Jul 31 23:46:20.087820 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:20.087737    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-4zdc5" event={"ID":"7d4b598c-cec0-4607-95bb-63f05f1eab65","Type":"ContainerStarted","Data":"c63c8e88f4f40fa9287a3a3ec5074593e3c4a9c978b003db398304b0884dae9c"}
Jul 31 23:46:20.089090 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:20.089041    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-2lwvf" event={"ID":"506a81ea-f212-4403-a5da-c92961c86683","Type":"ContainerStarted","Data":"bbd944af7016e12b91e06d11d64573c0e74d1f54fdcea3c6e460c84772c29b49"}
Jul 31 23:46:20.090723 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:20.090665    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-spblg" event={"ID":"c2e50946-aea4-4886-82ec-0bd5a31196cf","Type":"ContainerStarted","Data":"6482ecc734de32f1f275a2abf46ac66e70b718c26fb6da540f41c8ed0be02422"}
Jul 31 23:46:20.092081 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:20.092030    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-wvqkv" event={"ID":"ee2a1542-5293-4a53-a469-b2dc3317b711","Type":"ContainerStarted","Data":"fdda0de23721aea4b63d0e2cf565ca4298c26ac1ad17acb432528e84ba0e3e01"}
Jul 31 23:46:20.246176 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:20.246131    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/13eeae3e-dd1f-4829-8fd8-9a867df2630c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-vknnx\" (UID: \"13eeae3e-dd1f-4829-8fd8-9a867df2630c\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vknnx"
Jul 31 23:46:20.246358 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:20.246296    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 31 23:46:20.246429 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:20.246360    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13eeae3e-dd1f-4829-8fd8-9a867df2630c-cluster-monitoring-operator-tls podName:13eeae3e-dd1f-4829-8fd8-9a867df2630c nodeName:}" failed. No retries permitted until 2026-07-31 23:46:22.246342685 +0000 UTC m=+44.966142512 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/13eeae3e-dd1f-4829-8fd8-9a867df2630c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-vknnx" (UID: "13eeae3e-dd1f-4829-8fd8-9a867df2630c") : secret "cluster-monitoring-operator-tls" not found
Jul 31 23:46:20.347161 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:20.347079    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c33c7c85-f678-49ff-bcc0-5b1de21c563b-metrics-certs\") pod \"router-default-5579c88784-ntwhl\" (UID: \"c33c7c85-f678-49ff-bcc0-5b1de21c563b\") " pod="openshift-ingress/router-default-5579c88784-ntwhl"
Jul 31 23:46:20.347314 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:20.347263    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c33c7c85-f678-49ff-bcc0-5b1de21c563b-service-ca-bundle\") pod \"router-default-5579c88784-ntwhl\" (UID: \"c33c7c85-f678-49ff-bcc0-5b1de21c563b\") " pod="openshift-ingress/router-default-5579c88784-ntwhl"
Jul 31 23:46:20.347959 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:20.347408    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c33c7c85-f678-49ff-bcc0-5b1de21c563b-service-ca-bundle podName:c33c7c85-f678-49ff-bcc0-5b1de21c563b nodeName:}" failed. No retries permitted until 2026-07-31 23:46:22.347388287 +0000 UTC m=+45.067188128 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c33c7c85-f678-49ff-bcc0-5b1de21c563b-service-ca-bundle") pod "router-default-5579c88784-ntwhl" (UID: "c33c7c85-f678-49ff-bcc0-5b1de21c563b") : configmap references non-existent config key: service-ca.crt
Jul 31 23:46:20.347959 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:20.347702    2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 31 23:46:20.347959 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:20.347769    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c33c7c85-f678-49ff-bcc0-5b1de21c563b-metrics-certs podName:c33c7c85-f678-49ff-bcc0-5b1de21c563b nodeName:}" failed. No retries permitted until 2026-07-31 23:46:22.347753996 +0000 UTC m=+45.067553838 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c33c7c85-f678-49ff-bcc0-5b1de21c563b-metrics-certs") pod "router-default-5579c88784-ntwhl" (UID: "c33c7c85-f678-49ff-bcc0-5b1de21c563b") : secret "router-metrics-certs-default" not found
Jul 31 23:46:20.448206 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:20.448170    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ae80080-0f96-4a21-b13c-87738f73a578-metrics-tls\") pod \"dns-default-fxfvr\" (UID: \"0ae80080-0f96-4a21-b13c-87738f73a578\") " pod="openshift-dns/dns-default-fxfvr"
Jul 31 23:46:20.448385 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:20.448267    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7fb6400-a564-4c03-8381-b76eb7aac58b-registry-tls\") pod \"image-registry-5b46bd6c8c-m7dfb\" (UID: \"c7fb6400-a564-4c03-8381-b76eb7aac58b\") " pod="openshift-image-registry/image-registry-5b46bd6c8c-m7dfb"
Jul 31 23:46:20.448385 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:20.448303    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/87ffcd92-302a-4e74-bca9-b64311b2a0b1-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-nspll\" (UID: \"87ffcd92-302a-4e74-bca9-b64311b2a0b1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-nspll"
Jul 31 23:46:20.448492 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:20.448474    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 31 23:46:20.448555 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:20.448537    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87ffcd92-302a-4e74-bca9-b64311b2a0b1-samples-operator-tls podName:87ffcd92-302a-4e74-bca9-b64311b2a0b1 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:22.448519288 +0000 UTC m=+45.168319119 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/87ffcd92-302a-4e74-bca9-b64311b2a0b1-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-nspll" (UID: "87ffcd92-302a-4e74-bca9-b64311b2a0b1") : secret "samples-operator-tls" not found
Jul 31 23:46:20.448798 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:20.448777    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 31 23:46:20.448798 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:20.448799    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b46bd6c8c-m7dfb: secret "image-registry-tls" not found
Jul 31 23:46:20.448921 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:20.448816    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 23:46:20.448921 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:20.448839    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7fb6400-a564-4c03-8381-b76eb7aac58b-registry-tls podName:c7fb6400-a564-4c03-8381-b76eb7aac58b nodeName:}" failed. No retries permitted until 2026-07-31 23:46:28.448825767 +0000 UTC m=+51.168625608 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c7fb6400-a564-4c03-8381-b76eb7aac58b-registry-tls") pod "image-registry-5b46bd6c8c-m7dfb" (UID: "c7fb6400-a564-4c03-8381-b76eb7aac58b") : secret "image-registry-tls" not found
Jul 31 23:46:20.448921 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:20.448883    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ae80080-0f96-4a21-b13c-87738f73a578-metrics-tls podName:0ae80080-0f96-4a21-b13c-87738f73a578 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:28.448863372 +0000 UTC m=+51.168663319 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0ae80080-0f96-4a21-b13c-87738f73a578-metrics-tls") pod "dns-default-fxfvr" (UID: "0ae80080-0f96-4a21-b13c-87738f73a578") : secret "dns-default-metrics-tls" not found
Jul 31 23:46:21.198897 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:21.198866    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-fp7hq"]
Jul 31 23:46:21.216662 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:21.216614    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-fp7hq"]
Jul 31 23:46:21.216824 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:21.216760    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-f77f48c6d-fp7hq"
Jul 31 23:46:21.219756 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:21.219710    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 31 23:46:21.220015 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:21.219996    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 31 23:46:21.220361 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:21.220339    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-b9ls8\""
Jul 31 23:46:21.258246 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:21.258206    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7eeb5980-be60-42c9-b61b-639811f83942-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-fp7hq\" (UID: \"7eeb5980-be60-42c9-b61b-639811f83942\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-fp7hq"
Jul 31 23:46:21.258424 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:21.258260    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7eeb5980-be60-42c9-b61b-639811f83942-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-fp7hq\" (UID: \"7eeb5980-be60-42c9-b61b-639811f83942\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-fp7hq"
Jul 31 23:46:21.359764 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:21.359716    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7eeb5980-be60-42c9-b61b-639811f83942-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-fp7hq\" (UID: \"7eeb5980-be60-42c9-b61b-639811f83942\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-fp7hq"
Jul 31 23:46:21.359946 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:21.359791    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7eeb5980-be60-42c9-b61b-639811f83942-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-fp7hq\" (UID: \"7eeb5980-be60-42c9-b61b-639811f83942\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-fp7hq"
Jul 31 23:46:21.360061 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:21.360041    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 31 23:46:21.360121 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:21.360114    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7eeb5980-be60-42c9-b61b-639811f83942-networking-console-plugin-cert podName:7eeb5980-be60-42c9-b61b-639811f83942 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:21.860095497 +0000 UTC m=+44.579895338 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7eeb5980-be60-42c9-b61b-639811f83942-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-fp7hq" (UID: "7eeb5980-be60-42c9-b61b-639811f83942") : secret "networking-console-plugin-cert" not found
Jul 31 23:46:21.361091 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:21.360998    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7eeb5980-be60-42c9-b61b-639811f83942-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-fp7hq\" (UID: \"7eeb5980-be60-42c9-b61b-639811f83942\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-fp7hq"
Jul 31 23:46:21.865602 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:21.865566    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7eeb5980-be60-42c9-b61b-639811f83942-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-fp7hq\" (UID: \"7eeb5980-be60-42c9-b61b-639811f83942\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-fp7hq"
Jul 31 23:46:21.865771 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:21.865730    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 31 23:46:21.865817 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:21.865812    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7eeb5980-be60-42c9-b61b-639811f83942-networking-console-plugin-cert podName:7eeb5980-be60-42c9-b61b-639811f83942 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:22.865792418 +0000 UTC m=+45.585592242 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7eeb5980-be60-42c9-b61b-639811f83942-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-fp7hq" (UID: "7eeb5980-be60-42c9-b61b-639811f83942") : secret "networking-console-plugin-cert" not found
Jul 31 23:46:22.269119 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:22.269087    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/13eeae3e-dd1f-4829-8fd8-9a867df2630c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-vknnx\" (UID: \"13eeae3e-dd1f-4829-8fd8-9a867df2630c\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vknnx"
Jul 31 23:46:22.269598 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:22.269337    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 31 23:46:22.269598 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:22.269406    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13eeae3e-dd1f-4829-8fd8-9a867df2630c-cluster-monitoring-operator-tls podName:13eeae3e-dd1f-4829-8fd8-9a867df2630c nodeName:}" failed. No retries permitted until 2026-07-31 23:46:26.269387874 +0000 UTC m=+48.989187718 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/13eeae3e-dd1f-4829-8fd8-9a867df2630c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-vknnx" (UID: "13eeae3e-dd1f-4829-8fd8-9a867df2630c") : secret "cluster-monitoring-operator-tls" not found
Jul 31 23:46:22.370479 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:22.370441    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c33c7c85-f678-49ff-bcc0-5b1de21c563b-service-ca-bundle\") pod \"router-default-5579c88784-ntwhl\" (UID: \"c33c7c85-f678-49ff-bcc0-5b1de21c563b\") " pod="openshift-ingress/router-default-5579c88784-ntwhl"
Jul 31 23:46:22.370479 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:22.370484    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c33c7c85-f678-49ff-bcc0-5b1de21c563b-metrics-certs\") pod \"router-default-5579c88784-ntwhl\" (UID: \"c33c7c85-f678-49ff-bcc0-5b1de21c563b\") " pod="openshift-ingress/router-default-5579c88784-ntwhl"
Jul 31 23:46:22.370703 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:22.370668    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c33c7c85-f678-49ff-bcc0-5b1de21c563b-service-ca-bundle podName:c33c7c85-f678-49ff-bcc0-5b1de21c563b nodeName:}" failed. No retries permitted until 2026-07-31 23:46:26.370648033 +0000 UTC m=+49.090447886 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c33c7c85-f678-49ff-bcc0-5b1de21c563b-service-ca-bundle") pod "router-default-5579c88784-ntwhl" (UID: "c33c7c85-f678-49ff-bcc0-5b1de21c563b") : configmap references non-existent config key: service-ca.crt
Jul 31 23:46:22.370801 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:22.370732    2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 31 23:46:22.370857 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:22.370802    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c33c7c85-f678-49ff-bcc0-5b1de21c563b-metrics-certs podName:c33c7c85-f678-49ff-bcc0-5b1de21c563b nodeName:}" failed. No retries permitted until 2026-07-31 23:46:26.370789885 +0000 UTC m=+49.090589709 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c33c7c85-f678-49ff-bcc0-5b1de21c563b-metrics-certs") pod "router-default-5579c88784-ntwhl" (UID: "c33c7c85-f678-49ff-bcc0-5b1de21c563b") : secret "router-metrics-certs-default" not found
Jul 31 23:46:22.472139 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:22.472103    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/87ffcd92-302a-4e74-bca9-b64311b2a0b1-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-nspll\" (UID: \"87ffcd92-302a-4e74-bca9-b64311b2a0b1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-nspll"
Jul 31 23:46:22.472722 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:22.472664    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 31 23:46:22.472871 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:22.472801    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87ffcd92-302a-4e74-bca9-b64311b2a0b1-samples-operator-tls podName:87ffcd92-302a-4e74-bca9-b64311b2a0b1 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:26.472778615 +0000 UTC m=+49.192578455 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/87ffcd92-302a-4e74-bca9-b64311b2a0b1-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-nspll" (UID: "87ffcd92-302a-4e74-bca9-b64311b2a0b1") : secret "samples-operator-tls" not found
Jul 31 23:46:22.875768 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:22.875720    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7eeb5980-be60-42c9-b61b-639811f83942-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-fp7hq\" (UID: \"7eeb5980-be60-42c9-b61b-639811f83942\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-fp7hq"
Jul 31 23:46:22.875950 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:22.875860    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 31 23:46:22.875997 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:22.875959    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7eeb5980-be60-42c9-b61b-639811f83942-networking-console-plugin-cert podName:7eeb5980-be60-42c9-b61b-639811f83942 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:24.87593505 +0000 UTC m=+47.595734879 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7eeb5980-be60-42c9-b61b-639811f83942-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-fp7hq" (UID: "7eeb5980-be60-42c9-b61b-639811f83942") : secret "networking-console-plugin-cert" not found
Jul 31 23:46:24.891568 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:24.891531    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7eeb5980-be60-42c9-b61b-639811f83942-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-fp7hq\" (UID: \"7eeb5980-be60-42c9-b61b-639811f83942\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-fp7hq"
Jul 31 23:46:24.892021 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:24.891690    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 31 23:46:24.892021 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:24.891789    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7eeb5980-be60-42c9-b61b-639811f83942-networking-console-plugin-cert podName:7eeb5980-be60-42c9-b61b-639811f83942 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:28.891757449 +0000 UTC m=+51.611557290 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7eeb5980-be60-42c9-b61b-639811f83942-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-fp7hq" (UID: "7eeb5980-be60-42c9-b61b-639811f83942") : secret "networking-console-plugin-cert" not found
Jul 31 23:46:26.114066 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:26.114029    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-xjqzb" event={"ID":"967d9bc9-7bc0-496f-b18b-3b7b95cab4f2","Type":"ContainerStarted","Data":"911c915918bf4ae954e2381bb34c993830904fd65926be6e5c1e0e329afb7fc0"}
Jul 31 23:46:26.116953 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:26.116481    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-4zdc5" event={"ID":"7d4b598c-cec0-4607-95bb-63f05f1eab65","Type":"ContainerStarted","Data":"0125e7eea27a9f78f97091f6b0c8b1df9cee1a604f41a2c7e8c19de94008a223"}
Jul 31 23:46:26.119239 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:26.119212    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-slttz" event={"ID":"e709e147-60bb-46dc-be7d-3d5269dbcd8a","Type":"ContainerStarted","Data":"d42b65c68555a9e2e0d3a9a65b04e733fb7e42cd0a865dee327ccea4529a618f"}
Jul 31 23:46:26.121206 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:26.121163    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-2lwvf" event={"ID":"506a81ea-f212-4403-a5da-c92961c86683","Type":"ContainerStarted","Data":"049ab1886f26d71449641b7b4a2eba307ee6331789cb05446c3289ee6c7d19b0"}
Jul 31 23:46:26.124608 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:26.122850    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-spblg" event={"ID":"c2e50946-aea4-4886-82ec-0bd5a31196cf","Type":"ContainerStarted","Data":"fa47cd048a2b2b31655226440a2ac0570e318922f01260bf33e62f4416662810"}
Jul 31 23:46:26.124608 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:26.123542    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-spblg"
Jul 31 23:46:26.124773 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:26.124672    2559 patch_prober.go:28] interesting pod/console-operator-5ccd6b445f-spblg container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.12:8443/readyz\": dial tcp 10.132.0.12:8443: connect: connection refused" start-of-body=
Jul 31 23:46:26.124773 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:26.124717    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-5ccd6b445f-spblg" podUID="c2e50946-aea4-4886-82ec-0bd5a31196cf" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.12:8443/readyz\": dial tcp 10.132.0.12:8443: connect: connection refused"
Jul 31 23:46:26.126505 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:26.126136    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-wvqkv" event={"ID":"ee2a1542-5293-4a53-a469-b2dc3317b711","Type":"ContainerStarted","Data":"7e4989fed288d9a79992c1e3d91742b32bf330563bc308f763cc16301b3a6940"}
Jul 31 23:46:26.131184 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:26.131121    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-77696685b-xjqzb" podStartSLOduration=1.981140968 podStartE2EDuration="8.131105635s" totalImagesPullingTime="6.149964667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:26.130186093 +0000 UTC m=+48.849985958" watchObservedRunningTime="2026-07-31 23:46:26.131105635 +0000 UTC m=+48.850905483"
Jul 31 23:46:26.145667 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:26.145620    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-slttz" podStartSLOduration=1.545129269 podStartE2EDuration="8.145599843s" totalImagesPullingTime="6.600470574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:26.144400998 +0000 UTC m=+48.864200843" watchObservedRunningTime="2026-07-31 23:46:26.145599843 +0000 UTC m=+48.865399690"
Jul 31 23:46:26.172171 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:26.172114    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-5ccd6b445f-spblg" podStartSLOduration=1.659233445 podStartE2EDuration="8.17209138s" totalImagesPullingTime="6.512857935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:26.170116984 +0000 UTC m=+48.889916831" watchObservedRunningTime="2026-07-31 23:46:26.17209138 +0000 UTC m=+48.891891227"
Jul 31 23:46:26.185917 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:26.185852    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-wvqkv" podStartSLOduration=1.734248774 podStartE2EDuration="8.185830598s" totalImagesPullingTime="6.451581824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:26.184496173 +0000 UTC m=+48.904296020" watchObservedRunningTime="2026-07-31 23:46:26.185830598 +0000 UTC m=+48.905630447"
Jul 31 23:46:26.206444 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:26.206341    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-4zdc5" podStartSLOduration=1.809978638 podStartE2EDuration="8.206320919s" totalImagesPullingTime="6.396342281s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:26.204897581 +0000 UTC m=+48.924697428" watchObservedRunningTime="2026-07-31 23:46:26.206320919 +0000 UTC m=+48.926120768"
Jul 31 23:46:26.224134 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:26.224047    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-2lwvf" podStartSLOduration=1.471862798 podStartE2EDuration="8.224025793s" totalImagesPullingTime="6.752162995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:26.222399462 +0000 UTC m=+48.942199310" watchObservedRunningTime="2026-07-31 23:46:26.224025793 +0000 UTC m=+48.943825640"
Jul 31 23:46:26.303470 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:26.303439    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/13eeae3e-dd1f-4829-8fd8-9a867df2630c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-vknnx\" (UID: \"13eeae3e-dd1f-4829-8fd8-9a867df2630c\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vknnx"
Jul 31 23:46:26.303635 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:26.303571    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 31 23:46:26.303635 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:26.303622    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13eeae3e-dd1f-4829-8fd8-9a867df2630c-cluster-monitoring-operator-tls podName:13eeae3e-dd1f-4829-8fd8-9a867df2630c nodeName:}" failed. No retries permitted until 2026-07-31 23:46:34.303609183 +0000 UTC m=+57.023409007 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/13eeae3e-dd1f-4829-8fd8-9a867df2630c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-vknnx" (UID: "13eeae3e-dd1f-4829-8fd8-9a867df2630c") : secret "cluster-monitoring-operator-tls" not found
Jul 31 23:46:26.404439 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:26.404402    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c33c7c85-f678-49ff-bcc0-5b1de21c563b-service-ca-bundle\") pod \"router-default-5579c88784-ntwhl\" (UID: \"c33c7c85-f678-49ff-bcc0-5b1de21c563b\") " pod="openshift-ingress/router-default-5579c88784-ntwhl"
Jul 31 23:46:26.404605 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:26.404460    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c33c7c85-f678-49ff-bcc0-5b1de21c563b-metrics-certs\") pod \"router-default-5579c88784-ntwhl\" (UID: \"c33c7c85-f678-49ff-bcc0-5b1de21c563b\") " pod="openshift-ingress/router-default-5579c88784-ntwhl"
Jul 31 23:46:26.404659 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:26.404636    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c33c7c85-f678-49ff-bcc0-5b1de21c563b-service-ca-bundle podName:c33c7c85-f678-49ff-bcc0-5b1de21c563b nodeName:}" failed. No retries permitted until 2026-07-31 23:46:34.404613414 +0000 UTC m=+57.124413259 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c33c7c85-f678-49ff-bcc0-5b1de21c563b-service-ca-bundle") pod "router-default-5579c88784-ntwhl" (UID: "c33c7c85-f678-49ff-bcc0-5b1de21c563b") : configmap references non-existent config key: service-ca.crt
Jul 31 23:46:26.404659 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:26.404636    2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 31 23:46:26.404808 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:26.404685    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c33c7c85-f678-49ff-bcc0-5b1de21c563b-metrics-certs podName:c33c7c85-f678-49ff-bcc0-5b1de21c563b nodeName:}" failed. No retries permitted until 2026-07-31 23:46:34.404674982 +0000 UTC m=+57.124474813 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c33c7c85-f678-49ff-bcc0-5b1de21c563b-metrics-certs") pod "router-default-5579c88784-ntwhl" (UID: "c33c7c85-f678-49ff-bcc0-5b1de21c563b") : secret "router-metrics-certs-default" not found
Jul 31 23:46:26.506080 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:26.505992    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/87ffcd92-302a-4e74-bca9-b64311b2a0b1-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-nspll\" (UID: \"87ffcd92-302a-4e74-bca9-b64311b2a0b1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-nspll"
Jul 31 23:46:26.506224 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:26.506156    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 31 23:46:26.506285 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:26.506236    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87ffcd92-302a-4e74-bca9-b64311b2a0b1-samples-operator-tls podName:87ffcd92-302a-4e74-bca9-b64311b2a0b1 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:34.506215907 +0000 UTC m=+57.226015735 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/87ffcd92-302a-4e74-bca9-b64311b2a0b1-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-nspll" (UID: "87ffcd92-302a-4e74-bca9-b64311b2a0b1") : secret "samples-operator-tls" not found
Jul 31 23:46:26.752214 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:26.752184    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vqpfb"]
Jul 31 23:46:26.769714 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:26.769641    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vqpfb"]
Jul 31 23:46:26.769849 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:26.769766    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vqpfb"
Jul 31 23:46:26.772113 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:26.772091    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 31 23:46:26.772328 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:26.772090    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 31 23:46:26.772328 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:26.772279    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mw9xk\""
Jul 31 23:46:26.809953 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:26.809930    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfxzb\" (UniqueName: \"kubernetes.io/projected/41ab6526-f493-4ea8-881a-fda14abc3525-kube-api-access-vfxzb\") pod \"insights-runtime-extractor-vqpfb\" (UID: \"41ab6526-f493-4ea8-881a-fda14abc3525\") " pod="openshift-insights/insights-runtime-extractor-vqpfb"
Jul 31 23:46:26.810119 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:26.810072    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/41ab6526-f493-4ea8-881a-fda14abc3525-data-volume\") pod \"insights-runtime-extractor-vqpfb\" (UID: \"41ab6526-f493-4ea8-881a-fda14abc3525\") " pod="openshift-insights/insights-runtime-extractor-vqpfb"
Jul 31 23:46:26.810183 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:26.810139    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/41ab6526-f493-4ea8-881a-fda14abc3525-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vqpfb\" (UID: \"41ab6526-f493-4ea8-881a-fda14abc3525\") " pod="openshift-insights/insights-runtime-extractor-vqpfb"
Jul 31 23:46:26.810285 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:26.810262    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/41ab6526-f493-4ea8-881a-fda14abc3525-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vqpfb\" (UID: \"41ab6526-f493-4ea8-881a-fda14abc3525\") " pod="openshift-insights/insights-runtime-extractor-vqpfb"
Jul 31 23:46:26.810339 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:26.810324    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/41ab6526-f493-4ea8-881a-fda14abc3525-crio-socket\") pod \"insights-runtime-extractor-vqpfb\" (UID: \"41ab6526-f493-4ea8-881a-fda14abc3525\") " pod="openshift-insights/insights-runtime-extractor-vqpfb"
Jul 31 23:46:26.911723 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:26.911684    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/41ab6526-f493-4ea8-881a-fda14abc3525-data-volume\") pod \"insights-runtime-extractor-vqpfb\" (UID: \"41ab6526-f493-4ea8-881a-fda14abc3525\") " pod="openshift-insights/insights-runtime-extractor-vqpfb"
Jul 31 23:46:26.911909 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:26.911786    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/41ab6526-f493-4ea8-881a-fda14abc3525-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vqpfb\" (UID: \"41ab6526-f493-4ea8-881a-fda14abc3525\") " pod="openshift-insights/insights-runtime-extractor-vqpfb"
Jul 31 23:46:26.911909 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:26.911873    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/41ab6526-f493-4ea8-881a-fda14abc3525-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vqpfb\" (UID: \"41ab6526-f493-4ea8-881a-fda14abc3525\") " pod="openshift-insights/insights-runtime-extractor-vqpfb"
Jul 31 23:46:26.912030 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:26.911925    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/41ab6526-f493-4ea8-881a-fda14abc3525-crio-socket\") pod \"insights-runtime-extractor-vqpfb\" (UID: \"41ab6526-f493-4ea8-881a-fda14abc3525\") " pod="openshift-insights/insights-runtime-extractor-vqpfb"
Jul 31 23:46:26.912030 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:26.911927    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 23:46:26.912030 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:26.911966    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfxzb\" (UniqueName: \"kubernetes.io/projected/41ab6526-f493-4ea8-881a-fda14abc3525-kube-api-access-vfxzb\") pod \"insights-runtime-extractor-vqpfb\" (UID: \"41ab6526-f493-4ea8-881a-fda14abc3525\") " pod="openshift-insights/insights-runtime-extractor-vqpfb"
Jul 31 23:46:26.912030 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:26.912010    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/41ab6526-f493-4ea8-881a-fda14abc3525-kube-rbac-proxy-cm podName:41ab6526-f493-4ea8-881a-fda14abc3525 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:27.41198686 +0000 UTC m=+50.131786690 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/41ab6526-f493-4ea8-881a-fda14abc3525-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vqpfb" (UID: "41ab6526-f493-4ea8-881a-fda14abc3525") : configmap "kube-rbac-proxy" not found
Jul 31 23:46:26.912227 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:26.912052    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 23:46:26.912227 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:26.912075    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/41ab6526-f493-4ea8-881a-fda14abc3525-data-volume\") pod \"insights-runtime-extractor-vqpfb\" (UID: \"41ab6526-f493-4ea8-881a-fda14abc3525\") " pod="openshift-insights/insights-runtime-extractor-vqpfb"
Jul 31 23:46:26.912227 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:26.912183    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41ab6526-f493-4ea8-881a-fda14abc3525-insights-runtime-extractor-tls podName:41ab6526-f493-4ea8-881a-fda14abc3525 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:27.412165062 +0000 UTC m=+50.131964886 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/41ab6526-f493-4ea8-881a-fda14abc3525-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vqpfb" (UID: "41ab6526-f493-4ea8-881a-fda14abc3525") : secret "insights-runtime-extractor-tls" not found
Jul 31 23:46:26.912375 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:26.912294    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/41ab6526-f493-4ea8-881a-fda14abc3525-crio-socket\") pod \"insights-runtime-extractor-vqpfb\" (UID: \"41ab6526-f493-4ea8-881a-fda14abc3525\") " pod="openshift-insights/insights-runtime-extractor-vqpfb"
Jul 31 23:46:26.920735 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:26.920679    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfxzb\" (UniqueName: \"kubernetes.io/projected/41ab6526-f493-4ea8-881a-fda14abc3525-kube-api-access-vfxzb\") pod \"insights-runtime-extractor-vqpfb\" (UID: \"41ab6526-f493-4ea8-881a-fda14abc3525\") " pod="openshift-insights/insights-runtime-extractor-vqpfb"
Jul 31 23:46:27.131053 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:27.130989    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-spblg_c2e50946-aea4-4886-82ec-0bd5a31196cf/console-operator/0.log"
Jul 31 23:46:27.131053 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:27.131026    2559 generic.go:356] "Generic (PLEG): container finished" podID="c2e50946-aea4-4886-82ec-0bd5a31196cf" containerID="fa47cd048a2b2b31655226440a2ac0570e318922f01260bf33e62f4416662810" exitCode=255
Jul 31 23:46:27.131522 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:27.131099    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-spblg" event={"ID":"c2e50946-aea4-4886-82ec-0bd5a31196cf","Type":"ContainerDied","Data":"fa47cd048a2b2b31655226440a2ac0570e318922f01260bf33e62f4416662810"}
Jul 31 23:46:27.131522 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:27.131294    2559 scope.go:117] "RemoveContainer" containerID="fa47cd048a2b2b31655226440a2ac0570e318922f01260bf33e62f4416662810"
Jul 31 23:46:27.417136 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:27.417054    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/41ab6526-f493-4ea8-881a-fda14abc3525-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vqpfb\" (UID: \"41ab6526-f493-4ea8-881a-fda14abc3525\") " pod="openshift-insights/insights-runtime-extractor-vqpfb"
Jul 31 23:46:27.417276 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:27.417161    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/41ab6526-f493-4ea8-881a-fda14abc3525-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vqpfb\" (UID: \"41ab6526-f493-4ea8-881a-fda14abc3525\") " pod="openshift-insights/insights-runtime-extractor-vqpfb"
Jul 31 23:46:27.417276 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:27.417172    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 23:46:27.417276 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:27.417234    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/41ab6526-f493-4ea8-881a-fda14abc3525-kube-rbac-proxy-cm podName:41ab6526-f493-4ea8-881a-fda14abc3525 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:28.417219534 +0000 UTC m=+51.137019358 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/41ab6526-f493-4ea8-881a-fda14abc3525-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vqpfb" (UID: "41ab6526-f493-4ea8-881a-fda14abc3525") : configmap "kube-rbac-proxy" not found
Jul 31 23:46:27.417377 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:27.417306    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 23:46:27.417377 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:27.417359    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41ab6526-f493-4ea8-881a-fda14abc3525-insights-runtime-extractor-tls podName:41ab6526-f493-4ea8-881a-fda14abc3525 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:28.417344444 +0000 UTC m=+51.137144273 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/41ab6526-f493-4ea8-881a-fda14abc3525-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vqpfb" (UID: "41ab6526-f493-4ea8-881a-fda14abc3525") : secret "insights-runtime-extractor-tls" not found
Jul 31 23:46:28.118122 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:28.118090    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ld78r_96d86114-06a1-4c32-9d63-c7f4262fa871/dns-node-resolver/0.log"
Jul 31 23:46:28.135951 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:28.135924    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-spblg_c2e50946-aea4-4886-82ec-0bd5a31196cf/console-operator/1.log"
Jul 31 23:46:28.136399 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:28.136382    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-spblg_c2e50946-aea4-4886-82ec-0bd5a31196cf/console-operator/0.log"
Jul 31 23:46:28.136475 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:28.136425    2559 generic.go:356] "Generic (PLEG): container finished" podID="c2e50946-aea4-4886-82ec-0bd5a31196cf" containerID="dd39a0cb3d426d5b546a4be054ecb0201b5d332e804a65b559cd2dcf4d1ac6dd" exitCode=255
Jul 31 23:46:28.136475 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:28.136464    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-spblg" event={"ID":"c2e50946-aea4-4886-82ec-0bd5a31196cf","Type":"ContainerDied","Data":"dd39a0cb3d426d5b546a4be054ecb0201b5d332e804a65b559cd2dcf4d1ac6dd"}
Jul 31 23:46:28.136543 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:28.136534    2559 scope.go:117] "RemoveContainer" containerID="fa47cd048a2b2b31655226440a2ac0570e318922f01260bf33e62f4416662810"
Jul 31 23:46:28.136813 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:28.136797    2559 scope.go:117] "RemoveContainer" containerID="dd39a0cb3d426d5b546a4be054ecb0201b5d332e804a65b559cd2dcf4d1ac6dd"
Jul 31 23:46:28.136979 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:28.136960    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-spblg_openshift-console-operator(c2e50946-aea4-4886-82ec-0bd5a31196cf)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-spblg" podUID="c2e50946-aea4-4886-82ec-0bd5a31196cf"
Jul 31 23:46:28.189672 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:28.189633    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-swnwf"]
Jul 31 23:46:28.207754 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:28.207705    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-swnwf"]
Jul 31 23:46:28.207921 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:28.207846    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-swnwf"
Jul 31 23:46:28.210650 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:28.210620    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 31 23:46:28.210810 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:28.210706    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-p2hsz\""
Jul 31 23:46:28.210810 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:28.210793    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 31 23:46:28.325864 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:28.325828    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6gx5\" (UniqueName: \"kubernetes.io/projected/234c2a7e-69c4-4dff-8d02-8d0568136b0d-kube-api-access-l6gx5\") pod \"migrator-68bc745d87-swnwf\" (UID: \"234c2a7e-69c4-4dff-8d02-8d0568136b0d\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-swnwf"
Jul 31 23:46:28.426609 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:28.426526    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6gx5\" (UniqueName: \"kubernetes.io/projected/234c2a7e-69c4-4dff-8d02-8d0568136b0d-kube-api-access-l6gx5\") pod \"migrator-68bc745d87-swnwf\" (UID: \"234c2a7e-69c4-4dff-8d02-8d0568136b0d\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-swnwf"
Jul 31 23:46:28.426609 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:28.426575    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/41ab6526-f493-4ea8-881a-fda14abc3525-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vqpfb\" (UID: \"41ab6526-f493-4ea8-881a-fda14abc3525\") " pod="openshift-insights/insights-runtime-extractor-vqpfb"
Jul 31 23:46:28.426806 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:28.426619    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/41ab6526-f493-4ea8-881a-fda14abc3525-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vqpfb\" (UID: \"41ab6526-f493-4ea8-881a-fda14abc3525\") " pod="openshift-insights/insights-runtime-extractor-vqpfb"
Jul 31 23:46:28.426806 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:28.426705    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 23:46:28.426806 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:28.426717    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 23:46:28.426806 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:28.426789    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/41ab6526-f493-4ea8-881a-fda14abc3525-kube-rbac-proxy-cm podName:41ab6526-f493-4ea8-881a-fda14abc3525 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:30.426772753 +0000 UTC m=+53.146572577 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/41ab6526-f493-4ea8-881a-fda14abc3525-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vqpfb" (UID: "41ab6526-f493-4ea8-881a-fda14abc3525") : configmap "kube-rbac-proxy" not found
Jul 31 23:46:28.426806 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:28.426805    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41ab6526-f493-4ea8-881a-fda14abc3525-insights-runtime-extractor-tls podName:41ab6526-f493-4ea8-881a-fda14abc3525 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:30.426798566 +0000 UTC m=+53.146598390 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/41ab6526-f493-4ea8-881a-fda14abc3525-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vqpfb" (UID: "41ab6526-f493-4ea8-881a-fda14abc3525") : secret "insights-runtime-extractor-tls" not found
Jul 31 23:46:28.434393 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:28.434360    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6gx5\" (UniqueName: \"kubernetes.io/projected/234c2a7e-69c4-4dff-8d02-8d0568136b0d-kube-api-access-l6gx5\") pod \"migrator-68bc745d87-swnwf\" (UID: \"234c2a7e-69c4-4dff-8d02-8d0568136b0d\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-swnwf"
Jul 31 23:46:28.517543 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:28.517516    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-swnwf"
Jul 31 23:46:28.517802 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:28.517785    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7mgjz_482f5dcb-a14d-426f-93e6-b1ce4c9b89e6/node-ca/0.log"
Jul 31 23:46:28.527159 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:28.527134    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ae80080-0f96-4a21-b13c-87738f73a578-metrics-tls\") pod \"dns-default-fxfvr\" (UID: \"0ae80080-0f96-4a21-b13c-87738f73a578\") " pod="openshift-dns/dns-default-fxfvr"
Jul 31 23:46:28.527278 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:28.527216    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7fb6400-a564-4c03-8381-b76eb7aac58b-registry-tls\") pod \"image-registry-5b46bd6c8c-m7dfb\" (UID: \"c7fb6400-a564-4c03-8381-b76eb7aac58b\") " pod="openshift-image-registry/image-registry-5b46bd6c8c-m7dfb"
Jul 31 23:46:28.527333 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:28.527293    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 23:46:28.527333 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:28.527314    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 31 23:46:28.527333 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:28.527323    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b46bd6c8c-m7dfb: secret "image-registry-tls" not found
Jul 31 23:46:28.527478 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:28.527362    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ae80080-0f96-4a21-b13c-87738f73a578-metrics-tls podName:0ae80080-0f96-4a21-b13c-87738f73a578 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:44.527342104 +0000 UTC m=+67.247141945 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0ae80080-0f96-4a21-b13c-87738f73a578-metrics-tls") pod "dns-default-fxfvr" (UID: "0ae80080-0f96-4a21-b13c-87738f73a578") : secret "dns-default-metrics-tls" not found
Jul 31 23:46:28.527478 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:28.527383    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7fb6400-a564-4c03-8381-b76eb7aac58b-registry-tls podName:c7fb6400-a564-4c03-8381-b76eb7aac58b nodeName:}" failed. No retries permitted until 2026-07-31 23:46:44.527371739 +0000 UTC m=+67.247171569 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c7fb6400-a564-4c03-8381-b76eb7aac58b-registry-tls") pod "image-registry-5b46bd6c8c-m7dfb" (UID: "c7fb6400-a564-4c03-8381-b76eb7aac58b") : secret "image-registry-tls" not found
Jul 31 23:46:28.658158 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:28.658130    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-swnwf"]
Jul 31 23:46:28.660798 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:46:28.660767    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod234c2a7e_69c4_4dff_8d02_8d0568136b0d.slice/crio-f12c1caad5776d322320a1b44ada154a500b73f65f26ded8207591dfb06866b4 WatchSource:0}: Error finding container f12c1caad5776d322320a1b44ada154a500b73f65f26ded8207591dfb06866b4: Status 404 returned error can't find the container with id f12c1caad5776d322320a1b44ada154a500b73f65f26ded8207591dfb06866b4
Jul 31 23:46:28.930262 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:28.930226    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7eeb5980-be60-42c9-b61b-639811f83942-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-fp7hq\" (UID: \"7eeb5980-be60-42c9-b61b-639811f83942\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-fp7hq"
Jul 31 23:46:28.930423 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:28.930379    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 31 23:46:28.930465 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:28.930446    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7eeb5980-be60-42c9-b61b-639811f83942-networking-console-plugin-cert podName:7eeb5980-be60-42c9-b61b-639811f83942 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:36.930429265 +0000 UTC m=+59.650229093 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7eeb5980-be60-42c9-b61b-639811f83942-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-fp7hq" (UID: "7eeb5980-be60-42c9-b61b-639811f83942") : secret "networking-console-plugin-cert" not found
Jul 31 23:46:28.939651 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:28.939625    2559 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-5ccd6b445f-spblg"
Jul 31 23:46:29.141325 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:29.141298    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-spblg_c2e50946-aea4-4886-82ec-0bd5a31196cf/console-operator/1.log"
Jul 31 23:46:29.141774 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:29.141659    2559 scope.go:117] "RemoveContainer" containerID="dd39a0cb3d426d5b546a4be054ecb0201b5d332e804a65b559cd2dcf4d1ac6dd"
Jul 31 23:46:29.141880 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:29.141856    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-spblg_openshift-console-operator(c2e50946-aea4-4886-82ec-0bd5a31196cf)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-spblg" podUID="c2e50946-aea4-4886-82ec-0bd5a31196cf"
Jul 31 23:46:29.142602 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:29.142578    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-swnwf" event={"ID":"234c2a7e-69c4-4dff-8d02-8d0568136b0d","Type":"ContainerStarted","Data":"f12c1caad5776d322320a1b44ada154a500b73f65f26ded8207591dfb06866b4"}
Jul 31 23:46:29.318060 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:29.318028    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9zcz2_1d81c21b-5141-40fb-bb77-b676e9461c82/serve-healthcheck-canary/0.log"
Jul 31 23:46:30.145900 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:30.145876    2559 scope.go:117] "RemoveContainer" containerID="dd39a0cb3d426d5b546a4be054ecb0201b5d332e804a65b559cd2dcf4d1ac6dd"
Jul 31 23:46:30.146334 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:30.146029    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-spblg_openshift-console-operator(c2e50946-aea4-4886-82ec-0bd5a31196cf)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-spblg" podUID="c2e50946-aea4-4886-82ec-0bd5a31196cf"
Jul 31 23:46:30.444119 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:30.444088    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/41ab6526-f493-4ea8-881a-fda14abc3525-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vqpfb\" (UID: \"41ab6526-f493-4ea8-881a-fda14abc3525\") " pod="openshift-insights/insights-runtime-extractor-vqpfb"
Jul 31 23:46:30.444251 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:30.444147    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/41ab6526-f493-4ea8-881a-fda14abc3525-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vqpfb\" (UID: \"41ab6526-f493-4ea8-881a-fda14abc3525\") " pod="openshift-insights/insights-runtime-extractor-vqpfb"
Jul 31 23:46:30.444324 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:30.444250    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 23:46:30.444324 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:30.444304    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41ab6526-f493-4ea8-881a-fda14abc3525-insights-runtime-extractor-tls podName:41ab6526-f493-4ea8-881a-fda14abc3525 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:34.444287296 +0000 UTC m=+57.164087137 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/41ab6526-f493-4ea8-881a-fda14abc3525-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vqpfb" (UID: "41ab6526-f493-4ea8-881a-fda14abc3525") : secret "insights-runtime-extractor-tls" not found
Jul 31 23:46:30.444561 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:30.444544    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/41ab6526-f493-4ea8-881a-fda14abc3525-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vqpfb\" (UID: \"41ab6526-f493-4ea8-881a-fda14abc3525\") " pod="openshift-insights/insights-runtime-extractor-vqpfb"
Jul 31 23:46:30.502970 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:30.502936    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-h62vp"]
Jul 31 23:46:30.524885 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:30.524855    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-h62vp"]
Jul 31 23:46:30.525017 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:30.524914    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-98c88fbb5-h62vp"
Jul 31 23:46:30.527982 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:30.527961    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 31 23:46:30.528130 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:30.527961    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 31 23:46:30.528130 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:30.528008    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-8bgrj\""
Jul 31 23:46:30.529238 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:30.529220    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 31 23:46:30.529238 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:30.529229    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 31 23:46:30.645436 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:30.645402    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtmh7\" (UniqueName: \"kubernetes.io/projected/2c347211-7c73-4d97-914b-ea98511a01db-kube-api-access-gtmh7\") pod \"service-ca-98c88fbb5-h62vp\" (UID: \"2c347211-7c73-4d97-914b-ea98511a01db\") " pod="openshift-service-ca/service-ca-98c88fbb5-h62vp"
Jul 31 23:46:30.645597 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:30.645550    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2c347211-7c73-4d97-914b-ea98511a01db-signing-key\") pod \"service-ca-98c88fbb5-h62vp\" (UID: \"2c347211-7c73-4d97-914b-ea98511a01db\") " pod="openshift-service-ca/service-ca-98c88fbb5-h62vp"
Jul 31 23:46:30.645597 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:30.645591    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2c347211-7c73-4d97-914b-ea98511a01db-signing-cabundle\") pod \"service-ca-98c88fbb5-h62vp\" (UID: \"2c347211-7c73-4d97-914b-ea98511a01db\") " pod="openshift-service-ca/service-ca-98c88fbb5-h62vp"
Jul 31 23:46:30.746320 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:30.746282    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtmh7\" (UniqueName: \"kubernetes.io/projected/2c347211-7c73-4d97-914b-ea98511a01db-kube-api-access-gtmh7\") pod \"service-ca-98c88fbb5-h62vp\" (UID: \"2c347211-7c73-4d97-914b-ea98511a01db\") " pod="openshift-service-ca/service-ca-98c88fbb5-h62vp"
Jul 31 23:46:30.746488 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:30.746391    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2c347211-7c73-4d97-914b-ea98511a01db-signing-key\") pod \"service-ca-98c88fbb5-h62vp\" (UID: \"2c347211-7c73-4d97-914b-ea98511a01db\") " pod="openshift-service-ca/service-ca-98c88fbb5-h62vp"
Jul 31 23:46:30.746488 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:30.746425    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2c347211-7c73-4d97-914b-ea98511a01db-signing-cabundle\") pod \"service-ca-98c88fbb5-h62vp\" (UID: \"2c347211-7c73-4d97-914b-ea98511a01db\") " pod="openshift-service-ca/service-ca-98c88fbb5-h62vp"
Jul 31 23:46:30.747151 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:30.747133    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2c347211-7c73-4d97-914b-ea98511a01db-signing-cabundle\") pod \"service-ca-98c88fbb5-h62vp\" (UID: \"2c347211-7c73-4d97-914b-ea98511a01db\") " pod="openshift-service-ca/service-ca-98c88fbb5-h62vp"
Jul 31 23:46:30.748839 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:30.748817    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2c347211-7c73-4d97-914b-ea98511a01db-signing-key\") pod \"service-ca-98c88fbb5-h62vp\" (UID: \"2c347211-7c73-4d97-914b-ea98511a01db\") " pod="openshift-service-ca/service-ca-98c88fbb5-h62vp"
Jul 31 23:46:30.755414 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:30.755393    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtmh7\" (UniqueName: \"kubernetes.io/projected/2c347211-7c73-4d97-914b-ea98511a01db-kube-api-access-gtmh7\") pod \"service-ca-98c88fbb5-h62vp\" (UID: \"2c347211-7c73-4d97-914b-ea98511a01db\") " pod="openshift-service-ca/service-ca-98c88fbb5-h62vp"
Jul 31 23:46:30.833355 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:30.833324    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-98c88fbb5-h62vp"
Jul 31 23:46:30.978758 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:30.978715    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-h62vp"]
Jul 31 23:46:30.981918 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:46:30.981895    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c347211_7c73_4d97_914b_ea98511a01db.slice/crio-02c7dd24af39179a17f2ea149978fc2e86a81f8efbf013213de4c53c2ee46437 WatchSource:0}: Error finding container 02c7dd24af39179a17f2ea149978fc2e86a81f8efbf013213de4c53c2ee46437: Status 404 returned error can't find the container with id 02c7dd24af39179a17f2ea149978fc2e86a81f8efbf013213de4c53c2ee46437
Jul 31 23:46:31.152387 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:31.152135    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-swnwf" event={"ID":"234c2a7e-69c4-4dff-8d02-8d0568136b0d","Type":"ContainerStarted","Data":"5d86137accb5f7f8c376de329c0c3c537480eaf07ac5252f72243fb540866627"}
Jul 31 23:46:31.152387 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:31.152206    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-swnwf" event={"ID":"234c2a7e-69c4-4dff-8d02-8d0568136b0d","Type":"ContainerStarted","Data":"e5ed5aaec302ed125b814d83a9c937bdae74f8a622e6b0f7b0342dd388c10f2c"}
Jul 31 23:46:31.153756 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:31.153719    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-98c88fbb5-h62vp" event={"ID":"2c347211-7c73-4d97-914b-ea98511a01db","Type":"ContainerStarted","Data":"2722c1f8db21d78b8586871a754b032795bdd2dace30342dc6cd1153374f390f"}
Jul 31 23:46:31.153851 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:31.153776    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-98c88fbb5-h62vp" event={"ID":"2c347211-7c73-4d97-914b-ea98511a01db","Type":"ContainerStarted","Data":"02c7dd24af39179a17f2ea149978fc2e86a81f8efbf013213de4c53c2ee46437"}
Jul 31 23:46:31.173332 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:31.173261    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-swnwf" podStartSLOduration=1.6411183459999998 podStartE2EDuration="3.173245962s" totalImagesPullingTime="1.532127616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:31.17203048 +0000 UTC m=+53.891830337" watchObservedRunningTime="2026-07-31 23:46:31.173245962 +0000 UTC m=+53.893045807"
Jul 31 23:46:31.188929 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:31.188881    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-98c88fbb5-h62vp" podStartSLOduration=1.188867384 podStartE2EDuration="1.188867384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:31.187600302 +0000 UTC m=+53.907400149" watchObservedRunningTime="2026-07-31 23:46:31.188867384 +0000 UTC m=+53.908667230"
Jul 31 23:46:34.383864 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:34.383826    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/13eeae3e-dd1f-4829-8fd8-9a867df2630c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-vknnx\" (UID: \"13eeae3e-dd1f-4829-8fd8-9a867df2630c\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vknnx"
Jul 31 23:46:34.384311 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:34.383981    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 31 23:46:34.384311 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:34.384047    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13eeae3e-dd1f-4829-8fd8-9a867df2630c-cluster-monitoring-operator-tls podName:13eeae3e-dd1f-4829-8fd8-9a867df2630c nodeName:}" failed. No retries permitted until 2026-07-31 23:46:50.384031764 +0000 UTC m=+73.103831593 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/13eeae3e-dd1f-4829-8fd8-9a867df2630c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-vknnx" (UID: "13eeae3e-dd1f-4829-8fd8-9a867df2630c") : secret "cluster-monitoring-operator-tls" not found
Jul 31 23:46:34.485077 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:34.485045    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/41ab6526-f493-4ea8-881a-fda14abc3525-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vqpfb\" (UID: \"41ab6526-f493-4ea8-881a-fda14abc3525\") " pod="openshift-insights/insights-runtime-extractor-vqpfb"
Jul 31 23:46:34.485191 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:34.485084    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c33c7c85-f678-49ff-bcc0-5b1de21c563b-service-ca-bundle\") pod \"router-default-5579c88784-ntwhl\" (UID: \"c33c7c85-f678-49ff-bcc0-5b1de21c563b\") " pod="openshift-ingress/router-default-5579c88784-ntwhl"
Jul 31 23:46:34.485191 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:34.485111    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c33c7c85-f678-49ff-bcc0-5b1de21c563b-metrics-certs\") pod \"router-default-5579c88784-ntwhl\" (UID: \"c33c7c85-f678-49ff-bcc0-5b1de21c563b\") " pod="openshift-ingress/router-default-5579c88784-ntwhl"
Jul 31 23:46:34.485289 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:34.485194    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 23:46:34.485289 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:34.485233    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c33c7c85-f678-49ff-bcc0-5b1de21c563b-service-ca-bundle podName:c33c7c85-f678-49ff-bcc0-5b1de21c563b nodeName:}" failed. No retries permitted until 2026-07-31 23:46:50.485215795 +0000 UTC m=+73.205015643 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c33c7c85-f678-49ff-bcc0-5b1de21c563b-service-ca-bundle") pod "router-default-5579c88784-ntwhl" (UID: "c33c7c85-f678-49ff-bcc0-5b1de21c563b") : configmap references non-existent config key: service-ca.crt
Jul 31 23:46:34.485289 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:34.485251    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41ab6526-f493-4ea8-881a-fda14abc3525-insights-runtime-extractor-tls podName:41ab6526-f493-4ea8-881a-fda14abc3525 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:42.485245234 +0000 UTC m=+65.205045058 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/41ab6526-f493-4ea8-881a-fda14abc3525-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vqpfb" (UID: "41ab6526-f493-4ea8-881a-fda14abc3525") : secret "insights-runtime-extractor-tls" not found
Jul 31 23:46:34.485455 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:34.485324    2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 31 23:46:34.485455 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:34.485401    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c33c7c85-f678-49ff-bcc0-5b1de21c563b-metrics-certs podName:c33c7c85-f678-49ff-bcc0-5b1de21c563b nodeName:}" failed. No retries permitted until 2026-07-31 23:46:50.485382472 +0000 UTC m=+73.205182300 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c33c7c85-f678-49ff-bcc0-5b1de21c563b-metrics-certs") pod "router-default-5579c88784-ntwhl" (UID: "c33c7c85-f678-49ff-bcc0-5b1de21c563b") : secret "router-metrics-certs-default" not found
Jul 31 23:46:34.585979 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:34.585937    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/87ffcd92-302a-4e74-bca9-b64311b2a0b1-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-nspll\" (UID: \"87ffcd92-302a-4e74-bca9-b64311b2a0b1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-nspll"
Jul 31 23:46:34.588416 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:34.588390    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/87ffcd92-302a-4e74-bca9-b64311b2a0b1-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-nspll\" (UID: \"87ffcd92-302a-4e74-bca9-b64311b2a0b1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-nspll"
Jul 31 23:46:34.622738 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:34.622709    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-nspll"
Jul 31 23:46:34.748695 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:34.748660    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-nspll"]
Jul 31 23:46:35.167794 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:35.167733    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-nspll" event={"ID":"87ffcd92-302a-4e74-bca9-b64311b2a0b1","Type":"ContainerStarted","Data":"ae9aa00fcfe2321d62be906eaeaf85c9318947c850a8ace639219921c6cac1cd"}
Jul 31 23:46:36.123629 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:36.123590    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-spblg"
Jul 31 23:46:36.124180 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:36.124029    2559 scope.go:117] "RemoveContainer" containerID="dd39a0cb3d426d5b546a4be054ecb0201b5d332e804a65b559cd2dcf4d1ac6dd"
Jul 31 23:46:36.124238 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:36.124202    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-spblg_openshift-console-operator(c2e50946-aea4-4886-82ec-0bd5a31196cf)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-spblg" podUID="c2e50946-aea4-4886-82ec-0bd5a31196cf"
Jul 31 23:46:37.007277 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:37.007245    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7eeb5980-be60-42c9-b61b-639811f83942-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-fp7hq\" (UID: \"7eeb5980-be60-42c9-b61b-639811f83942\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-fp7hq"
Jul 31 23:46:37.007479 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:37.007400    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 31 23:46:37.007479 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:46:37.007465    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7eeb5980-be60-42c9-b61b-639811f83942-networking-console-plugin-cert podName:7eeb5980-be60-42c9-b61b-639811f83942 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:53.007449932 +0000 UTC m=+75.727249756 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7eeb5980-be60-42c9-b61b-639811f83942-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-fp7hq" (UID: "7eeb5980-be60-42c9-b61b-639811f83942") : secret "networking-console-plugin-cert" not found
Jul 31 23:46:37.175810 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:37.175766    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-nspll" event={"ID":"87ffcd92-302a-4e74-bca9-b64311b2a0b1","Type":"ContainerStarted","Data":"19edf3e9789460a91aa7019e5e54b42b22327b8a4ed379373144fb26e8b51286"}
Jul 31 23:46:37.176269 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:37.175833    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-nspll" event={"ID":"87ffcd92-302a-4e74-bca9-b64311b2a0b1","Type":"ContainerStarted","Data":"82b243ce87dc90ccf44430904ab7152a6238dfcb962404cd9baf5c65b9fa45ae"}
Jul 31 23:46:37.193959 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:37.193902    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-nspll" podStartSLOduration=17.459115514 podStartE2EDuration="19.193883605s" totalImagesPullingTime="1.734768091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:37.192587839 +0000 UTC m=+59.912387689" watchObservedRunningTime="2026-07-31 23:46:37.193883605 +0000 UTC m=+59.913683456"
Jul 31 23:46:38.051294 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:38.051262    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nmsz6"
Jul 31 23:46:42.556594 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:42.556559    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/41ab6526-f493-4ea8-881a-fda14abc3525-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vqpfb\" (UID: \"41ab6526-f493-4ea8-881a-fda14abc3525\") " pod="openshift-insights/insights-runtime-extractor-vqpfb"
Jul 31 23:46:42.558906 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:42.558882    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/41ab6526-f493-4ea8-881a-fda14abc3525-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vqpfb\" (UID: \"41ab6526-f493-4ea8-881a-fda14abc3525\") " pod="openshift-insights/insights-runtime-extractor-vqpfb"
Jul 31 23:46:42.657183 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:42.657151    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06-original-pull-secret\") pod \"global-pull-secret-syncer-jqggp\" (UID: \"e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06\") " pod="kube-system/global-pull-secret-syncer-jqggp"
Jul 31 23:46:42.657183 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:42.657186    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pgnt\" (UniqueName: \"kubernetes.io/projected/64e3d3a1-0422-4faa-a938-a523a0a7b827-kube-api-access-7pgnt\") pod \"network-check-target-s9j95\" (UID: \"64e3d3a1-0422-4faa-a938-a523a0a7b827\") " pod="openshift-network-diagnostics/network-check-target-s9j95"
Jul 31 23:46:42.657360 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:42.657210    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/450206d3-1b10-45ae-b6e9-6575c0d0be94-metrics-certs\") pod \"network-metrics-daemon-bpv44\" (UID: \"450206d3-1b10-45ae-b6e9-6575c0d0be94\") " pod="openshift-multus/network-metrics-daemon-bpv44"
Jul 31 23:46:42.659804 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:42.659764    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pgnt\" (UniqueName: \"kubernetes.io/projected/64e3d3a1-0422-4faa-a938-a523a0a7b827-kube-api-access-7pgnt\") pod \"network-check-target-s9j95\" (UID: \"64e3d3a1-0422-4faa-a938-a523a0a7b827\") " pod="openshift-network-diagnostics/network-check-target-s9j95"
Jul 31 23:46:42.660430 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:42.660412    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 31 23:46:42.660478 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:42.660417    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 31 23:46:42.669545 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:42.669525    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/450206d3-1b10-45ae-b6e9-6575c0d0be94-metrics-certs\") pod \"network-metrics-daemon-bpv44\" (UID: \"450206d3-1b10-45ae-b6e9-6575c0d0be94\") " pod="openshift-multus/network-metrics-daemon-bpv44"
Jul 31 23:46:42.669655 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:42.669638    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06-original-pull-secret\") pod \"global-pull-secret-syncer-jqggp\" (UID: \"e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06\") " pod="kube-system/global-pull-secret-syncer-jqggp"
Jul 31 23:46:42.682991 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:42.682967    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mw9xk\""
Jul 31 23:46:42.690808 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:42.690792    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vqpfb"
Jul 31 23:46:42.793898 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:42.793871    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-45dqd\""
Jul 31 23:46:42.797907 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:42.797885    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jqggp"
Jul 31 23:46:42.802188 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:42.802172    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s9j95"
Jul 31 23:46:42.805721 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:42.805707    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-c2rbb\""
Jul 31 23:46:42.811290 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:42.811271    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vqpfb"]
Jul 31 23:46:42.813585 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:42.813563    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpv44"
Jul 31 23:46:42.815166 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:46:42.815145    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41ab6526_f493_4ea8_881a_fda14abc3525.slice/crio-ace91c770af00a7436d29a9caf5835f09df973a2427a31e8242580059fd16ed6 WatchSource:0}: Error finding container ace91c770af00a7436d29a9caf5835f09df973a2427a31e8242580059fd16ed6: Status 404 returned error can't find the container with id ace91c770af00a7436d29a9caf5835f09df973a2427a31e8242580059fd16ed6
Jul 31 23:46:42.983771 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:42.983717    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bpv44"]
Jul 31 23:46:42.986337 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:46:42.986311    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod450206d3_1b10_45ae_b6e9_6575c0d0be94.slice/crio-e6f4864e77c4e75e77bb9eb697732d1a4f0344b1de0752100026e5626e399cd7 WatchSource:0}: Error finding container e6f4864e77c4e75e77bb9eb697732d1a4f0344b1de0752100026e5626e399cd7: Status 404 returned error can't find the container with id e6f4864e77c4e75e77bb9eb697732d1a4f0344b1de0752100026e5626e399cd7
Jul 31 23:46:43.153536 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:43.153409    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jqggp"]
Jul 31 23:46:43.156872 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:46:43.156837    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode31cdcb9_5ed4_4cc2_94d4_be29c89e8e06.slice/crio-02cf7db8fdfa3ab0024f43ebce6671f057b769a271f50239552b2dbf40045dca WatchSource:0}: Error finding container 02cf7db8fdfa3ab0024f43ebce6671f057b769a271f50239552b2dbf40045dca: Status 404 returned error can't find the container with id 02cf7db8fdfa3ab0024f43ebce6671f057b769a271f50239552b2dbf40045dca
Jul 31 23:46:43.159675 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:43.159648    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-s9j95"]
Jul 31 23:46:43.162964 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:46:43.162935    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64e3d3a1_0422_4faa_a938_a523a0a7b827.slice/crio-e2b5ff17979aa166cd9f2184485057a18e4b9efd269f4b92612db1e6836803d7 WatchSource:0}: Error finding container e2b5ff17979aa166cd9f2184485057a18e4b9efd269f4b92612db1e6836803d7: Status 404 returned error can't find the container with id e2b5ff17979aa166cd9f2184485057a18e4b9efd269f4b92612db1e6836803d7
Jul 31 23:46:43.195498 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:43.195472    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jqggp" event={"ID":"e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06","Type":"ContainerStarted","Data":"02cf7db8fdfa3ab0024f43ebce6671f057b769a271f50239552b2dbf40045dca"}
Jul 31 23:46:43.197195 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:43.197168    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vqpfb" event={"ID":"41ab6526-f493-4ea8-881a-fda14abc3525","Type":"ContainerStarted","Data":"99e268eb4baa25ffff02e735b2a6b3a071fba577c367b0976b2dbb03822e0737"}
Jul 31 23:46:43.197311 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:43.197232    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vqpfb" event={"ID":"41ab6526-f493-4ea8-881a-fda14abc3525","Type":"ContainerStarted","Data":"ace91c770af00a7436d29a9caf5835f09df973a2427a31e8242580059fd16ed6"}
Jul 31 23:46:43.198297 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:43.198271    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bpv44" event={"ID":"450206d3-1b10-45ae-b6e9-6575c0d0be94","Type":"ContainerStarted","Data":"e6f4864e77c4e75e77bb9eb697732d1a4f0344b1de0752100026e5626e399cd7"}
Jul 31 23:46:43.199329 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:43.199310    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-s9j95" event={"ID":"64e3d3a1-0422-4faa-a938-a523a0a7b827","Type":"ContainerStarted","Data":"e2b5ff17979aa166cd9f2184485057a18e4b9efd269f4b92612db1e6836803d7"}
Jul 31 23:46:44.204407 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:44.204378    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-s9j95" event={"ID":"64e3d3a1-0422-4faa-a938-a523a0a7b827","Type":"ContainerStarted","Data":"c12c40549410c38c4669608fbbc677e93786dd657c6ae2e5d05a1a98a81638ed"}
Jul 31 23:46:44.204809 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:44.204478    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-s9j95"
Jul 31 23:46:44.222281 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:44.222243    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-s9j95" podStartSLOduration=66.222227745 podStartE2EDuration="1m6.222227745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:45:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:44.220967529 +0000 UTC m=+66.940767374" watchObservedRunningTime="2026-07-31 23:46:44.222227745 +0000 UTC m=+66.942027603"
Jul 31 23:46:44.575124 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:44.575093    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ae80080-0f96-4a21-b13c-87738f73a578-metrics-tls\") pod \"dns-default-fxfvr\" (UID: \"0ae80080-0f96-4a21-b13c-87738f73a578\") " pod="openshift-dns/dns-default-fxfvr"
Jul 31 23:46:44.575285 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:44.575183    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7fb6400-a564-4c03-8381-b76eb7aac58b-registry-tls\") pod \"image-registry-5b46bd6c8c-m7dfb\" (UID: \"c7fb6400-a564-4c03-8381-b76eb7aac58b\") " pod="openshift-image-registry/image-registry-5b46bd6c8c-m7dfb"
Jul 31 23:46:44.577418 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:44.577391    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ae80080-0f96-4a21-b13c-87738f73a578-metrics-tls\") pod \"dns-default-fxfvr\" (UID: \"0ae80080-0f96-4a21-b13c-87738f73a578\") " pod="openshift-dns/dns-default-fxfvr"
Jul 31 23:46:44.577548 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:44.577499    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7fb6400-a564-4c03-8381-b76eb7aac58b-registry-tls\") pod \"image-registry-5b46bd6c8c-m7dfb\" (UID: \"c7fb6400-a564-4c03-8381-b76eb7aac58b\") " pod="openshift-image-registry/image-registry-5b46bd6c8c-m7dfb"
Jul 31 23:46:44.744256 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:44.744221    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mwlw9\""
Jul 31 23:46:44.752636 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:44.752616    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b46bd6c8c-m7dfb"
Jul 31 23:46:44.772724 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:44.772698    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tss5j\""
Jul 31 23:46:44.780253 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:44.780232    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fxfvr"
Jul 31 23:46:44.953871 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:44.953712    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b46bd6c8c-m7dfb"]
Jul 31 23:46:44.956218 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:46:44.956188    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7fb6400_a564_4c03_8381_b76eb7aac58b.slice/crio-3cca4a546729edc3c8854236430b5c27dadf8bc59dd0cac0e8474f59a4c6e23e WatchSource:0}: Error finding container 3cca4a546729edc3c8854236430b5c27dadf8bc59dd0cac0e8474f59a4c6e23e: Status 404 returned error can't find the container with id 3cca4a546729edc3c8854236430b5c27dadf8bc59dd0cac0e8474f59a4c6e23e
Jul 31 23:46:44.964121 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:44.964097    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fxfvr"]
Jul 31 23:46:44.966457 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:46:44.966436    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ae80080_0f96_4a21_b13c_87738f73a578.slice/crio-0b9fb8520c8f18f9a0d6e0206feb2d77879a035c0602fbc68721380a595dae5d WatchSource:0}: Error finding container 0b9fb8520c8f18f9a0d6e0206feb2d77879a035c0602fbc68721380a595dae5d: Status 404 returned error can't find the container with id 0b9fb8520c8f18f9a0d6e0206feb2d77879a035c0602fbc68721380a595dae5d
Jul 31 23:46:45.209898 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:45.209865    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fxfvr" event={"ID":"0ae80080-0f96-4a21-b13c-87738f73a578","Type":"ContainerStarted","Data":"0b9fb8520c8f18f9a0d6e0206feb2d77879a035c0602fbc68721380a595dae5d"}
Jul 31 23:46:45.211703 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:45.211678    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bpv44" event={"ID":"450206d3-1b10-45ae-b6e9-6575c0d0be94","Type":"ContainerStarted","Data":"0d8c7062bf89edb4775d1d8a1882dae88506a17ae94349117191fc1ae1db1ca7"}
Jul 31 23:46:45.211862 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:45.211726    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bpv44" event={"ID":"450206d3-1b10-45ae-b6e9-6575c0d0be94","Type":"ContainerStarted","Data":"070549b938bf0e65ce16bcb1419d932ff8f58b54a485b0b33cda6c7080de5c04"}
Jul 31 23:46:45.213245 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:45.213220    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b46bd6c8c-m7dfb" event={"ID":"c7fb6400-a564-4c03-8381-b76eb7aac58b","Type":"ContainerStarted","Data":"fc5ae69380ba47967f74fc112114188c3e7a9491a56d0de3e8c88d706e1aa909"}
Jul 31 23:46:45.213358 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:45.213259    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b46bd6c8c-m7dfb" event={"ID":"c7fb6400-a564-4c03-8381-b76eb7aac58b","Type":"ContainerStarted","Data":"3cca4a546729edc3c8854236430b5c27dadf8bc59dd0cac0e8474f59a4c6e23e"}
Jul 31 23:46:45.228169 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:45.228124    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-bpv44" podStartSLOduration=66.112851735 podStartE2EDuration="1m7.228109095s" totalImagesPullingTime="1.11525736s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:45:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:45.227139023 +0000 UTC m=+67.946938874" watchObservedRunningTime="2026-07-31 23:46:45.228109095 +0000 UTC m=+67.947908941"
Jul 31 23:46:45.247555 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:45.247512    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5b46bd6c8c-m7dfb" podStartSLOduration=66.247498433 podStartE2EDuration="1m6.247498433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:45:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:45.246085665 +0000 UTC m=+67.965885510" watchObservedRunningTime="2026-07-31 23:46:45.247498433 +0000 UTC m=+67.967298296"
Jul 31 23:46:46.217605 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:46.217576    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5b46bd6c8c-m7dfb"
Jul 31 23:46:48.225355 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:48.225322    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fxfvr" event={"ID":"0ae80080-0f96-4a21-b13c-87738f73a578","Type":"ContainerStarted","Data":"8cf01d6d2944eabe5b156df2592065f83def8f124bd44a7ed2ce8cd99f6ca274"}
Jul 31 23:46:48.225766 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:48.225371    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fxfvr" event={"ID":"0ae80080-0f96-4a21-b13c-87738f73a578","Type":"ContainerStarted","Data":"08de5387dc97adb4e5e4634309010074a226c88024a23cae61eb632126fe2cc2"}
Jul 31 23:46:48.225766 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:48.225428    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-fxfvr"
Jul 31 23:46:48.247047 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:48.247004    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-fxfvr" podStartSLOduration=34.094431424 podStartE2EDuration="36.246991495s" totalImagesPullingTime="2.152560071s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:48.246347224 +0000 UTC m=+70.966147069" watchObservedRunningTime="2026-07-31 23:46:48.246991495 +0000 UTC m=+70.966791340"
Jul 31 23:46:50.422927 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:50.422872    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/13eeae3e-dd1f-4829-8fd8-9a867df2630c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-vknnx\" (UID: \"13eeae3e-dd1f-4829-8fd8-9a867df2630c\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vknnx"
Jul 31 23:46:50.425246 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:50.425217    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/13eeae3e-dd1f-4829-8fd8-9a867df2630c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-vknnx\" (UID: \"13eeae3e-dd1f-4829-8fd8-9a867df2630c\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vknnx"
Jul 31 23:46:50.523500 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:50.523472    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c33c7c85-f678-49ff-bcc0-5b1de21c563b-service-ca-bundle\") pod \"router-default-5579c88784-ntwhl\" (UID: \"c33c7c85-f678-49ff-bcc0-5b1de21c563b\") " pod="openshift-ingress/router-default-5579c88784-ntwhl"
Jul 31 23:46:50.523648 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:50.523511    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c33c7c85-f678-49ff-bcc0-5b1de21c563b-metrics-certs\") pod \"router-default-5579c88784-ntwhl\" (UID: \"c33c7c85-f678-49ff-bcc0-5b1de21c563b\") " pod="openshift-ingress/router-default-5579c88784-ntwhl"
Jul 31 23:46:50.524785 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:50.524735    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c33c7c85-f678-49ff-bcc0-5b1de21c563b-service-ca-bundle\") pod \"router-default-5579c88784-ntwhl\" (UID: \"c33c7c85-f678-49ff-bcc0-5b1de21c563b\") " pod="openshift-ingress/router-default-5579c88784-ntwhl"
Jul 31 23:46:50.525736 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:50.525716    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c33c7c85-f678-49ff-bcc0-5b1de21c563b-metrics-certs\") pod \"router-default-5579c88784-ntwhl\" (UID: \"c33c7c85-f678-49ff-bcc0-5b1de21c563b\") " pod="openshift-ingress/router-default-5579c88784-ntwhl"
Jul 31 23:46:50.595718 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:50.595692    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-j8lfq\""
Jul 31 23:46:50.603087 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:50.603068    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vknnx"
Jul 31 23:46:50.714382 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:50.714357    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-gqffv\""
Jul 31 23:46:50.722470 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:50.722453    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5579c88784-ntwhl"
Jul 31 23:46:50.727200 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:50.727146    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vknnx"]
Jul 31 23:46:50.730418 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:46:50.730395    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13eeae3e_dd1f_4829_8fd8_9a867df2630c.slice/crio-415d230ca7578dc826654549786722f32b5d2ff9ffb64e1ba9edeec6c509cbbe WatchSource:0}: Error finding container 415d230ca7578dc826654549786722f32b5d2ff9ffb64e1ba9edeec6c509cbbe: Status 404 returned error can't find the container with id 415d230ca7578dc826654549786722f32b5d2ff9ffb64e1ba9edeec6c509cbbe
Jul 31 23:46:50.780945 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:50.780924    2559 scope.go:117] "RemoveContainer" containerID="dd39a0cb3d426d5b546a4be054ecb0201b5d332e804a65b559cd2dcf4d1ac6dd"
Jul 31 23:46:50.855816 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:50.855778    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5579c88784-ntwhl"]
Jul 31 23:46:50.858107 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:46:50.858076    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc33c7c85_f678_49ff_bcc0_5b1de21c563b.slice/crio-421fa66ffd62c9f0a70f1103c397fce39a16685dfcd82b0f591a3a1c63c16254 WatchSource:0}: Error finding container 421fa66ffd62c9f0a70f1103c397fce39a16685dfcd82b0f591a3a1c63c16254: Status 404 returned error can't find the container with id 421fa66ffd62c9f0a70f1103c397fce39a16685dfcd82b0f591a3a1c63c16254
Jul 31 23:46:51.236146 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:51.236109    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5579c88784-ntwhl" event={"ID":"c33c7c85-f678-49ff-bcc0-5b1de21c563b","Type":"ContainerStarted","Data":"fd5811a7e72b619b24ec5d88aa98026d909689791a17833e62c99083f4b2d411"}
Jul 31 23:46:51.236340 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:51.236165    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5579c88784-ntwhl" event={"ID":"c33c7c85-f678-49ff-bcc0-5b1de21c563b","Type":"ContainerStarted","Data":"421fa66ffd62c9f0a70f1103c397fce39a16685dfcd82b0f591a3a1c63c16254"}
Jul 31 23:46:51.237301 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:51.237276    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vknnx" event={"ID":"13eeae3e-dd1f-4829-8fd8-9a867df2630c","Type":"ContainerStarted","Data":"415d230ca7578dc826654549786722f32b5d2ff9ffb64e1ba9edeec6c509cbbe"}
Jul 31 23:46:51.238986 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:51.238970    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-spblg_c2e50946-aea4-4886-82ec-0bd5a31196cf/console-operator/1.log"
Jul 31 23:46:51.239073 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:51.239023    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-spblg" event={"ID":"c2e50946-aea4-4886-82ec-0bd5a31196cf","Type":"ContainerStarted","Data":"bcc2a60c696ee1a8c1e554a9b11e8c10bf41d17aba029879302e078ba00347e6"}
Jul 31 23:46:51.239283 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:51.239258    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-spblg"
Jul 31 23:46:51.259509 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:51.259468    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5579c88784-ntwhl" podStartSLOduration=33.259455732 podStartE2EDuration="33.259455732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:51.257851759 +0000 UTC m=+73.977651604" watchObservedRunningTime="2026-07-31 23:46:51.259455732 +0000 UTC m=+73.979255578"
Jul 31 23:46:51.272722 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:51.272701    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-5ccd6b445f-spblg"
Jul 31 23:46:51.723255 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:51.723218    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5579c88784-ntwhl"
Jul 31 23:46:51.726171 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:51.726146    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5579c88784-ntwhl"
Jul 31 23:46:52.243555 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.243525    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5579c88784-ntwhl"
Jul 31 23:46:52.244835 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.244812    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5579c88784-ntwhl"
Jul 31 23:46:52.562730 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.562703    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c5bbfd5d4-8twdj"]
Jul 31 23:46:52.565062 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.564818    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c5bbfd5d4-8twdj"
Jul 31 23:46:52.568071 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.567904    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 31 23:46:52.568071 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.567923    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 31 23:46:52.568285 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.568262    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 31 23:46:52.568285 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.568284    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77676cd765-gtc7v"]
Jul 31 23:46:52.573039 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.571883    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 31 23:46:52.574663 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.574642    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-658cd69646-vxkhb"]
Jul 31 23:46:52.574826 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.574812    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77676cd765-gtc7v"
Jul 31 23:46:52.577501 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.577479    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-658cd69646-vxkhb"
Jul 31 23:46:52.579498 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.579479    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 31 23:46:52.580300 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.580277    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 31 23:46:52.580403 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.580340    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 31 23:46:52.580403 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.580363    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 31 23:46:52.580882 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.580862    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-j28zq\""
Jul 31 23:46:52.580962 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.580899    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 31 23:46:52.592944 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.592904    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77676cd765-gtc7v"]
Jul 31 23:46:52.594563 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.594539    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c5bbfd5d4-8twdj"]
Jul 31 23:46:52.622252 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.622225    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-658cd69646-vxkhb"]
Jul 31 23:46:52.677659 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.677234    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-77c974ffcf-8znrf"]
Jul 31 23:46:52.684323 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.684302    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-77c974ffcf-8znrf"
Jul 31 23:46:52.690370 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.690348    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 31 23:46:52.690489 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.690394    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 31 23:46:52.690489 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.690437    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-c87jm\""
Jul 31 23:46:52.734760 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.734720    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-77c974ffcf-8znrf"]
Jul 31 23:46:52.743478 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.743442    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/35f64eaf-e8a3-40c7-baf0-4f64830c8347-hub\") pod \"cluster-proxy-proxy-agent-77676cd765-gtc7v\" (UID: \"35f64eaf-e8a3-40c7-baf0-4f64830c8347\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77676cd765-gtc7v"
Jul 31 23:46:52.743615 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.743476    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x56x6\" (UniqueName: \"kubernetes.io/projected/35f64eaf-e8a3-40c7-baf0-4f64830c8347-kube-api-access-x56x6\") pod \"cluster-proxy-proxy-agent-77676cd765-gtc7v\" (UID: \"35f64eaf-e8a3-40c7-baf0-4f64830c8347\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77676cd765-gtc7v"
Jul 31 23:46:52.743615 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.743505    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/34b3dd4c-a6bc-4fbe-b8e4-1a77e64ba06a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-658cd69646-vxkhb\" (UID: \"34b3dd4c-a6bc-4fbe-b8e4-1a77e64ba06a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-658cd69646-vxkhb"
Jul 31 23:46:52.743615 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.743526    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvhwg\" (UniqueName: \"kubernetes.io/projected/b6fd970f-01c9-4689-93fb-2fb84009fe0e-kube-api-access-dvhwg\") pod \"klusterlet-addon-workmgr-5c5bbfd5d4-8twdj\" (UID: \"b6fd970f-01c9-4689-93fb-2fb84009fe0e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c5bbfd5d4-8twdj"
Jul 31 23:46:52.743615 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.743600    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kds87\" (UniqueName: \"kubernetes.io/projected/34b3dd4c-a6bc-4fbe-b8e4-1a77e64ba06a-kube-api-access-kds87\") pod \"managed-serviceaccount-addon-agent-658cd69646-vxkhb\" (UID: \"34b3dd4c-a6bc-4fbe-b8e4-1a77e64ba06a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-658cd69646-vxkhb"
Jul 31 23:46:52.743879 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.743651    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/35f64eaf-e8a3-40c7-baf0-4f64830c8347-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-77676cd765-gtc7v\" (UID: \"35f64eaf-e8a3-40c7-baf0-4f64830c8347\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77676cd765-gtc7v"
Jul 31 23:46:52.743879 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.743696    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/35f64eaf-e8a3-40c7-baf0-4f64830c8347-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-77676cd765-gtc7v\" (UID: \"35f64eaf-e8a3-40c7-baf0-4f64830c8347\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77676cd765-gtc7v"
Jul 31 23:46:52.743879 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.743716    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/35f64eaf-e8a3-40c7-baf0-4f64830c8347-additional-service-ca\") pod \"cluster-proxy-proxy-agent-77676cd765-gtc7v\" (UID: \"35f64eaf-e8a3-40c7-baf0-4f64830c8347\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77676cd765-gtc7v"
Jul 31 23:46:52.743879 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.743773    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b6fd970f-01c9-4689-93fb-2fb84009fe0e-klusterlet-config\") pod \"klusterlet-addon-workmgr-5c5bbfd5d4-8twdj\" (UID: \"b6fd970f-01c9-4689-93fb-2fb84009fe0e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c5bbfd5d4-8twdj"
Jul 31 23:46:52.743879 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.743792    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6fd970f-01c9-4689-93fb-2fb84009fe0e-tmp\") pod \"klusterlet-addon-workmgr-5c5bbfd5d4-8twdj\" (UID: \"b6fd970f-01c9-4689-93fb-2fb84009fe0e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c5bbfd5d4-8twdj"
Jul 31 23:46:52.743879 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.743815    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/35f64eaf-e8a3-40c7-baf0-4f64830c8347-ca\") pod \"cluster-proxy-proxy-agent-77676cd765-gtc7v\" (UID: \"35f64eaf-e8a3-40c7-baf0-4f64830c8347\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77676cd765-gtc7v"
Jul 31 23:46:52.844940 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.844905    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/35f64eaf-e8a3-40c7-baf0-4f64830c8347-hub\") pod \"cluster-proxy-proxy-agent-77676cd765-gtc7v\" (UID: \"35f64eaf-e8a3-40c7-baf0-4f64830c8347\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77676cd765-gtc7v"
Jul 31 23:46:52.845117 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.844958    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x56x6\" (UniqueName: \"kubernetes.io/projected/35f64eaf-e8a3-40c7-baf0-4f64830c8347-kube-api-access-x56x6\") pod \"cluster-proxy-proxy-agent-77676cd765-gtc7v\" (UID: \"35f64eaf-e8a3-40c7-baf0-4f64830c8347\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77676cd765-gtc7v"
Jul 31 23:46:52.845117 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.844993    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/34b3dd4c-a6bc-4fbe-b8e4-1a77e64ba06a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-658cd69646-vxkhb\" (UID: \"34b3dd4c-a6bc-4fbe-b8e4-1a77e64ba06a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-658cd69646-vxkhb"
Jul 31 23:46:52.845117 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.845020    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvhwg\" (UniqueName: \"kubernetes.io/projected/b6fd970f-01c9-4689-93fb-2fb84009fe0e-kube-api-access-dvhwg\") pod \"klusterlet-addon-workmgr-5c5bbfd5d4-8twdj\" (UID: \"b6fd970f-01c9-4689-93fb-2fb84009fe0e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c5bbfd5d4-8twdj"
Jul 31 23:46:52.845117 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.845120    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk954\" (UniqueName: \"kubernetes.io/projected/08e9f54f-4554-4f00-8c74-5d45fc73b17f-kube-api-access-sk954\") pod \"downloads-77c974ffcf-8znrf\" (UID: \"08e9f54f-4554-4f00-8c74-5d45fc73b17f\") " pod="openshift-console/downloads-77c974ffcf-8znrf"
Jul 31 23:46:52.845360 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.845169    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kds87\" (UniqueName: \"kubernetes.io/projected/34b3dd4c-a6bc-4fbe-b8e4-1a77e64ba06a-kube-api-access-kds87\") pod \"managed-serviceaccount-addon-agent-658cd69646-vxkhb\" (UID: \"34b3dd4c-a6bc-4fbe-b8e4-1a77e64ba06a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-658cd69646-vxkhb"
Jul 31 23:46:52.845360 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.845247    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/35f64eaf-e8a3-40c7-baf0-4f64830c8347-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-77676cd765-gtc7v\" (UID: \"35f64eaf-e8a3-40c7-baf0-4f64830c8347\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77676cd765-gtc7v"
Jul 31 23:46:52.845360 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.845285    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/35f64eaf-e8a3-40c7-baf0-4f64830c8347-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-77676cd765-gtc7v\" (UID: \"35f64eaf-e8a3-40c7-baf0-4f64830c8347\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77676cd765-gtc7v"
Jul 31 23:46:52.845360 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.845321    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/35f64eaf-e8a3-40c7-baf0-4f64830c8347-additional-service-ca\") pod \"cluster-proxy-proxy-agent-77676cd765-gtc7v\" (UID: \"35f64eaf-e8a3-40c7-baf0-4f64830c8347\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77676cd765-gtc7v"
Jul 31 23:46:52.845542 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.845363    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b6fd970f-01c9-4689-93fb-2fb84009fe0e-klusterlet-config\") pod \"klusterlet-addon-workmgr-5c5bbfd5d4-8twdj\" (UID: \"b6fd970f-01c9-4689-93fb-2fb84009fe0e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c5bbfd5d4-8twdj"
Jul 31 23:46:52.845542 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.845389    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6fd970f-01c9-4689-93fb-2fb84009fe0e-tmp\") pod \"klusterlet-addon-workmgr-5c5bbfd5d4-8twdj\" (UID: \"b6fd970f-01c9-4689-93fb-2fb84009fe0e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c5bbfd5d4-8twdj"
Jul 31 23:46:52.845542 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.845411    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/35f64eaf-e8a3-40c7-baf0-4f64830c8347-ca\") pod \"cluster-proxy-proxy-agent-77676cd765-gtc7v\" (UID: \"35f64eaf-e8a3-40c7-baf0-4f64830c8347\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77676cd765-gtc7v"
Jul 31 23:46:52.846219 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.846188    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6fd970f-01c9-4689-93fb-2fb84009fe0e-tmp\") pod \"klusterlet-addon-workmgr-5c5bbfd5d4-8twdj\" (UID: \"b6fd970f-01c9-4689-93fb-2fb84009fe0e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c5bbfd5d4-8twdj"
Jul 31 23:46:52.846385 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.846276    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/35f64eaf-e8a3-40c7-baf0-4f64830c8347-additional-service-ca\") pod \"cluster-proxy-proxy-agent-77676cd765-gtc7v\" (UID: \"35f64eaf-e8a3-40c7-baf0-4f64830c8347\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77676cd765-gtc7v"
Jul 31 23:46:52.848169 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.848147    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/35f64eaf-e8a3-40c7-baf0-4f64830c8347-ca\") pod \"cluster-proxy-proxy-agent-77676cd765-gtc7v\" (UID: \"35f64eaf-e8a3-40c7-baf0-4f64830c8347\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77676cd765-gtc7v"
Jul 31 23:46:52.848294 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.848175    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/35f64eaf-e8a3-40c7-baf0-4f64830c8347-hub\") pod \"cluster-proxy-proxy-agent-77676cd765-gtc7v\" (UID: \"35f64eaf-e8a3-40c7-baf0-4f64830c8347\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77676cd765-gtc7v"
Jul 31 23:46:52.849073 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.849051    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/35f64eaf-e8a3-40c7-baf0-4f64830c8347-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-77676cd765-gtc7v\" (UID: \"35f64eaf-e8a3-40c7-baf0-4f64830c8347\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77676cd765-gtc7v"
Jul 31 23:46:52.849161 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.849085    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/34b3dd4c-a6bc-4fbe-b8e4-1a77e64ba06a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-658cd69646-vxkhb\" (UID: \"34b3dd4c-a6bc-4fbe-b8e4-1a77e64ba06a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-658cd69646-vxkhb"
Jul 31 23:46:52.849413 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.849396    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b6fd970f-01c9-4689-93fb-2fb84009fe0e-klusterlet-config\") pod \"klusterlet-addon-workmgr-5c5bbfd5d4-8twdj\" (UID: \"b6fd970f-01c9-4689-93fb-2fb84009fe0e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c5bbfd5d4-8twdj"
Jul 31 23:46:52.849456 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.849414    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/35f64eaf-e8a3-40c7-baf0-4f64830c8347-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-77676cd765-gtc7v\" (UID: \"35f64eaf-e8a3-40c7-baf0-4f64830c8347\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77676cd765-gtc7v"
Jul 31 23:46:52.863109 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.863088    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kds87\" (UniqueName: \"kubernetes.io/projected/34b3dd4c-a6bc-4fbe-b8e4-1a77e64ba06a-kube-api-access-kds87\") pod \"managed-serviceaccount-addon-agent-658cd69646-vxkhb\" (UID: \"34b3dd4c-a6bc-4fbe-b8e4-1a77e64ba06a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-658cd69646-vxkhb"
Jul 31 23:46:52.864763 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.864690    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvhwg\" (UniqueName: \"kubernetes.io/projected/b6fd970f-01c9-4689-93fb-2fb84009fe0e-kube-api-access-dvhwg\") pod \"klusterlet-addon-workmgr-5c5bbfd5d4-8twdj\" (UID: \"b6fd970f-01c9-4689-93fb-2fb84009fe0e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c5bbfd5d4-8twdj"
Jul 31 23:46:52.865316 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.865295    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x56x6\" (UniqueName: \"kubernetes.io/projected/35f64eaf-e8a3-40c7-baf0-4f64830c8347-kube-api-access-x56x6\") pod \"cluster-proxy-proxy-agent-77676cd765-gtc7v\" (UID: \"35f64eaf-e8a3-40c7-baf0-4f64830c8347\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77676cd765-gtc7v"
Jul 31 23:46:52.881984 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.881954    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c5bbfd5d4-8twdj"
Jul 31 23:46:52.893273 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.893253    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77676cd765-gtc7v"
Jul 31 23:46:52.900356 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.900337    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-658cd69646-vxkhb"
Jul 31 23:46:52.946427 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.946395    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk954\" (UniqueName: \"kubernetes.io/projected/08e9f54f-4554-4f00-8c74-5d45fc73b17f-kube-api-access-sk954\") pod \"downloads-77c974ffcf-8znrf\" (UID: \"08e9f54f-4554-4f00-8c74-5d45fc73b17f\") " pod="openshift-console/downloads-77c974ffcf-8znrf"
Jul 31 23:46:52.969798 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:52.969165    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk954\" (UniqueName: \"kubernetes.io/projected/08e9f54f-4554-4f00-8c74-5d45fc73b17f-kube-api-access-sk954\") pod \"downloads-77c974ffcf-8znrf\" (UID: \"08e9f54f-4554-4f00-8c74-5d45fc73b17f\") " pod="openshift-console/downloads-77c974ffcf-8znrf"
Jul 31 23:46:53.005170 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:53.004916    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-77c974ffcf-8znrf"
Jul 31 23:46:53.047393 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:53.047054    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7eeb5980-be60-42c9-b61b-639811f83942-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-fp7hq\" (UID: \"7eeb5980-be60-42c9-b61b-639811f83942\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-fp7hq"
Jul 31 23:46:53.051374 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:53.051312    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7eeb5980-be60-42c9-b61b-639811f83942-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-fp7hq\" (UID: \"7eeb5980-be60-42c9-b61b-639811f83942\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-fp7hq"
Jul 31 23:46:53.104513 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:53.101811    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c5bbfd5d4-8twdj"]
Jul 31 23:46:53.116189 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:46:53.113980    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6fd970f_01c9_4689_93fb_2fb84009fe0e.slice/crio-3e1dbba62ae63900c5dc54920b0805f42f7fcf8b3b52655d288cc1b0ac079b79 WatchSource:0}: Error finding container 3e1dbba62ae63900c5dc54920b0805f42f7fcf8b3b52655d288cc1b0ac079b79: Status 404 returned error can't find the container with id 3e1dbba62ae63900c5dc54920b0805f42f7fcf8b3b52655d288cc1b0ac079b79
Jul 31 23:46:53.163892 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:53.163842    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-658cd69646-vxkhb"]
Jul 31 23:46:53.165688 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:53.165660    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77676cd765-gtc7v"]
Jul 31 23:46:53.167841 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:46:53.167799    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod34b3dd4c_a6bc_4fbe_b8e4_1a77e64ba06a.slice/crio-e3fedd818928622d5224298712920cd3fefec18e3e83ba52aff3d63fc335474c WatchSource:0}: Error finding container e3fedd818928622d5224298712920cd3fefec18e3e83ba52aff3d63fc335474c: Status 404 returned error can't find the container with id e3fedd818928622d5224298712920cd3fefec18e3e83ba52aff3d63fc335474c
Jul 31 23:46:53.168229 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:46:53.168209    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35f64eaf_e8a3_40c7_baf0_4f64830c8347.slice/crio-7c0d17fcade2d4df545e11cec0616a42fea0d5d0853c2302b6a53156f49096ed WatchSource:0}: Error finding container 7c0d17fcade2d4df545e11cec0616a42fea0d5d0853c2302b6a53156f49096ed: Status 404 returned error can't find the container with id 7c0d17fcade2d4df545e11cec0616a42fea0d5d0853c2302b6a53156f49096ed
Jul 31 23:46:53.219834 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:53.219786    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-77c974ffcf-8znrf"]
Jul 31 23:46:53.223402 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:46:53.223376    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08e9f54f_4554_4f00_8c74_5d45fc73b17f.slice/crio-27220782e82af386a6234d3370023f7d7d85ad4c012a234ed2011bba5877b9b0 WatchSource:0}: Error finding container 27220782e82af386a6234d3370023f7d7d85ad4c012a234ed2011bba5877b9b0: Status 404 returned error can't find the container with id 27220782e82af386a6234d3370023f7d7d85ad4c012a234ed2011bba5877b9b0
Jul 31 23:46:53.248132 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:53.248097    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c5bbfd5d4-8twdj" event={"ID":"b6fd970f-01c9-4689-93fb-2fb84009fe0e","Type":"ContainerStarted","Data":"3e1dbba62ae63900c5dc54920b0805f42f7fcf8b3b52655d288cc1b0ac079b79"}
Jul 31 23:46:53.249626 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:53.249602    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vknnx" event={"ID":"13eeae3e-dd1f-4829-8fd8-9a867df2630c","Type":"ContainerStarted","Data":"59889bc81da42c51792f6923200a2a7ef9c4b6373bd437afeab3b082859a61a6"}
Jul 31 23:46:53.250794 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:53.250770    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-658cd69646-vxkhb" event={"ID":"34b3dd4c-a6bc-4fbe-b8e4-1a77e64ba06a","Type":"ContainerStarted","Data":"e3fedd818928622d5224298712920cd3fefec18e3e83ba52aff3d63fc335474c"}
Jul 31 23:46:53.251955 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:53.251929    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77676cd765-gtc7v" event={"ID":"35f64eaf-e8a3-40c7-baf0-4f64830c8347","Type":"ContainerStarted","Data":"7c0d17fcade2d4df545e11cec0616a42fea0d5d0853c2302b6a53156f49096ed"}
Jul 31 23:46:53.253195 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:53.253177    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-77c974ffcf-8znrf" event={"ID":"08e9f54f-4554-4f00-8c74-5d45fc73b17f","Type":"ContainerStarted","Data":"27220782e82af386a6234d3370023f7d7d85ad4c012a234ed2011bba5877b9b0"}
Jul 31 23:46:53.290988 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:53.290899    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vknnx" podStartSLOduration=33.533581526 podStartE2EDuration="35.290883086s" totalImagesPullingTime="1.75730156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:53.289224621 +0000 UTC m=+76.009024478" watchObservedRunningTime="2026-07-31 23:46:53.290883086 +0000 UTC m=+76.010682931"
Jul 31 23:46:53.332458 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:53.332432    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-b9ls8\""
Jul 31 23:46:53.340456 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:53.340432    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-f77f48c6d-fp7hq"
Jul 31 23:46:53.524222 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:53.524190    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-fp7hq"]
Jul 31 23:46:53.527764 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:46:53.527719    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eeb5980_be60_42c9_b61b_639811f83942.slice/crio-125fea738f652c8c6252eef473b25571b63476fd8a6da3d2d48f6abbce7a1d9b WatchSource:0}: Error finding container 125fea738f652c8c6252eef473b25571b63476fd8a6da3d2d48f6abbce7a1d9b: Status 404 returned error can't find the container with id 125fea738f652c8c6252eef473b25571b63476fd8a6da3d2d48f6abbce7a1d9b
Jul 31 23:46:54.276779 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:54.275426    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-f77f48c6d-fp7hq" event={"ID":"7eeb5980-be60-42c9-b61b-639811f83942","Type":"ContainerStarted","Data":"125fea738f652c8c6252eef473b25571b63476fd8a6da3d2d48f6abbce7a1d9b"}
Jul 31 23:46:58.231391 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:46:58.231146    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-fxfvr"
Jul 31 23:47:02.321954 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:02.321504    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77676cd765-gtc7v" event={"ID":"35f64eaf-e8a3-40c7-baf0-4f64830c8347","Type":"ContainerStarted","Data":"b577cad386cada50920617b8b1406cf7dd89aa3a7decaea62679c8c47b92b0d8"}
Jul 31 23:47:02.321954 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:02.321562    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77676cd765-gtc7v" event={"ID":"35f64eaf-e8a3-40c7-baf0-4f64830c8347","Type":"ContainerStarted","Data":"66dc0334f42bfef70dc3cb2b79cc3b9d61d4e7bf478a60ac6a2561671496c90b"}
Jul 31 23:47:02.321954 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:02.321577    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77676cd765-gtc7v" event={"ID":"35f64eaf-e8a3-40c7-baf0-4f64830c8347","Type":"ContainerStarted","Data":"3c960001d5d76dbf8ea23f0071be0766fbd301d6271ccda9c831c0fbe0399710"}
Jul 31 23:47:02.323669 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:02.323626    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-f77f48c6d-fp7hq" event={"ID":"7eeb5980-be60-42c9-b61b-639811f83942","Type":"ContainerStarted","Data":"36f15e02b2c3ef06f3325afec7efc118d24be88fabc86d2f94e880ec340f9f4e"}
Jul 31 23:47:02.325509 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:02.325483    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c5bbfd5d4-8twdj" event={"ID":"b6fd970f-01c9-4689-93fb-2fb84009fe0e","Type":"ContainerStarted","Data":"61a0c668b37532efee89c74195cb93b9174b20728eeba0d30be4f3d8ddb12bc0"}
Jul 31 23:47:02.325685 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:02.325667    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c5bbfd5d4-8twdj"
Jul 31 23:47:02.327125 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:02.327095    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-658cd69646-vxkhb" event={"ID":"34b3dd4c-a6bc-4fbe-b8e4-1a77e64ba06a","Type":"ContainerStarted","Data":"badf5c9d6e35b7f913a0ee2a8c909f2a4fe343932f34e2f7b35d7bce8f1da53a"}
Jul 31 23:47:02.327557 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:02.327535    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c5bbfd5d4-8twdj"
Jul 31 23:47:02.342031 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:02.341989    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77676cd765-gtc7v" podStartSLOduration=2.252661583 podStartE2EDuration="10.341977033s" totalImagesPullingTime="8.08931545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:02.340485071 +0000 UTC m=+85.060284917" watchObservedRunningTime="2026-07-31 23:47:02.341977033 +0000 UTC m=+85.061776878"
Jul 31 23:47:02.381534 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:02.381490    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-658cd69646-vxkhb" podStartSLOduration=2.2855729350000002 podStartE2EDuration="10.381475514s" totalImagesPullingTime="8.095902579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:02.379924532 +0000 UTC m=+85.099724377" watchObservedRunningTime="2026-07-31 23:47:02.381475514 +0000 UTC m=+85.101275359"
Jul 31 23:47:02.420163 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:02.420115    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-f77f48c6d-fp7hq" podStartSLOduration=33.691601084 podStartE2EDuration="41.420102347s" totalImagesPullingTime="7.728501263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:02.419642638 +0000 UTC m=+85.139442487" watchObservedRunningTime="2026-07-31 23:47:02.420102347 +0000 UTC m=+85.139902193"
Jul 31 23:47:03.806011 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:03.805955    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c5bbfd5d4-8twdj" podStartSLOduration=3.6342008359999998 podStartE2EDuration="11.805937488s" totalImagesPullingTime="8.171736652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:02.442057028 +0000 UTC m=+85.161856876" watchObservedRunningTime="2026-07-31 23:47:03.805937488 +0000 UTC m=+86.525737334"
Jul 31 23:47:03.806395 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:03.806352    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-6thpn"]
Jul 31 23:47:03.811939 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:03.811908    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6thpn"
Jul 31 23:47:03.816462 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:03.816436    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-f4bpr\""
Jul 31 23:47:03.816668 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:03.816652    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 31 23:47:03.816850 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:03.816835    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 31 23:47:03.817102 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:03.817085    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 31 23:47:03.817641 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:03.817625    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 31 23:47:03.839837 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:03.839795    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d3f711a4-1b8e-4e4a-b23e-ff0282a73571-node-exporter-tls\") pod \"node-exporter-6thpn\" (UID: \"d3f711a4-1b8e-4e4a-b23e-ff0282a73571\") " pod="openshift-monitoring/node-exporter-6thpn"
Jul 31 23:47:03.839951 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:03.839877    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d3f711a4-1b8e-4e4a-b23e-ff0282a73571-metrics-client-ca\") pod \"node-exporter-6thpn\" (UID: \"d3f711a4-1b8e-4e4a-b23e-ff0282a73571\") " pod="openshift-monitoring/node-exporter-6thpn"
Jul 31 23:47:03.839951 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:03.839920    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3f711a4-1b8e-4e4a-b23e-ff0282a73571-sys\") pod \"node-exporter-6thpn\" (UID: \"d3f711a4-1b8e-4e4a-b23e-ff0282a73571\") " pod="openshift-monitoring/node-exporter-6thpn"
Jul 31 23:47:03.839951 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:03.839941    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d3f711a4-1b8e-4e4a-b23e-ff0282a73571-node-exporter-accelerators-collector-config\") pod \"node-exporter-6thpn\" (UID: \"d3f711a4-1b8e-4e4a-b23e-ff0282a73571\") " pod="openshift-monitoring/node-exporter-6thpn"
Jul 31 23:47:03.840120 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:03.839966    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz7cn\" (UniqueName: \"kubernetes.io/projected/d3f711a4-1b8e-4e4a-b23e-ff0282a73571-kube-api-access-lz7cn\") pod \"node-exporter-6thpn\" (UID: \"d3f711a4-1b8e-4e4a-b23e-ff0282a73571\") " pod="openshift-monitoring/node-exporter-6thpn"
Jul 31 23:47:03.840120 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:03.840002    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d3f711a4-1b8e-4e4a-b23e-ff0282a73571-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6thpn\" (UID: \"d3f711a4-1b8e-4e4a-b23e-ff0282a73571\") " pod="openshift-monitoring/node-exporter-6thpn"
Jul 31 23:47:03.840120 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:03.840058    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d3f711a4-1b8e-4e4a-b23e-ff0282a73571-node-exporter-textfile\") pod \"node-exporter-6thpn\" (UID: \"d3f711a4-1b8e-4e4a-b23e-ff0282a73571\") " pod="openshift-monitoring/node-exporter-6thpn"
Jul 31 23:47:03.840120 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:03.840082    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d3f711a4-1b8e-4e4a-b23e-ff0282a73571-node-exporter-wtmp\") pod \"node-exporter-6thpn\" (UID: \"d3f711a4-1b8e-4e4a-b23e-ff0282a73571\") " pod="openshift-monitoring/node-exporter-6thpn"
Jul 31 23:47:03.840311 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:03.840119    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d3f711a4-1b8e-4e4a-b23e-ff0282a73571-root\") pod \"node-exporter-6thpn\" (UID: \"d3f711a4-1b8e-4e4a-b23e-ff0282a73571\") " pod="openshift-monitoring/node-exporter-6thpn"
Jul 31 23:47:03.941068 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:03.941033    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3f711a4-1b8e-4e4a-b23e-ff0282a73571-sys\") pod \"node-exporter-6thpn\" (UID: \"d3f711a4-1b8e-4e4a-b23e-ff0282a73571\") " pod="openshift-monitoring/node-exporter-6thpn"
Jul 31 23:47:03.941261 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:03.941079    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d3f711a4-1b8e-4e4a-b23e-ff0282a73571-node-exporter-accelerators-collector-config\") pod \"node-exporter-6thpn\" (UID: \"d3f711a4-1b8e-4e4a-b23e-ff0282a73571\") " pod="openshift-monitoring/node-exporter-6thpn"
Jul 31 23:47:03.941261 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:03.941105    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz7cn\" (UniqueName: \"kubernetes.io/projected/d3f711a4-1b8e-4e4a-b23e-ff0282a73571-kube-api-access-lz7cn\") pod \"node-exporter-6thpn\" (UID: \"d3f711a4-1b8e-4e4a-b23e-ff0282a73571\") " pod="openshift-monitoring/node-exporter-6thpn"
Jul 31 23:47:03.941261 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:03.941209    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d3f711a4-1b8e-4e4a-b23e-ff0282a73571-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6thpn\" (UID: \"d3f711a4-1b8e-4e4a-b23e-ff0282a73571\") " pod="openshift-monitoring/node-exporter-6thpn"
Jul 31 23:47:03.941261 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:03.941253    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d3f711a4-1b8e-4e4a-b23e-ff0282a73571-node-exporter-textfile\") pod \"node-exporter-6thpn\" (UID: \"d3f711a4-1b8e-4e4a-b23e-ff0282a73571\") " pod="openshift-monitoring/node-exporter-6thpn"
Jul 31 23:47:03.941477 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:03.941274    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d3f711a4-1b8e-4e4a-b23e-ff0282a73571-node-exporter-wtmp\") pod \"node-exporter-6thpn\" (UID: \"d3f711a4-1b8e-4e4a-b23e-ff0282a73571\") " pod="openshift-monitoring/node-exporter-6thpn"
Jul 31 23:47:03.941477 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:03.941310    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d3f711a4-1b8e-4e4a-b23e-ff0282a73571-root\") pod \"node-exporter-6thpn\" (UID: \"d3f711a4-1b8e-4e4a-b23e-ff0282a73571\") " pod="openshift-monitoring/node-exporter-6thpn"
Jul 31 23:47:03.941477 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:03.941379    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d3f711a4-1b8e-4e4a-b23e-ff0282a73571-node-exporter-tls\") pod \"node-exporter-6thpn\" (UID: \"d3f711a4-1b8e-4e4a-b23e-ff0282a73571\") " pod="openshift-monitoring/node-exporter-6thpn"
Jul 31 23:47:03.941477 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:03.941453    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d3f711a4-1b8e-4e4a-b23e-ff0282a73571-metrics-client-ca\") pod \"node-exporter-6thpn\" (UID: \"d3f711a4-1b8e-4e4a-b23e-ff0282a73571\") " pod="openshift-monitoring/node-exporter-6thpn"
Jul 31 23:47:03.942144 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:03.942118    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d3f711a4-1b8e-4e4a-b23e-ff0282a73571-metrics-client-ca\") pod \"node-exporter-6thpn\" (UID: \"d3f711a4-1b8e-4e4a-b23e-ff0282a73571\") " pod="openshift-monitoring/node-exporter-6thpn"
Jul 31 23:47:03.942270 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:03.942196    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3f711a4-1b8e-4e4a-b23e-ff0282a73571-sys\") pod \"node-exporter-6thpn\" (UID: \"d3f711a4-1b8e-4e4a-b23e-ff0282a73571\") " pod="openshift-monitoring/node-exporter-6thpn"
Jul 31 23:47:03.942590 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:03.942570    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d3f711a4-1b8e-4e4a-b23e-ff0282a73571-node-exporter-accelerators-collector-config\") pod \"node-exporter-6thpn\" (UID: \"d3f711a4-1b8e-4e4a-b23e-ff0282a73571\") " pod="openshift-monitoring/node-exporter-6thpn"
Jul 31 23:47:03.943275 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:03.942845    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d3f711a4-1b8e-4e4a-b23e-ff0282a73571-root\") pod \"node-exporter-6thpn\" (UID: \"d3f711a4-1b8e-4e4a-b23e-ff0282a73571\") " pod="openshift-monitoring/node-exporter-6thpn"
Jul 31 23:47:03.943275 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:47:03.942885    2559 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 31 23:47:03.943275 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:03.942896    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d3f711a4-1b8e-4e4a-b23e-ff0282a73571-node-exporter-wtmp\") pod \"node-exporter-6thpn\" (UID: \"d3f711a4-1b8e-4e4a-b23e-ff0282a73571\") " pod="openshift-monitoring/node-exporter-6thpn"
Jul 31 23:47:03.943275 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:47:03.942946    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3f711a4-1b8e-4e4a-b23e-ff0282a73571-node-exporter-tls podName:d3f711a4-1b8e-4e4a-b23e-ff0282a73571 nodeName:}" failed. No retries permitted until 2026-07-31 23:47:04.442927101 +0000 UTC m=+87.162726927 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/d3f711a4-1b8e-4e4a-b23e-ff0282a73571-node-exporter-tls") pod "node-exporter-6thpn" (UID: "d3f711a4-1b8e-4e4a-b23e-ff0282a73571") : secret "node-exporter-tls" not found
Jul 31 23:47:03.943275 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:03.943207    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d3f711a4-1b8e-4e4a-b23e-ff0282a73571-node-exporter-textfile\") pod \"node-exporter-6thpn\" (UID: \"d3f711a4-1b8e-4e4a-b23e-ff0282a73571\") " pod="openshift-monitoring/node-exporter-6thpn"
Jul 31 23:47:03.945772 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:03.945737    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d3f711a4-1b8e-4e4a-b23e-ff0282a73571-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6thpn\" (UID: \"d3f711a4-1b8e-4e4a-b23e-ff0282a73571\") " pod="openshift-monitoring/node-exporter-6thpn"
Jul 31 23:47:03.956066 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:03.955593    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz7cn\" (UniqueName: \"kubernetes.io/projected/d3f711a4-1b8e-4e4a-b23e-ff0282a73571-kube-api-access-lz7cn\") pod \"node-exporter-6thpn\" (UID: \"d3f711a4-1b8e-4e4a-b23e-ff0282a73571\") " pod="openshift-monitoring/node-exporter-6thpn"
Jul 31 23:47:04.446194 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:04.446162    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d3f711a4-1b8e-4e4a-b23e-ff0282a73571-node-exporter-tls\") pod \"node-exporter-6thpn\" (UID: \"d3f711a4-1b8e-4e4a-b23e-ff0282a73571\") " pod="openshift-monitoring/node-exporter-6thpn"
Jul 31 23:47:04.449065 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:04.449035    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d3f711a4-1b8e-4e4a-b23e-ff0282a73571-node-exporter-tls\") pod \"node-exporter-6thpn\" (UID: \"d3f711a4-1b8e-4e4a-b23e-ff0282a73571\") " pod="openshift-monitoring/node-exporter-6thpn"
Jul 31 23:47:04.728482 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:04.728175    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6thpn"
Jul 31 23:47:04.745903 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:47:04.745678    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3f711a4_1b8e_4e4a_b23e_ff0282a73571.slice/crio-b8fe8120d0258ae9b77f49bb84f971fa7cbfded426a620cb255acb2b74d5b26a WatchSource:0}: Error finding container b8fe8120d0258ae9b77f49bb84f971fa7cbfded426a620cb255acb2b74d5b26a: Status 404 returned error can't find the container with id b8fe8120d0258ae9b77f49bb84f971fa7cbfded426a620cb255acb2b74d5b26a
Jul 31 23:47:04.758604 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:04.758578    2559 patch_prober.go:28] interesting pod/image-registry-5b46bd6c8c-m7dfb container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 31 23:47:04.758715 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:04.758625    2559 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5b46bd6c8c-m7dfb" podUID="c7fb6400-a564-4c03-8381-b76eb7aac58b" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 23:47:05.342272 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:05.342201    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6thpn" event={"ID":"d3f711a4-1b8e-4e4a-b23e-ff0282a73571","Type":"ContainerStarted","Data":"b8fe8120d0258ae9b77f49bb84f971fa7cbfded426a620cb255acb2b74d5b26a"}
Jul 31 23:47:07.225996 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:07.225965    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5b46bd6c8c-m7dfb"
Jul 31 23:47:13.377309 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:13.377265    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-77c974ffcf-8znrf" event={"ID":"08e9f54f-4554-4f00-8c74-5d45fc73b17f","Type":"ContainerStarted","Data":"6356d3ed57ac5aeb116fb1d12370a3a4a89f52d047698070cd642316310f10c7"}
Jul 31 23:47:13.377793 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:13.377368    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-77c974ffcf-8znrf"
Jul 31 23:47:13.379505 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:13.379473    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vqpfb" event={"ID":"41ab6526-f493-4ea8-881a-fda14abc3525","Type":"ContainerStarted","Data":"6a343bfb05f1037ff93060aaa00a3eaf7993fc065f215f0833794baed182088e"}
Jul 31 23:47:13.381422 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:13.381399    2559 generic.go:356] "Generic (PLEG): container finished" podID="d3f711a4-1b8e-4e4a-b23e-ff0282a73571" containerID="42fc7267acc5380c2f5a345090eb0701c80d4ec1c0582491c3efbfced85fc856" exitCode=0
Jul 31 23:47:13.381523 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:13.381466    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6thpn" event={"ID":"d3f711a4-1b8e-4e4a-b23e-ff0282a73571","Type":"ContainerDied","Data":"42fc7267acc5380c2f5a345090eb0701c80d4ec1c0582491c3efbfced85fc856"}
Jul 31 23:47:13.391040 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:13.391019    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-77c974ffcf-8znrf"
Jul 31 23:47:13.401099 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:13.401053    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-77c974ffcf-8znrf" podStartSLOduration=1.585225929 podStartE2EDuration="21.401039365s" totalImagesPullingTime="19.815813436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:13.399434369 +0000 UTC m=+96.119234216" watchObservedRunningTime="2026-07-31 23:47:13.401039365 +0000 UTC m=+96.120839216"
Jul 31 23:47:14.389977 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:14.389935    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6thpn" event={"ID":"d3f711a4-1b8e-4e4a-b23e-ff0282a73571","Type":"ContainerStarted","Data":"9b5ac9f848c478ebc9aa106c5226bab0cab8819273dcdbaa517f3f296c1a99d1"}
Jul 31 23:47:14.390569 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:14.389992    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6thpn" event={"ID":"d3f711a4-1b8e-4e4a-b23e-ff0282a73571","Type":"ContainerStarted","Data":"65401aec3c8322cbd431c61e70ff0f84eafaa91c825a1e2ed2e401fff600cb7c"}
Jul 31 23:47:14.412084 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:14.412031    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-6thpn" podStartSLOduration=3.058414316 podStartE2EDuration="11.412015078s" totalImagesPullingTime="8.281066579s" totalInitContainerRuntime="72.534183ms" isStatefulPod=true podCreationTimestamp="2026-07-31 23:47:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:14.410938357 +0000 UTC m=+97.130738239" watchObservedRunningTime="2026-07-31 23:47:14.412015078 +0000 UTC m=+97.131814969"
Jul 31 23:47:15.217018 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:15.216985    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-s9j95"
Jul 31 23:47:15.300382 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:15.300354    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5b46bd6c8c-m7dfb"]
Jul 31 23:47:16.401973 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:16.401930    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vqpfb" event={"ID":"41ab6526-f493-4ea8-881a-fda14abc3525","Type":"ContainerStarted","Data":"ae3aef4fa06b4e0f7a77a26c33a32c288dfbee20788fbe5f59c144f0a65058dc"}
Jul 31 23:47:16.439015 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:16.438955    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-vqpfb" podStartSLOduration=18.130208232 podStartE2EDuration="50.438934842s" totalImagesPullingTime="32.30872661s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:16.437800261 +0000 UTC m=+99.157600108" watchObservedRunningTime="2026-07-31 23:47:16.438934842 +0000 UTC m=+99.158734689"
Jul 31 23:47:31.470508 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:31.470466    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jqggp" event={"ID":"e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06","Type":"ContainerStarted","Data":"d2597b62f46c3515c06fbd2491dc4ed8af81d348f54dab923aeb4aabad60e16b"}
Jul 31 23:47:31.486514 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:31.486465    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-jqggp" podStartSLOduration=65.054524839 podStartE2EDuration="1m52.486449986s" totalImagesPullingTime="47.431925147s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:45:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:31.485336697 +0000 UTC m=+114.205136555" watchObservedRunningTime="2026-07-31 23:47:31.486449986 +0000 UTC m=+114.206249832"
Jul 31 23:47:36.491909 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:36.491872    2559 generic.go:356] "Generic (PLEG): container finished" podID="967d9bc9-7bc0-496f-b18b-3b7b95cab4f2" containerID="911c915918bf4ae954e2381bb34c993830904fd65926be6e5c1e0e329afb7fc0" exitCode=0
Jul 31 23:47:36.492248 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:36.491947    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-xjqzb" event={"ID":"967d9bc9-7bc0-496f-b18b-3b7b95cab4f2","Type":"ContainerDied","Data":"911c915918bf4ae954e2381bb34c993830904fd65926be6e5c1e0e329afb7fc0"}
Jul 31 23:47:36.492311 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:36.492287    2559 scope.go:117] "RemoveContainer" containerID="911c915918bf4ae954e2381bb34c993830904fd65926be6e5c1e0e329afb7fc0"
Jul 31 23:47:37.497695 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:37.497662    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-xjqzb" event={"ID":"967d9bc9-7bc0-496f-b18b-3b7b95cab4f2","Type":"ContainerStarted","Data":"ee12b7b9125b65b2e02377dcb441f865b6fca2daac8066beb4dfcdc12929eaed"}
Jul 31 23:47:40.327263 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:40.327196    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5b46bd6c8c-m7dfb" podUID="c7fb6400-a564-4c03-8381-b76eb7aac58b" containerName="registry" containerID="cri-o://fc5ae69380ba47967f74fc112114188c3e7a9491a56d0de3e8c88d706e1aa909" gracePeriod=30
Jul 31 23:47:40.511195 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:40.511162    2559 generic.go:356] "Generic (PLEG): container finished" podID="c7fb6400-a564-4c03-8381-b76eb7aac58b" containerID="fc5ae69380ba47967f74fc112114188c3e7a9491a56d0de3e8c88d706e1aa909" exitCode=0
Jul 31 23:47:40.511341 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:40.511245    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b46bd6c8c-m7dfb" event={"ID":"c7fb6400-a564-4c03-8381-b76eb7aac58b","Type":"ContainerDied","Data":"fc5ae69380ba47967f74fc112114188c3e7a9491a56d0de3e8c88d706e1aa909"}
Jul 31 23:47:40.618465 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:40.618432    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b46bd6c8c-m7dfb"
Jul 31 23:47:40.765472 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:40.765440    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c7fb6400-a564-4c03-8381-b76eb7aac58b-bound-sa-token\") pod \"c7fb6400-a564-4c03-8381-b76eb7aac58b\" (UID: \"c7fb6400-a564-4c03-8381-b76eb7aac58b\") "
Jul 31 23:47:40.765624 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:40.765486    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c7fb6400-a564-4c03-8381-b76eb7aac58b-installation-pull-secrets\") pod \"c7fb6400-a564-4c03-8381-b76eb7aac58b\" (UID: \"c7fb6400-a564-4c03-8381-b76eb7aac58b\") "
Jul 31 23:47:40.765624 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:40.765519    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c7fb6400-a564-4c03-8381-b76eb7aac58b-image-registry-private-configuration\") pod \"c7fb6400-a564-4c03-8381-b76eb7aac58b\" (UID: \"c7fb6400-a564-4c03-8381-b76eb7aac58b\") "
Jul 31 23:47:40.765624 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:40.765574    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c7fb6400-a564-4c03-8381-b76eb7aac58b-trusted-ca\") pod \"c7fb6400-a564-4c03-8381-b76eb7aac58b\" (UID: \"c7fb6400-a564-4c03-8381-b76eb7aac58b\") "
Jul 31 23:47:40.765781 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:40.765627    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7fb6400-a564-4c03-8381-b76eb7aac58b-registry-tls\") pod \"c7fb6400-a564-4c03-8381-b76eb7aac58b\" (UID: \"c7fb6400-a564-4c03-8381-b76eb7aac58b\") "
Jul 31 23:47:40.765781 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:40.765652    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hc9r\" (UniqueName: \"kubernetes.io/projected/c7fb6400-a564-4c03-8381-b76eb7aac58b-kube-api-access-7hc9r\") pod \"c7fb6400-a564-4c03-8381-b76eb7aac58b\" (UID: \"c7fb6400-a564-4c03-8381-b76eb7aac58b\") "
Jul 31 23:47:40.765781 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:40.765689    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c7fb6400-a564-4c03-8381-b76eb7aac58b-ca-trust-extracted\") pod \"c7fb6400-a564-4c03-8381-b76eb7aac58b\" (UID: \"c7fb6400-a564-4c03-8381-b76eb7aac58b\") "
Jul 31 23:47:40.765781 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:40.765718    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c7fb6400-a564-4c03-8381-b76eb7aac58b-registry-certificates\") pod \"c7fb6400-a564-4c03-8381-b76eb7aac58b\" (UID: \"c7fb6400-a564-4c03-8381-b76eb7aac58b\") "
Jul 31 23:47:40.766325 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:40.766247    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7fb6400-a564-4c03-8381-b76eb7aac58b-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "c7fb6400-a564-4c03-8381-b76eb7aac58b" (UID: "c7fb6400-a564-4c03-8381-b76eb7aac58b"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:47:40.766458 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:40.766390    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7fb6400-a564-4c03-8381-b76eb7aac58b-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "c7fb6400-a564-4c03-8381-b76eb7aac58b" (UID: "c7fb6400-a564-4c03-8381-b76eb7aac58b"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:47:40.768195 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:40.768162    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7fb6400-a564-4c03-8381-b76eb7aac58b-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "c7fb6400-a564-4c03-8381-b76eb7aac58b" (UID: "c7fb6400-a564-4c03-8381-b76eb7aac58b"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:47:40.768325 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:40.768252    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7fb6400-a564-4c03-8381-b76eb7aac58b-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "c7fb6400-a564-4c03-8381-b76eb7aac58b" (UID: "c7fb6400-a564-4c03-8381-b76eb7aac58b"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:47:40.768400 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:40.768337    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7fb6400-a564-4c03-8381-b76eb7aac58b-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "c7fb6400-a564-4c03-8381-b76eb7aac58b" (UID: "c7fb6400-a564-4c03-8381-b76eb7aac58b"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:47:40.768524 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:40.768498    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7fb6400-a564-4c03-8381-b76eb7aac58b-kube-api-access-7hc9r" (OuterVolumeSpecName: "kube-api-access-7hc9r") pod "c7fb6400-a564-4c03-8381-b76eb7aac58b" (UID: "c7fb6400-a564-4c03-8381-b76eb7aac58b"). InnerVolumeSpecName "kube-api-access-7hc9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:47:40.768575 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:40.768556    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7fb6400-a564-4c03-8381-b76eb7aac58b-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "c7fb6400-a564-4c03-8381-b76eb7aac58b" (UID: "c7fb6400-a564-4c03-8381-b76eb7aac58b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:47:40.773863 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:40.773822    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7fb6400-a564-4c03-8381-b76eb7aac58b-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "c7fb6400-a564-4c03-8381-b76eb7aac58b" (UID: "c7fb6400-a564-4c03-8381-b76eb7aac58b"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 23:47:40.867253 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:40.867178    2559 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c7fb6400-a564-4c03-8381-b76eb7aac58b-bound-sa-token\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Jul 31 23:47:40.867253 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:40.867202    2559 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c7fb6400-a564-4c03-8381-b76eb7aac58b-installation-pull-secrets\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Jul 31 23:47:40.867253 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:40.867212    2559 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c7fb6400-a564-4c03-8381-b76eb7aac58b-image-registry-private-configuration\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Jul 31 23:47:40.867253 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:40.867222    2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c7fb6400-a564-4c03-8381-b76eb7aac58b-trusted-ca\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Jul 31 23:47:40.867253 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:40.867230    2559 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7fb6400-a564-4c03-8381-b76eb7aac58b-registry-tls\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Jul 31 23:47:40.867253 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:40.867237    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hc9r\" (UniqueName: \"kubernetes.io/projected/c7fb6400-a564-4c03-8381-b76eb7aac58b-kube-api-access-7hc9r\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Jul 31 23:47:40.867253 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:40.867245    2559 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c7fb6400-a564-4c03-8381-b76eb7aac58b-ca-trust-extracted\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Jul 31 23:47:40.867253 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:40.867254    2559 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c7fb6400-a564-4c03-8381-b76eb7aac58b-registry-certificates\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Jul 31 23:47:41.520893 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:41.520863    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b46bd6c8c-m7dfb"
Jul 31 23:47:41.520893 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:41.520872    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b46bd6c8c-m7dfb" event={"ID":"c7fb6400-a564-4c03-8381-b76eb7aac58b","Type":"ContainerDied","Data":"3cca4a546729edc3c8854236430b5c27dadf8bc59dd0cac0e8474f59a4c6e23e"}
Jul 31 23:47:41.521329 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:41.520961    2559 scope.go:117] "RemoveContainer" containerID="fc5ae69380ba47967f74fc112114188c3e7a9491a56d0de3e8c88d706e1aa909"
Jul 31 23:47:41.542314 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:41.542294    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5b46bd6c8c-m7dfb"]
Jul 31 23:47:41.546101 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:41.546082    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5b46bd6c8c-m7dfb"]
Jul 31 23:47:41.786034 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:41.785960    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7fb6400-a564-4c03-8381-b76eb7aac58b" path="/var/lib/kubelet/pods/c7fb6400-a564-4c03-8381-b76eb7aac58b/volumes"
Jul 31 23:47:56.585499 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:56.585473    2559 generic.go:356] "Generic (PLEG): container finished" podID="e709e147-60bb-46dc-be7d-3d5269dbcd8a" containerID="d42b65c68555a9e2e0d3a9a65b04e733fb7e42cd0a865dee327ccea4529a618f" exitCode=0
Jul 31 23:47:56.585846 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:56.585545    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-slttz" event={"ID":"e709e147-60bb-46dc-be7d-3d5269dbcd8a","Type":"ContainerDied","Data":"d42b65c68555a9e2e0d3a9a65b04e733fb7e42cd0a865dee327ccea4529a618f"}
Jul 31 23:47:56.585897 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:56.585884    2559 scope.go:117] "RemoveContainer" containerID="d42b65c68555a9e2e0d3a9a65b04e733fb7e42cd0a865dee327ccea4529a618f"
Jul 31 23:47:57.591340 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:47:57.591305    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-slttz" event={"ID":"e709e147-60bb-46dc-be7d-3d5269dbcd8a","Type":"ContainerStarted","Data":"3c926b00d8954dc12c07cf989042b3c686ea5db8ca0748071b67f09e1818b568"}
Jul 31 23:48:02.611201 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:48:02.611164    2559 generic.go:356] "Generic (PLEG): container finished" podID="ee2a1542-5293-4a53-a469-b2dc3317b711" containerID="7e4989fed288d9a79992c1e3d91742b32bf330563bc308f763cc16301b3a6940" exitCode=0
Jul 31 23:48:02.611201 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:48:02.611203    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-wvqkv" event={"ID":"ee2a1542-5293-4a53-a469-b2dc3317b711","Type":"ContainerDied","Data":"7e4989fed288d9a79992c1e3d91742b32bf330563bc308f763cc16301b3a6940"}
Jul 31 23:48:02.611629 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:48:02.611497    2559 scope.go:117] "RemoveContainer" containerID="7e4989fed288d9a79992c1e3d91742b32bf330563bc308f763cc16301b3a6940"
Jul 31 23:48:03.616469 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:48:03.616436    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-wvqkv" event={"ID":"ee2a1542-5293-4a53-a469-b2dc3317b711","Type":"ContainerStarted","Data":"a005dad8eb118aab013a15cd027d93c5d889b148be65b52706c6fa1cd1d0f334"}
Jul 31 23:50:37.690807 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:50:37.690779    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-spblg_c2e50946-aea4-4886-82ec-0bd5a31196cf/console-operator/1.log"
Jul 31 23:50:37.697213 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:50:37.697191    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-spblg_c2e50946-aea4-4886-82ec-0bd5a31196cf/console-operator/1.log"
Jul 31 23:50:37.699825 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:50:37.699807    2559 kubelet.go:1648] "Image garbage collection succeeded"
Jul 31 23:51:51.106309 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:51:51.106274    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6cb8c69f9-2q8vn"]
Jul 31 23:51:51.106756 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:51:51.106563    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7fb6400-a564-4c03-8381-b76eb7aac58b" containerName="registry"
Jul 31 23:51:51.106756 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:51:51.106573    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7fb6400-a564-4c03-8381-b76eb7aac58b" containerName="registry"
Jul 31 23:51:51.106756 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:51:51.106631    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7fb6400-a564-4c03-8381-b76eb7aac58b" containerName="registry"
Jul 31 23:51:51.108723 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:51:51.108707    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6cb8c69f9-2q8vn"
Jul 31 23:51:51.112730 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:51:51.112708    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 31 23:51:51.114030 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:51:51.113997    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 31 23:51:51.114333 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:51:51.114309    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 31 23:51:51.114401 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:51:51.114361    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-tlh7w\""
Jul 31 23:51:51.114659 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:51:51.114640    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 31 23:51:51.120703 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:51:51.120678    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 31 23:51:51.135356 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:51:51.135333    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6cb8c69f9-2q8vn"]
Jul 31 23:51:51.169333 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:51:51.169305    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da8cc73b-8926-4a78-8fde-6647144e1781-cert\") pod \"lws-controller-manager-6cb8c69f9-2q8vn\" (UID: \"da8cc73b-8926-4a78-8fde-6647144e1781\") " pod="openshift-lws-operator/lws-controller-manager-6cb8c69f9-2q8vn"
Jul 31 23:51:51.169440 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:51:51.169340    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggdpd\" (UniqueName: \"kubernetes.io/projected/da8cc73b-8926-4a78-8fde-6647144e1781-kube-api-access-ggdpd\") pod \"lws-controller-manager-6cb8c69f9-2q8vn\" (UID: \"da8cc73b-8926-4a78-8fde-6647144e1781\") " pod="openshift-lws-operator/lws-controller-manager-6cb8c69f9-2q8vn"
Jul 31 23:51:51.169440 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:51:51.169379    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/da8cc73b-8926-4a78-8fde-6647144e1781-metrics-cert\") pod \"lws-controller-manager-6cb8c69f9-2q8vn\" (UID: \"da8cc73b-8926-4a78-8fde-6647144e1781\") " pod="openshift-lws-operator/lws-controller-manager-6cb8c69f9-2q8vn"
Jul 31 23:51:51.169554 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:51:51.169472    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/da8cc73b-8926-4a78-8fde-6647144e1781-manager-config\") pod \"lws-controller-manager-6cb8c69f9-2q8vn\" (UID: \"da8cc73b-8926-4a78-8fde-6647144e1781\") " pod="openshift-lws-operator/lws-controller-manager-6cb8c69f9-2q8vn"
Jul 31 23:51:51.270347 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:51:51.270315    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da8cc73b-8926-4a78-8fde-6647144e1781-cert\") pod \"lws-controller-manager-6cb8c69f9-2q8vn\" (UID: \"da8cc73b-8926-4a78-8fde-6647144e1781\") " pod="openshift-lws-operator/lws-controller-manager-6cb8c69f9-2q8vn"
Jul 31 23:51:51.270511 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:51:51.270352    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggdpd\" (UniqueName: \"kubernetes.io/projected/da8cc73b-8926-4a78-8fde-6647144e1781-kube-api-access-ggdpd\") pod \"lws-controller-manager-6cb8c69f9-2q8vn\" (UID: \"da8cc73b-8926-4a78-8fde-6647144e1781\") " pod="openshift-lws-operator/lws-controller-manager-6cb8c69f9-2q8vn"
Jul 31 23:51:51.270511 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:51:51.270385    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/da8cc73b-8926-4a78-8fde-6647144e1781-metrics-cert\") pod \"lws-controller-manager-6cb8c69f9-2q8vn\" (UID: \"da8cc73b-8926-4a78-8fde-6647144e1781\") " pod="openshift-lws-operator/lws-controller-manager-6cb8c69f9-2q8vn"
Jul 31 23:51:51.270511 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:51:51.270437    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/da8cc73b-8926-4a78-8fde-6647144e1781-manager-config\") pod \"lws-controller-manager-6cb8c69f9-2q8vn\" (UID: \"da8cc73b-8926-4a78-8fde-6647144e1781\") " pod="openshift-lws-operator/lws-controller-manager-6cb8c69f9-2q8vn"
Jul 31 23:51:51.271137 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:51:51.271117    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/da8cc73b-8926-4a78-8fde-6647144e1781-manager-config\") pod \"lws-controller-manager-6cb8c69f9-2q8vn\" (UID: \"da8cc73b-8926-4a78-8fde-6647144e1781\") " pod="openshift-lws-operator/lws-controller-manager-6cb8c69f9-2q8vn"
Jul 31 23:51:51.272731 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:51:51.272710    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da8cc73b-8926-4a78-8fde-6647144e1781-cert\") pod \"lws-controller-manager-6cb8c69f9-2q8vn\" (UID: \"da8cc73b-8926-4a78-8fde-6647144e1781\") " pod="openshift-lws-operator/lws-controller-manager-6cb8c69f9-2q8vn"
Jul 31 23:51:51.272830 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:51:51.272792    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/da8cc73b-8926-4a78-8fde-6647144e1781-metrics-cert\") pod \"lws-controller-manager-6cb8c69f9-2q8vn\" (UID: \"da8cc73b-8926-4a78-8fde-6647144e1781\") " pod="openshift-lws-operator/lws-controller-manager-6cb8c69f9-2q8vn"
Jul 31 23:51:51.286136 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:51:51.286113    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggdpd\" (UniqueName: \"kubernetes.io/projected/da8cc73b-8926-4a78-8fde-6647144e1781-kube-api-access-ggdpd\") pod \"lws-controller-manager-6cb8c69f9-2q8vn\" (UID: \"da8cc73b-8926-4a78-8fde-6647144e1781\") " pod="openshift-lws-operator/lws-controller-manager-6cb8c69f9-2q8vn"
Jul 31 23:51:51.419249 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:51:51.419159    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6cb8c69f9-2q8vn"
Jul 31 23:51:51.568586 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:51:51.568562    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6cb8c69f9-2q8vn"]
Jul 31 23:51:51.570512 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:51:51.570483    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda8cc73b_8926_4a78_8fde_6647144e1781.slice/crio-f2071a0f2a16fb3a80e1d5492b3b6b102a39949a6ca55244dd1f69f637eb34e9 WatchSource:0}: Error finding container f2071a0f2a16fb3a80e1d5492b3b6b102a39949a6ca55244dd1f69f637eb34e9: Status 404 returned error can't find the container with id f2071a0f2a16fb3a80e1d5492b3b6b102a39949a6ca55244dd1f69f637eb34e9
Jul 31 23:51:51.572460 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:51:51.572442    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 23:51:52.479421 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:51:52.479374    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6cb8c69f9-2q8vn" event={"ID":"da8cc73b-8926-4a78-8fde-6647144e1781","Type":"ContainerStarted","Data":"f2071a0f2a16fb3a80e1d5492b3b6b102a39949a6ca55244dd1f69f637eb34e9"}
Jul 31 23:51:54.489400 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:51:54.489364    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6cb8c69f9-2q8vn" event={"ID":"da8cc73b-8926-4a78-8fde-6647144e1781","Type":"ContainerStarted","Data":"9fb669e57dc08a968a6b59b469113ac789e48031e56d3165fdc9397bd29f3dc3"}
Jul 31 23:51:54.489802 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:51:54.489427    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6cb8c69f9-2q8vn"
Jul 31 23:51:54.521880 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:51:54.521837    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6cb8c69f9-2q8vn" podStartSLOduration=1.055129016 podStartE2EDuration="3.521822786s" totalImagesPullingTime="2.46669377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:51:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:51:54.521250764 +0000 UTC m=+377.241050610" watchObservedRunningTime="2026-07-31 23:51:54.521822786 +0000 UTC m=+377.241622632"
Jul 31 23:52:05.495261 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:52:05.495231    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6cb8c69f9-2q8vn"
Jul 31 23:53:13.679910 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:53:13.679826    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-4pvmb"]
Jul 31 23:53:13.682987 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:53:13.682971    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-4pvmb"
Jul 31 23:53:13.685536 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:53:13.685504    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 31 23:53:13.685663 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:53:13.685585    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 31 23:53:13.685663 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:53:13.685585    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 31 23:53:13.685800 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:53:13.685593    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-wd7gf\""
Jul 31 23:53:13.691479 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:53:13.691459    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-4pvmb"]
Jul 31 23:53:13.722445 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:53:13.722423    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-4pvmb"]
Jul 31 23:53:13.810978 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:53:13.810954    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkp94\" (UniqueName: \"kubernetes.io/projected/12c3ad76-0e8b-4c7a-9402-f56d9519d156-kube-api-access-wkp94\") pod \"limitador-limitador-85d9db4d5d-4pvmb\" (UID: \"12c3ad76-0e8b-4c7a-9402-f56d9519d156\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-4pvmb"
Jul 31 23:53:13.811089 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:53:13.811008    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/12c3ad76-0e8b-4c7a-9402-f56d9519d156-config-file\") pod \"limitador-limitador-85d9db4d5d-4pvmb\" (UID: \"12c3ad76-0e8b-4c7a-9402-f56d9519d156\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-4pvmb"
Jul 31 23:53:13.911413 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:53:13.911383    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/12c3ad76-0e8b-4c7a-9402-f56d9519d156-config-file\") pod \"limitador-limitador-85d9db4d5d-4pvmb\" (UID: \"12c3ad76-0e8b-4c7a-9402-f56d9519d156\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-4pvmb"
Jul 31 23:53:13.911542 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:53:13.911448    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkp94\" (UniqueName: \"kubernetes.io/projected/12c3ad76-0e8b-4c7a-9402-f56d9519d156-kube-api-access-wkp94\") pod \"limitador-limitador-85d9db4d5d-4pvmb\" (UID: \"12c3ad76-0e8b-4c7a-9402-f56d9519d156\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-4pvmb"
Jul 31 23:53:13.912019 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:53:13.911999    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/12c3ad76-0e8b-4c7a-9402-f56d9519d156-config-file\") pod \"limitador-limitador-85d9db4d5d-4pvmb\" (UID: \"12c3ad76-0e8b-4c7a-9402-f56d9519d156\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-4pvmb"
Jul 31 23:53:13.919584 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:53:13.919562    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkp94\" (UniqueName: \"kubernetes.io/projected/12c3ad76-0e8b-4c7a-9402-f56d9519d156-kube-api-access-wkp94\") pod \"limitador-limitador-85d9db4d5d-4pvmb\" (UID: \"12c3ad76-0e8b-4c7a-9402-f56d9519d156\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-4pvmb"
Jul 31 23:53:13.993513 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:53:13.993482    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-4pvmb"
Jul 31 23:53:14.120680 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:53:14.120654    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-4pvmb"]
Jul 31 23:53:14.123068 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:53:14.123038    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12c3ad76_0e8b_4c7a_9402_f56d9519d156.slice/crio-26fbafdc4ffec1c7d6c357d061a6c20767487492a9aeddcfdcafeb27905d24b9 WatchSource:0}: Error finding container 26fbafdc4ffec1c7d6c357d061a6c20767487492a9aeddcfdcafeb27905d24b9: Status 404 returned error can't find the container with id 26fbafdc4ffec1c7d6c357d061a6c20767487492a9aeddcfdcafeb27905d24b9
Jul 31 23:53:14.791245 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:53:14.791196    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-4pvmb" event={"ID":"12c3ad76-0e8b-4c7a-9402-f56d9519d156","Type":"ContainerStarted","Data":"26fbafdc4ffec1c7d6c357d061a6c20767487492a9aeddcfdcafeb27905d24b9"}
Jul 31 23:53:16.802207 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:53:16.802170    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-4pvmb" event={"ID":"12c3ad76-0e8b-4c7a-9402-f56d9519d156","Type":"ContainerStarted","Data":"6277e6a971569b968137a7939c55bce4b599e95488e02d65e6ece60d3c08ee66"}
Jul 31 23:53:16.802680 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:53:16.802338    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-4pvmb"
Jul 31 23:53:16.818447 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:53:16.818406    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-85d9db4d5d-4pvmb" podStartSLOduration=1.605013108 podStartE2EDuration="3.818392986s" totalImagesPullingTime="2.213379878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:53:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:53:16.817019663 +0000 UTC m=+459.536819510" watchObservedRunningTime="2026-07-31 23:53:16.818392986 +0000 UTC m=+459.538192831"
Jul 31 23:53:27.806411 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:53:27.806380    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-4pvmb"
Jul 31 23:55:37.718429 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:55:37.718397    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-spblg_c2e50946-aea4-4886-82ec-0bd5a31196cf/console-operator/1.log"
Jul 31 23:55:37.721163 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:55:37.721138    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-spblg_c2e50946-aea4-4886-82ec-0bd5a31196cf/console-operator/1.log"
Jul 31 23:55:38.825673 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:55:38.825638    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-t7mpc"]
Jul 31 23:55:38.828925 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:55:38.828908    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-t7mpc"
Jul 31 23:55:38.831567 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:55:38.831544    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 31 23:55:38.831697 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:55:38.831613    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 31 23:55:38.832626 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:55:38.832608    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-wdd7c\""
Jul 31 23:55:38.832707 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:55:38.832637    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 31 23:55:38.836078 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:55:38.836055    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-t7mpc"]
Jul 31 23:55:38.942867 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:55:38.942835    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9whg\" (UniqueName: \"kubernetes.io/projected/e3f7c45f-1bee-41f1-8880-4820a748f7a7-kube-api-access-w9whg\") pod \"s3-init-t7mpc\" (UID: \"e3f7c45f-1bee-41f1-8880-4820a748f7a7\") " pod="kserve/s3-init-t7mpc"
Jul 31 23:55:38.943030 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:55:38.942894    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3f7c45f-1bee-41f1-8880-4820a748f7a7-model-cache\") pod \"s3-init-t7mpc\" (UID: \"e3f7c45f-1bee-41f1-8880-4820a748f7a7\") " pod="kserve/s3-init-t7mpc"
Jul 31 23:55:39.043811 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:55:39.043780    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9whg\" (UniqueName: \"kubernetes.io/projected/e3f7c45f-1bee-41f1-8880-4820a748f7a7-kube-api-access-w9whg\") pod \"s3-init-t7mpc\" (UID: \"e3f7c45f-1bee-41f1-8880-4820a748f7a7\") " pod="kserve/s3-init-t7mpc"
Jul 31 23:55:39.043977 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:55:39.043824    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3f7c45f-1bee-41f1-8880-4820a748f7a7-model-cache\") pod \"s3-init-t7mpc\" (UID: \"e3f7c45f-1bee-41f1-8880-4820a748f7a7\") " pod="kserve/s3-init-t7mpc"
Jul 31 23:55:39.044140 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:55:39.044123    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3f7c45f-1bee-41f1-8880-4820a748f7a7-model-cache\") pod \"s3-init-t7mpc\" (UID: \"e3f7c45f-1bee-41f1-8880-4820a748f7a7\") " pod="kserve/s3-init-t7mpc"
Jul 31 23:55:39.051833 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:55:39.051804    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9whg\" (UniqueName: \"kubernetes.io/projected/e3f7c45f-1bee-41f1-8880-4820a748f7a7-kube-api-access-w9whg\") pod \"s3-init-t7mpc\" (UID: \"e3f7c45f-1bee-41f1-8880-4820a748f7a7\") " pod="kserve/s3-init-t7mpc"
Jul 31 23:55:39.138430 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:55:39.138357    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-t7mpc"
Jul 31 23:55:39.267164 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:55:39.267065    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-t7mpc"]
Jul 31 23:55:39.269702 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:55:39.269674    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3f7c45f_1bee_41f1_8880_4820a748f7a7.slice/crio-3b6563ff74402fefb56c0955c2cd741feb76530cfdd0d8861bd75d84e7efc000 WatchSource:0}: Error finding container 3b6563ff74402fefb56c0955c2cd741feb76530cfdd0d8861bd75d84e7efc000: Status 404 returned error can't find the container with id 3b6563ff74402fefb56c0955c2cd741feb76530cfdd0d8861bd75d84e7efc000
Jul 31 23:55:39.352074 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:55:39.352044    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-t7mpc" event={"ID":"e3f7c45f-1bee-41f1-8880-4820a748f7a7","Type":"ContainerStarted","Data":"3b6563ff74402fefb56c0955c2cd741feb76530cfdd0d8861bd75d84e7efc000"}
Jul 31 23:55:43.374628 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:55:43.374590    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-t7mpc" event={"ID":"e3f7c45f-1bee-41f1-8880-4820a748f7a7","Type":"ContainerStarted","Data":"eb4cd7f33bfcd598fb9fe6aaa584c47865f5186630094a2b0dc5cfa2919c0d1b"}
Jul 31 23:55:44.380408 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:55:44.380374    2559 generic.go:356] "Generic (PLEG): container finished" podID="e3f7c45f-1bee-41f1-8880-4820a748f7a7" containerID="eb4cd7f33bfcd598fb9fe6aaa584c47865f5186630094a2b0dc5cfa2919c0d1b" exitCode=0
Jul 31 23:55:44.380816 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:55:44.380426    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-t7mpc" event={"ID":"e3f7c45f-1bee-41f1-8880-4820a748f7a7","Type":"ContainerDied","Data":"eb4cd7f33bfcd598fb9fe6aaa584c47865f5186630094a2b0dc5cfa2919c0d1b"}
Jul 31 23:55:49.406716 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:55:49.406672    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-t7mpc" event={"ID":"e3f7c45f-1bee-41f1-8880-4820a748f7a7","Type":"ContainerStarted","Data":"c235274c99146a3e414c0dfbdcd660026e1768dd9d5ebc48110cd0cf7b098dc6"}
Jul 31 23:55:49.428472 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:55:49.423846    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-t7mpc" podStartSLOduration=2.2015711 podStartE2EDuration="11.423818331s" totalImagesPullingTime="7.710749396s" totalInitContainerRuntime="1.511497835s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:55:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:55:49.422234677 +0000 UTC m=+612.142034559" watchObservedRunningTime="2026-07-31 23:55:49.423818331 +0000 UTC m=+612.143618179"
Jul 31 23:56:53.683947 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:56:53.683912    2559 generic.go:356] "Generic (PLEG): container finished" podID="e3f7c45f-1bee-41f1-8880-4820a748f7a7" containerID="c235274c99146a3e414c0dfbdcd660026e1768dd9d5ebc48110cd0cf7b098dc6" exitCode=0
Jul 31 23:56:53.684326 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:56:53.683980    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-t7mpc" event={"ID":"e3f7c45f-1bee-41f1-8880-4820a748f7a7","Type":"ContainerDied","Data":"c235274c99146a3e414c0dfbdcd660026e1768dd9d5ebc48110cd0cf7b098dc6"}
Jul 31 23:56:54.814801 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:56:54.814778    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-t7mpc"
Jul 31 23:56:54.952056 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:56:54.952027    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9whg\" (UniqueName: \"kubernetes.io/projected/e3f7c45f-1bee-41f1-8880-4820a748f7a7-kube-api-access-w9whg\") pod \"e3f7c45f-1bee-41f1-8880-4820a748f7a7\" (UID: \"e3f7c45f-1bee-41f1-8880-4820a748f7a7\") "
Jul 31 23:56:54.952213 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:56:54.952066    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3f7c45f-1bee-41f1-8880-4820a748f7a7-model-cache\") pod \"e3f7c45f-1bee-41f1-8880-4820a748f7a7\" (UID: \"e3f7c45f-1bee-41f1-8880-4820a748f7a7\") "
Jul 31 23:56:54.954282 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:56:54.954248    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3f7c45f-1bee-41f1-8880-4820a748f7a7-kube-api-access-w9whg" (OuterVolumeSpecName: "kube-api-access-w9whg") pod "e3f7c45f-1bee-41f1-8880-4820a748f7a7" (UID: "e3f7c45f-1bee-41f1-8880-4820a748f7a7"). InnerVolumeSpecName "kube-api-access-w9whg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:56:54.971308 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:56:54.971279    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f7c45f-1bee-41f1-8880-4820a748f7a7-model-cache" (OuterVolumeSpecName: "model-cache") pod "e3f7c45f-1bee-41f1-8880-4820a748f7a7" (UID: "e3f7c45f-1bee-41f1-8880-4820a748f7a7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 23:56:55.052819 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:56:55.052790    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9whg\" (UniqueName: \"kubernetes.io/projected/e3f7c45f-1bee-41f1-8880-4820a748f7a7-kube-api-access-w9whg\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Jul 31 23:56:55.052819 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:56:55.052817    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3f7c45f-1bee-41f1-8880-4820a748f7a7-model-cache\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Jul 31 23:56:55.693655 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:56:55.693628    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-t7mpc"
Jul 31 23:56:55.693829 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:56:55.693631    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-t7mpc" event={"ID":"e3f7c45f-1bee-41f1-8880-4820a748f7a7","Type":"ContainerDied","Data":"3b6563ff74402fefb56c0955c2cd741feb76530cfdd0d8861bd75d84e7efc000"}
Jul 31 23:56:55.693829 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:56:55.693737    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b6563ff74402fefb56c0955c2cd741feb76530cfdd0d8861bd75d84e7efc000"
Jul 31 23:57:14.520587 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:57:14.520546    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 31 23:57:14.521078 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:57:14.520995    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3f7c45f-1bee-41f1-8880-4820a748f7a7" containerName="download-hf-model"
Jul 31 23:57:14.521078 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:57:14.521015    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3f7c45f-1bee-41f1-8880-4820a748f7a7" containerName="download-hf-model"
Jul 31 23:57:14.521078 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:57:14.521041    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3f7c45f-1bee-41f1-8880-4820a748f7a7" containerName="s3-init"
Jul 31 23:57:14.521078 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:57:14.521049    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3f7c45f-1bee-41f1-8880-4820a748f7a7" containerName="s3-init"
Jul 31 23:57:14.521305 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:57:14.521148    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3f7c45f-1bee-41f1-8880-4820a748f7a7" containerName="s3-init"
Jul 31 23:57:14.524345 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:57:14.524323    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 23:57:14.528093 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:57:14.528067    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Jul 31 23:57:14.528233 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:57:14.528111    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Jul 31 23:57:14.528233 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:57:14.528128    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Jul 31 23:57:14.528233 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:57:14.528068    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-sv6zn\""
Jul 31 23:57:14.528233 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:57:14.528070    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-zbgmz\""
Jul 31 23:57:14.528233 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:57:14.528191    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Jul 31 23:57:14.528469 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:57:14.528375    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Jul 31 23:57:14.536515 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:57:14.536492    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 31 23:57:14.604263 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:57:14.604238    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7f2b302-5771-4b68-b6cd-e41c4959b146-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e7f2b302-5771-4b68-b6cd-e41c4959b146\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 23:57:14.604401 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:57:14.604269    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7f2b302-5771-4b68-b6cd-e41c4959b146-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e7f2b302-5771-4b68-b6cd-e41c4959b146\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 23:57:14.604401 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:57:14.604298    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7f2b302-5771-4b68-b6cd-e41c4959b146-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e7f2b302-5771-4b68-b6cd-e41c4959b146\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 23:57:14.604401 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:57:14.604318    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7f2b302-5771-4b68-b6cd-e41c4959b146-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e7f2b302-5771-4b68-b6cd-e41c4959b146\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 23:57:14.604401 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:57:14.604390    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7f2b302-5771-4b68-b6cd-e41c4959b146-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e7f2b302-5771-4b68-b6cd-e41c4959b146\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 23:57:14.604561 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:57:14.604411    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7f2b302-5771-4b68-b6cd-e41c4959b146-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e7f2b302-5771-4b68-b6cd-e41c4959b146\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 23:57:14.604561 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:57:14.604430    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5jjw\" (UniqueName: \"kubernetes.io/projected/e7f2b302-5771-4b68-b6cd-e41c4959b146-kube-api-access-n5jjw\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e7f2b302-5771-4b68-b6cd-e41c4959b146\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 23:57:14.604561 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:57:14.604446    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7f2b302-5771-4b68-b6cd-e41c4959b146-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e7f2b302-5771-4b68-b6cd-e41c4959b146\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 23:57:14.704894 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:57:14.704853    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7f2b302-5771-4b68-b6cd-e41c4959b146-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e7f2b302-5771-4b68-b6cd-e41c4959b146\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 23:57:14.704894 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:57:14.704898    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7f2b302-5771-4b68-b6cd-e41c4959b146-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e7f2b302-5771-4b68-b6cd-e41c4959b146\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 23:57:14.705121 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:57:14.704933    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7f2b302-5771-4b68-b6cd-e41c4959b146-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e7f2b302-5771-4b68-b6cd-e41c4959b146\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 23:57:14.705121 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:57:14.704953    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7f2b302-5771-4b68-b6cd-e41c4959b146-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e7f2b302-5771-4b68-b6cd-e41c4959b146\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 23:57:14.705121 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:57:14.704995    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7f2b302-5771-4b68-b6cd-e41c4959b146-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e7f2b302-5771-4b68-b6cd-e41c4959b146\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 23:57:14.705121 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:57:14.705012    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7f2b302-5771-4b68-b6cd-e41c4959b146-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e7f2b302-5771-4b68-b6cd-e41c4959b146\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 23:57:14.705121 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:57:14.705029    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5jjw\" (UniqueName: \"kubernetes.io/projected/e7f2b302-5771-4b68-b6cd-e41c4959b146-kube-api-access-n5jjw\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e7f2b302-5771-4b68-b6cd-e41c4959b146\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 23:57:14.705121 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:57:14.705102    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7f2b302-5771-4b68-b6cd-e41c4959b146-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e7f2b302-5771-4b68-b6cd-e41c4959b146\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 23:57:14.705358 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:57:14.705289    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7f2b302-5771-4b68-b6cd-e41c4959b146-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e7f2b302-5771-4b68-b6cd-e41c4959b146\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 23:57:14.705586 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:57:14.705563    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7f2b302-5771-4b68-b6cd-e41c4959b146-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e7f2b302-5771-4b68-b6cd-e41c4959b146\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 23:57:14.705656 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:57:14.705597    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7f2b302-5771-4b68-b6cd-e41c4959b146-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e7f2b302-5771-4b68-b6cd-e41c4959b146\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 23:57:14.705656 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:57:14.705626    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7f2b302-5771-4b68-b6cd-e41c4959b146-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e7f2b302-5771-4b68-b6cd-e41c4959b146\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 23:57:14.705719 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:57:14.705691    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7f2b302-5771-4b68-b6cd-e41c4959b146-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e7f2b302-5771-4b68-b6cd-e41c4959b146\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 23:57:14.707340 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:57:14.707314    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7f2b302-5771-4b68-b6cd-e41c4959b146-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e7f2b302-5771-4b68-b6cd-e41c4959b146\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 23:57:14.707658 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:57:14.707629    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7f2b302-5771-4b68-b6cd-e41c4959b146-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e7f2b302-5771-4b68-b6cd-e41c4959b146\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 23:57:14.720107 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:57:14.720081    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5jjw\" (UniqueName: \"kubernetes.io/projected/e7f2b302-5771-4b68-b6cd-e41c4959b146-kube-api-access-n5jjw\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e7f2b302-5771-4b68-b6cd-e41c4959b146\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 23:57:14.834718 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:57:14.834643    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 23:57:14.968352 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:57:14.968330    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 31 23:57:14.970243 ip-10-0-130-148 kubenswrapper[2559]: W0731 23:57:14.970215    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7f2b302_5771_4b68_b6cd_e41c4959b146.slice/crio-3a2cbcb2b87a5ec15766c50e44046f071244bdb9e2390b9bce135804f832074a WatchSource:0}: Error finding container 3a2cbcb2b87a5ec15766c50e44046f071244bdb9e2390b9bce135804f832074a: Status 404 returned error can't find the container with id 3a2cbcb2b87a5ec15766c50e44046f071244bdb9e2390b9bce135804f832074a
Jul 31 23:57:15.785137 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:57:15.785095    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"e7f2b302-5771-4b68-b6cd-e41c4959b146","Type":"ContainerStarted","Data":"c410cb2ef2149971d2cc9ea9733628a5c6be874b9d4f8d7cc4a26509df49e17f"}
Jul 31 23:57:15.785498 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:57:15.785155    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"e7f2b302-5771-4b68-b6cd-e41c4959b146","Type":"ContainerStarted","Data":"3a2cbcb2b87a5ec15766c50e44046f071244bdb9e2390b9bce135804f832074a"}
Jul 31 23:57:17.792115 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:57:17.792080    2559 generic.go:356] "Generic (PLEG): container finished" podID="e7f2b302-5771-4b68-b6cd-e41c4959b146" containerID="c410cb2ef2149971d2cc9ea9733628a5c6be874b9d4f8d7cc4a26509df49e17f" exitCode=0
Jul 31 23:57:17.792490 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:57:17.792151    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"e7f2b302-5771-4b68-b6cd-e41c4959b146","Type":"ContainerDied","Data":"c410cb2ef2149971d2cc9ea9733628a5c6be874b9d4f8d7cc4a26509df49e17f"}
Jul 31 23:57:17.793516 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:57:17.793502    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 23:57:45.940719 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:57:45.940681    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"e7f2b302-5771-4b68-b6cd-e41c4959b146","Type":"ContainerStarted","Data":"098f55b006873f81ff03e8f1a2bced8c76da11ce9141ca36429f2ebc0ebb36cb"}
Jul 31 23:57:45.964254 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:57:45.964204    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=2.647114686 podStartE2EDuration="31.964187802s" totalImagesPullingTime="27.365380451s" totalInitContainerRuntime="1.951692665s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:57:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:57:45.961479791 +0000 UTC m=+728.681279636" watchObservedRunningTime="2026-07-31 23:57:45.964187802 +0000 UTC m=+728.683987647"
Jul 31 23:57:46.834755 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:57:46.834706    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 23:57:54.835355 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:57:54.835313    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 23:57:54.836972 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:57:54.836944    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e7f2b302-5771-4b68-b6cd-e41c4959b146" containerName="main" probeResult="failure" output="Get \"https://10.132.0.29:8000/health\": dial tcp 10.132.0.29:8000: connect: connection refused"
Jul 31 23:58:04.835405 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:58:04.835357    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e7f2b302-5771-4b68-b6cd-e41c4959b146" containerName="main" probeResult="failure" output="Get \"https://10.132.0.29:8000/health\": dial tcp 10.132.0.29:8000: connect: connection refused"
Jul 31 23:58:14.835188 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:58:14.835136    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e7f2b302-5771-4b68-b6cd-e41c4959b146" containerName="main" probeResult="failure" output="Get \"https://10.132.0.29:8000/health\": dial tcp 10.132.0.29:8000: connect: connection refused"
Jul 31 23:58:24.835907 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:58:24.835866    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e7f2b302-5771-4b68-b6cd-e41c4959b146" containerName="main" probeResult="failure" output="Get \"https://10.132.0.29:8000/health\": dial tcp 10.132.0.29:8000: connect: connection refused"
Jul 31 23:58:34.835964 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:58:34.835918    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e7f2b302-5771-4b68-b6cd-e41c4959b146" containerName="main" probeResult="failure" output="Get \"https://10.132.0.29:8000/health\": dial tcp 10.132.0.29:8000: connect: connection refused"
Jul 31 23:58:44.835579 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:58:44.835524    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e7f2b302-5771-4b68-b6cd-e41c4959b146" containerName="main" probeResult="failure" output="Get \"https://10.132.0.29:8000/health\": dial tcp 10.132.0.29:8000: connect: connection refused"
Jul 31 23:58:54.836081 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:58:54.836034    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e7f2b302-5771-4b68-b6cd-e41c4959b146" containerName="main" probeResult="failure" output="Get \"https://10.132.0.29:8000/health\": dial tcp 10.132.0.29:8000: connect: connection refused"
Jul 31 23:59:04.836015 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:59:04.835917    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e7f2b302-5771-4b68-b6cd-e41c4959b146" containerName="main" probeResult="failure" output="Get \"https://10.132.0.29:8000/health\": dial tcp 10.132.0.29:8000: connect: connection refused"
Jul 31 23:59:14.835554 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:59:14.835504    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e7f2b302-5771-4b68-b6cd-e41c4959b146" containerName="main" probeResult="failure" output="Get \"https://10.132.0.29:8000/health\": dial tcp 10.132.0.29:8000: connect: connection refused"
Jul 31 23:59:24.845243 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:59:24.845206    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 23:59:24.852630 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:59:24.852603    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 23:59:30.166017 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:59:30.165979    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 31 23:59:45.188054 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:59:45.187987    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e7f2b302-5771-4b68-b6cd-e41c4959b146" containerName="main" containerID="cri-o://098f55b006873f81ff03e8f1a2bced8c76da11ce9141ca36429f2ebc0ebb36cb" gracePeriod=45
Jul 31 23:59:45.846074 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:59:45.846029    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e7f2b302-5771-4b68-b6cd-e41c4959b146" containerName="main" probeResult="failure" output="Get \"https://10.132.0.29:8000/health\": dial tcp 10.132.0.29:8000: connect: connection refused"
Jul 31 23:59:45.935099 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:59:45.935075    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 23:59:46.097818 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:59:46.097711    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n5jjw\" (UniqueName: \"kubernetes.io/projected/e7f2b302-5771-4b68-b6cd-e41c4959b146-kube-api-access-n5jjw\") pod \"e7f2b302-5771-4b68-b6cd-e41c4959b146\" (UID: \"e7f2b302-5771-4b68-b6cd-e41c4959b146\") "
Jul 31 23:59:46.097818 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:59:46.097804    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7f2b302-5771-4b68-b6cd-e41c4959b146-cabundle-cert\") pod \"e7f2b302-5771-4b68-b6cd-e41c4959b146\" (UID: \"e7f2b302-5771-4b68-b6cd-e41c4959b146\") "
Jul 31 23:59:46.098040 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:59:46.097837    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7f2b302-5771-4b68-b6cd-e41c4959b146-tmp-dir\") pod \"e7f2b302-5771-4b68-b6cd-e41c4959b146\" (UID: \"e7f2b302-5771-4b68-b6cd-e41c4959b146\") "
Jul 31 23:59:46.098040 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:59:46.097864    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7f2b302-5771-4b68-b6cd-e41c4959b146-model-cache\") pod \"e7f2b302-5771-4b68-b6cd-e41c4959b146\" (UID: \"e7f2b302-5771-4b68-b6cd-e41c4959b146\") "
Jul 31 23:59:46.098040 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:59:46.097921    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7f2b302-5771-4b68-b6cd-e41c4959b146-kserve-provision-location\") pod \"e7f2b302-5771-4b68-b6cd-e41c4959b146\" (UID: \"e7f2b302-5771-4b68-b6cd-e41c4959b146\") "
Jul 31 23:59:46.098040 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:59:46.097946    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7f2b302-5771-4b68-b6cd-e41c4959b146-tls-certs\") pod \"e7f2b302-5771-4b68-b6cd-e41c4959b146\" (UID: \"e7f2b302-5771-4b68-b6cd-e41c4959b146\") "
Jul 31 23:59:46.098040 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:59:46.097978    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7f2b302-5771-4b68-b6cd-e41c4959b146-dshm\") pod \"e7f2b302-5771-4b68-b6cd-e41c4959b146\" (UID: \"e7f2b302-5771-4b68-b6cd-e41c4959b146\") "
Jul 31 23:59:46.098286 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:59:46.098060    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7f2b302-5771-4b68-b6cd-e41c4959b146-home\") pod \"e7f2b302-5771-4b68-b6cd-e41c4959b146\" (UID: \"e7f2b302-5771-4b68-b6cd-e41c4959b146\") "
Jul 31 23:59:46.098286 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:59:46.098201    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7f2b302-5771-4b68-b6cd-e41c4959b146-model-cache" (OuterVolumeSpecName: "model-cache") pod "e7f2b302-5771-4b68-b6cd-e41c4959b146" (UID: "e7f2b302-5771-4b68-b6cd-e41c4959b146"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 23:59:46.098394 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:59:46.098285    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7f2b302-5771-4b68-b6cd-e41c4959b146-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e7f2b302-5771-4b68-b6cd-e41c4959b146" (UID: "e7f2b302-5771-4b68-b6cd-e41c4959b146"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:59:46.098394 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:59:46.098314    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7f2b302-5771-4b68-b6cd-e41c4959b146-model-cache\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Jul 31 23:59:46.098865 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:59:46.098838    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7f2b302-5771-4b68-b6cd-e41c4959b146-home" (OuterVolumeSpecName: "home") pod "e7f2b302-5771-4b68-b6cd-e41c4959b146" (UID: "e7f2b302-5771-4b68-b6cd-e41c4959b146"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 23:59:46.100096 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:59:46.100067    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7f2b302-5771-4b68-b6cd-e41c4959b146-kube-api-access-n5jjw" (OuterVolumeSpecName: "kube-api-access-n5jjw") pod "e7f2b302-5771-4b68-b6cd-e41c4959b146" (UID: "e7f2b302-5771-4b68-b6cd-e41c4959b146"). InnerVolumeSpecName "kube-api-access-n5jjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:59:46.100270 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:59:46.100246    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7f2b302-5771-4b68-b6cd-e41c4959b146-dshm" (OuterVolumeSpecName: "dshm") pod "e7f2b302-5771-4b68-b6cd-e41c4959b146" (UID: "e7f2b302-5771-4b68-b6cd-e41c4959b146"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 23:59:46.100413 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:59:46.100384    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7f2b302-5771-4b68-b6cd-e41c4959b146-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e7f2b302-5771-4b68-b6cd-e41c4959b146" (UID: "e7f2b302-5771-4b68-b6cd-e41c4959b146"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:59:46.110569 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:59:46.110530    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7f2b302-5771-4b68-b6cd-e41c4959b146-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e7f2b302-5771-4b68-b6cd-e41c4959b146" (UID: "e7f2b302-5771-4b68-b6cd-e41c4959b146"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 23:59:46.119395 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:59:46.119372    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7f2b302-5771-4b68-b6cd-e41c4959b146-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e7f2b302-5771-4b68-b6cd-e41c4959b146" (UID: "e7f2b302-5771-4b68-b6cd-e41c4959b146"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 23:59:46.198837 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:59:46.198794    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7f2b302-5771-4b68-b6cd-e41c4959b146-home\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Jul 31 23:59:46.198837 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:59:46.198836    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5jjw\" (UniqueName: \"kubernetes.io/projected/e7f2b302-5771-4b68-b6cd-e41c4959b146-kube-api-access-n5jjw\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Jul 31 23:59:46.198837 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:59:46.198847    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7f2b302-5771-4b68-b6cd-e41c4959b146-cabundle-cert\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Jul 31 23:59:46.199243 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:59:46.198861    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7f2b302-5771-4b68-b6cd-e41c4959b146-tmp-dir\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Jul 31 23:59:46.199243 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:59:46.198872    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7f2b302-5771-4b68-b6cd-e41c4959b146-kserve-provision-location\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Jul 31 23:59:46.199243 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:59:46.198880    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7f2b302-5771-4b68-b6cd-e41c4959b146-tls-certs\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Jul 31 23:59:46.199243 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:59:46.198888    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7f2b302-5771-4b68-b6cd-e41c4959b146-dshm\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Jul 31 23:59:46.488133 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:59:46.488102    2559 generic.go:356] "Generic (PLEG): container finished" podID="e7f2b302-5771-4b68-b6cd-e41c4959b146" containerID="098f55b006873f81ff03e8f1a2bced8c76da11ce9141ca36429f2ebc0ebb36cb" exitCode=0
Jul 31 23:59:46.488315 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:59:46.488158    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 23:59:46.488315 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:59:46.488187    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"e7f2b302-5771-4b68-b6cd-e41c4959b146","Type":"ContainerDied","Data":"098f55b006873f81ff03e8f1a2bced8c76da11ce9141ca36429f2ebc0ebb36cb"}
Jul 31 23:59:46.488315 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:59:46.488241    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"e7f2b302-5771-4b68-b6cd-e41c4959b146","Type":"ContainerDied","Data":"3a2cbcb2b87a5ec15766c50e44046f071244bdb9e2390b9bce135804f832074a"}
Jul 31 23:59:46.488315 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:59:46.488263    2559 scope.go:117] "RemoveContainer" containerID="098f55b006873f81ff03e8f1a2bced8c76da11ce9141ca36429f2ebc0ebb36cb"
Jul 31 23:59:46.496688 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:59:46.496664    2559 scope.go:117] "RemoveContainer" containerID="c410cb2ef2149971d2cc9ea9733628a5c6be874b9d4f8d7cc4a26509df49e17f"
Jul 31 23:59:46.504015 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:59:46.503995    2559 scope.go:117] "RemoveContainer" containerID="098f55b006873f81ff03e8f1a2bced8c76da11ce9141ca36429f2ebc0ebb36cb"
Jul 31 23:59:46.504262 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:59:46.504241    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"098f55b006873f81ff03e8f1a2bced8c76da11ce9141ca36429f2ebc0ebb36cb\": container with ID starting with 098f55b006873f81ff03e8f1a2bced8c76da11ce9141ca36429f2ebc0ebb36cb not found: ID does not exist" containerID="098f55b006873f81ff03e8f1a2bced8c76da11ce9141ca36429f2ebc0ebb36cb"
Jul 31 23:59:46.504306 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:59:46.504271    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"098f55b006873f81ff03e8f1a2bced8c76da11ce9141ca36429f2ebc0ebb36cb"} err="failed to get container status \"098f55b006873f81ff03e8f1a2bced8c76da11ce9141ca36429f2ebc0ebb36cb\": rpc error: code = NotFound desc = could not find container \"098f55b006873f81ff03e8f1a2bced8c76da11ce9141ca36429f2ebc0ebb36cb\": container with ID starting with 098f55b006873f81ff03e8f1a2bced8c76da11ce9141ca36429f2ebc0ebb36cb not found: ID does not exist"
Jul 31 23:59:46.504306 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:59:46.504300    2559 scope.go:117] "RemoveContainer" containerID="c410cb2ef2149971d2cc9ea9733628a5c6be874b9d4f8d7cc4a26509df49e17f"
Jul 31 23:59:46.504537 ip-10-0-130-148 kubenswrapper[2559]: E0731 23:59:46.504522    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c410cb2ef2149971d2cc9ea9733628a5c6be874b9d4f8d7cc4a26509df49e17f\": container with ID starting with c410cb2ef2149971d2cc9ea9733628a5c6be874b9d4f8d7cc4a26509df49e17f not found: ID does not exist" containerID="c410cb2ef2149971d2cc9ea9733628a5c6be874b9d4f8d7cc4a26509df49e17f"
Jul 31 23:59:46.504601 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:59:46.504542    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c410cb2ef2149971d2cc9ea9733628a5c6be874b9d4f8d7cc4a26509df49e17f"} err="failed to get container status \"c410cb2ef2149971d2cc9ea9733628a5c6be874b9d4f8d7cc4a26509df49e17f\": rpc error: code = NotFound desc = could not find container \"c410cb2ef2149971d2cc9ea9733628a5c6be874b9d4f8d7cc4a26509df49e17f\": container with ID starting with c410cb2ef2149971d2cc9ea9733628a5c6be874b9d4f8d7cc4a26509df49e17f not found: ID does not exist"
Jul 31 23:59:46.510603 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:59:46.510579    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 31 23:59:46.513442 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:59:46.513424    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 31 23:59:47.786412 ip-10-0-130-148 kubenswrapper[2559]: I0731 23:59:47.786380    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7f2b302-5771-4b68-b6cd-e41c4959b146" path="/var/lib/kubelet/pods/e7f2b302-5771-4b68-b6cd-e41c4959b146/volumes"
Aug 01 00:00:00.150236 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:00.150205    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-pruner-29759040-9x7p9"]
Aug 01 00:00:00.150700 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:00.150639    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7f2b302-5771-4b68-b6cd-e41c4959b146" containerName="storage-initializer"
Aug 01 00:00:00.150700 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:00.150658    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7f2b302-5771-4b68-b6cd-e41c4959b146" containerName="storage-initializer"
Aug 01 00:00:00.150700 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:00.150674    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7f2b302-5771-4b68-b6cd-e41c4959b146" containerName="main"
Aug 01 00:00:00.150700 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:00.150682    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7f2b302-5771-4b68-b6cd-e41c4959b146" containerName="main"
Aug 01 00:00:00.150928 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:00.150808    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7f2b302-5771-4b68-b6cd-e41c4959b146" containerName="main"
Aug 01 00:00:00.154969 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:00.154946    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29759040-9x7p9"
Aug 01 00:00:00.158117 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:00.158094    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"pruner-dockercfg-cmwjq\""
Aug 01 00:00:00.158255 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:00.158130    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"serviceca\""
Aug 01 00:00:00.164357 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:00.164330    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-pruner-29759040-9x7p9"]
Aug 01 00:00:00.203954 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:00.203924    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlnnq\" (UniqueName: \"kubernetes.io/projected/8243c84a-74f8-424f-833d-8eee80046f7a-kube-api-access-vlnnq\") pod \"image-pruner-29759040-9x7p9\" (UID: \"8243c84a-74f8-424f-833d-8eee80046f7a\") " pod="openshift-image-registry/image-pruner-29759040-9x7p9"
Aug 01 00:00:00.204116 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:00.203986    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8243c84a-74f8-424f-833d-8eee80046f7a-serviceca\") pod \"image-pruner-29759040-9x7p9\" (UID: \"8243c84a-74f8-424f-833d-8eee80046f7a\") " pod="openshift-image-registry/image-pruner-29759040-9x7p9"
Aug 01 00:00:00.304584 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:00.304550    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8243c84a-74f8-424f-833d-8eee80046f7a-serviceca\") pod \"image-pruner-29759040-9x7p9\" (UID: \"8243c84a-74f8-424f-833d-8eee80046f7a\") " pod="openshift-image-registry/image-pruner-29759040-9x7p9"
Aug 01 00:00:00.304734 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:00.304610    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlnnq\" (UniqueName: \"kubernetes.io/projected/8243c84a-74f8-424f-833d-8eee80046f7a-kube-api-access-vlnnq\") pod \"image-pruner-29759040-9x7p9\" (UID: \"8243c84a-74f8-424f-833d-8eee80046f7a\") " pod="openshift-image-registry/image-pruner-29759040-9x7p9"
Aug 01 00:00:00.305272 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:00.305252    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8243c84a-74f8-424f-833d-8eee80046f7a-serviceca\") pod \"image-pruner-29759040-9x7p9\" (UID: \"8243c84a-74f8-424f-833d-8eee80046f7a\") " pod="openshift-image-registry/image-pruner-29759040-9x7p9"
Aug 01 00:00:00.313111 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:00.313088    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlnnq\" (UniqueName: \"kubernetes.io/projected/8243c84a-74f8-424f-833d-8eee80046f7a-kube-api-access-vlnnq\") pod \"image-pruner-29759040-9x7p9\" (UID: \"8243c84a-74f8-424f-833d-8eee80046f7a\") " pod="openshift-image-registry/image-pruner-29759040-9x7p9"
Aug 01 00:00:00.482195 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:00.482166    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29759040-9x7p9"
Aug 01 00:00:00.610532 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:00.610507    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-pruner-29759040-9x7p9"]
Aug 01 00:00:00.613055 ip-10-0-130-148 kubenswrapper[2559]: W0801 00:00:00.613025    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8243c84a_74f8_424f_833d_8eee80046f7a.slice/crio-0a66ef1806f8481cc333a13775c2654bbc2dc8e079af6d189d941b5e7750454d WatchSource:0}: Error finding container 0a66ef1806f8481cc333a13775c2654bbc2dc8e079af6d189d941b5e7750454d: Status 404 returned error can't find the container with id 0a66ef1806f8481cc333a13775c2654bbc2dc8e079af6d189d941b5e7750454d
Aug 01 00:00:01.557438 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:01.557406    2559 generic.go:356] "Generic (PLEG): container finished" podID="8243c84a-74f8-424f-833d-8eee80046f7a" containerID="582cb065cd0ab422f6c65b8e6503ceaddd7ba93c155ff55e548c6b0037c5d5ed" exitCode=0
Aug 01 00:00:01.557847 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:01.557455    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29759040-9x7p9" event={"ID":"8243c84a-74f8-424f-833d-8eee80046f7a","Type":"ContainerDied","Data":"582cb065cd0ab422f6c65b8e6503ceaddd7ba93c155ff55e548c6b0037c5d5ed"}
Aug 01 00:00:01.557847 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:01.557497    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29759040-9x7p9" event={"ID":"8243c84a-74f8-424f-833d-8eee80046f7a","Type":"ContainerStarted","Data":"0a66ef1806f8481cc333a13775c2654bbc2dc8e079af6d189d941b5e7750454d"}
Aug 01 00:00:02.697372 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:02.697348    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29759040-9x7p9"
Aug 01 00:00:02.722806 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:02.722774    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8243c84a-74f8-424f-833d-8eee80046f7a-serviceca\") pod \"8243c84a-74f8-424f-833d-8eee80046f7a\" (UID: \"8243c84a-74f8-424f-833d-8eee80046f7a\") "
Aug 01 00:00:02.722976 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:02.722822    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vlnnq\" (UniqueName: \"kubernetes.io/projected/8243c84a-74f8-424f-833d-8eee80046f7a-kube-api-access-vlnnq\") pod \"8243c84a-74f8-424f-833d-8eee80046f7a\" (UID: \"8243c84a-74f8-424f-833d-8eee80046f7a\") "
Aug 01 00:00:02.723146 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:02.723126    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8243c84a-74f8-424f-833d-8eee80046f7a-serviceca" (OuterVolumeSpecName: "serviceca") pod "8243c84a-74f8-424f-833d-8eee80046f7a" (UID: "8243c84a-74f8-424f-833d-8eee80046f7a"). InnerVolumeSpecName "serviceca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:00:02.724729 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:02.724703    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8243c84a-74f8-424f-833d-8eee80046f7a-kube-api-access-vlnnq" (OuterVolumeSpecName: "kube-api-access-vlnnq") pod "8243c84a-74f8-424f-833d-8eee80046f7a" (UID: "8243c84a-74f8-424f-833d-8eee80046f7a"). InnerVolumeSpecName "kube-api-access-vlnnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:00:02.824411 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:02.824323    2559 reconciler_common.go:299] "Volume detached for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8243c84a-74f8-424f-833d-8eee80046f7a-serviceca\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:00:02.824411 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:02.824357    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlnnq\" (UniqueName: \"kubernetes.io/projected/8243c84a-74f8-424f-833d-8eee80046f7a-kube-api-access-vlnnq\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:00:03.567419 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:03.567389    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29759040-9x7p9"
Aug 01 00:00:03.567419 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:03.567399    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29759040-9x7p9" event={"ID":"8243c84a-74f8-424f-833d-8eee80046f7a","Type":"ContainerDied","Data":"0a66ef1806f8481cc333a13775c2654bbc2dc8e079af6d189d941b5e7750454d"}
Aug 01 00:00:03.567636 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:03.567445    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a66ef1806f8481cc333a13775c2654bbc2dc8e079af6d189d941b5e7750454d"
Aug 01 00:00:24.956608 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:24.956569    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-qn7mx"]
Aug 01 00:00:24.957072 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:24.956905    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8243c84a-74f8-424f-833d-8eee80046f7a" containerName="image-pruner"
Aug 01 00:00:24.957072 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:24.956920    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8243c84a-74f8-424f-833d-8eee80046f7a" containerName="image-pruner"
Aug 01 00:00:24.957072 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:24.956984    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8243c84a-74f8-424f-833d-8eee80046f7a" containerName="image-pruner"
Aug 01 00:00:24.960120 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:24.960099    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-qn7mx"
Aug 01 00:00:24.962834 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:24.962736    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Aug 01 00:00:24.962941 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:24.962849    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Aug 01 00:00:24.964054 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:24.964034    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"default-dockercfg-bxp4t\""
Aug 01 00:00:24.964125 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:24.964108    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
Aug 01 00:00:24.969212 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:24.969191    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-qn7mx"]
Aug 01 00:00:24.985692 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:24.985658    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1099f877-f9ea-4db9-bf5b-9b57bfa1fb34-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-qn7mx\" (UID: \"1099f877-f9ea-4db9-bf5b-9b57bfa1fb34\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-qn7mx"
Aug 01 00:00:24.985839 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:24.985780    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1099f877-f9ea-4db9-bf5b-9b57bfa1fb34-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-qn7mx\" (UID: \"1099f877-f9ea-4db9-bf5b-9b57bfa1fb34\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-qn7mx"
Aug 01 00:00:24.985839 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:24.985812    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1099f877-f9ea-4db9-bf5b-9b57bfa1fb34-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-qn7mx\" (UID: \"1099f877-f9ea-4db9-bf5b-9b57bfa1fb34\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-qn7mx"
Aug 01 00:00:24.985957 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:24.985848    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1099f877-f9ea-4db9-bf5b-9b57bfa1fb34-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-qn7mx\" (UID: \"1099f877-f9ea-4db9-bf5b-9b57bfa1fb34\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-qn7mx"
Aug 01 00:00:24.985957 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:24.985873    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1099f877-f9ea-4db9-bf5b-9b57bfa1fb34-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-qn7mx\" (UID: \"1099f877-f9ea-4db9-bf5b-9b57bfa1fb34\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-qn7mx"
Aug 01 00:00:24.985957 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:24.985913    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8xpd\" (UniqueName: \"kubernetes.io/projected/1099f877-f9ea-4db9-bf5b-9b57bfa1fb34-kube-api-access-w8xpd\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-qn7mx\" (UID: \"1099f877-f9ea-4db9-bf5b-9b57bfa1fb34\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-qn7mx"
Aug 01 00:00:25.086980 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:25.086949    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1099f877-f9ea-4db9-bf5b-9b57bfa1fb34-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-qn7mx\" (UID: \"1099f877-f9ea-4db9-bf5b-9b57bfa1fb34\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-qn7mx"
Aug 01 00:00:25.087110 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:25.086995    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1099f877-f9ea-4db9-bf5b-9b57bfa1fb34-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-qn7mx\" (UID: \"1099f877-f9ea-4db9-bf5b-9b57bfa1fb34\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-qn7mx"
Aug 01 00:00:25.087110 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:25.087013    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1099f877-f9ea-4db9-bf5b-9b57bfa1fb34-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-qn7mx\" (UID: \"1099f877-f9ea-4db9-bf5b-9b57bfa1fb34\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-qn7mx"
Aug 01 00:00:25.087191 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:25.087127    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1099f877-f9ea-4db9-bf5b-9b57bfa1fb34-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-qn7mx\" (UID: \"1099f877-f9ea-4db9-bf5b-9b57bfa1fb34\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-qn7mx"
Aug 01 00:00:25.087191 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:25.087163    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1099f877-f9ea-4db9-bf5b-9b57bfa1fb34-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-qn7mx\" (UID: \"1099f877-f9ea-4db9-bf5b-9b57bfa1fb34\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-qn7mx"
Aug 01 00:00:25.087271 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:25.087193    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8xpd\" (UniqueName: \"kubernetes.io/projected/1099f877-f9ea-4db9-bf5b-9b57bfa1fb34-kube-api-access-w8xpd\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-qn7mx\" (UID: \"1099f877-f9ea-4db9-bf5b-9b57bfa1fb34\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-qn7mx"
Aug 01 00:00:25.087439 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:25.087415    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1099f877-f9ea-4db9-bf5b-9b57bfa1fb34-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-qn7mx\" (UID: \"1099f877-f9ea-4db9-bf5b-9b57bfa1fb34\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-qn7mx"
Aug 01 00:00:25.087439 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:25.087438    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1099f877-f9ea-4db9-bf5b-9b57bfa1fb34-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-qn7mx\" (UID: \"1099f877-f9ea-4db9-bf5b-9b57bfa1fb34\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-qn7mx"
Aug 01 00:00:25.087597 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:25.087497    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1099f877-f9ea-4db9-bf5b-9b57bfa1fb34-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-qn7mx\" (UID: \"1099f877-f9ea-4db9-bf5b-9b57bfa1fb34\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-qn7mx"
Aug 01 00:00:25.089192 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:25.089175    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1099f877-f9ea-4db9-bf5b-9b57bfa1fb34-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-qn7mx\" (UID: \"1099f877-f9ea-4db9-bf5b-9b57bfa1fb34\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-qn7mx"
Aug 01 00:00:25.089585 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:25.089565    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1099f877-f9ea-4db9-bf5b-9b57bfa1fb34-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-qn7mx\" (UID: \"1099f877-f9ea-4db9-bf5b-9b57bfa1fb34\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-qn7mx"
Aug 01 00:00:25.095350 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:25.095329    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8xpd\" (UniqueName: \"kubernetes.io/projected/1099f877-f9ea-4db9-bf5b-9b57bfa1fb34-kube-api-access-w8xpd\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-qn7mx\" (UID: \"1099f877-f9ea-4db9-bf5b-9b57bfa1fb34\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-qn7mx"
Aug 01 00:00:25.292709 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:25.292629    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-qn7mx"
Aug 01 00:00:25.426633 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:25.426601    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-qn7mx"]
Aug 01 00:00:25.429782 ip-10-0-130-148 kubenswrapper[2559]: W0801 00:00:25.429731    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1099f877_f9ea_4db9_bf5b_9b57bfa1fb34.slice/crio-0ea30bb189a1d365e35a924cf080002f191cb6fd4feb713518e93fbb4e43544c WatchSource:0}: Error finding container 0ea30bb189a1d365e35a924cf080002f191cb6fd4feb713518e93fbb4e43544c: Status 404 returned error can't find the container with id 0ea30bb189a1d365e35a924cf080002f191cb6fd4feb713518e93fbb4e43544c
Aug 01 00:00:25.663252 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:25.663162    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-qn7mx" event={"ID":"1099f877-f9ea-4db9-bf5b-9b57bfa1fb34","Type":"ContainerStarted","Data":"0ea30bb189a1d365e35a924cf080002f191cb6fd4feb713518e93fbb4e43544c"}
Aug 01 00:00:32.287085 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:32.287055    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-qn7mx"]
Aug 01 00:00:37.745255 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:37.745224    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-spblg_c2e50946-aea4-4886-82ec-0bd5a31196cf/console-operator/1.log"
Aug 01 00:00:37.745716 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:37.745350    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-spblg_c2e50946-aea4-4886-82ec-0bd5a31196cf/console-operator/1.log"
Aug 01 00:00:39.732022 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:39.731983    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-qn7mx" event={"ID":"1099f877-f9ea-4db9-bf5b-9b57bfa1fb34","Type":"ContainerStarted","Data":"05a012b227d2d766c1cca793c57e2dfa8f019bfde63787821abc11f680590a1d"}
Aug 01 00:00:39.740789 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:00:39.740735    2559 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 01 00:00:39.740789 ip-10-0-130-148 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:00:39.740789 ip-10-0-130-148 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Aug 01 00:00:39.740789 ip-10-0-130-148 kubenswrapper[2559]:  > containerID="05a012b227d2d766c1cca793c57e2dfa8f019bfde63787821abc11f680590a1d" cmd=["/bin/sleep","15"]
Aug 01 00:00:39.741012 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:39.740802    2559 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-qn7mx" message="" err=<
Aug 01 00:00:39.741012 ip-10-0-130-148 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:00:39.741012 ip-10-0-130-148 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Aug 01 00:00:39.741012 ip-10-0-130-148 kubenswrapper[2559]:  >
Aug 01 00:00:39.741012 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:00:39.740815    2559 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 01 00:00:39.741012 ip-10-0-130-148 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:00:39.741012 ip-10-0-130-148 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Aug 01 00:00:39.741012 ip-10-0-130-148 kubenswrapper[2559]:  > pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-qn7mx" podUID="1099f877-f9ea-4db9-bf5b-9b57bfa1fb34" containerName="main" containerID="cri-o://05a012b227d2d766c1cca793c57e2dfa8f019bfde63787821abc11f680590a1d"
Aug 01 00:00:39.741012 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:39.740854    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-qn7mx" podUID="1099f877-f9ea-4db9-bf5b-9b57bfa1fb34" containerName="main" containerID="cri-o://05a012b227d2d766c1cca793c57e2dfa8f019bfde63787821abc11f680590a1d" gracePeriod=60
Aug 01 00:00:39.749156 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:39.749106    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-qn7mx" podStartSLOduration=2.313325279 podStartE2EDuration="15.749089637s" totalImagesPullingTime="13.435764358s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:00:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:00:39.747069845 +0000 UTC m=+902.466869704" watchObservedRunningTime="2026-08-01 00:00:39.749089637 +0000 UTC m=+902.468889484"
Aug 01 00:00:39.985356 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:39.985304    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-qn7mx"
Aug 01 00:00:40.114476 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:40.114446    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1099f877-f9ea-4db9-bf5b-9b57bfa1fb34-model-cache\") pod \"1099f877-f9ea-4db9-bf5b-9b57bfa1fb34\" (UID: \"1099f877-f9ea-4db9-bf5b-9b57bfa1fb34\") "
Aug 01 00:00:40.114633 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:40.114482    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w8xpd\" (UniqueName: \"kubernetes.io/projected/1099f877-f9ea-4db9-bf5b-9b57bfa1fb34-kube-api-access-w8xpd\") pod \"1099f877-f9ea-4db9-bf5b-9b57bfa1fb34\" (UID: \"1099f877-f9ea-4db9-bf5b-9b57bfa1fb34\") "
Aug 01 00:00:40.114633 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:40.114515    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1099f877-f9ea-4db9-bf5b-9b57bfa1fb34-dshm\") pod \"1099f877-f9ea-4db9-bf5b-9b57bfa1fb34\" (UID: \"1099f877-f9ea-4db9-bf5b-9b57bfa1fb34\") "
Aug 01 00:00:40.114633 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:40.114546    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1099f877-f9ea-4db9-bf5b-9b57bfa1fb34-tls-certs\") pod \"1099f877-f9ea-4db9-bf5b-9b57bfa1fb34\" (UID: \"1099f877-f9ea-4db9-bf5b-9b57bfa1fb34\") "
Aug 01 00:00:40.114633 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:40.114564    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1099f877-f9ea-4db9-bf5b-9b57bfa1fb34-tmp-dir\") pod \"1099f877-f9ea-4db9-bf5b-9b57bfa1fb34\" (UID: \"1099f877-f9ea-4db9-bf5b-9b57bfa1fb34\") "
Aug 01 00:00:40.114855 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:40.114635    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1099f877-f9ea-4db9-bf5b-9b57bfa1fb34-home\") pod \"1099f877-f9ea-4db9-bf5b-9b57bfa1fb34\" (UID: \"1099f877-f9ea-4db9-bf5b-9b57bfa1fb34\") "
Aug 01 00:00:40.114855 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:40.114702    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1099f877-f9ea-4db9-bf5b-9b57bfa1fb34-model-cache" (OuterVolumeSpecName: "model-cache") pod "1099f877-f9ea-4db9-bf5b-9b57bfa1fb34" (UID: "1099f877-f9ea-4db9-bf5b-9b57bfa1fb34"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:00:40.114957 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:40.114865    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1099f877-f9ea-4db9-bf5b-9b57bfa1fb34-model-cache\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:00:40.114957 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:40.114940    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1099f877-f9ea-4db9-bf5b-9b57bfa1fb34-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1099f877-f9ea-4db9-bf5b-9b57bfa1fb34" (UID: "1099f877-f9ea-4db9-bf5b-9b57bfa1fb34"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:00:40.115035 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:40.114952    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1099f877-f9ea-4db9-bf5b-9b57bfa1fb34-home" (OuterVolumeSpecName: "home") pod "1099f877-f9ea-4db9-bf5b-9b57bfa1fb34" (UID: "1099f877-f9ea-4db9-bf5b-9b57bfa1fb34"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:00:40.116781 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:40.116754    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1099f877-f9ea-4db9-bf5b-9b57bfa1fb34-dshm" (OuterVolumeSpecName: "dshm") pod "1099f877-f9ea-4db9-bf5b-9b57bfa1fb34" (UID: "1099f877-f9ea-4db9-bf5b-9b57bfa1fb34"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:00:40.116898 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:40.116828    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1099f877-f9ea-4db9-bf5b-9b57bfa1fb34-kube-api-access-w8xpd" (OuterVolumeSpecName: "kube-api-access-w8xpd") pod "1099f877-f9ea-4db9-bf5b-9b57bfa1fb34" (UID: "1099f877-f9ea-4db9-bf5b-9b57bfa1fb34"). InnerVolumeSpecName "kube-api-access-w8xpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:00:40.116898 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:40.116842    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1099f877-f9ea-4db9-bf5b-9b57bfa1fb34-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1099f877-f9ea-4db9-bf5b-9b57bfa1fb34" (UID: "1099f877-f9ea-4db9-bf5b-9b57bfa1fb34"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:00:40.215986 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:40.215960    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1099f877-f9ea-4db9-bf5b-9b57bfa1fb34-home\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:00:40.215986 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:40.215981    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8xpd\" (UniqueName: \"kubernetes.io/projected/1099f877-f9ea-4db9-bf5b-9b57bfa1fb34-kube-api-access-w8xpd\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:00:40.215986 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:40.215991    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1099f877-f9ea-4db9-bf5b-9b57bfa1fb34-dshm\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:00:40.216168 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:40.216000    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1099f877-f9ea-4db9-bf5b-9b57bfa1fb34-tls-certs\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:00:40.216168 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:40.216009    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1099f877-f9ea-4db9-bf5b-9b57bfa1fb34-tmp-dir\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:00:40.738104 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:40.738074    2559 generic.go:356] "Generic (PLEG): container finished" podID="1099f877-f9ea-4db9-bf5b-9b57bfa1fb34" containerID="05a012b227d2d766c1cca793c57e2dfa8f019bfde63787821abc11f680590a1d" exitCode=0
Aug 01 00:00:40.738517 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:40.738110    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-qn7mx" event={"ID":"1099f877-f9ea-4db9-bf5b-9b57bfa1fb34","Type":"ContainerDied","Data":"05a012b227d2d766c1cca793c57e2dfa8f019bfde63787821abc11f680590a1d"}
Aug 01 00:00:40.738517 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:40.738118    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-qn7mx"
Aug 01 00:00:40.738517 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:40.738152    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-qn7mx" event={"ID":"1099f877-f9ea-4db9-bf5b-9b57bfa1fb34","Type":"ContainerDied","Data":"0ea30bb189a1d365e35a924cf080002f191cb6fd4feb713518e93fbb4e43544c"}
Aug 01 00:00:40.738517 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:40.738175    2559 scope.go:117] "RemoveContainer" containerID="05a012b227d2d766c1cca793c57e2dfa8f019bfde63787821abc11f680590a1d"
Aug 01 00:00:40.747372 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:40.747356    2559 scope.go:117] "RemoveContainer" containerID="05a012b227d2d766c1cca793c57e2dfa8f019bfde63787821abc11f680590a1d"
Aug 01 00:00:40.747610 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:00:40.747592    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05a012b227d2d766c1cca793c57e2dfa8f019bfde63787821abc11f680590a1d\": container with ID starting with 05a012b227d2d766c1cca793c57e2dfa8f019bfde63787821abc11f680590a1d not found: ID does not exist" containerID="05a012b227d2d766c1cca793c57e2dfa8f019bfde63787821abc11f680590a1d"
Aug 01 00:00:40.747674 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:40.747618    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05a012b227d2d766c1cca793c57e2dfa8f019bfde63787821abc11f680590a1d"} err="failed to get container status \"05a012b227d2d766c1cca793c57e2dfa8f019bfde63787821abc11f680590a1d\": rpc error: code = NotFound desc = could not find container \"05a012b227d2d766c1cca793c57e2dfa8f019bfde63787821abc11f680590a1d\": container with ID starting with 05a012b227d2d766c1cca793c57e2dfa8f019bfde63787821abc11f680590a1d not found: ID does not exist"
Aug 01 00:00:40.759092 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:40.759049    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-qn7mx"]
Aug 01 00:00:40.760900 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:40.760881    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-qn7mx"]
Aug 01 00:00:41.785855 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:41.785821    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1099f877-f9ea-4db9-bf5b-9b57bfa1fb34" path="/var/lib/kubelet/pods/1099f877-f9ea-4db9-bf5b-9b57bfa1fb34/volumes"
Aug 01 00:00:53.354907 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:53.354873    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfn7zzf"]
Aug 01 00:00:53.355299 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:53.355215    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1099f877-f9ea-4db9-bf5b-9b57bfa1fb34" containerName="main"
Aug 01 00:00:53.355299 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:53.355226    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1099f877-f9ea-4db9-bf5b-9b57bfa1fb34" containerName="main"
Aug 01 00:00:53.355299 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:53.355297    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1099f877-f9ea-4db9-bf5b-9b57bfa1fb34" containerName="main"
Aug 01 00:00:53.359458 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:53.359439    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfn7zzf"
Aug 01 00:00:53.363287 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:53.363265    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"default-dockercfg-lmxk7\""
Aug 01 00:00:53.363393 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:53.363283    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\""
Aug 01 00:00:53.363393 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:53.363296    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Aug 01 00:00:53.363393 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:53.363322    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Aug 01 00:00:53.366362 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:53.366186    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfn7zzf"]
Aug 01 00:00:53.522531 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:53.522490    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2651c02-536e-4519-825b-3b3901e4e3a2-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfn7zzf\" (UID: \"f2651c02-536e-4519-825b-3b3901e4e3a2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfn7zzf"
Aug 01 00:00:53.522705 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:53.522540    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8cbh\" (UniqueName: \"kubernetes.io/projected/f2651c02-536e-4519-825b-3b3901e4e3a2-kube-api-access-v8cbh\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfn7zzf\" (UID: \"f2651c02-536e-4519-825b-3b3901e4e3a2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfn7zzf"
Aug 01 00:00:53.522705 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:53.522616    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2651c02-536e-4519-825b-3b3901e4e3a2-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfn7zzf\" (UID: \"f2651c02-536e-4519-825b-3b3901e4e3a2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfn7zzf"
Aug 01 00:00:53.522705 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:53.522672    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2651c02-536e-4519-825b-3b3901e4e3a2-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfn7zzf\" (UID: \"f2651c02-536e-4519-825b-3b3901e4e3a2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfn7zzf"
Aug 01 00:00:53.522848 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:53.522705    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2651c02-536e-4519-825b-3b3901e4e3a2-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfn7zzf\" (UID: \"f2651c02-536e-4519-825b-3b3901e4e3a2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfn7zzf"
Aug 01 00:00:53.522848 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:53.522726    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2651c02-536e-4519-825b-3b3901e4e3a2-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfn7zzf\" (UID: \"f2651c02-536e-4519-825b-3b3901e4e3a2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfn7zzf"
Aug 01 00:00:53.623883 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:53.623802    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8cbh\" (UniqueName: \"kubernetes.io/projected/f2651c02-536e-4519-825b-3b3901e4e3a2-kube-api-access-v8cbh\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfn7zzf\" (UID: \"f2651c02-536e-4519-825b-3b3901e4e3a2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfn7zzf"
Aug 01 00:00:53.624057 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:53.623887    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2651c02-536e-4519-825b-3b3901e4e3a2-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfn7zzf\" (UID: \"f2651c02-536e-4519-825b-3b3901e4e3a2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfn7zzf"
Aug 01 00:00:53.624057 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:53.623938    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2651c02-536e-4519-825b-3b3901e4e3a2-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfn7zzf\" (UID: \"f2651c02-536e-4519-825b-3b3901e4e3a2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfn7zzf"
Aug 01 00:00:53.624057 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:53.623958    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2651c02-536e-4519-825b-3b3901e4e3a2-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfn7zzf\" (UID: \"f2651c02-536e-4519-825b-3b3901e4e3a2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfn7zzf"
Aug 01 00:00:53.624057 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:53.623975    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2651c02-536e-4519-825b-3b3901e4e3a2-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfn7zzf\" (UID: \"f2651c02-536e-4519-825b-3b3901e4e3a2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfn7zzf"
Aug 01 00:00:53.624057 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:53.624021    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2651c02-536e-4519-825b-3b3901e4e3a2-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfn7zzf\" (UID: \"f2651c02-536e-4519-825b-3b3901e4e3a2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfn7zzf"
Aug 01 00:00:53.624401 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:53.624378    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2651c02-536e-4519-825b-3b3901e4e3a2-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfn7zzf\" (UID: \"f2651c02-536e-4519-825b-3b3901e4e3a2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfn7zzf"
Aug 01 00:00:53.624458 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:53.624415    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2651c02-536e-4519-825b-3b3901e4e3a2-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfn7zzf\" (UID: \"f2651c02-536e-4519-825b-3b3901e4e3a2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfn7zzf"
Aug 01 00:00:53.624458 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:53.624450    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2651c02-536e-4519-825b-3b3901e4e3a2-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfn7zzf\" (UID: \"f2651c02-536e-4519-825b-3b3901e4e3a2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfn7zzf"
Aug 01 00:00:53.626274 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:53.626253    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2651c02-536e-4519-825b-3b3901e4e3a2-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfn7zzf\" (UID: \"f2651c02-536e-4519-825b-3b3901e4e3a2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfn7zzf"
Aug 01 00:00:53.626569 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:53.626554    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2651c02-536e-4519-825b-3b3901e4e3a2-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfn7zzf\" (UID: \"f2651c02-536e-4519-825b-3b3901e4e3a2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfn7zzf"
Aug 01 00:00:53.631305 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:53.631287    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8cbh\" (UniqueName: \"kubernetes.io/projected/f2651c02-536e-4519-825b-3b3901e4e3a2-kube-api-access-v8cbh\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfn7zzf\" (UID: \"f2651c02-536e-4519-825b-3b3901e4e3a2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfn7zzf"
Aug 01 00:00:53.670132 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:53.670105    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfn7zzf"
Aug 01 00:00:53.800121 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:53.800096    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfn7zzf"]
Aug 01 00:00:53.802507 ip-10-0-130-148 kubenswrapper[2559]: W0801 00:00:53.802472    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2651c02_536e_4519_825b_3b3901e4e3a2.slice/crio-ae71cad1e525a77549856e53d231008aa23ae8d808ac2b0b5bd510bcf3bdbfc9 WatchSource:0}: Error finding container ae71cad1e525a77549856e53d231008aa23ae8d808ac2b0b5bd510bcf3bdbfc9: Status 404 returned error can't find the container with id ae71cad1e525a77549856e53d231008aa23ae8d808ac2b0b5bd510bcf3bdbfc9
Aug 01 00:00:54.801143 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:54.801112    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfn7zzf" event={"ID":"f2651c02-536e-4519-825b-3b3901e4e3a2","Type":"ContainerStarted","Data":"19f1947512105e37d5127ff9202301f995c38ecb97ff6b31b1fa67c13e0b2477"}
Aug 01 00:00:54.801518 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:54.801156    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfn7zzf" event={"ID":"f2651c02-536e-4519-825b-3b3901e4e3a2","Type":"ContainerStarted","Data":"ae71cad1e525a77549856e53d231008aa23ae8d808ac2b0b5bd510bcf3bdbfc9"}
Aug 01 00:00:54.829572 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:54.829519    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfn7zzf" podStartSLOduration=1.8295032409999998 podStartE2EDuration="1.829503241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:00:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:00:54.826852486 +0000 UTC m=+917.546652375" watchObservedRunningTime="2026-08-01 00:00:54.829503241 +0000 UTC m=+917.549303086"
Aug 01 00:00:55.670228 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:00:55.670196    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfn7zzf"
Aug 01 00:01:01.176006 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:01.175964    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfn7zzf"]
Aug 01 00:01:01.184949 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:01:01.184920    2559 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 01 00:01:01.184949 ip-10-0-130-148 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:01:01.184949 ip-10-0-130-148 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Aug 01 00:01:01.184949 ip-10-0-130-148 kubenswrapper[2559]:  > containerID="19f1947512105e37d5127ff9202301f995c38ecb97ff6b31b1fa67c13e0b2477" cmd=["/bin/sleep","15"]
Aug 01 00:01:01.185173 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:01.184964    2559 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfn7zzf" message="" err=<
Aug 01 00:01:01.185173 ip-10-0-130-148 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:01:01.185173 ip-10-0-130-148 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Aug 01 00:01:01.185173 ip-10-0-130-148 kubenswrapper[2559]:  >
Aug 01 00:01:01.185173 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:01:01.184977    2559 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 01 00:01:01.185173 ip-10-0-130-148 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:01:01.185173 ip-10-0-130-148 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Aug 01 00:01:01.185173 ip-10-0-130-148 kubenswrapper[2559]:  > pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfn7zzf" podUID="f2651c02-536e-4519-825b-3b3901e4e3a2" containerName="main" containerID="cri-o://19f1947512105e37d5127ff9202301f995c38ecb97ff6b31b1fa67c13e0b2477"
Aug 01 00:01:01.185173 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:01.185012    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfn7zzf" podUID="f2651c02-536e-4519-825b-3b3901e4e3a2" containerName="main" containerID="cri-o://19f1947512105e37d5127ff9202301f995c38ecb97ff6b31b1fa67c13e0b2477" gracePeriod=60
Aug 01 00:01:01.435498 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:01.435432    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfn7zzf"
Aug 01 00:01:01.590063 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:01.590016    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2651c02-536e-4519-825b-3b3901e4e3a2-model-cache\") pod \"f2651c02-536e-4519-825b-3b3901e4e3a2\" (UID: \"f2651c02-536e-4519-825b-3b3901e4e3a2\") "
Aug 01 00:01:01.590063 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:01.590079    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2651c02-536e-4519-825b-3b3901e4e3a2-dshm\") pod \"f2651c02-536e-4519-825b-3b3901e4e3a2\" (UID: \"f2651c02-536e-4519-825b-3b3901e4e3a2\") "
Aug 01 00:01:01.590308 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:01.590109    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8cbh\" (UniqueName: \"kubernetes.io/projected/f2651c02-536e-4519-825b-3b3901e4e3a2-kube-api-access-v8cbh\") pod \"f2651c02-536e-4519-825b-3b3901e4e3a2\" (UID: \"f2651c02-536e-4519-825b-3b3901e4e3a2\") "
Aug 01 00:01:01.590308 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:01.590126    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2651c02-536e-4519-825b-3b3901e4e3a2-tmp-dir\") pod \"f2651c02-536e-4519-825b-3b3901e4e3a2\" (UID: \"f2651c02-536e-4519-825b-3b3901e4e3a2\") "
Aug 01 00:01:01.590308 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:01.590172    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2651c02-536e-4519-825b-3b3901e4e3a2-home\") pod \"f2651c02-536e-4519-825b-3b3901e4e3a2\" (UID: \"f2651c02-536e-4519-825b-3b3901e4e3a2\") "
Aug 01 00:01:01.590308 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:01.590194    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2651c02-536e-4519-825b-3b3901e4e3a2-tls-certs\") pod \"f2651c02-536e-4519-825b-3b3901e4e3a2\" (UID: \"f2651c02-536e-4519-825b-3b3901e4e3a2\") "
Aug 01 00:01:01.590504 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:01.590332    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2651c02-536e-4519-825b-3b3901e4e3a2-model-cache" (OuterVolumeSpecName: "model-cache") pod "f2651c02-536e-4519-825b-3b3901e4e3a2" (UID: "f2651c02-536e-4519-825b-3b3901e4e3a2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:01:01.590504 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:01.590440    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2651c02-536e-4519-825b-3b3901e4e3a2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f2651c02-536e-4519-825b-3b3901e4e3a2" (UID: "f2651c02-536e-4519-825b-3b3901e4e3a2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:01:01.590504 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:01.590450    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2651c02-536e-4519-825b-3b3901e4e3a2-home" (OuterVolumeSpecName: "home") pod "f2651c02-536e-4519-825b-3b3901e4e3a2" (UID: "f2651c02-536e-4519-825b-3b3901e4e3a2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:01:01.592411 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:01.592384    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2651c02-536e-4519-825b-3b3901e4e3a2-dshm" (OuterVolumeSpecName: "dshm") pod "f2651c02-536e-4519-825b-3b3901e4e3a2" (UID: "f2651c02-536e-4519-825b-3b3901e4e3a2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:01:01.592411 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:01.592391    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2651c02-536e-4519-825b-3b3901e4e3a2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f2651c02-536e-4519-825b-3b3901e4e3a2" (UID: "f2651c02-536e-4519-825b-3b3901e4e3a2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:01:01.592592 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:01.592454    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2651c02-536e-4519-825b-3b3901e4e3a2-kube-api-access-v8cbh" (OuterVolumeSpecName: "kube-api-access-v8cbh") pod "f2651c02-536e-4519-825b-3b3901e4e3a2" (UID: "f2651c02-536e-4519-825b-3b3901e4e3a2"). InnerVolumeSpecName "kube-api-access-v8cbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:01:01.691079 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:01.690994    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2651c02-536e-4519-825b-3b3901e4e3a2-model-cache\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:01:01.691079 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:01.691024    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2651c02-536e-4519-825b-3b3901e4e3a2-dshm\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:01:01.691079 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:01.691034    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8cbh\" (UniqueName: \"kubernetes.io/projected/f2651c02-536e-4519-825b-3b3901e4e3a2-kube-api-access-v8cbh\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:01:01.691079 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:01.691042    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2651c02-536e-4519-825b-3b3901e4e3a2-tmp-dir\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:01:01.691079 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:01.691052    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2651c02-536e-4519-825b-3b3901e4e3a2-home\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:01:01.691079 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:01.691060    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2651c02-536e-4519-825b-3b3901e4e3a2-tls-certs\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:01:01.840953 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:01.840921    2559 generic.go:356] "Generic (PLEG): container finished" podID="f2651c02-536e-4519-825b-3b3901e4e3a2" containerID="19f1947512105e37d5127ff9202301f995c38ecb97ff6b31b1fa67c13e0b2477" exitCode=0
Aug 01 00:01:01.841130 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:01.840971    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfn7zzf"
Aug 01 00:01:01.841130 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:01.840991    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfn7zzf" event={"ID":"f2651c02-536e-4519-825b-3b3901e4e3a2","Type":"ContainerDied","Data":"19f1947512105e37d5127ff9202301f995c38ecb97ff6b31b1fa67c13e0b2477"}
Aug 01 00:01:01.841130 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:01.841053    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfn7zzf" event={"ID":"f2651c02-536e-4519-825b-3b3901e4e3a2","Type":"ContainerDied","Data":"ae71cad1e525a77549856e53d231008aa23ae8d808ac2b0b5bd510bcf3bdbfc9"}
Aug 01 00:01:01.841130 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:01.841073    2559 scope.go:117] "RemoveContainer" containerID="19f1947512105e37d5127ff9202301f995c38ecb97ff6b31b1fa67c13e0b2477"
Aug 01 00:01:01.849643 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:01.849628    2559 scope.go:117] "RemoveContainer" containerID="19f1947512105e37d5127ff9202301f995c38ecb97ff6b31b1fa67c13e0b2477"
Aug 01 00:01:01.849937 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:01:01.849918    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19f1947512105e37d5127ff9202301f995c38ecb97ff6b31b1fa67c13e0b2477\": container with ID starting with 19f1947512105e37d5127ff9202301f995c38ecb97ff6b31b1fa67c13e0b2477 not found: ID does not exist" containerID="19f1947512105e37d5127ff9202301f995c38ecb97ff6b31b1fa67c13e0b2477"
Aug 01 00:01:01.850000 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:01.849945    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19f1947512105e37d5127ff9202301f995c38ecb97ff6b31b1fa67c13e0b2477"} err="failed to get container status \"19f1947512105e37d5127ff9202301f995c38ecb97ff6b31b1fa67c13e0b2477\": rpc error: code = NotFound desc = could not find container \"19f1947512105e37d5127ff9202301f995c38ecb97ff6b31b1fa67c13e0b2477\": container with ID starting with 19f1947512105e37d5127ff9202301f995c38ecb97ff6b31b1fa67c13e0b2477 not found: ID does not exist"
Aug 01 00:01:01.857929 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:01.857901    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfn7zzf"]
Aug 01 00:01:01.859640 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:01.859621    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfn7zzf"]
Aug 01 00:01:03.785718 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:03.785684    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2651c02-536e-4519-825b-3b3901e4e3a2" path="/var/lib/kubelet/pods/f2651c02-536e-4519-825b-3b3901e4e3a2/volumes"
Aug 01 00:01:18.254787 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:18.254733    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7ddd76df96-djb28"]
Aug 01 00:01:18.255324 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:18.255072    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2651c02-536e-4519-825b-3b3901e4e3a2" containerName="main"
Aug 01 00:01:18.255324 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:18.255083    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2651c02-536e-4519-825b-3b3901e4e3a2" containerName="main"
Aug 01 00:01:18.255324 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:18.255138    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2651c02-536e-4519-825b-3b3901e4e3a2" containerName="main"
Aug 01 00:01:18.258100 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:18.258078    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7ddd76df96-djb28"
Aug 01 00:01:18.260806 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:18.260776    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"odh-kserve-custom-ca-bundle\""
Aug 01 00:01:18.260930 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:18.260777    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"seaweedfs-s3-creds\""
Aug 01 00:01:18.260930 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:18.260828    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-jsbbp\""
Aug 01 00:01:18.260930 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:18.260784    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Aug 01 00:01:18.260930 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:18.260888    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Aug 01 00:01:18.261177 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:18.261159    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Aug 01 00:01:18.268676 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:18.268657    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7ddd76df96-djb28"]
Aug 01 00:01:18.327852 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:18.327827    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db8e3776-295a-4128-9d02-16190e7e7167-tmp-dir\") pod \"auth-enabled-test-kserve-7ddd76df96-djb28\" (UID: \"db8e3776-295a-4128-9d02-16190e7e7167\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7ddd76df96-djb28"
Aug 01 00:01:18.327852 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:18.327853    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db8e3776-295a-4128-9d02-16190e7e7167-dshm\") pod \"auth-enabled-test-kserve-7ddd76df96-djb28\" (UID: \"db8e3776-295a-4128-9d02-16190e7e7167\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7ddd76df96-djb28"
Aug 01 00:01:18.328028 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:18.327870    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db8e3776-295a-4128-9d02-16190e7e7167-kserve-provision-location\") pod \"auth-enabled-test-kserve-7ddd76df96-djb28\" (UID: \"db8e3776-295a-4128-9d02-16190e7e7167\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7ddd76df96-djb28"
Aug 01 00:01:18.328028 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:18.327888    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db8e3776-295a-4128-9d02-16190e7e7167-cabundle-cert\") pod \"auth-enabled-test-kserve-7ddd76df96-djb28\" (UID: \"db8e3776-295a-4128-9d02-16190e7e7167\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7ddd76df96-djb28"
Aug 01 00:01:18.328028 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:18.327945    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db8e3776-295a-4128-9d02-16190e7e7167-home\") pod \"auth-enabled-test-kserve-7ddd76df96-djb28\" (UID: \"db8e3776-295a-4128-9d02-16190e7e7167\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7ddd76df96-djb28"
Aug 01 00:01:18.328028 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:18.328018    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7hbl\" (UniqueName: \"kubernetes.io/projected/db8e3776-295a-4128-9d02-16190e7e7167-kube-api-access-h7hbl\") pod \"auth-enabled-test-kserve-7ddd76df96-djb28\" (UID: \"db8e3776-295a-4128-9d02-16190e7e7167\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7ddd76df96-djb28"
Aug 01 00:01:18.328160 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:18.328041    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db8e3776-295a-4128-9d02-16190e7e7167-model-cache\") pod \"auth-enabled-test-kserve-7ddd76df96-djb28\" (UID: \"db8e3776-295a-4128-9d02-16190e7e7167\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7ddd76df96-djb28"
Aug 01 00:01:18.328160 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:18.328056    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db8e3776-295a-4128-9d02-16190e7e7167-tls-certs\") pod \"auth-enabled-test-kserve-7ddd76df96-djb28\" (UID: \"db8e3776-295a-4128-9d02-16190e7e7167\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7ddd76df96-djb28"
Aug 01 00:01:18.429240 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:18.429207    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7hbl\" (UniqueName: \"kubernetes.io/projected/db8e3776-295a-4128-9d02-16190e7e7167-kube-api-access-h7hbl\") pod \"auth-enabled-test-kserve-7ddd76df96-djb28\" (UID: \"db8e3776-295a-4128-9d02-16190e7e7167\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7ddd76df96-djb28"
Aug 01 00:01:18.429380 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:18.429246    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db8e3776-295a-4128-9d02-16190e7e7167-model-cache\") pod \"auth-enabled-test-kserve-7ddd76df96-djb28\" (UID: \"db8e3776-295a-4128-9d02-16190e7e7167\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7ddd76df96-djb28"
Aug 01 00:01:18.429380 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:18.429265    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db8e3776-295a-4128-9d02-16190e7e7167-tls-certs\") pod \"auth-enabled-test-kserve-7ddd76df96-djb28\" (UID: \"db8e3776-295a-4128-9d02-16190e7e7167\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7ddd76df96-djb28"
Aug 01 00:01:18.429380 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:18.429311    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db8e3776-295a-4128-9d02-16190e7e7167-tmp-dir\") pod \"auth-enabled-test-kserve-7ddd76df96-djb28\" (UID: \"db8e3776-295a-4128-9d02-16190e7e7167\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7ddd76df96-djb28"
Aug 01 00:01:18.429380 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:18.429328    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db8e3776-295a-4128-9d02-16190e7e7167-dshm\") pod \"auth-enabled-test-kserve-7ddd76df96-djb28\" (UID: \"db8e3776-295a-4128-9d02-16190e7e7167\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7ddd76df96-djb28"
Aug 01 00:01:18.429380 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:18.429342    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db8e3776-295a-4128-9d02-16190e7e7167-kserve-provision-location\") pod \"auth-enabled-test-kserve-7ddd76df96-djb28\" (UID: \"db8e3776-295a-4128-9d02-16190e7e7167\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7ddd76df96-djb28"
Aug 01 00:01:18.429380 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:18.429365    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db8e3776-295a-4128-9d02-16190e7e7167-cabundle-cert\") pod \"auth-enabled-test-kserve-7ddd76df96-djb28\" (UID: \"db8e3776-295a-4128-9d02-16190e7e7167\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7ddd76df96-djb28"
Aug 01 00:01:18.429661 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:18.429462    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db8e3776-295a-4128-9d02-16190e7e7167-home\") pod \"auth-enabled-test-kserve-7ddd76df96-djb28\" (UID: \"db8e3776-295a-4128-9d02-16190e7e7167\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7ddd76df96-djb28"
Aug 01 00:01:18.429717 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:18.429663    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db8e3776-295a-4128-9d02-16190e7e7167-model-cache\") pod \"auth-enabled-test-kserve-7ddd76df96-djb28\" (UID: \"db8e3776-295a-4128-9d02-16190e7e7167\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7ddd76df96-djb28"
Aug 01 00:01:18.429822 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:18.429716    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db8e3776-295a-4128-9d02-16190e7e7167-tmp-dir\") pod \"auth-enabled-test-kserve-7ddd76df96-djb28\" (UID: \"db8e3776-295a-4128-9d02-16190e7e7167\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7ddd76df96-djb28"
Aug 01 00:01:18.429822 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:18.429784    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db8e3776-295a-4128-9d02-16190e7e7167-kserve-provision-location\") pod \"auth-enabled-test-kserve-7ddd76df96-djb28\" (UID: \"db8e3776-295a-4128-9d02-16190e7e7167\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7ddd76df96-djb28"
Aug 01 00:01:18.429935 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:18.429879    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db8e3776-295a-4128-9d02-16190e7e7167-home\") pod \"auth-enabled-test-kserve-7ddd76df96-djb28\" (UID: \"db8e3776-295a-4128-9d02-16190e7e7167\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7ddd76df96-djb28"
Aug 01 00:01:18.430209 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:18.430187    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db8e3776-295a-4128-9d02-16190e7e7167-cabundle-cert\") pod \"auth-enabled-test-kserve-7ddd76df96-djb28\" (UID: \"db8e3776-295a-4128-9d02-16190e7e7167\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7ddd76df96-djb28"
Aug 01 00:01:18.431638 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:18.431617    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db8e3776-295a-4128-9d02-16190e7e7167-dshm\") pod \"auth-enabled-test-kserve-7ddd76df96-djb28\" (UID: \"db8e3776-295a-4128-9d02-16190e7e7167\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7ddd76df96-djb28"
Aug 01 00:01:18.431917 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:18.431903    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db8e3776-295a-4128-9d02-16190e7e7167-tls-certs\") pod \"auth-enabled-test-kserve-7ddd76df96-djb28\" (UID: \"db8e3776-295a-4128-9d02-16190e7e7167\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7ddd76df96-djb28"
Aug 01 00:01:18.437875 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:18.437858    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7hbl\" (UniqueName: \"kubernetes.io/projected/db8e3776-295a-4128-9d02-16190e7e7167-kube-api-access-h7hbl\") pod \"auth-enabled-test-kserve-7ddd76df96-djb28\" (UID: \"db8e3776-295a-4128-9d02-16190e7e7167\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7ddd76df96-djb28"
Aug 01 00:01:18.569856 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:18.569771    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7ddd76df96-djb28"
Aug 01 00:01:18.704244 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:18.704216    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7ddd76df96-djb28"]
Aug 01 00:01:18.706526 ip-10-0-130-148 kubenswrapper[2559]: W0801 00:01:18.706481    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb8e3776_295a_4128_9d02_16190e7e7167.slice/crio-cb04259304a7589ce6e61f6ae5121e2689b12c9b3ae0ea1ad887252c11943fdd WatchSource:0}: Error finding container cb04259304a7589ce6e61f6ae5121e2689b12c9b3ae0ea1ad887252c11943fdd: Status 404 returned error can't find the container with id cb04259304a7589ce6e61f6ae5121e2689b12c9b3ae0ea1ad887252c11943fdd
Aug 01 00:01:18.915592 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:18.915509    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7ddd76df96-djb28" event={"ID":"db8e3776-295a-4128-9d02-16190e7e7167","Type":"ContainerStarted","Data":"d1605e9e7e0f3eba48df57e872950dcda9468e23aa5724f2997bb55ef3c4815f"}
Aug 01 00:01:18.915592 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:18.915568    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7ddd76df96-djb28" event={"ID":"db8e3776-295a-4128-9d02-16190e7e7167","Type":"ContainerStarted","Data":"cb04259304a7589ce6e61f6ae5121e2689b12c9b3ae0ea1ad887252c11943fdd"}
Aug 01 00:01:20.924856 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:20.924820    2559 generic.go:356] "Generic (PLEG): container finished" podID="db8e3776-295a-4128-9d02-16190e7e7167" containerID="d1605e9e7e0f3eba48df57e872950dcda9468e23aa5724f2997bb55ef3c4815f" exitCode=0
Aug 01 00:01:20.925250 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:20.924900    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7ddd76df96-djb28" event={"ID":"db8e3776-295a-4128-9d02-16190e7e7167","Type":"ContainerDied","Data":"d1605e9e7e0f3eba48df57e872950dcda9468e23aa5724f2997bb55ef3c4815f"}
Aug 01 00:01:21.931457 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:21.931422    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7ddd76df96-djb28" event={"ID":"db8e3776-295a-4128-9d02-16190e7e7167","Type":"ContainerStarted","Data":"39c1cb595c5af73fb50fbf97421fb23be1a9dc43886733cf524eddc0c6e4ebe3"}
Aug 01 00:01:21.954661 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:21.954604    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7ddd76df96-djb28" podStartSLOduration=1.91580767 podStartE2EDuration="3.954589686s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.038782016s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:01:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:01:21.952293866 +0000 UTC m=+944.672093713" watchObservedRunningTime="2026-08-01 00:01:21.954589686 +0000 UTC m=+944.674389531"
Aug 01 00:01:22.570513 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:22.570479    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7ddd76df96-djb28"
Aug 01 00:01:28.570335 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:28.570296    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7ddd76df96-djb28"
Aug 01 00:01:28.571924 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:28.571895    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7ddd76df96-djb28" podUID="db8e3776-295a-4128-9d02-16190e7e7167" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/health\": dial tcp 10.132.0.33:8000: connect: connection refused"
Aug 01 00:01:38.570677 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:38.570621    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7ddd76df96-djb28" podUID="db8e3776-295a-4128-9d02-16190e7e7167" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/health\": dial tcp 10.132.0.33:8000: connect: connection refused"
Aug 01 00:01:48.570581 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:48.570535    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7ddd76df96-djb28" podUID="db8e3776-295a-4128-9d02-16190e7e7167" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/health\": dial tcp 10.132.0.33:8000: connect: connection refused"
Aug 01 00:01:58.571237 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:01:58.571186    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7ddd76df96-djb28" podUID="db8e3776-295a-4128-9d02-16190e7e7167" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/health\": dial tcp 10.132.0.33:8000: connect: connection refused"
Aug 01 00:02:08.570453 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:02:08.570359    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7ddd76df96-djb28" podUID="db8e3776-295a-4128-9d02-16190e7e7167" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/health\": dial tcp 10.132.0.33:8000: connect: connection refused"
Aug 01 00:02:18.571058 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:02:18.571004    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7ddd76df96-djb28" podUID="db8e3776-295a-4128-9d02-16190e7e7167" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/health\": dial tcp 10.132.0.33:8000: connect: connection refused"
Aug 01 00:02:28.571156 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:02:28.571111    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7ddd76df96-djb28" podUID="db8e3776-295a-4128-9d02-16190e7e7167" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/health\": dial tcp 10.132.0.33:8000: connect: connection refused"
Aug 01 00:02:38.570548 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:02:38.570501    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7ddd76df96-djb28" podUID="db8e3776-295a-4128-9d02-16190e7e7167" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/health\": dial tcp 10.132.0.33:8000: connect: connection refused"
Aug 01 00:02:48.571126 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:02:48.571078    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7ddd76df96-djb28" podUID="db8e3776-295a-4128-9d02-16190e7e7167" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/health\": dial tcp 10.132.0.33:8000: connect: connection refused"
Aug 01 00:02:58.570632 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:02:58.570584    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7ddd76df96-djb28" podUID="db8e3776-295a-4128-9d02-16190e7e7167" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/health\": dial tcp 10.132.0.33:8000: connect: connection refused"
Aug 01 00:03:08.571100 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:03:08.571053    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7ddd76df96-djb28" podUID="db8e3776-295a-4128-9d02-16190e7e7167" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/health\": dial tcp 10.132.0.33:8000: connect: connection refused"
Aug 01 00:03:18.582495 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:03:18.582456    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7ddd76df96-djb28"
Aug 01 00:03:18.591585 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:03:18.591560    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7ddd76df96-djb28"
Aug 01 00:03:25.399523 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:03:25.399490    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7ddd76df96-djb28"]
Aug 01 00:03:40.421581 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:03:40.421518    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7ddd76df96-djb28" podUID="db8e3776-295a-4128-9d02-16190e7e7167" containerName="main" containerID="cri-o://39c1cb595c5af73fb50fbf97421fb23be1a9dc43886733cf524eddc0c6e4ebe3" gracePeriod=45
Aug 01 00:03:40.584217 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:03:40.584176    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7ddd76df96-djb28" podUID="db8e3776-295a-4128-9d02-16190e7e7167" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/health\": dial tcp 10.132.0.33:8000: connect: connection refused"
Aug 01 00:03:41.081002 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:03:41.080978    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7ddd76df96-djb28"
Aug 01 00:03:41.138035 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:03:41.138005    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db8e3776-295a-4128-9d02-16190e7e7167-dshm\") pod \"db8e3776-295a-4128-9d02-16190e7e7167\" (UID: \"db8e3776-295a-4128-9d02-16190e7e7167\") "
Aug 01 00:03:41.138206 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:03:41.138046    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db8e3776-295a-4128-9d02-16190e7e7167-tmp-dir\") pod \"db8e3776-295a-4128-9d02-16190e7e7167\" (UID: \"db8e3776-295a-4128-9d02-16190e7e7167\") "
Aug 01 00:03:41.138206 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:03:41.138069    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db8e3776-295a-4128-9d02-16190e7e7167-model-cache\") pod \"db8e3776-295a-4128-9d02-16190e7e7167\" (UID: \"db8e3776-295a-4128-9d02-16190e7e7167\") "
Aug 01 00:03:41.138206 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:03:41.138109    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db8e3776-295a-4128-9d02-16190e7e7167-home\") pod \"db8e3776-295a-4128-9d02-16190e7e7167\" (UID: \"db8e3776-295a-4128-9d02-16190e7e7167\") "
Aug 01 00:03:41.138206 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:03:41.138191    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db8e3776-295a-4128-9d02-16190e7e7167-tls-certs\") pod \"db8e3776-295a-4128-9d02-16190e7e7167\" (UID: \"db8e3776-295a-4128-9d02-16190e7e7167\") "
Aug 01 00:03:41.138413 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:03:41.138223    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db8e3776-295a-4128-9d02-16190e7e7167-cabundle-cert\") pod \"db8e3776-295a-4128-9d02-16190e7e7167\" (UID: \"db8e3776-295a-4128-9d02-16190e7e7167\") "
Aug 01 00:03:41.138413 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:03:41.138259    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db8e3776-295a-4128-9d02-16190e7e7167-kserve-provision-location\") pod \"db8e3776-295a-4128-9d02-16190e7e7167\" (UID: \"db8e3776-295a-4128-9d02-16190e7e7167\") "
Aug 01 00:03:41.138413 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:03:41.138315    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h7hbl\" (UniqueName: \"kubernetes.io/projected/db8e3776-295a-4128-9d02-16190e7e7167-kube-api-access-h7hbl\") pod \"db8e3776-295a-4128-9d02-16190e7e7167\" (UID: \"db8e3776-295a-4128-9d02-16190e7e7167\") "
Aug 01 00:03:41.138413 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:03:41.138377    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db8e3776-295a-4128-9d02-16190e7e7167-model-cache" (OuterVolumeSpecName: "model-cache") pod "db8e3776-295a-4128-9d02-16190e7e7167" (UID: "db8e3776-295a-4128-9d02-16190e7e7167"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:03:41.138667 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:03:41.138646    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db8e3776-295a-4128-9d02-16190e7e7167-model-cache\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:03:41.139131 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:03:41.138919    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db8e3776-295a-4128-9d02-16190e7e7167-home" (OuterVolumeSpecName: "home") pod "db8e3776-295a-4128-9d02-16190e7e7167" (UID: "db8e3776-295a-4128-9d02-16190e7e7167"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:03:41.139333 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:03:41.139304    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db8e3776-295a-4128-9d02-16190e7e7167-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "db8e3776-295a-4128-9d02-16190e7e7167" (UID: "db8e3776-295a-4128-9d02-16190e7e7167"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:03:41.140585 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:03:41.140556    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db8e3776-295a-4128-9d02-16190e7e7167-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "db8e3776-295a-4128-9d02-16190e7e7167" (UID: "db8e3776-295a-4128-9d02-16190e7e7167"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:03:41.140585 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:03:41.140567    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db8e3776-295a-4128-9d02-16190e7e7167-kube-api-access-h7hbl" (OuterVolumeSpecName: "kube-api-access-h7hbl") pod "db8e3776-295a-4128-9d02-16190e7e7167" (UID: "db8e3776-295a-4128-9d02-16190e7e7167"). InnerVolumeSpecName "kube-api-access-h7hbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:03:41.140766 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:03:41.140623    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db8e3776-295a-4128-9d02-16190e7e7167-dshm" (OuterVolumeSpecName: "dshm") pod "db8e3776-295a-4128-9d02-16190e7e7167" (UID: "db8e3776-295a-4128-9d02-16190e7e7167"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:03:41.151014 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:03:41.150989    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db8e3776-295a-4128-9d02-16190e7e7167-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "db8e3776-295a-4128-9d02-16190e7e7167" (UID: "db8e3776-295a-4128-9d02-16190e7e7167"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:03:41.160805 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:03:41.160781    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db8e3776-295a-4128-9d02-16190e7e7167-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "db8e3776-295a-4128-9d02-16190e7e7167" (UID: "db8e3776-295a-4128-9d02-16190e7e7167"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:03:41.240074 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:03:41.240039    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7hbl\" (UniqueName: \"kubernetes.io/projected/db8e3776-295a-4128-9d02-16190e7e7167-kube-api-access-h7hbl\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:03:41.240074 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:03:41.240068    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db8e3776-295a-4128-9d02-16190e7e7167-dshm\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:03:41.240264 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:03:41.240081    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db8e3776-295a-4128-9d02-16190e7e7167-tmp-dir\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:03:41.240264 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:03:41.240092    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db8e3776-295a-4128-9d02-16190e7e7167-home\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:03:41.240264 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:03:41.240102    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db8e3776-295a-4128-9d02-16190e7e7167-tls-certs\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:03:41.240264 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:03:41.240113    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db8e3776-295a-4128-9d02-16190e7e7167-cabundle-cert\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:03:41.240264 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:03:41.240123    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db8e3776-295a-4128-9d02-16190e7e7167-kserve-provision-location\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:03:41.574142 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:03:41.574056    2559 generic.go:356] "Generic (PLEG): container finished" podID="db8e3776-295a-4128-9d02-16190e7e7167" containerID="39c1cb595c5af73fb50fbf97421fb23be1a9dc43886733cf524eddc0c6e4ebe3" exitCode=0
Aug 01 00:03:41.574142 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:03:41.574098    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7ddd76df96-djb28" event={"ID":"db8e3776-295a-4128-9d02-16190e7e7167","Type":"ContainerDied","Data":"39c1cb595c5af73fb50fbf97421fb23be1a9dc43886733cf524eddc0c6e4ebe3"}
Aug 01 00:03:41.574142 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:03:41.574109    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7ddd76df96-djb28"
Aug 01 00:03:41.574670 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:03:41.574147    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7ddd76df96-djb28" event={"ID":"db8e3776-295a-4128-9d02-16190e7e7167","Type":"ContainerDied","Data":"cb04259304a7589ce6e61f6ae5121e2689b12c9b3ae0ea1ad887252c11943fdd"}
Aug 01 00:03:41.574670 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:03:41.574173    2559 scope.go:117] "RemoveContainer" containerID="39c1cb595c5af73fb50fbf97421fb23be1a9dc43886733cf524eddc0c6e4ebe3"
Aug 01 00:03:41.583524 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:03:41.583506    2559 scope.go:117] "RemoveContainer" containerID="d1605e9e7e0f3eba48df57e872950dcda9468e23aa5724f2997bb55ef3c4815f"
Aug 01 00:03:41.590845 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:03:41.590827    2559 scope.go:117] "RemoveContainer" containerID="39c1cb595c5af73fb50fbf97421fb23be1a9dc43886733cf524eddc0c6e4ebe3"
Aug 01 00:03:41.591123 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:03:41.591102    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39c1cb595c5af73fb50fbf97421fb23be1a9dc43886733cf524eddc0c6e4ebe3\": container with ID starting with 39c1cb595c5af73fb50fbf97421fb23be1a9dc43886733cf524eddc0c6e4ebe3 not found: ID does not exist" containerID="39c1cb595c5af73fb50fbf97421fb23be1a9dc43886733cf524eddc0c6e4ebe3"
Aug 01 00:03:41.591164 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:03:41.591133    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39c1cb595c5af73fb50fbf97421fb23be1a9dc43886733cf524eddc0c6e4ebe3"} err="failed to get container status \"39c1cb595c5af73fb50fbf97421fb23be1a9dc43886733cf524eddc0c6e4ebe3\": rpc error: code = NotFound desc = could not find container \"39c1cb595c5af73fb50fbf97421fb23be1a9dc43886733cf524eddc0c6e4ebe3\": container with ID starting with 39c1cb595c5af73fb50fbf97421fb23be1a9dc43886733cf524eddc0c6e4ebe3 not found: ID does not exist"
Aug 01 00:03:41.591164 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:03:41.591151    2559 scope.go:117] "RemoveContainer" containerID="d1605e9e7e0f3eba48df57e872950dcda9468e23aa5724f2997bb55ef3c4815f"
Aug 01 00:03:41.591366 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:03:41.591348    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1605e9e7e0f3eba48df57e872950dcda9468e23aa5724f2997bb55ef3c4815f\": container with ID starting with d1605e9e7e0f3eba48df57e872950dcda9468e23aa5724f2997bb55ef3c4815f not found: ID does not exist" containerID="d1605e9e7e0f3eba48df57e872950dcda9468e23aa5724f2997bb55ef3c4815f"
Aug 01 00:03:41.591421 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:03:41.591374    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1605e9e7e0f3eba48df57e872950dcda9468e23aa5724f2997bb55ef3c4815f"} err="failed to get container status \"d1605e9e7e0f3eba48df57e872950dcda9468e23aa5724f2997bb55ef3c4815f\": rpc error: code = NotFound desc = could not find container \"d1605e9e7e0f3eba48df57e872950dcda9468e23aa5724f2997bb55ef3c4815f\": container with ID starting with d1605e9e7e0f3eba48df57e872950dcda9468e23aa5724f2997bb55ef3c4815f not found: ID does not exist"
Aug 01 00:03:41.597242 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:03:41.597220    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7ddd76df96-djb28"]
Aug 01 00:03:41.600354 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:03:41.600333    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7ddd76df96-djb28"]
Aug 01 00:03:41.785356 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:03:41.785321    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db8e3776-295a-4128-9d02-16190e7e7167" path="/var/lib/kubelet/pods/db8e3776-295a-4128-9d02-16190e7e7167/volumes"
Aug 01 00:04:22.217609 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:04:22.217577    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f775b9785-qpfwc"]
Aug 01 00:04:22.218044 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:04:22.217891    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db8e3776-295a-4128-9d02-16190e7e7167" containerName="main"
Aug 01 00:04:22.218044 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:04:22.217902    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="db8e3776-295a-4128-9d02-16190e7e7167" containerName="main"
Aug 01 00:04:22.218044 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:04:22.217914    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db8e3776-295a-4128-9d02-16190e7e7167" containerName="storage-initializer"
Aug 01 00:04:22.218044 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:04:22.217920    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="db8e3776-295a-4128-9d02-16190e7e7167" containerName="storage-initializer"
Aug 01 00:04:22.218044 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:04:22.217988    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="db8e3776-295a-4128-9d02-16190e7e7167" containerName="main"
Aug 01 00:04:22.220770 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:04:22.220734    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f775b9785-qpfwc"
Aug 01 00:04:22.227758 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:04:22.227717    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Aug 01 00:04:22.227758 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:04:22.227717    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"odh-kserve-custom-ca-bundle\""
Aug 01 00:04:22.227951 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:04:22.227768    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"seaweedfs-s3-creds\""
Aug 01 00:04:22.227951 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:04:22.227787    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-phjk2\""
Aug 01 00:04:22.227951 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:04:22.227772    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Aug 01 00:04:22.232891 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:04:22.232871    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Aug 01 00:04:22.245494 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:04:22.245471    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f775b9785-qpfwc"]
Aug 01 00:04:22.263557 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:04:22.263536    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5j8b\" (UniqueName: \"kubernetes.io/projected/a5305039-e9fb-4ea3-a726-f5a26219a294-kube-api-access-j5j8b\") pod \"auth-invalid-token-test-kserve-5f775b9785-qpfwc\" (UID: \"a5305039-e9fb-4ea3-a726-f5a26219a294\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f775b9785-qpfwc"
Aug 01 00:04:22.263674 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:04:22.263566    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5305039-e9fb-4ea3-a726-f5a26219a294-model-cache\") pod \"auth-invalid-token-test-kserve-5f775b9785-qpfwc\" (UID: \"a5305039-e9fb-4ea3-a726-f5a26219a294\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f775b9785-qpfwc"
Aug 01 00:04:22.263674 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:04:22.263586    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5305039-e9fb-4ea3-a726-f5a26219a294-dshm\") pod \"auth-invalid-token-test-kserve-5f775b9785-qpfwc\" (UID: \"a5305039-e9fb-4ea3-a726-f5a26219a294\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f775b9785-qpfwc"
Aug 01 00:04:22.263674 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:04:22.263605    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5305039-e9fb-4ea3-a726-f5a26219a294-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-5f775b9785-qpfwc\" (UID: \"a5305039-e9fb-4ea3-a726-f5a26219a294\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f775b9785-qpfwc"
Aug 01 00:04:22.263674 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:04:22.263651    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5305039-e9fb-4ea3-a726-f5a26219a294-cabundle-cert\") pod \"auth-invalid-token-test-kserve-5f775b9785-qpfwc\" (UID: \"a5305039-e9fb-4ea3-a726-f5a26219a294\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f775b9785-qpfwc"
Aug 01 00:04:22.263674 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:04:22.263674    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5305039-e9fb-4ea3-a726-f5a26219a294-tls-certs\") pod \"auth-invalid-token-test-kserve-5f775b9785-qpfwc\" (UID: \"a5305039-e9fb-4ea3-a726-f5a26219a294\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f775b9785-qpfwc"
Aug 01 00:04:22.263862 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:04:22.263693    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5305039-e9fb-4ea3-a726-f5a26219a294-tmp-dir\") pod \"auth-invalid-token-test-kserve-5f775b9785-qpfwc\" (UID: \"a5305039-e9fb-4ea3-a726-f5a26219a294\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f775b9785-qpfwc"
Aug 01 00:04:22.263862 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:04:22.263756    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5305039-e9fb-4ea3-a726-f5a26219a294-home\") pod \"auth-invalid-token-test-kserve-5f775b9785-qpfwc\" (UID: \"a5305039-e9fb-4ea3-a726-f5a26219a294\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f775b9785-qpfwc"
Aug 01 00:04:22.364536 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:04:22.364506    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5305039-e9fb-4ea3-a726-f5a26219a294-home\") pod \"auth-invalid-token-test-kserve-5f775b9785-qpfwc\" (UID: \"a5305039-e9fb-4ea3-a726-f5a26219a294\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f775b9785-qpfwc"
Aug 01 00:04:22.364721 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:04:22.364544    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5j8b\" (UniqueName: \"kubernetes.io/projected/a5305039-e9fb-4ea3-a726-f5a26219a294-kube-api-access-j5j8b\") pod \"auth-invalid-token-test-kserve-5f775b9785-qpfwc\" (UID: \"a5305039-e9fb-4ea3-a726-f5a26219a294\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f775b9785-qpfwc"
Aug 01 00:04:22.364721 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:04:22.364567    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5305039-e9fb-4ea3-a726-f5a26219a294-model-cache\") pod \"auth-invalid-token-test-kserve-5f775b9785-qpfwc\" (UID: \"a5305039-e9fb-4ea3-a726-f5a26219a294\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f775b9785-qpfwc"
Aug 01 00:04:22.364721 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:04:22.364624    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5305039-e9fb-4ea3-a726-f5a26219a294-dshm\") pod \"auth-invalid-token-test-kserve-5f775b9785-qpfwc\" (UID: \"a5305039-e9fb-4ea3-a726-f5a26219a294\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f775b9785-qpfwc"
Aug 01 00:04:22.364721 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:04:22.364662    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5305039-e9fb-4ea3-a726-f5a26219a294-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-5f775b9785-qpfwc\" (UID: \"a5305039-e9fb-4ea3-a726-f5a26219a294\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f775b9785-qpfwc"
Aug 01 00:04:22.364721 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:04:22.364689    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5305039-e9fb-4ea3-a726-f5a26219a294-cabundle-cert\") pod \"auth-invalid-token-test-kserve-5f775b9785-qpfwc\" (UID: \"a5305039-e9fb-4ea3-a726-f5a26219a294\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f775b9785-qpfwc"
Aug 01 00:04:22.365050 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:04:22.364728    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5305039-e9fb-4ea3-a726-f5a26219a294-tls-certs\") pod \"auth-invalid-token-test-kserve-5f775b9785-qpfwc\" (UID: \"a5305039-e9fb-4ea3-a726-f5a26219a294\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f775b9785-qpfwc"
Aug 01 00:04:22.365050 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:04:22.364775    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5305039-e9fb-4ea3-a726-f5a26219a294-tmp-dir\") pod \"auth-invalid-token-test-kserve-5f775b9785-qpfwc\" (UID: \"a5305039-e9fb-4ea3-a726-f5a26219a294\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f775b9785-qpfwc"
Aug 01 00:04:22.365050 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:04:22.364991    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5305039-e9fb-4ea3-a726-f5a26219a294-home\") pod \"auth-invalid-token-test-kserve-5f775b9785-qpfwc\" (UID: \"a5305039-e9fb-4ea3-a726-f5a26219a294\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f775b9785-qpfwc"
Aug 01 00:04:22.365050 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:04:22.365011    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5305039-e9fb-4ea3-a726-f5a26219a294-model-cache\") pod \"auth-invalid-token-test-kserve-5f775b9785-qpfwc\" (UID: \"a5305039-e9fb-4ea3-a726-f5a26219a294\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f775b9785-qpfwc"
Aug 01 00:04:22.365310 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:04:22.365291    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5305039-e9fb-4ea3-a726-f5a26219a294-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-5f775b9785-qpfwc\" (UID: \"a5305039-e9fb-4ea3-a726-f5a26219a294\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f775b9785-qpfwc"
Aug 01 00:04:22.365400 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:04:22.365373    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5305039-e9fb-4ea3-a726-f5a26219a294-tmp-dir\") pod \"auth-invalid-token-test-kserve-5f775b9785-qpfwc\" (UID: \"a5305039-e9fb-4ea3-a726-f5a26219a294\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f775b9785-qpfwc"
Aug 01 00:04:22.365457 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:04:22.365410    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5305039-e9fb-4ea3-a726-f5a26219a294-cabundle-cert\") pod \"auth-invalid-token-test-kserve-5f775b9785-qpfwc\" (UID: \"a5305039-e9fb-4ea3-a726-f5a26219a294\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f775b9785-qpfwc"
Aug 01 00:04:22.367102 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:04:22.367066    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5305039-e9fb-4ea3-a726-f5a26219a294-dshm\") pod \"auth-invalid-token-test-kserve-5f775b9785-qpfwc\" (UID: \"a5305039-e9fb-4ea3-a726-f5a26219a294\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f775b9785-qpfwc"
Aug 01 00:04:22.367373 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:04:22.367354    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5305039-e9fb-4ea3-a726-f5a26219a294-tls-certs\") pod \"auth-invalid-token-test-kserve-5f775b9785-qpfwc\" (UID: \"a5305039-e9fb-4ea3-a726-f5a26219a294\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f775b9785-qpfwc"
Aug 01 00:04:22.378468 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:04:22.378444    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5j8b\" (UniqueName: \"kubernetes.io/projected/a5305039-e9fb-4ea3-a726-f5a26219a294-kube-api-access-j5j8b\") pod \"auth-invalid-token-test-kserve-5f775b9785-qpfwc\" (UID: \"a5305039-e9fb-4ea3-a726-f5a26219a294\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f775b9785-qpfwc"
Aug 01 00:04:22.531152 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:04:22.531084    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f775b9785-qpfwc"
Aug 01 00:04:22.667729 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:04:22.667705    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f775b9785-qpfwc"]
Aug 01 00:04:22.669930 ip-10-0-130-148 kubenswrapper[2559]: W0801 00:04:22.669907    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5305039_e9fb_4ea3_a726_f5a26219a294.slice/crio-826b74e19f29bacf78b47042eed6a27b5dcac8af62e1ee0ac94fa9db448e5e6e WatchSource:0}: Error finding container 826b74e19f29bacf78b47042eed6a27b5dcac8af62e1ee0ac94fa9db448e5e6e: Status 404 returned error can't find the container with id 826b74e19f29bacf78b47042eed6a27b5dcac8af62e1ee0ac94fa9db448e5e6e
Aug 01 00:04:22.756156 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:04:22.756121    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f775b9785-qpfwc" event={"ID":"a5305039-e9fb-4ea3-a726-f5a26219a294","Type":"ContainerStarted","Data":"2bda99664cb3f156cf4b9be9e95d2713259d78c3bb22e6a8939f5a5072734d18"}
Aug 01 00:04:22.756275 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:04:22.756174    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f775b9785-qpfwc" event={"ID":"a5305039-e9fb-4ea3-a726-f5a26219a294","Type":"ContainerStarted","Data":"826b74e19f29bacf78b47042eed6a27b5dcac8af62e1ee0ac94fa9db448e5e6e"}
Aug 01 00:04:24.767104 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:04:24.767074    2559 generic.go:356] "Generic (PLEG): container finished" podID="a5305039-e9fb-4ea3-a726-f5a26219a294" containerID="2bda99664cb3f156cf4b9be9e95d2713259d78c3bb22e6a8939f5a5072734d18" exitCode=0
Aug 01 00:04:24.767436 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:04:24.767118    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f775b9785-qpfwc" event={"ID":"a5305039-e9fb-4ea3-a726-f5a26219a294","Type":"ContainerDied","Data":"2bda99664cb3f156cf4b9be9e95d2713259d78c3bb22e6a8939f5a5072734d18"}
Aug 01 00:04:25.773622 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:04:25.773586    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f775b9785-qpfwc" event={"ID":"a5305039-e9fb-4ea3-a726-f5a26219a294","Type":"ContainerStarted","Data":"2a1ac31538fa1edc690c52c35dab9dcac81c75f27797c921609d9a2068b2c48d"}
Aug 01 00:04:25.798399 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:04:25.798355    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f775b9785-qpfwc" podStartSLOduration=1.841458289 podStartE2EDuration="3.798336826s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.956878537s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:04:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:04:25.796297538 +0000 UTC m=+1128.516097384" watchObservedRunningTime="2026-08-01 00:04:25.798336826 +0000 UTC m=+1128.518136721"
Aug 01 00:04:26.532082 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:04:26.532048    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f775b9785-qpfwc"
Aug 01 00:04:32.531926 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:04:32.531886    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f775b9785-qpfwc"
Aug 01 00:04:32.533433 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:04:32.533403    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f775b9785-qpfwc" podUID="a5305039-e9fb-4ea3-a726-f5a26219a294" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8000/health\": dial tcp 10.132.0.34:8000: connect: connection refused"
Aug 01 00:04:42.532333 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:04:42.532292    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f775b9785-qpfwc" podUID="a5305039-e9fb-4ea3-a726-f5a26219a294" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8000/health\": dial tcp 10.132.0.34:8000: connect: connection refused"
Aug 01 00:04:52.532331 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:04:52.532278    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f775b9785-qpfwc" podUID="a5305039-e9fb-4ea3-a726-f5a26219a294" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8000/health\": dial tcp 10.132.0.34:8000: connect: connection refused"
Aug 01 00:05:02.532068 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:05:02.531971    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f775b9785-qpfwc" podUID="a5305039-e9fb-4ea3-a726-f5a26219a294" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8000/health\": dial tcp 10.132.0.34:8000: connect: connection refused"
Aug 01 00:05:12.531590 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:05:12.531546    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f775b9785-qpfwc" podUID="a5305039-e9fb-4ea3-a726-f5a26219a294" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8000/health\": dial tcp 10.132.0.34:8000: connect: connection refused"
Aug 01 00:05:22.531636 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:05:22.531585    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f775b9785-qpfwc" podUID="a5305039-e9fb-4ea3-a726-f5a26219a294" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8000/health\": dial tcp 10.132.0.34:8000: connect: connection refused"
Aug 01 00:05:32.532210 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:05:32.532157    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f775b9785-qpfwc" podUID="a5305039-e9fb-4ea3-a726-f5a26219a294" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8000/health\": dial tcp 10.132.0.34:8000: connect: connection refused"
Aug 01 00:05:37.772145 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:05:37.772112    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-spblg_c2e50946-aea4-4886-82ec-0bd5a31196cf/console-operator/1.log"
Aug 01 00:05:37.780358 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:05:37.780329    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-spblg_c2e50946-aea4-4886-82ec-0bd5a31196cf/console-operator/1.log"
Aug 01 00:05:42.532490 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:05:42.532444    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f775b9785-qpfwc" podUID="a5305039-e9fb-4ea3-a726-f5a26219a294" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8000/health\": dial tcp 10.132.0.34:8000: connect: connection refused"
Aug 01 00:05:52.531956 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:05:52.531915    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f775b9785-qpfwc" podUID="a5305039-e9fb-4ea3-a726-f5a26219a294" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8000/health\": dial tcp 10.132.0.34:8000: connect: connection refused"
Aug 01 00:06:02.531759 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:02.531687    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f775b9785-qpfwc" podUID="a5305039-e9fb-4ea3-a726-f5a26219a294" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8000/health\": dial tcp 10.132.0.34:8000: connect: connection refused"
Aug 01 00:06:12.532102 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:12.532047    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f775b9785-qpfwc" podUID="a5305039-e9fb-4ea3-a726-f5a26219a294" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8000/health\": dial tcp 10.132.0.34:8000: connect: connection refused"
Aug 01 00:06:22.541730 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:22.541699    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f775b9785-qpfwc"
Aug 01 00:06:22.549757 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:22.549724    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f775b9785-qpfwc"
Aug 01 00:06:24.191952 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:24.191921    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f775b9785-qpfwc"]
Aug 01 00:06:39.352054 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:39.352011    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f775b9785-qpfwc" podUID="a5305039-e9fb-4ea3-a726-f5a26219a294" containerName="main" containerID="cri-o://2a1ac31538fa1edc690c52c35dab9dcac81c75f27797c921609d9a2068b2c48d" gracePeriod=45
Aug 01 00:06:39.542703 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:39.542653    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f775b9785-qpfwc" podUID="a5305039-e9fb-4ea3-a726-f5a26219a294" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8000/health\": dial tcp 10.132.0.34:8000: connect: connection refused"
Aug 01 00:06:40.007664 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:40.007639    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f775b9785-qpfwc"
Aug 01 00:06:40.105693 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:40.105588    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5305039-e9fb-4ea3-a726-f5a26219a294-kserve-provision-location\") pod \"a5305039-e9fb-4ea3-a726-f5a26219a294\" (UID: \"a5305039-e9fb-4ea3-a726-f5a26219a294\") "
Aug 01 00:06:40.105693 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:40.105639    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5305039-e9fb-4ea3-a726-f5a26219a294-dshm\") pod \"a5305039-e9fb-4ea3-a726-f5a26219a294\" (UID: \"a5305039-e9fb-4ea3-a726-f5a26219a294\") "
Aug 01 00:06:40.105693 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:40.105662    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5305039-e9fb-4ea3-a726-f5a26219a294-tmp-dir\") pod \"a5305039-e9fb-4ea3-a726-f5a26219a294\" (UID: \"a5305039-e9fb-4ea3-a726-f5a26219a294\") "
Aug 01 00:06:40.106009 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:40.105702    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5305039-e9fb-4ea3-a726-f5a26219a294-model-cache\") pod \"a5305039-e9fb-4ea3-a726-f5a26219a294\" (UID: \"a5305039-e9fb-4ea3-a726-f5a26219a294\") "
Aug 01 00:06:40.106009 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:40.105795    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5305039-e9fb-4ea3-a726-f5a26219a294-tls-certs\") pod \"a5305039-e9fb-4ea3-a726-f5a26219a294\" (UID: \"a5305039-e9fb-4ea3-a726-f5a26219a294\") "
Aug 01 00:06:40.106009 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:40.105843    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5305039-e9fb-4ea3-a726-f5a26219a294-cabundle-cert\") pod \"a5305039-e9fb-4ea3-a726-f5a26219a294\" (UID: \"a5305039-e9fb-4ea3-a726-f5a26219a294\") "
Aug 01 00:06:40.106009 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:40.105865    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5305039-e9fb-4ea3-a726-f5a26219a294-home\") pod \"a5305039-e9fb-4ea3-a726-f5a26219a294\" (UID: \"a5305039-e9fb-4ea3-a726-f5a26219a294\") "
Aug 01 00:06:40.106009 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:40.105910    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5j8b\" (UniqueName: \"kubernetes.io/projected/a5305039-e9fb-4ea3-a726-f5a26219a294-kube-api-access-j5j8b\") pod \"a5305039-e9fb-4ea3-a726-f5a26219a294\" (UID: \"a5305039-e9fb-4ea3-a726-f5a26219a294\") "
Aug 01 00:06:40.106250 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:40.106210    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5305039-e9fb-4ea3-a726-f5a26219a294-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a5305039-e9fb-4ea3-a726-f5a26219a294" (UID: "a5305039-e9fb-4ea3-a726-f5a26219a294"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:06:40.106414 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:40.106333    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5305039-e9fb-4ea3-a726-f5a26219a294-cabundle-cert\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:06:40.106572 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:40.106541    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5305039-e9fb-4ea3-a726-f5a26219a294-model-cache" (OuterVolumeSpecName: "model-cache") pod "a5305039-e9fb-4ea3-a726-f5a26219a294" (UID: "a5305039-e9fb-4ea3-a726-f5a26219a294"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:06:40.106654 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:40.106633    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5305039-e9fb-4ea3-a726-f5a26219a294-home" (OuterVolumeSpecName: "home") pod "a5305039-e9fb-4ea3-a726-f5a26219a294" (UID: "a5305039-e9fb-4ea3-a726-f5a26219a294"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:06:40.108347 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:40.108323    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5305039-e9fb-4ea3-a726-f5a26219a294-kube-api-access-j5j8b" (OuterVolumeSpecName: "kube-api-access-j5j8b") pod "a5305039-e9fb-4ea3-a726-f5a26219a294" (UID: "a5305039-e9fb-4ea3-a726-f5a26219a294"). InnerVolumeSpecName "kube-api-access-j5j8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:06:40.108488 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:40.108466    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5305039-e9fb-4ea3-a726-f5a26219a294-dshm" (OuterVolumeSpecName: "dshm") pod "a5305039-e9fb-4ea3-a726-f5a26219a294" (UID: "a5305039-e9fb-4ea3-a726-f5a26219a294"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:06:40.108550 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:40.108500    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5305039-e9fb-4ea3-a726-f5a26219a294-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a5305039-e9fb-4ea3-a726-f5a26219a294" (UID: "a5305039-e9fb-4ea3-a726-f5a26219a294"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:06:40.125651 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:40.125615    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5305039-e9fb-4ea3-a726-f5a26219a294-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a5305039-e9fb-4ea3-a726-f5a26219a294" (UID: "a5305039-e9fb-4ea3-a726-f5a26219a294"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:06:40.134880 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:40.134854    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5305039-e9fb-4ea3-a726-f5a26219a294-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a5305039-e9fb-4ea3-a726-f5a26219a294" (UID: "a5305039-e9fb-4ea3-a726-f5a26219a294"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:06:40.207357 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:40.207325    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5305039-e9fb-4ea3-a726-f5a26219a294-kserve-provision-location\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:06:40.207357 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:40.207355    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5305039-e9fb-4ea3-a726-f5a26219a294-dshm\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:06:40.207539 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:40.207366    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5305039-e9fb-4ea3-a726-f5a26219a294-tmp-dir\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:06:40.207539 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:40.207376    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5305039-e9fb-4ea3-a726-f5a26219a294-model-cache\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:06:40.207539 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:40.207384    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5305039-e9fb-4ea3-a726-f5a26219a294-tls-certs\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:06:40.207539 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:40.207391    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5305039-e9fb-4ea3-a726-f5a26219a294-home\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:06:40.207539 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:40.207399    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5j8b\" (UniqueName: \"kubernetes.io/projected/a5305039-e9fb-4ea3-a726-f5a26219a294-kube-api-access-j5j8b\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:06:40.400452 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:40.400368    2559 generic.go:356] "Generic (PLEG): container finished" podID="a5305039-e9fb-4ea3-a726-f5a26219a294" containerID="2a1ac31538fa1edc690c52c35dab9dcac81c75f27797c921609d9a2068b2c48d" exitCode=0
Aug 01 00:06:40.400452 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:40.400417    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f775b9785-qpfwc"
Aug 01 00:06:40.400909 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:40.400414    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f775b9785-qpfwc" event={"ID":"a5305039-e9fb-4ea3-a726-f5a26219a294","Type":"ContainerDied","Data":"2a1ac31538fa1edc690c52c35dab9dcac81c75f27797c921609d9a2068b2c48d"}
Aug 01 00:06:40.400909 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:40.400536    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f775b9785-qpfwc" event={"ID":"a5305039-e9fb-4ea3-a726-f5a26219a294","Type":"ContainerDied","Data":"826b74e19f29bacf78b47042eed6a27b5dcac8af62e1ee0ac94fa9db448e5e6e"}
Aug 01 00:06:40.400909 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:40.400568    2559 scope.go:117] "RemoveContainer" containerID="2a1ac31538fa1edc690c52c35dab9dcac81c75f27797c921609d9a2068b2c48d"
Aug 01 00:06:40.409224 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:40.409209    2559 scope.go:117] "RemoveContainer" containerID="2bda99664cb3f156cf4b9be9e95d2713259d78c3bb22e6a8939f5a5072734d18"
Aug 01 00:06:40.416525 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:40.416510    2559 scope.go:117] "RemoveContainer" containerID="2a1ac31538fa1edc690c52c35dab9dcac81c75f27797c921609d9a2068b2c48d"
Aug 01 00:06:40.416801 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:06:40.416778    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a1ac31538fa1edc690c52c35dab9dcac81c75f27797c921609d9a2068b2c48d\": container with ID starting with 2a1ac31538fa1edc690c52c35dab9dcac81c75f27797c921609d9a2068b2c48d not found: ID does not exist" containerID="2a1ac31538fa1edc690c52c35dab9dcac81c75f27797c921609d9a2068b2c48d"
Aug 01 00:06:40.416862 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:40.416812    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a1ac31538fa1edc690c52c35dab9dcac81c75f27797c921609d9a2068b2c48d"} err="failed to get container status \"2a1ac31538fa1edc690c52c35dab9dcac81c75f27797c921609d9a2068b2c48d\": rpc error: code = NotFound desc = could not find container \"2a1ac31538fa1edc690c52c35dab9dcac81c75f27797c921609d9a2068b2c48d\": container with ID starting with 2a1ac31538fa1edc690c52c35dab9dcac81c75f27797c921609d9a2068b2c48d not found: ID does not exist"
Aug 01 00:06:40.416862 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:40.416830    2559 scope.go:117] "RemoveContainer" containerID="2bda99664cb3f156cf4b9be9e95d2713259d78c3bb22e6a8939f5a5072734d18"
Aug 01 00:06:40.417068 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:06:40.417049    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bda99664cb3f156cf4b9be9e95d2713259d78c3bb22e6a8939f5a5072734d18\": container with ID starting with 2bda99664cb3f156cf4b9be9e95d2713259d78c3bb22e6a8939f5a5072734d18 not found: ID does not exist" containerID="2bda99664cb3f156cf4b9be9e95d2713259d78c3bb22e6a8939f5a5072734d18"
Aug 01 00:06:40.417118 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:40.417075    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bda99664cb3f156cf4b9be9e95d2713259d78c3bb22e6a8939f5a5072734d18"} err="failed to get container status \"2bda99664cb3f156cf4b9be9e95d2713259d78c3bb22e6a8939f5a5072734d18\": rpc error: code = NotFound desc = could not find container \"2bda99664cb3f156cf4b9be9e95d2713259d78c3bb22e6a8939f5a5072734d18\": container with ID starting with 2bda99664cb3f156cf4b9be9e95d2713259d78c3bb22e6a8939f5a5072734d18 not found: ID does not exist"
Aug 01 00:06:40.425572 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:40.425548    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f775b9785-qpfwc"]
Aug 01 00:06:40.429678 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:40.429660    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f775b9785-qpfwc"]
Aug 01 00:06:41.785399 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:41.785357    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5305039-e9fb-4ea3-a726-f5a26219a294" path="/var/lib/kubelet/pods/a5305039-e9fb-4ea3-a726-f5a26219a294/volumes"
Aug 01 00:06:59.058084 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:59.058036    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-d78cb5c5d-5pc22"]
Aug 01 00:06:59.058674 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:59.058492    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5305039-e9fb-4ea3-a726-f5a26219a294" containerName="storage-initializer"
Aug 01 00:06:59.058674 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:59.058506    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5305039-e9fb-4ea3-a726-f5a26219a294" containerName="storage-initializer"
Aug 01 00:06:59.058674 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:59.058520    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5305039-e9fb-4ea3-a726-f5a26219a294" containerName="main"
Aug 01 00:06:59.058674 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:59.058529    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5305039-e9fb-4ea3-a726-f5a26219a294" containerName="main"
Aug 01 00:06:59.058674 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:59.058633    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5305039-e9fb-4ea3-a726-f5a26219a294" containerName="main"
Aug 01 00:06:59.062236 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:59.062214    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-d78cb5c5d-5pc22"
Aug 01 00:06:59.064922 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:59.064895    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"odh-kserve-custom-ca-bundle\""
Aug 01 00:06:59.064922 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:59.064910    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Aug 01 00:06:59.065116 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:59.064901    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-cnfsn\""
Aug 01 00:06:59.065116 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:59.064905    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"seaweedfs-s3-creds\""
Aug 01 00:06:59.065228 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:59.065122    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Aug 01 00:06:59.065228 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:59.065179    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Aug 01 00:06:59.073523 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:59.073501    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-d78cb5c5d-5pc22"]
Aug 01 00:06:59.157462 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:59.157410    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b909e30-8459-41ba-97ec-d35b27d86320-cabundle-cert\") pod \"auth-disabled-test-kserve-d78cb5c5d-5pc22\" (UID: \"8b909e30-8459-41ba-97ec-d35b27d86320\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-d78cb5c5d-5pc22"
Aug 01 00:06:59.157462 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:59.157462    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b909e30-8459-41ba-97ec-d35b27d86320-model-cache\") pod \"auth-disabled-test-kserve-d78cb5c5d-5pc22\" (UID: \"8b909e30-8459-41ba-97ec-d35b27d86320\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-d78cb5c5d-5pc22"
Aug 01 00:06:59.157678 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:59.157494    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b909e30-8459-41ba-97ec-d35b27d86320-dshm\") pod \"auth-disabled-test-kserve-d78cb5c5d-5pc22\" (UID: \"8b909e30-8459-41ba-97ec-d35b27d86320\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-d78cb5c5d-5pc22"
Aug 01 00:06:59.157678 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:59.157550    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b909e30-8459-41ba-97ec-d35b27d86320-tls-certs\") pod \"auth-disabled-test-kserve-d78cb5c5d-5pc22\" (UID: \"8b909e30-8459-41ba-97ec-d35b27d86320\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-d78cb5c5d-5pc22"
Aug 01 00:06:59.157678 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:59.157583    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b909e30-8459-41ba-97ec-d35b27d86320-home\") pod \"auth-disabled-test-kserve-d78cb5c5d-5pc22\" (UID: \"8b909e30-8459-41ba-97ec-d35b27d86320\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-d78cb5c5d-5pc22"
Aug 01 00:06:59.157678 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:59.157622    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k4ft\" (UniqueName: \"kubernetes.io/projected/8b909e30-8459-41ba-97ec-d35b27d86320-kube-api-access-4k4ft\") pod \"auth-disabled-test-kserve-d78cb5c5d-5pc22\" (UID: \"8b909e30-8459-41ba-97ec-d35b27d86320\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-d78cb5c5d-5pc22"
Aug 01 00:06:59.157849 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:59.157690    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b909e30-8459-41ba-97ec-d35b27d86320-kserve-provision-location\") pod \"auth-disabled-test-kserve-d78cb5c5d-5pc22\" (UID: \"8b909e30-8459-41ba-97ec-d35b27d86320\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-d78cb5c5d-5pc22"
Aug 01 00:06:59.157849 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:59.157736    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b909e30-8459-41ba-97ec-d35b27d86320-tmp-dir\") pod \"auth-disabled-test-kserve-d78cb5c5d-5pc22\" (UID: \"8b909e30-8459-41ba-97ec-d35b27d86320\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-d78cb5c5d-5pc22"
Aug 01 00:06:59.258966 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:59.258937    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b909e30-8459-41ba-97ec-d35b27d86320-kserve-provision-location\") pod \"auth-disabled-test-kserve-d78cb5c5d-5pc22\" (UID: \"8b909e30-8459-41ba-97ec-d35b27d86320\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-d78cb5c5d-5pc22"
Aug 01 00:06:59.259118 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:59.258981    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b909e30-8459-41ba-97ec-d35b27d86320-tmp-dir\") pod \"auth-disabled-test-kserve-d78cb5c5d-5pc22\" (UID: \"8b909e30-8459-41ba-97ec-d35b27d86320\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-d78cb5c5d-5pc22"
Aug 01 00:06:59.259118 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:59.259001    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b909e30-8459-41ba-97ec-d35b27d86320-cabundle-cert\") pod \"auth-disabled-test-kserve-d78cb5c5d-5pc22\" (UID: \"8b909e30-8459-41ba-97ec-d35b27d86320\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-d78cb5c5d-5pc22"
Aug 01 00:06:59.259192 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:59.259115    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b909e30-8459-41ba-97ec-d35b27d86320-model-cache\") pod \"auth-disabled-test-kserve-d78cb5c5d-5pc22\" (UID: \"8b909e30-8459-41ba-97ec-d35b27d86320\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-d78cb5c5d-5pc22"
Aug 01 00:06:59.259192 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:59.259174    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b909e30-8459-41ba-97ec-d35b27d86320-dshm\") pod \"auth-disabled-test-kserve-d78cb5c5d-5pc22\" (UID: \"8b909e30-8459-41ba-97ec-d35b27d86320\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-d78cb5c5d-5pc22"
Aug 01 00:06:59.259285 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:59.259206    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b909e30-8459-41ba-97ec-d35b27d86320-tls-certs\") pod \"auth-disabled-test-kserve-d78cb5c5d-5pc22\" (UID: \"8b909e30-8459-41ba-97ec-d35b27d86320\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-d78cb5c5d-5pc22"
Aug 01 00:06:59.259285 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:59.259259    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b909e30-8459-41ba-97ec-d35b27d86320-home\") pod \"auth-disabled-test-kserve-d78cb5c5d-5pc22\" (UID: \"8b909e30-8459-41ba-97ec-d35b27d86320\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-d78cb5c5d-5pc22"
Aug 01 00:06:59.259393 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:59.259322    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k4ft\" (UniqueName: \"kubernetes.io/projected/8b909e30-8459-41ba-97ec-d35b27d86320-kube-api-access-4k4ft\") pod \"auth-disabled-test-kserve-d78cb5c5d-5pc22\" (UID: \"8b909e30-8459-41ba-97ec-d35b27d86320\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-d78cb5c5d-5pc22"
Aug 01 00:06:59.259451 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:59.259383    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b909e30-8459-41ba-97ec-d35b27d86320-tmp-dir\") pod \"auth-disabled-test-kserve-d78cb5c5d-5pc22\" (UID: \"8b909e30-8459-41ba-97ec-d35b27d86320\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-d78cb5c5d-5pc22"
Aug 01 00:06:59.259451 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:59.259404    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b909e30-8459-41ba-97ec-d35b27d86320-kserve-provision-location\") pod \"auth-disabled-test-kserve-d78cb5c5d-5pc22\" (UID: \"8b909e30-8459-41ba-97ec-d35b27d86320\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-d78cb5c5d-5pc22"
Aug 01 00:06:59.259571 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:59.259547    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b909e30-8459-41ba-97ec-d35b27d86320-model-cache\") pod \"auth-disabled-test-kserve-d78cb5c5d-5pc22\" (UID: \"8b909e30-8459-41ba-97ec-d35b27d86320\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-d78cb5c5d-5pc22"
Aug 01 00:06:59.259668 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:59.259645    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b909e30-8459-41ba-97ec-d35b27d86320-home\") pod \"auth-disabled-test-kserve-d78cb5c5d-5pc22\" (UID: \"8b909e30-8459-41ba-97ec-d35b27d86320\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-d78cb5c5d-5pc22"
Aug 01 00:06:59.259778 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:59.259761    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b909e30-8459-41ba-97ec-d35b27d86320-cabundle-cert\") pod \"auth-disabled-test-kserve-d78cb5c5d-5pc22\" (UID: \"8b909e30-8459-41ba-97ec-d35b27d86320\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-d78cb5c5d-5pc22"
Aug 01 00:06:59.261441 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:59.261425    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b909e30-8459-41ba-97ec-d35b27d86320-dshm\") pod \"auth-disabled-test-kserve-d78cb5c5d-5pc22\" (UID: \"8b909e30-8459-41ba-97ec-d35b27d86320\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-d78cb5c5d-5pc22"
Aug 01 00:06:59.261594 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:59.261578    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b909e30-8459-41ba-97ec-d35b27d86320-tls-certs\") pod \"auth-disabled-test-kserve-d78cb5c5d-5pc22\" (UID: \"8b909e30-8459-41ba-97ec-d35b27d86320\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-d78cb5c5d-5pc22"
Aug 01 00:06:59.270960 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:59.270939    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k4ft\" (UniqueName: \"kubernetes.io/projected/8b909e30-8459-41ba-97ec-d35b27d86320-kube-api-access-4k4ft\") pod \"auth-disabled-test-kserve-d78cb5c5d-5pc22\" (UID: \"8b909e30-8459-41ba-97ec-d35b27d86320\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-d78cb5c5d-5pc22"
Aug 01 00:06:59.373689 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:59.373607    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-d78cb5c5d-5pc22"
Aug 01 00:06:59.504659 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:06:59.504628    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-d78cb5c5d-5pc22"]
Aug 01 00:06:59.507758 ip-10-0-130-148 kubenswrapper[2559]: W0801 00:06:59.507708    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b909e30_8459_41ba_97ec_d35b27d86320.slice/crio-b0b92d710b817a59ed2025670a913a3d38d27b875ecd20f66e8dafebc65388b0 WatchSource:0}: Error finding container b0b92d710b817a59ed2025670a913a3d38d27b875ecd20f66e8dafebc65388b0: Status 404 returned error can't find the container with id b0b92d710b817a59ed2025670a913a3d38d27b875ecd20f66e8dafebc65388b0
Aug 01 00:07:00.489391 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:07:00.489354    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-d78cb5c5d-5pc22" event={"ID":"8b909e30-8459-41ba-97ec-d35b27d86320","Type":"ContainerStarted","Data":"bbb356323b2e12d848e02c13debe05edafe38340d5293d58ffadad31028a37bc"}
Aug 01 00:07:00.489780 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:07:00.489410    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-d78cb5c5d-5pc22" event={"ID":"8b909e30-8459-41ba-97ec-d35b27d86320","Type":"ContainerStarted","Data":"b0b92d710b817a59ed2025670a913a3d38d27b875ecd20f66e8dafebc65388b0"}
Aug 01 00:07:02.498977 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:07:02.498946    2559 generic.go:356] "Generic (PLEG): container finished" podID="8b909e30-8459-41ba-97ec-d35b27d86320" containerID="bbb356323b2e12d848e02c13debe05edafe38340d5293d58ffadad31028a37bc" exitCode=0
Aug 01 00:07:02.498977 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:07:02.498980    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-d78cb5c5d-5pc22" event={"ID":"8b909e30-8459-41ba-97ec-d35b27d86320","Type":"ContainerDied","Data":"bbb356323b2e12d848e02c13debe05edafe38340d5293d58ffadad31028a37bc"}
Aug 01 00:07:03.506041 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:07:03.506001    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-d78cb5c5d-5pc22" event={"ID":"8b909e30-8459-41ba-97ec-d35b27d86320","Type":"ContainerStarted","Data":"892ff4588b2b699b9bf9cf313c7a49e3913479fe93582b0b13242b402cd4ab39"}
Aug 01 00:07:03.526725 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:07:03.526660    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-d78cb5c5d-5pc22" podStartSLOduration=2.444830691 podStartE2EDuration="4.526640767s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.081810076s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:06:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:07:03.526247355 +0000 UTC m=+1286.246047204" watchObservedRunningTime="2026-08-01 00:07:03.526640767 +0000 UTC m=+1286.246440615"
Aug 01 00:07:04.373826 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:07:04.373785    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-d78cb5c5d-5pc22"
Aug 01 00:07:09.374459 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:07:09.374419    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-d78cb5c5d-5pc22"
Aug 01 00:07:09.375946 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:07:09.375918    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-d78cb5c5d-5pc22" podUID="8b909e30-8459-41ba-97ec-d35b27d86320" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Aug 01 00:07:19.374691 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:07:19.374645    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-d78cb5c5d-5pc22" podUID="8b909e30-8459-41ba-97ec-d35b27d86320" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Aug 01 00:07:29.374396 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:07:29.374344    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-d78cb5c5d-5pc22" podUID="8b909e30-8459-41ba-97ec-d35b27d86320" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Aug 01 00:07:35.405841 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:07:35.405806    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fqbxgn"]
Aug 01 00:07:35.408323 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:07:35.408301    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fqbxgn"
Aug 01 00:07:35.411217 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:07:35.411192    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Aug 01 00:07:35.411376 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:07:35.411276    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-kqv2f\""
Aug 01 00:07:35.411376 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:07:35.411311    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Aug 01 00:07:35.412473 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:07:35.412454    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Aug 01 00:07:35.419833 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:07:35.419813    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fqbxgn"]
Aug 01 00:07:35.457991 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:07:35.457949    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e9ba5a4-d36f-4611-8871-7061468d61ce-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fqbxgn\" (UID: \"4e9ba5a4-d36f-4611-8871-7061468d61ce\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fqbxgn"
Aug 01 00:07:35.458175 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:07:35.458007    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e9ba5a4-d36f-4611-8871-7061468d61ce-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fqbxgn\" (UID: \"4e9ba5a4-d36f-4611-8871-7061468d61ce\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fqbxgn"
Aug 01 00:07:35.458175 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:07:35.458050    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqtb5\" (UniqueName: \"kubernetes.io/projected/4e9ba5a4-d36f-4611-8871-7061468d61ce-kube-api-access-bqtb5\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fqbxgn\" (UID: \"4e9ba5a4-d36f-4611-8871-7061468d61ce\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fqbxgn"
Aug 01 00:07:35.458175 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:07:35.458077    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e9ba5a4-d36f-4611-8871-7061468d61ce-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fqbxgn\" (UID: \"4e9ba5a4-d36f-4611-8871-7061468d61ce\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fqbxgn"
Aug 01 00:07:35.458175 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:07:35.458128    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e9ba5a4-d36f-4611-8871-7061468d61ce-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fqbxgn\" (UID: \"4e9ba5a4-d36f-4611-8871-7061468d61ce\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fqbxgn"
Aug 01 00:07:35.458402 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:07:35.458225    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e9ba5a4-d36f-4611-8871-7061468d61ce-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fqbxgn\" (UID: \"4e9ba5a4-d36f-4611-8871-7061468d61ce\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fqbxgn"
Aug 01 00:07:35.558979 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:07:35.558936    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqtb5\" (UniqueName: \"kubernetes.io/projected/4e9ba5a4-d36f-4611-8871-7061468d61ce-kube-api-access-bqtb5\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fqbxgn\" (UID: \"4e9ba5a4-d36f-4611-8871-7061468d61ce\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fqbxgn"
Aug 01 00:07:35.558979 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:07:35.558971    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e9ba5a4-d36f-4611-8871-7061468d61ce-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fqbxgn\" (UID: \"4e9ba5a4-d36f-4611-8871-7061468d61ce\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fqbxgn"
Aug 01 00:07:35.559235 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:07:35.558994    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e9ba5a4-d36f-4611-8871-7061468d61ce-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fqbxgn\" (UID: \"4e9ba5a4-d36f-4611-8871-7061468d61ce\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fqbxgn"
Aug 01 00:07:35.559235 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:07:35.559039    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e9ba5a4-d36f-4611-8871-7061468d61ce-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fqbxgn\" (UID: \"4e9ba5a4-d36f-4611-8871-7061468d61ce\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fqbxgn"
Aug 01 00:07:35.559235 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:07:35.559073    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e9ba5a4-d36f-4611-8871-7061468d61ce-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fqbxgn\" (UID: \"4e9ba5a4-d36f-4611-8871-7061468d61ce\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fqbxgn"
Aug 01 00:07:35.559235 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:07:35.559093    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e9ba5a4-d36f-4611-8871-7061468d61ce-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fqbxgn\" (UID: \"4e9ba5a4-d36f-4611-8871-7061468d61ce\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fqbxgn"
Aug 01 00:07:35.559462 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:07:35.559405    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e9ba5a4-d36f-4611-8871-7061468d61ce-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fqbxgn\" (UID: \"4e9ba5a4-d36f-4611-8871-7061468d61ce\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fqbxgn"
Aug 01 00:07:35.559462 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:07:35.559433    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e9ba5a4-d36f-4611-8871-7061468d61ce-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fqbxgn\" (UID: \"4e9ba5a4-d36f-4611-8871-7061468d61ce\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fqbxgn"
Aug 01 00:07:35.559569 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:07:35.559479    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e9ba5a4-d36f-4611-8871-7061468d61ce-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fqbxgn\" (UID: \"4e9ba5a4-d36f-4611-8871-7061468d61ce\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fqbxgn"
Aug 01 00:07:35.561434 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:07:35.561406    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e9ba5a4-d36f-4611-8871-7061468d61ce-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fqbxgn\" (UID: \"4e9ba5a4-d36f-4611-8871-7061468d61ce\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fqbxgn"
Aug 01 00:07:35.561654 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:07:35.561632    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e9ba5a4-d36f-4611-8871-7061468d61ce-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fqbxgn\" (UID: \"4e9ba5a4-d36f-4611-8871-7061468d61ce\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fqbxgn"
Aug 01 00:07:35.571109 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:07:35.571082    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqtb5\" (UniqueName: \"kubernetes.io/projected/4e9ba5a4-d36f-4611-8871-7061468d61ce-kube-api-access-bqtb5\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fqbxgn\" (UID: \"4e9ba5a4-d36f-4611-8871-7061468d61ce\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fqbxgn"
Aug 01 00:07:35.719562 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:07:35.719521    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fqbxgn"
Aug 01 00:07:35.859069 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:07:35.859041    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fqbxgn"]
Aug 01 00:07:35.861045 ip-10-0-130-148 kubenswrapper[2559]: W0801 00:07:35.861018    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e9ba5a4_d36f_4611_8871_7061468d61ce.slice/crio-5ded8a2530ed3de03be51491504e6013a6fb4da1e0183481e642b7cb92e6ce90 WatchSource:0}: Error finding container 5ded8a2530ed3de03be51491504e6013a6fb4da1e0183481e642b7cb92e6ce90: Status 404 returned error can't find the container with id 5ded8a2530ed3de03be51491504e6013a6fb4da1e0183481e642b7cb92e6ce90
Aug 01 00:07:36.656676 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:07:36.656638    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fqbxgn" event={"ID":"4e9ba5a4-d36f-4611-8871-7061468d61ce","Type":"ContainerStarted","Data":"b7e5d8bfbf1cc1b27400f276d8e01362c9d47de12bde935c38ee7f5be46e3792"}
Aug 01 00:07:36.657045 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:07:36.656723    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fqbxgn" event={"ID":"4e9ba5a4-d36f-4611-8871-7061468d61ce","Type":"ContainerStarted","Data":"5ded8a2530ed3de03be51491504e6013a6fb4da1e0183481e642b7cb92e6ce90"}
Aug 01 00:07:36.673915 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:07:36.673862    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fqbxgn" podStartSLOduration=1.673848333 podStartE2EDuration="1.673848333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:07:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:07:36.672802955 +0000 UTC m=+1319.392602827" watchObservedRunningTime="2026-08-01 00:07:36.673848333 +0000 UTC m=+1319.393648178"
Aug 01 00:07:36.719912 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:07:36.719877    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fqbxgn"
Aug 01 00:07:39.374625 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:07:39.374581    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-d78cb5c5d-5pc22" podUID="8b909e30-8459-41ba-97ec-d35b27d86320" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Aug 01 00:07:45.720513 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:07:45.720480    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fqbxgn"
Aug 01 00:07:45.733079 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:07:45.733055    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fqbxgn"
Aug 01 00:07:46.720170 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:07:46.720133    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fqbxgn"
Aug 01 00:07:49.374255 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:07:49.374206    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-d78cb5c5d-5pc22" podUID="8b909e30-8459-41ba-97ec-d35b27d86320" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Aug 01 00:07:59.374987 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:07:59.374935    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-d78cb5c5d-5pc22" podUID="8b909e30-8459-41ba-97ec-d35b27d86320" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Aug 01 00:08:09.283385 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:09.283290    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fqbxgn"]
Aug 01 00:08:09.293331 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:08:09.293292    2559 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 01 00:08:09.293331 ip-10-0-130-148 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:08:09.293331 ip-10-0-130-148 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Aug 01 00:08:09.293331 ip-10-0-130-148 kubenswrapper[2559]:  > containerID="b7e5d8bfbf1cc1b27400f276d8e01362c9d47de12bde935c38ee7f5be46e3792" cmd=["/bin/sleep","15"]
Aug 01 00:08:09.293636 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:09.293354    2559 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fqbxgn" message="" err=<
Aug 01 00:08:09.293636 ip-10-0-130-148 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:08:09.293636 ip-10-0-130-148 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Aug 01 00:08:09.293636 ip-10-0-130-148 kubenswrapper[2559]:  >
Aug 01 00:08:09.293636 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:08:09.293370    2559 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 01 00:08:09.293636 ip-10-0-130-148 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:08:09.293636 ip-10-0-130-148 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Aug 01 00:08:09.293636 ip-10-0-130-148 kubenswrapper[2559]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fqbxgn" podUID="4e9ba5a4-d36f-4611-8871-7061468d61ce" containerName="main" containerID="cri-o://b7e5d8bfbf1cc1b27400f276d8e01362c9d47de12bde935c38ee7f5be46e3792"
Aug 01 00:08:09.293636 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:09.293426    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fqbxgn" podUID="4e9ba5a4-d36f-4611-8871-7061468d61ce" containerName="main" containerID="cri-o://b7e5d8bfbf1cc1b27400f276d8e01362c9d47de12bde935c38ee7f5be46e3792" gracePeriod=60
Aug 01 00:08:09.374377 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:09.374336    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-d78cb5c5d-5pc22" podUID="8b909e30-8459-41ba-97ec-d35b27d86320" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Aug 01 00:08:09.634146 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:09.634119    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fqbxgn"
Aug 01 00:08:09.664504 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:09.664474    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bqtb5\" (UniqueName: \"kubernetes.io/projected/4e9ba5a4-d36f-4611-8871-7061468d61ce-kube-api-access-bqtb5\") pod \"4e9ba5a4-d36f-4611-8871-7061468d61ce\" (UID: \"4e9ba5a4-d36f-4611-8871-7061468d61ce\") "
Aug 01 00:08:09.664640 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:09.664543    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e9ba5a4-d36f-4611-8871-7061468d61ce-model-cache\") pod \"4e9ba5a4-d36f-4611-8871-7061468d61ce\" (UID: \"4e9ba5a4-d36f-4611-8871-7061468d61ce\") "
Aug 01 00:08:09.664640 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:09.664604    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e9ba5a4-d36f-4611-8871-7061468d61ce-dshm\") pod \"4e9ba5a4-d36f-4611-8871-7061468d61ce\" (UID: \"4e9ba5a4-d36f-4611-8871-7061468d61ce\") "
Aug 01 00:08:09.664734 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:09.664652    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e9ba5a4-d36f-4611-8871-7061468d61ce-home\") pod \"4e9ba5a4-d36f-4611-8871-7061468d61ce\" (UID: \"4e9ba5a4-d36f-4611-8871-7061468d61ce\") "
Aug 01 00:08:09.664734 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:09.664681    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e9ba5a4-d36f-4611-8871-7061468d61ce-tls-certs\") pod \"4e9ba5a4-d36f-4611-8871-7061468d61ce\" (UID: \"4e9ba5a4-d36f-4611-8871-7061468d61ce\") "
Aug 01 00:08:09.664734 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:09.664703    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e9ba5a4-d36f-4611-8871-7061468d61ce-tmp-dir\") pod \"4e9ba5a4-d36f-4611-8871-7061468d61ce\" (UID: \"4e9ba5a4-d36f-4611-8871-7061468d61ce\") "
Aug 01 00:08:09.664919 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:09.664867    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e9ba5a4-d36f-4611-8871-7061468d61ce-model-cache" (OuterVolumeSpecName: "model-cache") pod "4e9ba5a4-d36f-4611-8871-7061468d61ce" (UID: "4e9ba5a4-d36f-4611-8871-7061468d61ce"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:08:09.664978 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:09.664949    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e9ba5a4-d36f-4611-8871-7061468d61ce-home" (OuterVolumeSpecName: "home") pod "4e9ba5a4-d36f-4611-8871-7061468d61ce" (UID: "4e9ba5a4-d36f-4611-8871-7061468d61ce"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:08:09.665032 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:09.665007    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e9ba5a4-d36f-4611-8871-7061468d61ce-model-cache\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:08:09.665127 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:09.665106    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e9ba5a4-d36f-4611-8871-7061468d61ce-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4e9ba5a4-d36f-4611-8871-7061468d61ce" (UID: "4e9ba5a4-d36f-4611-8871-7061468d61ce"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:08:09.666975 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:09.666949    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e9ba5a4-d36f-4611-8871-7061468d61ce-kube-api-access-bqtb5" (OuterVolumeSpecName: "kube-api-access-bqtb5") pod "4e9ba5a4-d36f-4611-8871-7061468d61ce" (UID: "4e9ba5a4-d36f-4611-8871-7061468d61ce"). InnerVolumeSpecName "kube-api-access-bqtb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:08:09.667254 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:09.667223    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e9ba5a4-d36f-4611-8871-7061468d61ce-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4e9ba5a4-d36f-4611-8871-7061468d61ce" (UID: "4e9ba5a4-d36f-4611-8871-7061468d61ce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:08:09.667342 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:09.667263    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e9ba5a4-d36f-4611-8871-7061468d61ce-dshm" (OuterVolumeSpecName: "dshm") pod "4e9ba5a4-d36f-4611-8871-7061468d61ce" (UID: "4e9ba5a4-d36f-4611-8871-7061468d61ce"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:08:09.765971 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:09.765935    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e9ba5a4-d36f-4611-8871-7061468d61ce-dshm\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:08:09.765971 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:09.765965    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e9ba5a4-d36f-4611-8871-7061468d61ce-home\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:08:09.765971 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:09.765974    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e9ba5a4-d36f-4611-8871-7061468d61ce-tls-certs\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:08:09.766193 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:09.766004    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e9ba5a4-d36f-4611-8871-7061468d61ce-tmp-dir\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:08:09.766193 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:09.766014    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqtb5\" (UniqueName: \"kubernetes.io/projected/4e9ba5a4-d36f-4611-8871-7061468d61ce-kube-api-access-bqtb5\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:08:09.814875 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:09.814784    2559 generic.go:356] "Generic (PLEG): container finished" podID="4e9ba5a4-d36f-4611-8871-7061468d61ce" containerID="b7e5d8bfbf1cc1b27400f276d8e01362c9d47de12bde935c38ee7f5be46e3792" exitCode=0
Aug 01 00:08:09.814875 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:09.814836    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fqbxgn"
Aug 01 00:08:09.815075 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:09.814874    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fqbxgn" event={"ID":"4e9ba5a4-d36f-4611-8871-7061468d61ce","Type":"ContainerDied","Data":"b7e5d8bfbf1cc1b27400f276d8e01362c9d47de12bde935c38ee7f5be46e3792"}
Aug 01 00:08:09.815075 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:09.814924    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fqbxgn" event={"ID":"4e9ba5a4-d36f-4611-8871-7061468d61ce","Type":"ContainerDied","Data":"5ded8a2530ed3de03be51491504e6013a6fb4da1e0183481e642b7cb92e6ce90"}
Aug 01 00:08:09.815075 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:09.814945    2559 scope.go:117] "RemoveContainer" containerID="b7e5d8bfbf1cc1b27400f276d8e01362c9d47de12bde935c38ee7f5be46e3792"
Aug 01 00:08:09.823806 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:09.823785    2559 scope.go:117] "RemoveContainer" containerID="b7e5d8bfbf1cc1b27400f276d8e01362c9d47de12bde935c38ee7f5be46e3792"
Aug 01 00:08:09.824076 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:08:09.824058    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7e5d8bfbf1cc1b27400f276d8e01362c9d47de12bde935c38ee7f5be46e3792\": container with ID starting with b7e5d8bfbf1cc1b27400f276d8e01362c9d47de12bde935c38ee7f5be46e3792 not found: ID does not exist" containerID="b7e5d8bfbf1cc1b27400f276d8e01362c9d47de12bde935c38ee7f5be46e3792"
Aug 01 00:08:09.824144 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:09.824089    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7e5d8bfbf1cc1b27400f276d8e01362c9d47de12bde935c38ee7f5be46e3792"} err="failed to get container status \"b7e5d8bfbf1cc1b27400f276d8e01362c9d47de12bde935c38ee7f5be46e3792\": rpc error: code = NotFound desc = could not find container \"b7e5d8bfbf1cc1b27400f276d8e01362c9d47de12bde935c38ee7f5be46e3792\": container with ID starting with b7e5d8bfbf1cc1b27400f276d8e01362c9d47de12bde935c38ee7f5be46e3792 not found: ID does not exist"
Aug 01 00:08:09.831177 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:09.831156    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fqbxgn"]
Aug 01 00:08:09.839297 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:09.839277    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fqbxgn"]
Aug 01 00:08:11.786924 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:11.786887    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e9ba5a4-d36f-4611-8871-7061468d61ce" path="/var/lib/kubelet/pods/4e9ba5a4-d36f-4611-8871-7061468d61ce/volumes"
Aug 01 00:08:19.374641 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:19.374595    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-d78cb5c5d-5pc22" podUID="8b909e30-8459-41ba-97ec-d35b27d86320" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Aug 01 00:08:29.374110 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:29.374061    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-d78cb5c5d-5pc22" podUID="8b909e30-8459-41ba-97ec-d35b27d86320" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Aug 01 00:08:39.374709 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:39.374652    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-d78cb5c5d-5pc22" podUID="8b909e30-8459-41ba-97ec-d35b27d86320" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Aug 01 00:08:44.370869 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:44.370824    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2r46n"]
Aug 01 00:08:44.371323 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:44.371298    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e9ba5a4-d36f-4611-8871-7061468d61ce" containerName="main"
Aug 01 00:08:44.371323 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:44.371320    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e9ba5a4-d36f-4611-8871-7061468d61ce" containerName="main"
Aug 01 00:08:44.371439 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:44.371427    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e9ba5a4-d36f-4611-8871-7061468d61ce" containerName="main"
Aug 01 00:08:44.374727 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:44.374701    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2r46n"
Aug 01 00:08:44.377510 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:44.377476    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Aug 01 00:08:44.377637 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:44.377535    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Aug 01 00:08:44.377637 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:44.377475    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Aug 01 00:08:44.377637 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:44.377570    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Aug 01 00:08:44.377840 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:44.377711    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Aug 01 00:08:44.378868 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:44.378848    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-9gzgn\""
Aug 01 00:08:44.385288 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:44.385262    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2r46n"]
Aug 01 00:08:44.463865 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:44.463834    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89e04bde-4256-4539-837e-1961d441b74f-home\") pod \"tokenizer-clean-path-test-kserve-567447fd4-2r46n\" (UID: \"89e04bde-4256-4539-837e-1961d441b74f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2r46n"
Aug 01 00:08:44.463865 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:44.463866    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89e04bde-4256-4539-837e-1961d441b74f-model-cache\") pod \"tokenizer-clean-path-test-kserve-567447fd4-2r46n\" (UID: \"89e04bde-4256-4539-837e-1961d441b74f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2r46n"
Aug 01 00:08:44.464085 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:44.463897    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89e04bde-4256-4539-837e-1961d441b74f-dshm\") pod \"tokenizer-clean-path-test-kserve-567447fd4-2r46n\" (UID: \"89e04bde-4256-4539-837e-1961d441b74f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2r46n"
Aug 01 00:08:44.464085 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:44.463958    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89e04bde-4256-4539-837e-1961d441b74f-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-567447fd4-2r46n\" (UID: \"89e04bde-4256-4539-837e-1961d441b74f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2r46n"
Aug 01 00:08:44.464085 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:44.463997    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89e04bde-4256-4539-837e-1961d441b74f-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-567447fd4-2r46n\" (UID: \"89e04bde-4256-4539-837e-1961d441b74f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2r46n"
Aug 01 00:08:44.464085 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:44.464018    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89e04bde-4256-4539-837e-1961d441b74f-tls-certs\") pod \"tokenizer-clean-path-test-kserve-567447fd4-2r46n\" (UID: \"89e04bde-4256-4539-837e-1961d441b74f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2r46n"
Aug 01 00:08:44.464085 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:44.464067    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6jj7\" (UniqueName: \"kubernetes.io/projected/89e04bde-4256-4539-837e-1961d441b74f-kube-api-access-h6jj7\") pod \"tokenizer-clean-path-test-kserve-567447fd4-2r46n\" (UID: \"89e04bde-4256-4539-837e-1961d441b74f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2r46n"
Aug 01 00:08:44.464251 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:44.464089    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89e04bde-4256-4539-837e-1961d441b74f-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-567447fd4-2r46n\" (UID: \"89e04bde-4256-4539-837e-1961d441b74f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2r46n"
Aug 01 00:08:44.564511 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:44.564472    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6jj7\" (UniqueName: \"kubernetes.io/projected/89e04bde-4256-4539-837e-1961d441b74f-kube-api-access-h6jj7\") pod \"tokenizer-clean-path-test-kserve-567447fd4-2r46n\" (UID: \"89e04bde-4256-4539-837e-1961d441b74f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2r46n"
Aug 01 00:08:44.564511 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:44.564517    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89e04bde-4256-4539-837e-1961d441b74f-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-567447fd4-2r46n\" (UID: \"89e04bde-4256-4539-837e-1961d441b74f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2r46n"
Aug 01 00:08:44.564773 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:44.564541    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89e04bde-4256-4539-837e-1961d441b74f-home\") pod \"tokenizer-clean-path-test-kserve-567447fd4-2r46n\" (UID: \"89e04bde-4256-4539-837e-1961d441b74f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2r46n"
Aug 01 00:08:44.564773 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:44.564558    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89e04bde-4256-4539-837e-1961d441b74f-model-cache\") pod \"tokenizer-clean-path-test-kserve-567447fd4-2r46n\" (UID: \"89e04bde-4256-4539-837e-1961d441b74f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2r46n"
Aug 01 00:08:44.564773 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:44.564589    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89e04bde-4256-4539-837e-1961d441b74f-dshm\") pod \"tokenizer-clean-path-test-kserve-567447fd4-2r46n\" (UID: \"89e04bde-4256-4539-837e-1961d441b74f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2r46n"
Aug 01 00:08:44.564773 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:44.564621    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89e04bde-4256-4539-837e-1961d441b74f-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-567447fd4-2r46n\" (UID: \"89e04bde-4256-4539-837e-1961d441b74f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2r46n"
Aug 01 00:08:44.564773 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:44.564652    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89e04bde-4256-4539-837e-1961d441b74f-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-567447fd4-2r46n\" (UID: \"89e04bde-4256-4539-837e-1961d441b74f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2r46n"
Aug 01 00:08:44.564773 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:44.564680    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89e04bde-4256-4539-837e-1961d441b74f-tls-certs\") pod \"tokenizer-clean-path-test-kserve-567447fd4-2r46n\" (UID: \"89e04bde-4256-4539-837e-1961d441b74f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2r46n"
Aug 01 00:08:44.565153 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:44.565040    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89e04bde-4256-4539-837e-1961d441b74f-home\") pod \"tokenizer-clean-path-test-kserve-567447fd4-2r46n\" (UID: \"89e04bde-4256-4539-837e-1961d441b74f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2r46n"
Aug 01 00:08:44.565153 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:44.565079    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89e04bde-4256-4539-837e-1961d441b74f-model-cache\") pod \"tokenizer-clean-path-test-kserve-567447fd4-2r46n\" (UID: \"89e04bde-4256-4539-837e-1961d441b74f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2r46n"
Aug 01 00:08:44.565153 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:44.565125    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89e04bde-4256-4539-837e-1961d441b74f-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-567447fd4-2r46n\" (UID: \"89e04bde-4256-4539-837e-1961d441b74f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2r46n"
Aug 01 00:08:44.565310 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:44.565260    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89e04bde-4256-4539-837e-1961d441b74f-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-567447fd4-2r46n\" (UID: \"89e04bde-4256-4539-837e-1961d441b74f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2r46n"
Aug 01 00:08:44.565453 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:44.565427    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89e04bde-4256-4539-837e-1961d441b74f-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-567447fd4-2r46n\" (UID: \"89e04bde-4256-4539-837e-1961d441b74f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2r46n"
Aug 01 00:08:44.567083 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:44.567063    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89e04bde-4256-4539-837e-1961d441b74f-dshm\") pod \"tokenizer-clean-path-test-kserve-567447fd4-2r46n\" (UID: \"89e04bde-4256-4539-837e-1961d441b74f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2r46n"
Aug 01 00:08:44.567212 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:44.567195    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89e04bde-4256-4539-837e-1961d441b74f-tls-certs\") pod \"tokenizer-clean-path-test-kserve-567447fd4-2r46n\" (UID: \"89e04bde-4256-4539-837e-1961d441b74f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2r46n"
Aug 01 00:08:44.577526 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:44.577495    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6jj7\" (UniqueName: \"kubernetes.io/projected/89e04bde-4256-4539-837e-1961d441b74f-kube-api-access-h6jj7\") pod \"tokenizer-clean-path-test-kserve-567447fd4-2r46n\" (UID: \"89e04bde-4256-4539-837e-1961d441b74f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2r46n"
Aug 01 00:08:44.596158 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:44.596129    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c56f55965-6bbn8"]
Aug 01 00:08:44.599228 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:44.599205    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c56f55965-6bbn8"
Aug 01 00:08:44.610275 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:44.610239    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c56f55965-6bbn8"]
Aug 01 00:08:44.665200 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:44.665108    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41c739a2-ec92-4fd4-8e17-d444d3062108-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-c56f55965-6bbn8\" (UID: \"41c739a2-ec92-4fd4-8e17-d444d3062108\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c56f55965-6bbn8"
Aug 01 00:08:44.665200 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:44.665169    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41c739a2-ec92-4fd4-8e17-d444d3062108-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-c56f55965-6bbn8\" (UID: \"41c739a2-ec92-4fd4-8e17-d444d3062108\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c56f55965-6bbn8"
Aug 01 00:08:44.665379 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:44.665250    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41c739a2-ec92-4fd4-8e17-d444d3062108-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-c56f55965-6bbn8\" (UID: \"41c739a2-ec92-4fd4-8e17-d444d3062108\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c56f55965-6bbn8"
Aug 01 00:08:44.665379 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:44.665290    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41c739a2-ec92-4fd4-8e17-d444d3062108-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-c56f55965-6bbn8\" (UID: \"41c739a2-ec92-4fd4-8e17-d444d3062108\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c56f55965-6bbn8"
Aug 01 00:08:44.689137 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:44.689094    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2r46n"
Aug 01 00:08:44.765736 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:44.765704    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41c739a2-ec92-4fd4-8e17-d444d3062108-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-c56f55965-6bbn8\" (UID: \"41c739a2-ec92-4fd4-8e17-d444d3062108\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c56f55965-6bbn8"
Aug 01 00:08:44.765959 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:44.765858    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41c739a2-ec92-4fd4-8e17-d444d3062108-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-c56f55965-6bbn8\" (UID: \"41c739a2-ec92-4fd4-8e17-d444d3062108\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c56f55965-6bbn8"
Aug 01 00:08:44.765959 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:44.765942    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41c739a2-ec92-4fd4-8e17-d444d3062108-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-c56f55965-6bbn8\" (UID: \"41c739a2-ec92-4fd4-8e17-d444d3062108\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c56f55965-6bbn8"
Aug 01 00:08:44.766265 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:44.766242    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41c739a2-ec92-4fd4-8e17-d444d3062108-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-c56f55965-6bbn8\" (UID: \"41c739a2-ec92-4fd4-8e17-d444d3062108\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c56f55965-6bbn8"
Aug 01 00:08:44.766265 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:44.766257    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41c739a2-ec92-4fd4-8e17-d444d3062108-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-c56f55965-6bbn8\" (UID: \"41c739a2-ec92-4fd4-8e17-d444d3062108\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c56f55965-6bbn8"
Aug 01 00:08:44.766389 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:44.766355    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41c739a2-ec92-4fd4-8e17-d444d3062108-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-c56f55965-6bbn8\" (UID: \"41c739a2-ec92-4fd4-8e17-d444d3062108\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c56f55965-6bbn8"
Aug 01 00:08:44.766526 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:44.766504    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41c739a2-ec92-4fd4-8e17-d444d3062108-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-c56f55965-6bbn8\" (UID: \"41c739a2-ec92-4fd4-8e17-d444d3062108\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c56f55965-6bbn8"
Aug 01 00:08:44.768529 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:44.768508    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41c739a2-ec92-4fd4-8e17-d444d3062108-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-c56f55965-6bbn8\" (UID: \"41c739a2-ec92-4fd4-8e17-d444d3062108\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c56f55965-6bbn8"
Aug 01 00:08:44.823953 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:44.823780    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2r46n"]
Aug 01 00:08:44.826142 ip-10-0-130-148 kubenswrapper[2559]: W0801 00:08:44.826115    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89e04bde_4256_4539_837e_1961d441b74f.slice/crio-cb4be894c8057af44386f275d8bef2298df0f42683a7d601becb84ae4b468e61 WatchSource:0}: Error finding container cb4be894c8057af44386f275d8bef2298df0f42683a7d601becb84ae4b468e61: Status 404 returned error can't find the container with id cb4be894c8057af44386f275d8bef2298df0f42683a7d601becb84ae4b468e61
Aug 01 00:08:44.910028 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:44.909994    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c56f55965-6bbn8"
Aug 01 00:08:44.979382 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:44.979340    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2r46n" event={"ID":"89e04bde-4256-4539-837e-1961d441b74f","Type":"ContainerStarted","Data":"b23c2b74ae445213d0a6eafc8592f39e83e9ad73ba873d6eda4ec9cc37a5babe"}
Aug 01 00:08:44.979545 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:44.979392    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2r46n" event={"ID":"89e04bde-4256-4539-837e-1961d441b74f","Type":"ContainerStarted","Data":"cb4be894c8057af44386f275d8bef2298df0f42683a7d601becb84ae4b468e61"}
Aug 01 00:08:45.057039 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:45.057009    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c56f55965-6bbn8"]
Aug 01 00:08:45.059525 ip-10-0-130-148 kubenswrapper[2559]: W0801 00:08:45.059480    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41c739a2_ec92_4fd4_8e17_d444d3062108.slice/crio-2772ceb7143656762d21429f9cc9d211808d2bef504450675081d010bb709336 WatchSource:0}: Error finding container 2772ceb7143656762d21429f9cc9d211808d2bef504450675081d010bb709336: Status 404 returned error can't find the container with id 2772ceb7143656762d21429f9cc9d211808d2bef504450675081d010bb709336
Aug 01 00:08:45.986085 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:45.986038    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c56f55965-6bbn8" event={"ID":"41c739a2-ec92-4fd4-8e17-d444d3062108","Type":"ContainerStarted","Data":"2b5583c12d72014994b5be0c545cc90033326aa746ace8d5c81853b88b84f37d"}
Aug 01 00:08:45.986472 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:45.986176    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c56f55965-6bbn8" event={"ID":"41c739a2-ec92-4fd4-8e17-d444d3062108","Type":"ContainerStarted","Data":"2772ceb7143656762d21429f9cc9d211808d2bef504450675081d010bb709336"}
Aug 01 00:08:47.998456 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:47.998425    2559 generic.go:356] "Generic (PLEG): container finished" podID="89e04bde-4256-4539-837e-1961d441b74f" containerID="b23c2b74ae445213d0a6eafc8592f39e83e9ad73ba873d6eda4ec9cc37a5babe" exitCode=0
Aug 01 00:08:47.999001 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:47.998497    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2r46n" event={"ID":"89e04bde-4256-4539-837e-1961d441b74f","Type":"ContainerDied","Data":"b23c2b74ae445213d0a6eafc8592f39e83e9ad73ba873d6eda4ec9cc37a5babe"}
Aug 01 00:08:48.000050 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:48.000030    2559 generic.go:356] "Generic (PLEG): container finished" podID="41c739a2-ec92-4fd4-8e17-d444d3062108" containerID="2b5583c12d72014994b5be0c545cc90033326aa746ace8d5c81853b88b84f37d" exitCode=0
Aug 01 00:08:48.000155 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:48.000079    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c56f55965-6bbn8" event={"ID":"41c739a2-ec92-4fd4-8e17-d444d3062108","Type":"ContainerDied","Data":"2b5583c12d72014994b5be0c545cc90033326aa746ace8d5c81853b88b84f37d"}
Aug 01 00:08:48.001125 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:48.001106    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 01 00:08:49.009604 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:49.009566    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2r46n" event={"ID":"89e04bde-4256-4539-837e-1961d441b74f","Type":"ContainerStarted","Data":"83d336ba3bf2cb183a98755290350c4088f21ef4c919788718ea04be2c06cafa"}
Aug 01 00:08:49.040575 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:49.040520    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2r46n" podStartSLOduration=2.663251352 podStartE2EDuration="5.040499355s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.377248003s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:08:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:08:49.038694916 +0000 UTC m=+1391.758494786" watchObservedRunningTime="2026-08-01 00:08:49.040499355 +0000 UTC m=+1391.760299205"
Aug 01 00:08:49.374644 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:49.374416    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-d78cb5c5d-5pc22" podUID="8b909e30-8459-41ba-97ec-d35b27d86320" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Aug 01 00:08:49.689404 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:49.689263    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2r46n"
Aug 01 00:08:54.690074 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:54.690033    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2r46n"
Aug 01 00:08:54.706064 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:54.706035    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2r46n"
Aug 01 00:08:55.053882 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:55.053855    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2r46n"
Aug 01 00:08:59.386510 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:59.386473    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-d78cb5c5d-5pc22"
Aug 01 00:08:59.398079 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:08:59.398050    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-d78cb5c5d-5pc22"
Aug 01 00:09:12.923266 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:09:12.923230    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-d78cb5c5d-5pc22"]
Aug 01 00:09:27.216285 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:09:27.216231    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c56f55965-6bbn8" event={"ID":"41c739a2-ec92-4fd4-8e17-d444d3062108","Type":"ContainerStarted","Data":"162414d17f9315371991f4c48e89ba6e74ef375ea82de41eaf7c4951507e3baa"}
Aug 01 00:09:27.216803 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:09:27.216491    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c56f55965-6bbn8"
Aug 01 00:09:27.237552 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:09:27.237493    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c56f55965-6bbn8" podStartSLOduration=2.256816542 podStartE2EDuration="43.237473499s" totalImagesPullingTime="38.677021792s" totalInitContainerRuntime="2.303635165s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:08:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:09:27.236228804 +0000 UTC m=+1429.956028711" watchObservedRunningTime="2026-08-01 00:09:27.237473499 +0000 UTC m=+1429.957273347"
Aug 01 00:09:27.947996 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:09:27.947958    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-d78cb5c5d-5pc22" podUID="8b909e30-8459-41ba-97ec-d35b27d86320" containerName="main" containerID="cri-o://892ff4588b2b699b9bf9cf313c7a49e3913479fe93582b0b13242b402cd4ab39" gracePeriod=45
Aug 01 00:09:28.388086 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:09:28.387987    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-d78cb5c5d-5pc22" podUID="8b909e30-8459-41ba-97ec-d35b27d86320" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Aug 01 00:09:28.760236 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:09:28.760200    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-d78cb5c5d-5pc22"
Aug 01 00:09:28.874858 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:09:28.874825    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b909e30-8459-41ba-97ec-d35b27d86320-tmp-dir\") pod \"8b909e30-8459-41ba-97ec-d35b27d86320\" (UID: \"8b909e30-8459-41ba-97ec-d35b27d86320\") "
Aug 01 00:09:28.875050 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:09:28.874884    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4k4ft\" (UniqueName: \"kubernetes.io/projected/8b909e30-8459-41ba-97ec-d35b27d86320-kube-api-access-4k4ft\") pod \"8b909e30-8459-41ba-97ec-d35b27d86320\" (UID: \"8b909e30-8459-41ba-97ec-d35b27d86320\") "
Aug 01 00:09:28.875050 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:09:28.874954    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b909e30-8459-41ba-97ec-d35b27d86320-cabundle-cert\") pod \"8b909e30-8459-41ba-97ec-d35b27d86320\" (UID: \"8b909e30-8459-41ba-97ec-d35b27d86320\") "
Aug 01 00:09:28.875050 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:09:28.874978    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b909e30-8459-41ba-97ec-d35b27d86320-kserve-provision-location\") pod \"8b909e30-8459-41ba-97ec-d35b27d86320\" (UID: \"8b909e30-8459-41ba-97ec-d35b27d86320\") "
Aug 01 00:09:28.875050 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:09:28.875010    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b909e30-8459-41ba-97ec-d35b27d86320-dshm\") pod \"8b909e30-8459-41ba-97ec-d35b27d86320\" (UID: \"8b909e30-8459-41ba-97ec-d35b27d86320\") "
Aug 01 00:09:28.875050 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:09:28.875031    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b909e30-8459-41ba-97ec-d35b27d86320-tls-certs\") pod \"8b909e30-8459-41ba-97ec-d35b27d86320\" (UID: \"8b909e30-8459-41ba-97ec-d35b27d86320\") "
Aug 01 00:09:28.875321 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:09:28.875085    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b909e30-8459-41ba-97ec-d35b27d86320-home\") pod \"8b909e30-8459-41ba-97ec-d35b27d86320\" (UID: \"8b909e30-8459-41ba-97ec-d35b27d86320\") "
Aug 01 00:09:28.875321 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:09:28.875133    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b909e30-8459-41ba-97ec-d35b27d86320-model-cache\") pod \"8b909e30-8459-41ba-97ec-d35b27d86320\" (UID: \"8b909e30-8459-41ba-97ec-d35b27d86320\") "
Aug 01 00:09:28.875321 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:09:28.875301    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b909e30-8459-41ba-97ec-d35b27d86320-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8b909e30-8459-41ba-97ec-d35b27d86320" (UID: "8b909e30-8459-41ba-97ec-d35b27d86320"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:09:28.875473 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:09:28.875392    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b909e30-8459-41ba-97ec-d35b27d86320-cabundle-cert\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:09:28.875574 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:09:28.875545    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b909e30-8459-41ba-97ec-d35b27d86320-model-cache" (OuterVolumeSpecName: "model-cache") pod "8b909e30-8459-41ba-97ec-d35b27d86320" (UID: "8b909e30-8459-41ba-97ec-d35b27d86320"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:09:28.876068 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:09:28.875931    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b909e30-8459-41ba-97ec-d35b27d86320-home" (OuterVolumeSpecName: "home") pod "8b909e30-8459-41ba-97ec-d35b27d86320" (UID: "8b909e30-8459-41ba-97ec-d35b27d86320"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:09:28.877787 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:09:28.877758    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b909e30-8459-41ba-97ec-d35b27d86320-kube-api-access-4k4ft" (OuterVolumeSpecName: "kube-api-access-4k4ft") pod "8b909e30-8459-41ba-97ec-d35b27d86320" (UID: "8b909e30-8459-41ba-97ec-d35b27d86320"). InnerVolumeSpecName "kube-api-access-4k4ft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:09:28.877970 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:09:28.877949    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b909e30-8459-41ba-97ec-d35b27d86320-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8b909e30-8459-41ba-97ec-d35b27d86320" (UID: "8b909e30-8459-41ba-97ec-d35b27d86320"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:09:28.878572 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:09:28.878549    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b909e30-8459-41ba-97ec-d35b27d86320-dshm" (OuterVolumeSpecName: "dshm") pod "8b909e30-8459-41ba-97ec-d35b27d86320" (UID: "8b909e30-8459-41ba-97ec-d35b27d86320"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:09:28.886698 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:09:28.886675    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b909e30-8459-41ba-97ec-d35b27d86320-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8b909e30-8459-41ba-97ec-d35b27d86320" (UID: "8b909e30-8459-41ba-97ec-d35b27d86320"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:09:28.904817 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:09:28.904733    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b909e30-8459-41ba-97ec-d35b27d86320-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8b909e30-8459-41ba-97ec-d35b27d86320" (UID: "8b909e30-8459-41ba-97ec-d35b27d86320"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:09:28.976779 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:09:28.976734    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b909e30-8459-41ba-97ec-d35b27d86320-kserve-provision-location\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:09:28.976779 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:09:28.976772    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b909e30-8459-41ba-97ec-d35b27d86320-dshm\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:09:28.976779 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:09:28.976782    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b909e30-8459-41ba-97ec-d35b27d86320-tls-certs\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:09:28.976985 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:09:28.976791    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b909e30-8459-41ba-97ec-d35b27d86320-home\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:09:28.976985 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:09:28.976799    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b909e30-8459-41ba-97ec-d35b27d86320-model-cache\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:09:28.976985 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:09:28.976808    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b909e30-8459-41ba-97ec-d35b27d86320-tmp-dir\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:09:28.976985 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:09:28.976816    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4k4ft\" (UniqueName: \"kubernetes.io/projected/8b909e30-8459-41ba-97ec-d35b27d86320-kube-api-access-4k4ft\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:09:29.234535 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:09:29.234497    2559 generic.go:356] "Generic (PLEG): container finished" podID="8b909e30-8459-41ba-97ec-d35b27d86320" containerID="892ff4588b2b699b9bf9cf313c7a49e3913479fe93582b0b13242b402cd4ab39" exitCode=0
Aug 01 00:09:29.234705 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:09:29.234577    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-d78cb5c5d-5pc22"
Aug 01 00:09:29.234705 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:09:29.234570    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-d78cb5c5d-5pc22" event={"ID":"8b909e30-8459-41ba-97ec-d35b27d86320","Type":"ContainerDied","Data":"892ff4588b2b699b9bf9cf313c7a49e3913479fe93582b0b13242b402cd4ab39"}
Aug 01 00:09:29.234850 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:09:29.234711    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-d78cb5c5d-5pc22" event={"ID":"8b909e30-8459-41ba-97ec-d35b27d86320","Type":"ContainerDied","Data":"b0b92d710b817a59ed2025670a913a3d38d27b875ecd20f66e8dafebc65388b0"}
Aug 01 00:09:29.234850 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:09:29.234781    2559 scope.go:117] "RemoveContainer" containerID="892ff4588b2b699b9bf9cf313c7a49e3913479fe93582b0b13242b402cd4ab39"
Aug 01 00:09:29.244110 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:09:29.244087    2559 scope.go:117] "RemoveContainer" containerID="bbb356323b2e12d848e02c13debe05edafe38340d5293d58ffadad31028a37bc"
Aug 01 00:09:29.251888 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:09:29.251868    2559 scope.go:117] "RemoveContainer" containerID="892ff4588b2b699b9bf9cf313c7a49e3913479fe93582b0b13242b402cd4ab39"
Aug 01 00:09:29.252372 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:09:29.252350    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"892ff4588b2b699b9bf9cf313c7a49e3913479fe93582b0b13242b402cd4ab39\": container with ID starting with 892ff4588b2b699b9bf9cf313c7a49e3913479fe93582b0b13242b402cd4ab39 not found: ID does not exist" containerID="892ff4588b2b699b9bf9cf313c7a49e3913479fe93582b0b13242b402cd4ab39"
Aug 01 00:09:29.252485 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:09:29.252380    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"892ff4588b2b699b9bf9cf313c7a49e3913479fe93582b0b13242b402cd4ab39"} err="failed to get container status \"892ff4588b2b699b9bf9cf313c7a49e3913479fe93582b0b13242b402cd4ab39\": rpc error: code = NotFound desc = could not find container \"892ff4588b2b699b9bf9cf313c7a49e3913479fe93582b0b13242b402cd4ab39\": container with ID starting with 892ff4588b2b699b9bf9cf313c7a49e3913479fe93582b0b13242b402cd4ab39 not found: ID does not exist"
Aug 01 00:09:29.252485 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:09:29.252398    2559 scope.go:117] "RemoveContainer" containerID="bbb356323b2e12d848e02c13debe05edafe38340d5293d58ffadad31028a37bc"
Aug 01 00:09:29.252651 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:09:29.252630    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbb356323b2e12d848e02c13debe05edafe38340d5293d58ffadad31028a37bc\": container with ID starting with bbb356323b2e12d848e02c13debe05edafe38340d5293d58ffadad31028a37bc not found: ID does not exist" containerID="bbb356323b2e12d848e02c13debe05edafe38340d5293d58ffadad31028a37bc"
Aug 01 00:09:29.252691 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:09:29.252661    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbb356323b2e12d848e02c13debe05edafe38340d5293d58ffadad31028a37bc"} err="failed to get container status \"bbb356323b2e12d848e02c13debe05edafe38340d5293d58ffadad31028a37bc\": rpc error: code = NotFound desc = could not find container \"bbb356323b2e12d848e02c13debe05edafe38340d5293d58ffadad31028a37bc\": container with ID starting with bbb356323b2e12d848e02c13debe05edafe38340d5293d58ffadad31028a37bc not found: ID does not exist"
Aug 01 00:09:29.260040 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:09:29.260016    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-d78cb5c5d-5pc22"]
Aug 01 00:09:29.262648 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:09:29.262628    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-d78cb5c5d-5pc22"]
Aug 01 00:09:29.786401 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:09:29.786365    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b909e30-8459-41ba-97ec-d35b27d86320" path="/var/lib/kubelet/pods/8b909e30-8459-41ba-97ec-d35b27d86320/volumes"
Aug 01 00:09:58.233088 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:09:58.233059    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c56f55965-6bbn8"
Aug 01 00:10:08.589453 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:08.589421    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-cf47448b_tokenizer-clean-path-test-tokenizer-c56f55965-6bbn8_41c739a2-ec92-4fd4-8e17-d444d3062108/main/0.log"
Aug 01 00:10:13.777020 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:13.776988    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c56f55965-6bbn8"]
Aug 01 00:10:13.777581 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:13.777195    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c56f55965-6bbn8" podUID="41c739a2-ec92-4fd4-8e17-d444d3062108" containerName="main" containerID="cri-o://162414d17f9315371991f4c48e89ba6e74ef375ea82de41eaf7c4951507e3baa" gracePeriod=30
Aug 01 00:10:13.788750 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:13.788724    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2r46n"]
Aug 01 00:10:13.797296 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:10:13.797261    2559 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 01 00:10:13.797296 ip-10-0-130-148 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:10:13.797296 ip-10-0-130-148 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Aug 01 00:10:13.797296 ip-10-0-130-148 kubenswrapper[2559]:  > containerID="83d336ba3bf2cb183a98755290350c4088f21ef4c919788718ea04be2c06cafa" cmd=["/bin/sleep","15"]
Aug 01 00:10:13.797555 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:13.797315    2559 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2r46n" message="" err=<
Aug 01 00:10:13.797555 ip-10-0-130-148 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:10:13.797555 ip-10-0-130-148 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Aug 01 00:10:13.797555 ip-10-0-130-148 kubenswrapper[2559]:  >
Aug 01 00:10:13.797555 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:10:13.797327    2559 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 01 00:10:13.797555 ip-10-0-130-148 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:10:13.797555 ip-10-0-130-148 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Aug 01 00:10:13.797555 ip-10-0-130-148 kubenswrapper[2559]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2r46n" podUID="89e04bde-4256-4539-837e-1961d441b74f" containerName="main" containerID="cri-o://83d336ba3bf2cb183a98755290350c4088f21ef4c919788718ea04be2c06cafa"
Aug 01 00:10:13.797555 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:13.797365    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2r46n" podUID="89e04bde-4256-4539-837e-1961d441b74f" containerName="main" containerID="cri-o://83d336ba3bf2cb183a98755290350c4088f21ef4c919788718ea04be2c06cafa" gracePeriod=60
Aug 01 00:10:14.042966 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.042858    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2r46n" podUID="89e04bde-4256-4539-837e-1961d441b74f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
Aug 01 00:10:14.143806 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.143783    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2r46n"
Aug 01 00:10:14.264841 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.264808    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89e04bde-4256-4539-837e-1961d441b74f-home\") pod \"89e04bde-4256-4539-837e-1961d441b74f\" (UID: \"89e04bde-4256-4539-837e-1961d441b74f\") "
Aug 01 00:10:14.265002 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.264894    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89e04bde-4256-4539-837e-1961d441b74f-kserve-provision-location\") pod \"89e04bde-4256-4539-837e-1961d441b74f\" (UID: \"89e04bde-4256-4539-837e-1961d441b74f\") "
Aug 01 00:10:14.265002 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.264922    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89e04bde-4256-4539-837e-1961d441b74f-tmp-dir\") pod \"89e04bde-4256-4539-837e-1961d441b74f\" (UID: \"89e04bde-4256-4539-837e-1961d441b74f\") "
Aug 01 00:10:14.265002 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.264948    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h6jj7\" (UniqueName: \"kubernetes.io/projected/89e04bde-4256-4539-837e-1961d441b74f-kube-api-access-h6jj7\") pod \"89e04bde-4256-4539-837e-1961d441b74f\" (UID: \"89e04bde-4256-4539-837e-1961d441b74f\") "
Aug 01 00:10:14.265002 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.264984    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89e04bde-4256-4539-837e-1961d441b74f-tls-certs\") pod \"89e04bde-4256-4539-837e-1961d441b74f\" (UID: \"89e04bde-4256-4539-837e-1961d441b74f\") "
Aug 01 00:10:14.265232 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.265014    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89e04bde-4256-4539-837e-1961d441b74f-dshm\") pod \"89e04bde-4256-4539-837e-1961d441b74f\" (UID: \"89e04bde-4256-4539-837e-1961d441b74f\") "
Aug 01 00:10:14.265232 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.265072    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89e04bde-4256-4539-837e-1961d441b74f-model-cache\") pod \"89e04bde-4256-4539-837e-1961d441b74f\" (UID: \"89e04bde-4256-4539-837e-1961d441b74f\") "
Aug 01 00:10:14.265232 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.265102    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89e04bde-4256-4539-837e-1961d441b74f-cabundle-cert\") pod \"89e04bde-4256-4539-837e-1961d441b74f\" (UID: \"89e04bde-4256-4539-837e-1961d441b74f\") "
Aug 01 00:10:14.265232 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.265066    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89e04bde-4256-4539-837e-1961d441b74f-home" (OuterVolumeSpecName: "home") pod "89e04bde-4256-4539-837e-1961d441b74f" (UID: "89e04bde-4256-4539-837e-1961d441b74f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:10:14.265232 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.265139    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89e04bde-4256-4539-837e-1961d441b74f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "89e04bde-4256-4539-837e-1961d441b74f" (UID: "89e04bde-4256-4539-837e-1961d441b74f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:10:14.265502 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.265356    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89e04bde-4256-4539-837e-1961d441b74f-home\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:10:14.265502 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.265337    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89e04bde-4256-4539-837e-1961d441b74f-model-cache" (OuterVolumeSpecName: "model-cache") pod "89e04bde-4256-4539-837e-1961d441b74f" (UID: "89e04bde-4256-4539-837e-1961d441b74f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:10:14.265502 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.265373    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89e04bde-4256-4539-837e-1961d441b74f-tmp-dir\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:10:14.265613 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.265501    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89e04bde-4256-4539-837e-1961d441b74f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "89e04bde-4256-4539-837e-1961d441b74f" (UID: "89e04bde-4256-4539-837e-1961d441b74f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:10:14.267318 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.267290    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89e04bde-4256-4539-837e-1961d441b74f-dshm" (OuterVolumeSpecName: "dshm") pod "89e04bde-4256-4539-837e-1961d441b74f" (UID: "89e04bde-4256-4539-837e-1961d441b74f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:10:14.267421 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.267363    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89e04bde-4256-4539-837e-1961d441b74f-kube-api-access-h6jj7" (OuterVolumeSpecName: "kube-api-access-h6jj7") pod "89e04bde-4256-4539-837e-1961d441b74f" (UID: "89e04bde-4256-4539-837e-1961d441b74f"). InnerVolumeSpecName "kube-api-access-h6jj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:10:14.267464 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.267430    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89e04bde-4256-4539-837e-1961d441b74f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "89e04bde-4256-4539-837e-1961d441b74f" (UID: "89e04bde-4256-4539-837e-1961d441b74f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:10:14.287667 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.287632    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89e04bde-4256-4539-837e-1961d441b74f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "89e04bde-4256-4539-837e-1961d441b74f" (UID: "89e04bde-4256-4539-837e-1961d441b74f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:10:14.367771 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.366867    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89e04bde-4256-4539-837e-1961d441b74f-kserve-provision-location\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:10:14.367771 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.366922    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6jj7\" (UniqueName: \"kubernetes.io/projected/89e04bde-4256-4539-837e-1961d441b74f-kube-api-access-h6jj7\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:10:14.367771 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.366939    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89e04bde-4256-4539-837e-1961d441b74f-tls-certs\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:10:14.367771 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.366954    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89e04bde-4256-4539-837e-1961d441b74f-dshm\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:10:14.367771 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.367010    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89e04bde-4256-4539-837e-1961d441b74f-model-cache\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:10:14.367771 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.367042    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89e04bde-4256-4539-837e-1961d441b74f-cabundle-cert\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:10:14.414685 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.414659    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c56f55965-6bbn8"
Aug 01 00:10:14.455987 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.455952    2559 generic.go:356] "Generic (PLEG): container finished" podID="89e04bde-4256-4539-837e-1961d441b74f" containerID="83d336ba3bf2cb183a98755290350c4088f21ef4c919788718ea04be2c06cafa" exitCode=0
Aug 01 00:10:14.456181 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.455989    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2r46n" event={"ID":"89e04bde-4256-4539-837e-1961d441b74f","Type":"ContainerDied","Data":"83d336ba3bf2cb183a98755290350c4088f21ef4c919788718ea04be2c06cafa"}
Aug 01 00:10:14.456181 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.456034    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2r46n"
Aug 01 00:10:14.456181 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.456077    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2r46n" event={"ID":"89e04bde-4256-4539-837e-1961d441b74f","Type":"ContainerDied","Data":"cb4be894c8057af44386f275d8bef2298df0f42683a7d601becb84ae4b468e61"}
Aug 01 00:10:14.456181 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.456110    2559 scope.go:117] "RemoveContainer" containerID="83d336ba3bf2cb183a98755290350c4088f21ef4c919788718ea04be2c06cafa"
Aug 01 00:10:14.458042 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.458019    2559 generic.go:356] "Generic (PLEG): container finished" podID="41c739a2-ec92-4fd4-8e17-d444d3062108" containerID="162414d17f9315371991f4c48e89ba6e74ef375ea82de41eaf7c4951507e3baa" exitCode=0
Aug 01 00:10:14.458179 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.458078    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c56f55965-6bbn8" event={"ID":"41c739a2-ec92-4fd4-8e17-d444d3062108","Type":"ContainerDied","Data":"162414d17f9315371991f4c48e89ba6e74ef375ea82de41eaf7c4951507e3baa"}
Aug 01 00:10:14.458179 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.458146    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c56f55965-6bbn8" event={"ID":"41c739a2-ec92-4fd4-8e17-d444d3062108","Type":"ContainerDied","Data":"2772ceb7143656762d21429f9cc9d211808d2bef504450675081d010bb709336"}
Aug 01 00:10:14.458179 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.458081    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c56f55965-6bbn8"
Aug 01 00:10:14.465165 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.465146    2559 scope.go:117] "RemoveContainer" containerID="b23c2b74ae445213d0a6eafc8592f39e83e9ad73ba873d6eda4ec9cc37a5babe"
Aug 01 00:10:14.473103 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.473090    2559 scope.go:117] "RemoveContainer" containerID="83d336ba3bf2cb183a98755290350c4088f21ef4c919788718ea04be2c06cafa"
Aug 01 00:10:14.473326 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:10:14.473311    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83d336ba3bf2cb183a98755290350c4088f21ef4c919788718ea04be2c06cafa\": container with ID starting with 83d336ba3bf2cb183a98755290350c4088f21ef4c919788718ea04be2c06cafa not found: ID does not exist" containerID="83d336ba3bf2cb183a98755290350c4088f21ef4c919788718ea04be2c06cafa"
Aug 01 00:10:14.473383 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.473332    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83d336ba3bf2cb183a98755290350c4088f21ef4c919788718ea04be2c06cafa"} err="failed to get container status \"83d336ba3bf2cb183a98755290350c4088f21ef4c919788718ea04be2c06cafa\": rpc error: code = NotFound desc = could not find container \"83d336ba3bf2cb183a98755290350c4088f21ef4c919788718ea04be2c06cafa\": container with ID starting with 83d336ba3bf2cb183a98755290350c4088f21ef4c919788718ea04be2c06cafa not found: ID does not exist"
Aug 01 00:10:14.473383 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.473347    2559 scope.go:117] "RemoveContainer" containerID="b23c2b74ae445213d0a6eafc8592f39e83e9ad73ba873d6eda4ec9cc37a5babe"
Aug 01 00:10:14.473561 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:10:14.473542    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b23c2b74ae445213d0a6eafc8592f39e83e9ad73ba873d6eda4ec9cc37a5babe\": container with ID starting with b23c2b74ae445213d0a6eafc8592f39e83e9ad73ba873d6eda4ec9cc37a5babe not found: ID does not exist" containerID="b23c2b74ae445213d0a6eafc8592f39e83e9ad73ba873d6eda4ec9cc37a5babe"
Aug 01 00:10:14.473627 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.473571    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b23c2b74ae445213d0a6eafc8592f39e83e9ad73ba873d6eda4ec9cc37a5babe"} err="failed to get container status \"b23c2b74ae445213d0a6eafc8592f39e83e9ad73ba873d6eda4ec9cc37a5babe\": rpc error: code = NotFound desc = could not find container \"b23c2b74ae445213d0a6eafc8592f39e83e9ad73ba873d6eda4ec9cc37a5babe\": container with ID starting with b23c2b74ae445213d0a6eafc8592f39e83e9ad73ba873d6eda4ec9cc37a5babe not found: ID does not exist"
Aug 01 00:10:14.473627 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.473594    2559 scope.go:117] "RemoveContainer" containerID="162414d17f9315371991f4c48e89ba6e74ef375ea82de41eaf7c4951507e3baa"
Aug 01 00:10:14.480683 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.480661    2559 scope.go:117] "RemoveContainer" containerID="2b5583c12d72014994b5be0c545cc90033326aa746ace8d5c81853b88b84f37d"
Aug 01 00:10:14.485080 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.485062    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2r46n"]
Aug 01 00:10:14.488270 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.488254    2559 scope.go:117] "RemoveContainer" containerID="162414d17f9315371991f4c48e89ba6e74ef375ea82de41eaf7c4951507e3baa"
Aug 01 00:10:14.488568 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:10:14.488550    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"162414d17f9315371991f4c48e89ba6e74ef375ea82de41eaf7c4951507e3baa\": container with ID starting with 162414d17f9315371991f4c48e89ba6e74ef375ea82de41eaf7c4951507e3baa not found: ID does not exist" containerID="162414d17f9315371991f4c48e89ba6e74ef375ea82de41eaf7c4951507e3baa"
Aug 01 00:10:14.488621 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.488576    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"162414d17f9315371991f4c48e89ba6e74ef375ea82de41eaf7c4951507e3baa"} err="failed to get container status \"162414d17f9315371991f4c48e89ba6e74ef375ea82de41eaf7c4951507e3baa\": rpc error: code = NotFound desc = could not find container \"162414d17f9315371991f4c48e89ba6e74ef375ea82de41eaf7c4951507e3baa\": container with ID starting with 162414d17f9315371991f4c48e89ba6e74ef375ea82de41eaf7c4951507e3baa not found: ID does not exist"
Aug 01 00:10:14.488621 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.488594    2559 scope.go:117] "RemoveContainer" containerID="2b5583c12d72014994b5be0c545cc90033326aa746ace8d5c81853b88b84f37d"
Aug 01 00:10:14.488895 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:10:14.488867    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b5583c12d72014994b5be0c545cc90033326aa746ace8d5c81853b88b84f37d\": container with ID starting with 2b5583c12d72014994b5be0c545cc90033326aa746ace8d5c81853b88b84f37d not found: ID does not exist" containerID="2b5583c12d72014994b5be0c545cc90033326aa746ace8d5c81853b88b84f37d"
Aug 01 00:10:14.488965 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.488893    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b5583c12d72014994b5be0c545cc90033326aa746ace8d5c81853b88b84f37d"} err="failed to get container status \"2b5583c12d72014994b5be0c545cc90033326aa746ace8d5c81853b88b84f37d\": rpc error: code = NotFound desc = could not find container \"2b5583c12d72014994b5be0c545cc90033326aa746ace8d5c81853b88b84f37d\": container with ID starting with 2b5583c12d72014994b5be0c545cc90033326aa746ace8d5c81853b88b84f37d not found: ID does not exist"
Aug 01 00:10:14.489589 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.489571    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-567447fd4-2r46n"]
Aug 01 00:10:14.568049 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.567964    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41c739a2-ec92-4fd4-8e17-d444d3062108-cabundle-cert\") pod \"41c739a2-ec92-4fd4-8e17-d444d3062108\" (UID: \"41c739a2-ec92-4fd4-8e17-d444d3062108\") "
Aug 01 00:10:14.568049 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.568010    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41c739a2-ec92-4fd4-8e17-d444d3062108-tls-certs\") pod \"41c739a2-ec92-4fd4-8e17-d444d3062108\" (UID: \"41c739a2-ec92-4fd4-8e17-d444d3062108\") "
Aug 01 00:10:14.568049 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.568047    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41c739a2-ec92-4fd4-8e17-d444d3062108-tmp-dir\") pod \"41c739a2-ec92-4fd4-8e17-d444d3062108\" (UID: \"41c739a2-ec92-4fd4-8e17-d444d3062108\") "
Aug 01 00:10:14.568263 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.568074    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41c739a2-ec92-4fd4-8e17-d444d3062108-kserve-provision-location\") pod \"41c739a2-ec92-4fd4-8e17-d444d3062108\" (UID: \"41c739a2-ec92-4fd4-8e17-d444d3062108\") "
Aug 01 00:10:14.568406 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.568377    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41c739a2-ec92-4fd4-8e17-d444d3062108-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "41c739a2-ec92-4fd4-8e17-d444d3062108" (UID: "41c739a2-ec92-4fd4-8e17-d444d3062108"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:10:14.568455 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.568431    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41c739a2-ec92-4fd4-8e17-d444d3062108-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "41c739a2-ec92-4fd4-8e17-d444d3062108" (UID: "41c739a2-ec92-4fd4-8e17-d444d3062108"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:10:14.570095 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.570076    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41c739a2-ec92-4fd4-8e17-d444d3062108-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "41c739a2-ec92-4fd4-8e17-d444d3062108" (UID: "41c739a2-ec92-4fd4-8e17-d444d3062108"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:10:14.587986 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.587957    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41c739a2-ec92-4fd4-8e17-d444d3062108-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "41c739a2-ec92-4fd4-8e17-d444d3062108" (UID: "41c739a2-ec92-4fd4-8e17-d444d3062108"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:10:14.669079 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.669038    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41c739a2-ec92-4fd4-8e17-d444d3062108-cabundle-cert\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:10:14.669079 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.669075    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41c739a2-ec92-4fd4-8e17-d444d3062108-tls-certs\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:10:14.669079 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.669084    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41c739a2-ec92-4fd4-8e17-d444d3062108-tmp-dir\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:10:14.669257 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.669093    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41c739a2-ec92-4fd4-8e17-d444d3062108-kserve-provision-location\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:10:14.782318 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.782291    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c56f55965-6bbn8"]
Aug 01 00:10:14.787934 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:14.787913    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-c56f55965-6bbn8"]
Aug 01 00:10:15.785271 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:15.785240    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41c739a2-ec92-4fd4-8e17-d444d3062108" path="/var/lib/kubelet/pods/41c739a2-ec92-4fd4-8e17-d444d3062108/volumes"
Aug 01 00:10:15.785730 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:15.785715    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89e04bde-4256-4539-837e-1961d441b74f" path="/var/lib/kubelet/pods/89e04bde-4256-4539-837e-1961d441b74f/volumes"
Aug 01 00:10:37.805957 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:37.805923    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-spblg_c2e50946-aea4-4886-82ec-0bd5a31196cf/console-operator/1.log"
Aug 01 00:10:37.809080 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:37.809060    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-spblg_c2e50946-aea4-4886-82ec-0bd5a31196cf/console-operator/1.log"
Aug 01 00:10:50.380891 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.380856    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt"]
Aug 01 00:10:50.381281 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.381173    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41c739a2-ec92-4fd4-8e17-d444d3062108" containerName="main"
Aug 01 00:10:50.381281 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.381184    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="41c739a2-ec92-4fd4-8e17-d444d3062108" containerName="main"
Aug 01 00:10:50.381281 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.381195    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89e04bde-4256-4539-837e-1961d441b74f" containerName="storage-initializer"
Aug 01 00:10:50.381281 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.381201    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="89e04bde-4256-4539-837e-1961d441b74f" containerName="storage-initializer"
Aug 01 00:10:50.381281 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.381209    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89e04bde-4256-4539-837e-1961d441b74f" containerName="main"
Aug 01 00:10:50.381281 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.381215    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="89e04bde-4256-4539-837e-1961d441b74f" containerName="main"
Aug 01 00:10:50.381281 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.381223    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b909e30-8459-41ba-97ec-d35b27d86320" containerName="main"
Aug 01 00:10:50.381281 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.381227    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b909e30-8459-41ba-97ec-d35b27d86320" containerName="main"
Aug 01 00:10:50.381281 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.381237    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b909e30-8459-41ba-97ec-d35b27d86320" containerName="storage-initializer"
Aug 01 00:10:50.381281 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.381242    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b909e30-8459-41ba-97ec-d35b27d86320" containerName="storage-initializer"
Aug 01 00:10:50.381281 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.381251    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41c739a2-ec92-4fd4-8e17-d444d3062108" containerName="storage-initializer"
Aug 01 00:10:50.381281 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.381256    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="41c739a2-ec92-4fd4-8e17-d444d3062108" containerName="storage-initializer"
Aug 01 00:10:50.381686 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.381303    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="41c739a2-ec92-4fd4-8e17-d444d3062108" containerName="main"
Aug 01 00:10:50.381686 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.381312    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="89e04bde-4256-4539-837e-1961d441b74f" containerName="main"
Aug 01 00:10:50.381686 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.381318    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b909e30-8459-41ba-97ec-d35b27d86320" containerName="main"
Aug 01 00:10:50.383464 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.383444    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt"
Aug 01 00:10:50.386114 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.386094    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Aug 01 00:10:50.386221 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.386100    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Aug 01 00:10:50.386221 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.386149    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-9cc8q\""
Aug 01 00:10:50.386327 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.386221    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Aug 01 00:10:50.387158 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.387143    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Aug 01 00:10:50.387229 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.387174    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Aug 01 00:10:50.397092 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.397072    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt"]
Aug 01 00:10:50.439082 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.439058    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b60bddd2-fece-42a1-a73e-ad4f06558575-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt\" (UID: \"b60bddd2-fece-42a1-a73e-ad4f06558575\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt"
Aug 01 00:10:50.439204 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.439088    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b60bddd2-fece-42a1-a73e-ad4f06558575-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt\" (UID: \"b60bddd2-fece-42a1-a73e-ad4f06558575\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt"
Aug 01 00:10:50.439204 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.439113    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b60bddd2-fece-42a1-a73e-ad4f06558575-dshm\") pod \"tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt\" (UID: \"b60bddd2-fece-42a1-a73e-ad4f06558575\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt"
Aug 01 00:10:50.439204 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.439146    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b60bddd2-fece-42a1-a73e-ad4f06558575-home\") pod \"tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt\" (UID: \"b60bddd2-fece-42a1-a73e-ad4f06558575\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt"
Aug 01 00:10:50.439204 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.439176    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b60bddd2-fece-42a1-a73e-ad4f06558575-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt\" (UID: \"b60bddd2-fece-42a1-a73e-ad4f06558575\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt"
Aug 01 00:10:50.439344 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.439218    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xbnv\" (UniqueName: \"kubernetes.io/projected/b60bddd2-fece-42a1-a73e-ad4f06558575-kube-api-access-4xbnv\") pod \"tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt\" (UID: \"b60bddd2-fece-42a1-a73e-ad4f06558575\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt"
Aug 01 00:10:50.439344 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.439245    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b60bddd2-fece-42a1-a73e-ad4f06558575-tls-certs\") pod \"tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt\" (UID: \"b60bddd2-fece-42a1-a73e-ad4f06558575\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt"
Aug 01 00:10:50.439344 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.439265    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b60bddd2-fece-42a1-a73e-ad4f06558575-model-cache\") pod \"tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt\" (UID: \"b60bddd2-fece-42a1-a73e-ad4f06558575\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt"
Aug 01 00:10:50.540019 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.539980    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xbnv\" (UniqueName: \"kubernetes.io/projected/b60bddd2-fece-42a1-a73e-ad4f06558575-kube-api-access-4xbnv\") pod \"tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt\" (UID: \"b60bddd2-fece-42a1-a73e-ad4f06558575\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt"
Aug 01 00:10:50.540019 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.540027    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b60bddd2-fece-42a1-a73e-ad4f06558575-tls-certs\") pod \"tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt\" (UID: \"b60bddd2-fece-42a1-a73e-ad4f06558575\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt"
Aug 01 00:10:50.540240 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.540049    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b60bddd2-fece-42a1-a73e-ad4f06558575-model-cache\") pod \"tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt\" (UID: \"b60bddd2-fece-42a1-a73e-ad4f06558575\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt"
Aug 01 00:10:50.540240 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.540077    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b60bddd2-fece-42a1-a73e-ad4f06558575-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt\" (UID: \"b60bddd2-fece-42a1-a73e-ad4f06558575\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt"
Aug 01 00:10:50.540240 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.540094    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b60bddd2-fece-42a1-a73e-ad4f06558575-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt\" (UID: \"b60bddd2-fece-42a1-a73e-ad4f06558575\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt"
Aug 01 00:10:50.540240 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.540113    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b60bddd2-fece-42a1-a73e-ad4f06558575-dshm\") pod \"tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt\" (UID: \"b60bddd2-fece-42a1-a73e-ad4f06558575\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt"
Aug 01 00:10:50.540240 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.540134    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b60bddd2-fece-42a1-a73e-ad4f06558575-home\") pod \"tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt\" (UID: \"b60bddd2-fece-42a1-a73e-ad4f06558575\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt"
Aug 01 00:10:50.540240 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.540156    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b60bddd2-fece-42a1-a73e-ad4f06558575-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt\" (UID: \"b60bddd2-fece-42a1-a73e-ad4f06558575\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt"
Aug 01 00:10:50.540555 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.540524    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b60bddd2-fece-42a1-a73e-ad4f06558575-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt\" (UID: \"b60bddd2-fece-42a1-a73e-ad4f06558575\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt"
Aug 01 00:10:50.540645 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.540625    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b60bddd2-fece-42a1-a73e-ad4f06558575-model-cache\") pod \"tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt\" (UID: \"b60bddd2-fece-42a1-a73e-ad4f06558575\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt"
Aug 01 00:10:50.540703 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.540689    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b60bddd2-fece-42a1-a73e-ad4f06558575-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt\" (UID: \"b60bddd2-fece-42a1-a73e-ad4f06558575\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt"
Aug 01 00:10:50.540787 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.540739    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b60bddd2-fece-42a1-a73e-ad4f06558575-home\") pod \"tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt\" (UID: \"b60bddd2-fece-42a1-a73e-ad4f06558575\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt"
Aug 01 00:10:50.540940 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.540922    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b60bddd2-fece-42a1-a73e-ad4f06558575-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt\" (UID: \"b60bddd2-fece-42a1-a73e-ad4f06558575\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt"
Aug 01 00:10:50.542387 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.542369    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b60bddd2-fece-42a1-a73e-ad4f06558575-dshm\") pod \"tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt\" (UID: \"b60bddd2-fece-42a1-a73e-ad4f06558575\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt"
Aug 01 00:10:50.542626 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.542611    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b60bddd2-fece-42a1-a73e-ad4f06558575-tls-certs\") pod \"tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt\" (UID: \"b60bddd2-fece-42a1-a73e-ad4f06558575\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt"
Aug 01 00:10:50.550857 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.550834    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xbnv\" (UniqueName: \"kubernetes.io/projected/b60bddd2-fece-42a1-a73e-ad4f06558575-kube-api-access-4xbnv\") pod \"tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt\" (UID: \"b60bddd2-fece-42a1-a73e-ad4f06558575\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt"
Aug 01 00:10:50.621467 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.621437    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-64f689b7c5-xcpw2"]
Aug 01 00:10:50.624382 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.624362    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-64f689b7c5-xcpw2"
Aug 01 00:10:50.634564 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.634537    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-64f689b7c5-xcpw2"]
Aug 01 00:10:50.693878 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.693833    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt"
Aug 01 00:10:50.742421 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.742387    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16936fc4-2f5d-44ef-8da0-744e480f48a0-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-64f689b7c5-xcpw2\" (UID: \"16936fc4-2f5d-44ef-8da0-744e480f48a0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-64f689b7c5-xcpw2"
Aug 01 00:10:50.742583 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.742442    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16936fc4-2f5d-44ef-8da0-744e480f48a0-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-64f689b7c5-xcpw2\" (UID: \"16936fc4-2f5d-44ef-8da0-744e480f48a0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-64f689b7c5-xcpw2"
Aug 01 00:10:50.742583 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.742530    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/16936fc4-2f5d-44ef-8da0-744e480f48a0-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-64f689b7c5-xcpw2\" (UID: \"16936fc4-2f5d-44ef-8da0-744e480f48a0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-64f689b7c5-xcpw2"
Aug 01 00:10:50.742583 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.742565    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16936fc4-2f5d-44ef-8da0-744e480f48a0-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-64f689b7c5-xcpw2\" (UID: \"16936fc4-2f5d-44ef-8da0-744e480f48a0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-64f689b7c5-xcpw2"
Aug 01 00:10:50.823003 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.822955    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt"]
Aug 01 00:10:50.825181 ip-10-0-130-148 kubenswrapper[2559]: W0801 00:10:50.825156    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb60bddd2_fece_42a1_a73e_ad4f06558575.slice/crio-04cba2a58c4aad2783dd7c7aee4139bcc09f86f012132143397ea3ec8859a47d WatchSource:0}: Error finding container 04cba2a58c4aad2783dd7c7aee4139bcc09f86f012132143397ea3ec8859a47d: Status 404 returned error can't find the container with id 04cba2a58c4aad2783dd7c7aee4139bcc09f86f012132143397ea3ec8859a47d
Aug 01 00:10:50.843867 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.843845    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/16936fc4-2f5d-44ef-8da0-744e480f48a0-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-64f689b7c5-xcpw2\" (UID: \"16936fc4-2f5d-44ef-8da0-744e480f48a0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-64f689b7c5-xcpw2"
Aug 01 00:10:50.843958 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.843894    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16936fc4-2f5d-44ef-8da0-744e480f48a0-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-64f689b7c5-xcpw2\" (UID: \"16936fc4-2f5d-44ef-8da0-744e480f48a0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-64f689b7c5-xcpw2"
Aug 01 00:10:50.843958 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.843947    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16936fc4-2f5d-44ef-8da0-744e480f48a0-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-64f689b7c5-xcpw2\" (UID: \"16936fc4-2f5d-44ef-8da0-744e480f48a0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-64f689b7c5-xcpw2"
Aug 01 00:10:50.844062 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.843995    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16936fc4-2f5d-44ef-8da0-744e480f48a0-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-64f689b7c5-xcpw2\" (UID: \"16936fc4-2f5d-44ef-8da0-744e480f48a0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-64f689b7c5-xcpw2"
Aug 01 00:10:50.844895 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.844643    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/16936fc4-2f5d-44ef-8da0-744e480f48a0-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-64f689b7c5-xcpw2\" (UID: \"16936fc4-2f5d-44ef-8da0-744e480f48a0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-64f689b7c5-xcpw2"
Aug 01 00:10:50.844895 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.844844    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16936fc4-2f5d-44ef-8da0-744e480f48a0-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-64f689b7c5-xcpw2\" (UID: \"16936fc4-2f5d-44ef-8da0-744e480f48a0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-64f689b7c5-xcpw2"
Aug 01 00:10:50.844895 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.844876    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16936fc4-2f5d-44ef-8da0-744e480f48a0-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-64f689b7c5-xcpw2\" (UID: \"16936fc4-2f5d-44ef-8da0-744e480f48a0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-64f689b7c5-xcpw2"
Aug 01 00:10:50.847049 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.847029    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16936fc4-2f5d-44ef-8da0-744e480f48a0-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-64f689b7c5-xcpw2\" (UID: \"16936fc4-2f5d-44ef-8da0-744e480f48a0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-64f689b7c5-xcpw2"
Aug 01 00:10:50.936012 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:50.935981    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-64f689b7c5-xcpw2"
Aug 01 00:10:51.063652 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:51.063616    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-64f689b7c5-xcpw2"]
Aug 01 00:10:51.066055 ip-10-0-130-148 kubenswrapper[2559]: W0801 00:10:51.066026    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16936fc4_2f5d_44ef_8da0_744e480f48a0.slice/crio-e936a69ee194fc135f84f0fe1048d121e320755d800beb630ed136c2603cbb04 WatchSource:0}: Error finding container e936a69ee194fc135f84f0fe1048d121e320755d800beb630ed136c2603cbb04: Status 404 returned error can't find the container with id e936a69ee194fc135f84f0fe1048d121e320755d800beb630ed136c2603cbb04
Aug 01 00:10:51.629154 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:51.629114    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-64f689b7c5-xcpw2" event={"ID":"16936fc4-2f5d-44ef-8da0-744e480f48a0","Type":"ContainerStarted","Data":"60a447c61b980f6f0ec3875f2bf026a8b93257f9fc1ba78f56011041b3096819"}
Aug 01 00:10:51.629621 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:51.629182    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-64f689b7c5-xcpw2" event={"ID":"16936fc4-2f5d-44ef-8da0-744e480f48a0","Type":"ContainerStarted","Data":"e936a69ee194fc135f84f0fe1048d121e320755d800beb630ed136c2603cbb04"}
Aug 01 00:10:51.630729 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:51.630688    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt" event={"ID":"b60bddd2-fece-42a1-a73e-ad4f06558575","Type":"ContainerStarted","Data":"d9ee11e3573ef420d1e2960813ec340e46b4c19d173bd2a6edeb66e539f68246"}
Aug 01 00:10:51.630892 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:51.630761    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt" event={"ID":"b60bddd2-fece-42a1-a73e-ad4f06558575","Type":"ContainerStarted","Data":"04cba2a58c4aad2783dd7c7aee4139bcc09f86f012132143397ea3ec8859a47d"}
Aug 01 00:10:53.644983 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:53.644899    2559 generic.go:356] "Generic (PLEG): container finished" podID="b60bddd2-fece-42a1-a73e-ad4f06558575" containerID="d9ee11e3573ef420d1e2960813ec340e46b4c19d173bd2a6edeb66e539f68246" exitCode=0
Aug 01 00:10:53.645389 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:53.644978    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt" event={"ID":"b60bddd2-fece-42a1-a73e-ad4f06558575","Type":"ContainerDied","Data":"d9ee11e3573ef420d1e2960813ec340e46b4c19d173bd2a6edeb66e539f68246"}
Aug 01 00:10:53.646583 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:53.646562    2559 generic.go:356] "Generic (PLEG): container finished" podID="16936fc4-2f5d-44ef-8da0-744e480f48a0" containerID="60a447c61b980f6f0ec3875f2bf026a8b93257f9fc1ba78f56011041b3096819" exitCode=0
Aug 01 00:10:53.646672 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:53.646598    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-64f689b7c5-xcpw2" event={"ID":"16936fc4-2f5d-44ef-8da0-744e480f48a0","Type":"ContainerDied","Data":"60a447c61b980f6f0ec3875f2bf026a8b93257f9fc1ba78f56011041b3096819"}
Aug 01 00:10:54.652974 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:54.652935    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt" event={"ID":"b60bddd2-fece-42a1-a73e-ad4f06558575","Type":"ContainerStarted","Data":"1dd7b0267632818a9fee04638706ed1cceb35776e8b6bb98ff17c74a655aec82"}
Aug 01 00:10:54.654582 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:54.654558    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-64f689b7c5-xcpw2" event={"ID":"16936fc4-2f5d-44ef-8da0-744e480f48a0","Type":"ContainerStarted","Data":"d67c5c7e2b05eab43d42deec1bcc457e9933518585858bd95b82a6c6912eea51"}
Aug 01 00:10:54.654784 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:54.654764    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-64f689b7c5-xcpw2"
Aug 01 00:10:54.672783 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:54.672724    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt" podStartSLOduration=2.306616574 podStartE2EDuration="4.672709666s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.366093092s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:10:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:10:54.671778222 +0000 UTC m=+1517.391578083" watchObservedRunningTime="2026-08-01 00:10:54.672709666 +0000 UTC m=+1517.392509739"
Aug 01 00:10:54.689352 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:54.689301    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-64f689b7c5-xcpw2" podStartSLOduration=2.370029333 podStartE2EDuration="4.689290037s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.319260704s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:10:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:10:54.688050456 +0000 UTC m=+1517.407850305" watchObservedRunningTime="2026-08-01 00:10:54.689290037 +0000 UTC m=+1517.409089884"
Aug 01 00:10:54.696674 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:10:54.695703    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt"
Aug 01 00:11:00.694888 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:00.694802    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt"
Aug 01 00:11:00.707698 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:00.707673    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt"
Aug 01 00:11:01.699354 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:01.699321    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt"
Aug 01 00:11:25.671378 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:25.671348    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-64f689b7c5-xcpw2"
Aug 01 00:11:26.700499 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:26.700445    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_tokenizer-migration-path-test-tokenizer-64f689b7c5-xcpw2_16936fc4-2f5d-44ef-8da0-744e480f48a0/main/0.log"
Aug 01 00:11:26.971494 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:26.971404    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-64f689b7c5-xcpw2"]
Aug 01 00:11:26.971687 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:26.971629    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-64f689b7c5-xcpw2" podUID="16936fc4-2f5d-44ef-8da0-744e480f48a0" containerName="main" containerID="cri-o://d67c5c7e2b05eab43d42deec1bcc457e9933518585858bd95b82a6c6912eea51" gracePeriod=30
Aug 01 00:11:26.978193 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:26.978164    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt"]
Aug 01 00:11:26.986575 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:11:26.986540    2559 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 01 00:11:26.986575 ip-10-0-130-148 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:11:26.986575 ip-10-0-130-148 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Aug 01 00:11:26.986575 ip-10-0-130-148 kubenswrapper[2559]:  > containerID="1dd7b0267632818a9fee04638706ed1cceb35776e8b6bb98ff17c74a655aec82" cmd=["/bin/sleep","15"]
Aug 01 00:11:26.986873 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:26.986601    2559 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt" message="" err=<
Aug 01 00:11:26.986873 ip-10-0-130-148 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:11:26.986873 ip-10-0-130-148 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Aug 01 00:11:26.986873 ip-10-0-130-148 kubenswrapper[2559]:  >
Aug 01 00:11:26.986873 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:11:26.986620    2559 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 01 00:11:26.986873 ip-10-0-130-148 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:11:26.986873 ip-10-0-130-148 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Aug 01 00:11:26.986873 ip-10-0-130-148 kubenswrapper[2559]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt" podUID="b60bddd2-fece-42a1-a73e-ad4f06558575" containerName="main" containerID="cri-o://1dd7b0267632818a9fee04638706ed1cceb35776e8b6bb98ff17c74a655aec82"
Aug 01 00:11:26.986873 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:26.986674    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt" podUID="b60bddd2-fece-42a1-a73e-ad4f06558575" containerName="main" containerID="cri-o://1dd7b0267632818a9fee04638706ed1cceb35776e8b6bb98ff17c74a655aec82" gracePeriod=60
Aug 01 00:11:27.332344 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.332321    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt"
Aug 01 00:11:27.467062 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.467033    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b60bddd2-fece-42a1-a73e-ad4f06558575-dshm\") pod \"b60bddd2-fece-42a1-a73e-ad4f06558575\" (UID: \"b60bddd2-fece-42a1-a73e-ad4f06558575\") "
Aug 01 00:11:27.467223 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.467071    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b60bddd2-fece-42a1-a73e-ad4f06558575-tmp-dir\") pod \"b60bddd2-fece-42a1-a73e-ad4f06558575\" (UID: \"b60bddd2-fece-42a1-a73e-ad4f06558575\") "
Aug 01 00:11:27.467223 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.467102    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b60bddd2-fece-42a1-a73e-ad4f06558575-home\") pod \"b60bddd2-fece-42a1-a73e-ad4f06558575\" (UID: \"b60bddd2-fece-42a1-a73e-ad4f06558575\") "
Aug 01 00:11:27.467223 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.467121    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b60bddd2-fece-42a1-a73e-ad4f06558575-tls-certs\") pod \"b60bddd2-fece-42a1-a73e-ad4f06558575\" (UID: \"b60bddd2-fece-42a1-a73e-ad4f06558575\") "
Aug 01 00:11:27.467223 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.467169    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4xbnv\" (UniqueName: \"kubernetes.io/projected/b60bddd2-fece-42a1-a73e-ad4f06558575-kube-api-access-4xbnv\") pod \"b60bddd2-fece-42a1-a73e-ad4f06558575\" (UID: \"b60bddd2-fece-42a1-a73e-ad4f06558575\") "
Aug 01 00:11:27.467223 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.467186    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b60bddd2-fece-42a1-a73e-ad4f06558575-model-cache\") pod \"b60bddd2-fece-42a1-a73e-ad4f06558575\" (UID: \"b60bddd2-fece-42a1-a73e-ad4f06558575\") "
Aug 01 00:11:27.467223 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.467216    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b60bddd2-fece-42a1-a73e-ad4f06558575-cabundle-cert\") pod \"b60bddd2-fece-42a1-a73e-ad4f06558575\" (UID: \"b60bddd2-fece-42a1-a73e-ad4f06558575\") "
Aug 01 00:11:27.467550 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.467304    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b60bddd2-fece-42a1-a73e-ad4f06558575-home" (OuterVolumeSpecName: "home") pod "b60bddd2-fece-42a1-a73e-ad4f06558575" (UID: "b60bddd2-fece-42a1-a73e-ad4f06558575"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:11:27.467550 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.467341    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b60bddd2-fece-42a1-a73e-ad4f06558575-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b60bddd2-fece-42a1-a73e-ad4f06558575" (UID: "b60bddd2-fece-42a1-a73e-ad4f06558575"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:11:27.467550 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.467363    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b60bddd2-fece-42a1-a73e-ad4f06558575-kserve-provision-location\") pod \"b60bddd2-fece-42a1-a73e-ad4f06558575\" (UID: \"b60bddd2-fece-42a1-a73e-ad4f06558575\") "
Aug 01 00:11:27.467550 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.467413    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b60bddd2-fece-42a1-a73e-ad4f06558575-model-cache" (OuterVolumeSpecName: "model-cache") pod "b60bddd2-fece-42a1-a73e-ad4f06558575" (UID: "b60bddd2-fece-42a1-a73e-ad4f06558575"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:11:27.467550 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.467534    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b60bddd2-fece-42a1-a73e-ad4f06558575-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b60bddd2-fece-42a1-a73e-ad4f06558575" (UID: "b60bddd2-fece-42a1-a73e-ad4f06558575"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:11:27.467843 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.467656    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b60bddd2-fece-42a1-a73e-ad4f06558575-tmp-dir\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:11:27.467843 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.467674    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b60bddd2-fece-42a1-a73e-ad4f06558575-home\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:11:27.467843 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.467686    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b60bddd2-fece-42a1-a73e-ad4f06558575-model-cache\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:11:27.467843 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.467694    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b60bddd2-fece-42a1-a73e-ad4f06558575-cabundle-cert\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:11:27.469431 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.469402    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b60bddd2-fece-42a1-a73e-ad4f06558575-kube-api-access-4xbnv" (OuterVolumeSpecName: "kube-api-access-4xbnv") pod "b60bddd2-fece-42a1-a73e-ad4f06558575" (UID: "b60bddd2-fece-42a1-a73e-ad4f06558575"). InnerVolumeSpecName "kube-api-access-4xbnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:11:27.469431 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.469407    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b60bddd2-fece-42a1-a73e-ad4f06558575-dshm" (OuterVolumeSpecName: "dshm") pod "b60bddd2-fece-42a1-a73e-ad4f06558575" (UID: "b60bddd2-fece-42a1-a73e-ad4f06558575"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:11:27.469627 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.469464    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b60bddd2-fece-42a1-a73e-ad4f06558575-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b60bddd2-fece-42a1-a73e-ad4f06558575" (UID: "b60bddd2-fece-42a1-a73e-ad4f06558575"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:11:27.491010 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.490983    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b60bddd2-fece-42a1-a73e-ad4f06558575-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b60bddd2-fece-42a1-a73e-ad4f06558575" (UID: "b60bddd2-fece-42a1-a73e-ad4f06558575"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:11:27.568301 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.568263    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b60bddd2-fece-42a1-a73e-ad4f06558575-tls-certs\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:11:27.568301 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.568288    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xbnv\" (UniqueName: \"kubernetes.io/projected/b60bddd2-fece-42a1-a73e-ad4f06558575-kube-api-access-4xbnv\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:11:27.568301 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.568298    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b60bddd2-fece-42a1-a73e-ad4f06558575-kserve-provision-location\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:11:27.568301 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.568307    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b60bddd2-fece-42a1-a73e-ad4f06558575-dshm\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:11:27.609730 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.609707    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-64f689b7c5-xcpw2"
Aug 01 00:11:27.769584 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.769540    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16936fc4-2f5d-44ef-8da0-744e480f48a0-tmp-dir\") pod \"16936fc4-2f5d-44ef-8da0-744e480f48a0\" (UID: \"16936fc4-2f5d-44ef-8da0-744e480f48a0\") "
Aug 01 00:11:27.769584 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.769589    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16936fc4-2f5d-44ef-8da0-744e480f48a0-tls-certs\") pod \"16936fc4-2f5d-44ef-8da0-744e480f48a0\" (UID: \"16936fc4-2f5d-44ef-8da0-744e480f48a0\") "
Aug 01 00:11:27.770113 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.769619    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16936fc4-2f5d-44ef-8da0-744e480f48a0-kserve-provision-location\") pod \"16936fc4-2f5d-44ef-8da0-744e480f48a0\" (UID: \"16936fc4-2f5d-44ef-8da0-744e480f48a0\") "
Aug 01 00:11:27.770113 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.769671    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/16936fc4-2f5d-44ef-8da0-744e480f48a0-cabundle-cert\") pod \"16936fc4-2f5d-44ef-8da0-744e480f48a0\" (UID: \"16936fc4-2f5d-44ef-8da0-744e480f48a0\") "
Aug 01 00:11:27.770113 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.769965    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16936fc4-2f5d-44ef-8da0-744e480f48a0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "16936fc4-2f5d-44ef-8da0-744e480f48a0" (UID: "16936fc4-2f5d-44ef-8da0-744e480f48a0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:11:27.770113 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.769988    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16936fc4-2f5d-44ef-8da0-744e480f48a0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "16936fc4-2f5d-44ef-8da0-744e480f48a0" (UID: "16936fc4-2f5d-44ef-8da0-744e480f48a0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:11:27.771638 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.771617    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16936fc4-2f5d-44ef-8da0-744e480f48a0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "16936fc4-2f5d-44ef-8da0-744e480f48a0" (UID: "16936fc4-2f5d-44ef-8da0-744e480f48a0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:11:27.793507 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.793430    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16936fc4-2f5d-44ef-8da0-744e480f48a0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "16936fc4-2f5d-44ef-8da0-744e480f48a0" (UID: "16936fc4-2f5d-44ef-8da0-744e480f48a0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:11:27.810958 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.810934    2559 generic.go:356] "Generic (PLEG): container finished" podID="16936fc4-2f5d-44ef-8da0-744e480f48a0" containerID="d67c5c7e2b05eab43d42deec1bcc457e9933518585858bd95b82a6c6912eea51" exitCode=0
Aug 01 00:11:27.811079 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.810985    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-64f689b7c5-xcpw2"
Aug 01 00:11:27.811079 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.811020    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-64f689b7c5-xcpw2" event={"ID":"16936fc4-2f5d-44ef-8da0-744e480f48a0","Type":"ContainerDied","Data":"d67c5c7e2b05eab43d42deec1bcc457e9933518585858bd95b82a6c6912eea51"}
Aug 01 00:11:27.811079 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.811071    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-64f689b7c5-xcpw2" event={"ID":"16936fc4-2f5d-44ef-8da0-744e480f48a0","Type":"ContainerDied","Data":"e936a69ee194fc135f84f0fe1048d121e320755d800beb630ed136c2603cbb04"}
Aug 01 00:11:27.811220 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.811102    2559 scope.go:117] "RemoveContainer" containerID="d67c5c7e2b05eab43d42deec1bcc457e9933518585858bd95b82a6c6912eea51"
Aug 01 00:11:27.812622 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.812602    2559 generic.go:356] "Generic (PLEG): container finished" podID="b60bddd2-fece-42a1-a73e-ad4f06558575" containerID="1dd7b0267632818a9fee04638706ed1cceb35776e8b6bb98ff17c74a655aec82" exitCode=0
Aug 01 00:11:27.812716 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.812651    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt" event={"ID":"b60bddd2-fece-42a1-a73e-ad4f06558575","Type":"ContainerDied","Data":"1dd7b0267632818a9fee04638706ed1cceb35776e8b6bb98ff17c74a655aec82"}
Aug 01 00:11:27.812716 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.812655    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt"
Aug 01 00:11:27.812716 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.812683    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt" event={"ID":"b60bddd2-fece-42a1-a73e-ad4f06558575","Type":"ContainerDied","Data":"04cba2a58c4aad2783dd7c7aee4139bcc09f86f012132143397ea3ec8859a47d"}
Aug 01 00:11:27.820401 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.820384    2559 scope.go:117] "RemoveContainer" containerID="60a447c61b980f6f0ec3875f2bf026a8b93257f9fc1ba78f56011041b3096819"
Aug 01 00:11:27.827668 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.827642    2559 scope.go:117] "RemoveContainer" containerID="d67c5c7e2b05eab43d42deec1bcc457e9933518585858bd95b82a6c6912eea51"
Aug 01 00:11:27.827998 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:11:27.827970    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d67c5c7e2b05eab43d42deec1bcc457e9933518585858bd95b82a6c6912eea51\": container with ID starting with d67c5c7e2b05eab43d42deec1bcc457e9933518585858bd95b82a6c6912eea51 not found: ID does not exist" containerID="d67c5c7e2b05eab43d42deec1bcc457e9933518585858bd95b82a6c6912eea51"
Aug 01 00:11:27.827998 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.828005    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d67c5c7e2b05eab43d42deec1bcc457e9933518585858bd95b82a6c6912eea51"} err="failed to get container status \"d67c5c7e2b05eab43d42deec1bcc457e9933518585858bd95b82a6c6912eea51\": rpc error: code = NotFound desc = could not find container \"d67c5c7e2b05eab43d42deec1bcc457e9933518585858bd95b82a6c6912eea51\": container with ID starting with d67c5c7e2b05eab43d42deec1bcc457e9933518585858bd95b82a6c6912eea51 not found: ID does not exist"
Aug 01 00:11:27.828187 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.828023    2559 scope.go:117] "RemoveContainer" containerID="60a447c61b980f6f0ec3875f2bf026a8b93257f9fc1ba78f56011041b3096819"
Aug 01 00:11:27.828277 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:11:27.828258    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60a447c61b980f6f0ec3875f2bf026a8b93257f9fc1ba78f56011041b3096819\": container with ID starting with 60a447c61b980f6f0ec3875f2bf026a8b93257f9fc1ba78f56011041b3096819 not found: ID does not exist" containerID="60a447c61b980f6f0ec3875f2bf026a8b93257f9fc1ba78f56011041b3096819"
Aug 01 00:11:27.828340 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.828286    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60a447c61b980f6f0ec3875f2bf026a8b93257f9fc1ba78f56011041b3096819"} err="failed to get container status \"60a447c61b980f6f0ec3875f2bf026a8b93257f9fc1ba78f56011041b3096819\": rpc error: code = NotFound desc = could not find container \"60a447c61b980f6f0ec3875f2bf026a8b93257f9fc1ba78f56011041b3096819\": container with ID starting with 60a447c61b980f6f0ec3875f2bf026a8b93257f9fc1ba78f56011041b3096819 not found: ID does not exist"
Aug 01 00:11:27.828340 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.828314    2559 scope.go:117] "RemoveContainer" containerID="1dd7b0267632818a9fee04638706ed1cceb35776e8b6bb98ff17c74a655aec82"
Aug 01 00:11:27.830016 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.829997    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt"]
Aug 01 00:11:27.838969 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.834312    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-jtjnt"]
Aug 01 00:11:27.839156 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.839021    2559 scope.go:117] "RemoveContainer" containerID="d9ee11e3573ef420d1e2960813ec340e46b4c19d173bd2a6edeb66e539f68246"
Aug 01 00:11:27.846465 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.846446    2559 scope.go:117] "RemoveContainer" containerID="1dd7b0267632818a9fee04638706ed1cceb35776e8b6bb98ff17c74a655aec82"
Aug 01 00:11:27.846732 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:11:27.846707    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dd7b0267632818a9fee04638706ed1cceb35776e8b6bb98ff17c74a655aec82\": container with ID starting with 1dd7b0267632818a9fee04638706ed1cceb35776e8b6bb98ff17c74a655aec82 not found: ID does not exist" containerID="1dd7b0267632818a9fee04638706ed1cceb35776e8b6bb98ff17c74a655aec82"
Aug 01 00:11:27.846888 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.846862    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dd7b0267632818a9fee04638706ed1cceb35776e8b6bb98ff17c74a655aec82"} err="failed to get container status \"1dd7b0267632818a9fee04638706ed1cceb35776e8b6bb98ff17c74a655aec82\": rpc error: code = NotFound desc = could not find container \"1dd7b0267632818a9fee04638706ed1cceb35776e8b6bb98ff17c74a655aec82\": container with ID starting with 1dd7b0267632818a9fee04638706ed1cceb35776e8b6bb98ff17c74a655aec82 not found: ID does not exist"
Aug 01 00:11:27.846944 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.846893    2559 scope.go:117] "RemoveContainer" containerID="d9ee11e3573ef420d1e2960813ec340e46b4c19d173bd2a6edeb66e539f68246"
Aug 01 00:11:27.847280 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:11:27.847253    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9ee11e3573ef420d1e2960813ec340e46b4c19d173bd2a6edeb66e539f68246\": container with ID starting with d9ee11e3573ef420d1e2960813ec340e46b4c19d173bd2a6edeb66e539f68246 not found: ID does not exist" containerID="d9ee11e3573ef420d1e2960813ec340e46b4c19d173bd2a6edeb66e539f68246"
Aug 01 00:11:27.847374 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.847284    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9ee11e3573ef420d1e2960813ec340e46b4c19d173bd2a6edeb66e539f68246"} err="failed to get container status \"d9ee11e3573ef420d1e2960813ec340e46b4c19d173bd2a6edeb66e539f68246\": rpc error: code = NotFound desc = could not find container \"d9ee11e3573ef420d1e2960813ec340e46b4c19d173bd2a6edeb66e539f68246\": container with ID starting with d9ee11e3573ef420d1e2960813ec340e46b4c19d173bd2a6edeb66e539f68246 not found: ID does not exist"
Aug 01 00:11:27.849166 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.849145    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-64f689b7c5-xcpw2"]
Aug 01 00:11:27.850500 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.850483    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-64f689b7c5-xcpw2"]
Aug 01 00:11:27.870258 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.870231    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16936fc4-2f5d-44ef-8da0-744e480f48a0-tmp-dir\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:11:27.870258 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.870260    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16936fc4-2f5d-44ef-8da0-744e480f48a0-tls-certs\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:11:27.870373 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.870269    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16936fc4-2f5d-44ef-8da0-744e480f48a0-kserve-provision-location\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:11:27.870373 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:27.870278    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/16936fc4-2f5d-44ef-8da0-744e480f48a0-cabundle-cert\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:11:29.785478 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:29.785441    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16936fc4-2f5d-44ef-8da0-744e480f48a0" path="/var/lib/kubelet/pods/16936fc4-2f5d-44ef-8da0-744e480f48a0/volumes"
Aug 01 00:11:29.785926 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:29.785911    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b60bddd2-fece-42a1-a73e-ad4f06558575" path="/var/lib/kubelet/pods/b60bddd2-fece-42a1-a73e-ad4f06558575/volumes"
Aug 01 00:11:38.565421 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:38.565389    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hplv2"]
Aug 01 00:11:38.565801 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:38.565706    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16936fc4-2f5d-44ef-8da0-744e480f48a0" containerName="main"
Aug 01 00:11:38.565801 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:38.565718    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="16936fc4-2f5d-44ef-8da0-744e480f48a0" containerName="main"
Aug 01 00:11:38.565801 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:38.565727    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16936fc4-2f5d-44ef-8da0-744e480f48a0" containerName="storage-initializer"
Aug 01 00:11:38.565801 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:38.565733    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="16936fc4-2f5d-44ef-8da0-744e480f48a0" containerName="storage-initializer"
Aug 01 00:11:38.565801 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:38.565767    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b60bddd2-fece-42a1-a73e-ad4f06558575" containerName="storage-initializer"
Aug 01 00:11:38.565801 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:38.565774    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b60bddd2-fece-42a1-a73e-ad4f06558575" containerName="storage-initializer"
Aug 01 00:11:38.565801 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:38.565785    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b60bddd2-fece-42a1-a73e-ad4f06558575" containerName="main"
Aug 01 00:11:38.565801 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:38.565790    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b60bddd2-fece-42a1-a73e-ad4f06558575" containerName="main"
Aug 01 00:11:38.566059 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:38.565845    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="16936fc4-2f5d-44ef-8da0-744e480f48a0" containerName="main"
Aug 01 00:11:38.566059 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:38.565853    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b60bddd2-fece-42a1-a73e-ad4f06558575" containerName="main"
Aug 01 00:11:38.570508 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:38.570485    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hplv2"
Aug 01 00:11:38.574197 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:38.574170    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"test-div-v1-kserve-self-signed-certs\""
Aug 01 00:11:38.574197 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:38.574182    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"default-dockercfg-z9g4d\""
Aug 01 00:11:38.574388 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:38.574197    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"openshift-service-ca.crt\""
Aug 01 00:11:38.574388 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:38.574205    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"kube-root-ca.crt\""
Aug 01 00:11:38.578581 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:38.578558    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hplv2"]
Aug 01 00:11:38.648220 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:38.648183    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/007f751b-4f41-46f1-90a3-007e369556bf-home\") pod \"test-div-v1-kserve-fb566d667-hplv2\" (UID: \"007f751b-4f41-46f1-90a3-007e369556bf\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hplv2"
Aug 01 00:11:38.648399 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:38.648245    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrx9l\" (UniqueName: \"kubernetes.io/projected/007f751b-4f41-46f1-90a3-007e369556bf-kube-api-access-hrx9l\") pod \"test-div-v1-kserve-fb566d667-hplv2\" (UID: \"007f751b-4f41-46f1-90a3-007e369556bf\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hplv2"
Aug 01 00:11:38.648399 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:38.648277    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/007f751b-4f41-46f1-90a3-007e369556bf-dshm\") pod \"test-div-v1-kserve-fb566d667-hplv2\" (UID: \"007f751b-4f41-46f1-90a3-007e369556bf\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hplv2"
Aug 01 00:11:38.648399 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:38.648302    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/007f751b-4f41-46f1-90a3-007e369556bf-model-cache\") pod \"test-div-v1-kserve-fb566d667-hplv2\" (UID: \"007f751b-4f41-46f1-90a3-007e369556bf\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hplv2"
Aug 01 00:11:38.648399 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:38.648332    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/007f751b-4f41-46f1-90a3-007e369556bf-tls-certs\") pod \"test-div-v1-kserve-fb566d667-hplv2\" (UID: \"007f751b-4f41-46f1-90a3-007e369556bf\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hplv2"
Aug 01 00:11:38.648399 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:38.648388    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/007f751b-4f41-46f1-90a3-007e369556bf-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-hplv2\" (UID: \"007f751b-4f41-46f1-90a3-007e369556bf\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hplv2"
Aug 01 00:11:38.748873 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:38.748836    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/007f751b-4f41-46f1-90a3-007e369556bf-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-hplv2\" (UID: \"007f751b-4f41-46f1-90a3-007e369556bf\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hplv2"
Aug 01 00:11:38.748873 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:38.748878    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/007f751b-4f41-46f1-90a3-007e369556bf-home\") pod \"test-div-v1-kserve-fb566d667-hplv2\" (UID: \"007f751b-4f41-46f1-90a3-007e369556bf\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hplv2"
Aug 01 00:11:38.749085 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:38.748911    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrx9l\" (UniqueName: \"kubernetes.io/projected/007f751b-4f41-46f1-90a3-007e369556bf-kube-api-access-hrx9l\") pod \"test-div-v1-kserve-fb566d667-hplv2\" (UID: \"007f751b-4f41-46f1-90a3-007e369556bf\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hplv2"
Aug 01 00:11:38.749085 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:38.748932    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/007f751b-4f41-46f1-90a3-007e369556bf-dshm\") pod \"test-div-v1-kserve-fb566d667-hplv2\" (UID: \"007f751b-4f41-46f1-90a3-007e369556bf\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hplv2"
Aug 01 00:11:38.749085 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:38.748955    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/007f751b-4f41-46f1-90a3-007e369556bf-model-cache\") pod \"test-div-v1-kserve-fb566d667-hplv2\" (UID: \"007f751b-4f41-46f1-90a3-007e369556bf\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hplv2"
Aug 01 00:11:38.749085 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:38.748988    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/007f751b-4f41-46f1-90a3-007e369556bf-tls-certs\") pod \"test-div-v1-kserve-fb566d667-hplv2\" (UID: \"007f751b-4f41-46f1-90a3-007e369556bf\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hplv2"
Aug 01 00:11:38.749305 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:38.749203    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/007f751b-4f41-46f1-90a3-007e369556bf-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-hplv2\" (UID: \"007f751b-4f41-46f1-90a3-007e369556bf\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hplv2"
Aug 01 00:11:38.749305 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:38.749279    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/007f751b-4f41-46f1-90a3-007e369556bf-home\") pod \"test-div-v1-kserve-fb566d667-hplv2\" (UID: \"007f751b-4f41-46f1-90a3-007e369556bf\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hplv2"
Aug 01 00:11:38.749384 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:38.749364    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/007f751b-4f41-46f1-90a3-007e369556bf-model-cache\") pod \"test-div-v1-kserve-fb566d667-hplv2\" (UID: \"007f751b-4f41-46f1-90a3-007e369556bf\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hplv2"
Aug 01 00:11:38.751275 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:38.751247    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/007f751b-4f41-46f1-90a3-007e369556bf-dshm\") pod \"test-div-v1-kserve-fb566d667-hplv2\" (UID: \"007f751b-4f41-46f1-90a3-007e369556bf\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hplv2"
Aug 01 00:11:38.751464 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:38.751450    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/007f751b-4f41-46f1-90a3-007e369556bf-tls-certs\") pod \"test-div-v1-kserve-fb566d667-hplv2\" (UID: \"007f751b-4f41-46f1-90a3-007e369556bf\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hplv2"
Aug 01 00:11:38.756791 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:38.756736    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrx9l\" (UniqueName: \"kubernetes.io/projected/007f751b-4f41-46f1-90a3-007e369556bf-kube-api-access-hrx9l\") pod \"test-div-v1-kserve-fb566d667-hplv2\" (UID: \"007f751b-4f41-46f1-90a3-007e369556bf\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hplv2"
Aug 01 00:11:38.881355 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:38.881266    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hplv2"
Aug 01 00:11:39.012566 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:39.012541    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hplv2"]
Aug 01 00:11:39.015044 ip-10-0-130-148 kubenswrapper[2559]: W0801 00:11:39.015018    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod007f751b_4f41_46f1_90a3_007e369556bf.slice/crio-f8067e4f5ec9f9c1665368df0467855c927c7790c157e3ca6bb58c2ef442048a WatchSource:0}: Error finding container f8067e4f5ec9f9c1665368df0467855c927c7790c157e3ca6bb58c2ef442048a: Status 404 returned error can't find the container with id f8067e4f5ec9f9c1665368df0467855c927c7790c157e3ca6bb58c2ef442048a
Aug 01 00:11:39.866646 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:39.866603    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hplv2" event={"ID":"007f751b-4f41-46f1-90a3-007e369556bf","Type":"ContainerStarted","Data":"f8067e4f5ec9f9c1665368df0467855c927c7790c157e3ca6bb58c2ef442048a"}
Aug 01 00:11:40.872379 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:40.872340    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hplv2" event={"ID":"007f751b-4f41-46f1-90a3-007e369556bf","Type":"ContainerStarted","Data":"89972181fc15575238b5167d49aa73cf8c7e7bcf338092de967292570ad6c8c1"}
Aug 01 00:11:40.881705 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:40.881683    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hplv2"
Aug 01 00:11:40.889773 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:40.889717    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hplv2" podStartSLOduration=1.794430712 podStartE2EDuration="2.889703976s" totalImagesPullingTime="1.095273264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:11:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:11:40.888351348 +0000 UTC m=+1563.608151195" watchObservedRunningTime="2026-08-01 00:11:40.889703976 +0000 UTC m=+1563.609503821"
Aug 01 00:11:48.882243 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:48.882208    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hplv2"
Aug 01 00:11:48.894592 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:48.894563    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hplv2"
Aug 01 00:11:48.917119 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:11:48.917092    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hplv2"
Aug 01 00:12:25.786799 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:25.786768    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-nhzcl"]
Aug 01 00:12:25.792069 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:25.792051    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-nhzcl"
Aug 01 00:12:25.794655 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:25.794631    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Aug 01 00:12:25.794764 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:25.794638    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Aug 01 00:12:25.794764 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:25.794668    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Aug 01 00:12:25.795929 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:25.795913    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-7mfhv\""
Aug 01 00:12:25.795988 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:25.795954    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Aug 01 00:12:25.796110 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:25.796093    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Aug 01 00:12:25.801875 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:25.801857    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-nhzcl"]
Aug 01 00:12:25.920668 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:25.920639    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29a5095a-8dfd-4f90-84d6-9332ffb8c188-dshm\") pod \"tokenizer-template-args-test-kserve-57f54c4c47-nhzcl\" (UID: \"29a5095a-8dfd-4f90-84d6-9332ffb8c188\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-nhzcl"
Aug 01 00:12:25.920837 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:25.920675    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29a5095a-8dfd-4f90-84d6-9332ffb8c188-tls-certs\") pod \"tokenizer-template-args-test-kserve-57f54c4c47-nhzcl\" (UID: \"29a5095a-8dfd-4f90-84d6-9332ffb8c188\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-nhzcl"
Aug 01 00:12:25.920837 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:25.920731    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29a5095a-8dfd-4f90-84d6-9332ffb8c188-tmp-dir\") pod \"tokenizer-template-args-test-kserve-57f54c4c47-nhzcl\" (UID: \"29a5095a-8dfd-4f90-84d6-9332ffb8c188\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-nhzcl"
Aug 01 00:12:25.920964 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:25.920850    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29a5095a-8dfd-4f90-84d6-9332ffb8c188-home\") pod \"tokenizer-template-args-test-kserve-57f54c4c47-nhzcl\" (UID: \"29a5095a-8dfd-4f90-84d6-9332ffb8c188\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-nhzcl"
Aug 01 00:12:25.920964 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:25.920891    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29a5095a-8dfd-4f90-84d6-9332ffb8c188-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-57f54c4c47-nhzcl\" (UID: \"29a5095a-8dfd-4f90-84d6-9332ffb8c188\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-nhzcl"
Aug 01 00:12:25.920964 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:25.920926    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8klb\" (UniqueName: \"kubernetes.io/projected/29a5095a-8dfd-4f90-84d6-9332ffb8c188-kube-api-access-g8klb\") pod \"tokenizer-template-args-test-kserve-57f54c4c47-nhzcl\" (UID: \"29a5095a-8dfd-4f90-84d6-9332ffb8c188\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-nhzcl"
Aug 01 00:12:25.921083 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:25.920972    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29a5095a-8dfd-4f90-84d6-9332ffb8c188-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-57f54c4c47-nhzcl\" (UID: \"29a5095a-8dfd-4f90-84d6-9332ffb8c188\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-nhzcl"
Aug 01 00:12:25.921083 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:25.921028    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29a5095a-8dfd-4f90-84d6-9332ffb8c188-model-cache\") pod \"tokenizer-template-args-test-kserve-57f54c4c47-nhzcl\" (UID: \"29a5095a-8dfd-4f90-84d6-9332ffb8c188\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-nhzcl"
Aug 01 00:12:26.022195 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:26.022169    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29a5095a-8dfd-4f90-84d6-9332ffb8c188-model-cache\") pod \"tokenizer-template-args-test-kserve-57f54c4c47-nhzcl\" (UID: \"29a5095a-8dfd-4f90-84d6-9332ffb8c188\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-nhzcl"
Aug 01 00:12:26.022539 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:26.022217    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29a5095a-8dfd-4f90-84d6-9332ffb8c188-dshm\") pod \"tokenizer-template-args-test-kserve-57f54c4c47-nhzcl\" (UID: \"29a5095a-8dfd-4f90-84d6-9332ffb8c188\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-nhzcl"
Aug 01 00:12:26.022539 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:26.022249    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29a5095a-8dfd-4f90-84d6-9332ffb8c188-tls-certs\") pod \"tokenizer-template-args-test-kserve-57f54c4c47-nhzcl\" (UID: \"29a5095a-8dfd-4f90-84d6-9332ffb8c188\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-nhzcl"
Aug 01 00:12:26.022539 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:26.022273    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29a5095a-8dfd-4f90-84d6-9332ffb8c188-tmp-dir\") pod \"tokenizer-template-args-test-kserve-57f54c4c47-nhzcl\" (UID: \"29a5095a-8dfd-4f90-84d6-9332ffb8c188\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-nhzcl"
Aug 01 00:12:26.022539 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:26.022299    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29a5095a-8dfd-4f90-84d6-9332ffb8c188-home\") pod \"tokenizer-template-args-test-kserve-57f54c4c47-nhzcl\" (UID: \"29a5095a-8dfd-4f90-84d6-9332ffb8c188\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-nhzcl"
Aug 01 00:12:26.022539 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:26.022322    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29a5095a-8dfd-4f90-84d6-9332ffb8c188-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-57f54c4c47-nhzcl\" (UID: \"29a5095a-8dfd-4f90-84d6-9332ffb8c188\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-nhzcl"
Aug 01 00:12:26.022539 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:26.022352    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8klb\" (UniqueName: \"kubernetes.io/projected/29a5095a-8dfd-4f90-84d6-9332ffb8c188-kube-api-access-g8klb\") pod \"tokenizer-template-args-test-kserve-57f54c4c47-nhzcl\" (UID: \"29a5095a-8dfd-4f90-84d6-9332ffb8c188\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-nhzcl"
Aug 01 00:12:26.022539 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:26.022396    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29a5095a-8dfd-4f90-84d6-9332ffb8c188-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-57f54c4c47-nhzcl\" (UID: \"29a5095a-8dfd-4f90-84d6-9332ffb8c188\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-nhzcl"
Aug 01 00:12:26.022948 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:26.022635    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29a5095a-8dfd-4f90-84d6-9332ffb8c188-model-cache\") pod \"tokenizer-template-args-test-kserve-57f54c4c47-nhzcl\" (UID: \"29a5095a-8dfd-4f90-84d6-9332ffb8c188\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-nhzcl"
Aug 01 00:12:26.022948 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:26.022695    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29a5095a-8dfd-4f90-84d6-9332ffb8c188-tmp-dir\") pod \"tokenizer-template-args-test-kserve-57f54c4c47-nhzcl\" (UID: \"29a5095a-8dfd-4f90-84d6-9332ffb8c188\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-nhzcl"
Aug 01 00:12:26.022948 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:26.022763    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29a5095a-8dfd-4f90-84d6-9332ffb8c188-home\") pod \"tokenizer-template-args-test-kserve-57f54c4c47-nhzcl\" (UID: \"29a5095a-8dfd-4f90-84d6-9332ffb8c188\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-nhzcl"
Aug 01 00:12:26.023139 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:26.023122    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29a5095a-8dfd-4f90-84d6-9332ffb8c188-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-57f54c4c47-nhzcl\" (UID: \"29a5095a-8dfd-4f90-84d6-9332ffb8c188\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-nhzcl"
Aug 01 00:12:26.023431 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:26.023414    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29a5095a-8dfd-4f90-84d6-9332ffb8c188-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-57f54c4c47-nhzcl\" (UID: \"29a5095a-8dfd-4f90-84d6-9332ffb8c188\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-nhzcl"
Aug 01 00:12:26.025072 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:26.025050    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29a5095a-8dfd-4f90-84d6-9332ffb8c188-dshm\") pod \"tokenizer-template-args-test-kserve-57f54c4c47-nhzcl\" (UID: \"29a5095a-8dfd-4f90-84d6-9332ffb8c188\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-nhzcl"
Aug 01 00:12:26.025413 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:26.025393    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29a5095a-8dfd-4f90-84d6-9332ffb8c188-tls-certs\") pod \"tokenizer-template-args-test-kserve-57f54c4c47-nhzcl\" (UID: \"29a5095a-8dfd-4f90-84d6-9332ffb8c188\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-nhzcl"
Aug 01 00:12:26.033108 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:26.033084    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8klb\" (UniqueName: \"kubernetes.io/projected/29a5095a-8dfd-4f90-84d6-9332ffb8c188-kube-api-access-g8klb\") pod \"tokenizer-template-args-test-kserve-57f54c4c47-nhzcl\" (UID: \"29a5095a-8dfd-4f90-84d6-9332ffb8c188\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-nhzcl"
Aug 01 00:12:26.034500 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:26.034474    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-754fb5455-wt65r"]
Aug 01 00:12:26.038471 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:26.038425    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-754fb5455-wt65r"
Aug 01 00:12:26.050049 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:26.049820    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-754fb5455-wt65r"]
Aug 01 00:12:26.102528 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:26.102494    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-nhzcl"
Aug 01 00:12:26.123856 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:26.123829    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51358fba-bbd5-4131-94ee-e99e1dc28ed7-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-754fb5455-wt65r\" (UID: \"51358fba-bbd5-4131-94ee-e99e1dc28ed7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-754fb5455-wt65r"
Aug 01 00:12:26.123959 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:26.123882    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51358fba-bbd5-4131-94ee-e99e1dc28ed7-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-754fb5455-wt65r\" (UID: \"51358fba-bbd5-4131-94ee-e99e1dc28ed7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-754fb5455-wt65r"
Aug 01 00:12:26.123959 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:26.123941    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51358fba-bbd5-4131-94ee-e99e1dc28ed7-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-754fb5455-wt65r\" (UID: \"51358fba-bbd5-4131-94ee-e99e1dc28ed7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-754fb5455-wt65r"
Aug 01 00:12:26.124044 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:26.123997    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51358fba-bbd5-4131-94ee-e99e1dc28ed7-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-754fb5455-wt65r\" (UID: \"51358fba-bbd5-4131-94ee-e99e1dc28ed7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-754fb5455-wt65r"
Aug 01 00:12:26.225240 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:26.225210    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51358fba-bbd5-4131-94ee-e99e1dc28ed7-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-754fb5455-wt65r\" (UID: \"51358fba-bbd5-4131-94ee-e99e1dc28ed7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-754fb5455-wt65r"
Aug 01 00:12:26.225390 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:26.225250    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51358fba-bbd5-4131-94ee-e99e1dc28ed7-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-754fb5455-wt65r\" (UID: \"51358fba-bbd5-4131-94ee-e99e1dc28ed7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-754fb5455-wt65r"
Aug 01 00:12:26.225390 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:26.225273    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51358fba-bbd5-4131-94ee-e99e1dc28ed7-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-754fb5455-wt65r\" (UID: \"51358fba-bbd5-4131-94ee-e99e1dc28ed7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-754fb5455-wt65r"
Aug 01 00:12:26.225390 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:26.225310    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51358fba-bbd5-4131-94ee-e99e1dc28ed7-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-754fb5455-wt65r\" (UID: \"51358fba-bbd5-4131-94ee-e99e1dc28ed7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-754fb5455-wt65r"
Aug 01 00:12:26.225618 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:26.225597    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51358fba-bbd5-4131-94ee-e99e1dc28ed7-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-754fb5455-wt65r\" (UID: \"51358fba-bbd5-4131-94ee-e99e1dc28ed7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-754fb5455-wt65r"
Aug 01 00:12:26.225690 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:26.225617    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51358fba-bbd5-4131-94ee-e99e1dc28ed7-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-754fb5455-wt65r\" (UID: \"51358fba-bbd5-4131-94ee-e99e1dc28ed7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-754fb5455-wt65r"
Aug 01 00:12:26.226003 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:26.225981    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51358fba-bbd5-4131-94ee-e99e1dc28ed7-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-754fb5455-wt65r\" (UID: \"51358fba-bbd5-4131-94ee-e99e1dc28ed7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-754fb5455-wt65r"
Aug 01 00:12:26.227658 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:26.227641    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51358fba-bbd5-4131-94ee-e99e1dc28ed7-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-754fb5455-wt65r\" (UID: \"51358fba-bbd5-4131-94ee-e99e1dc28ed7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-754fb5455-wt65r"
Aug 01 00:12:26.235403 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:26.235385    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-nhzcl"]
Aug 01 00:12:26.237350 ip-10-0-130-148 kubenswrapper[2559]: W0801 00:12:26.237326    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29a5095a_8dfd_4f90_84d6_9332ffb8c188.slice/crio-4f08bd2253ceab29bc8a28ccd4638d57a6b222f2705cafd8db50602e30af3105 WatchSource:0}: Error finding container 4f08bd2253ceab29bc8a28ccd4638d57a6b222f2705cafd8db50602e30af3105: Status 404 returned error can't find the container with id 4f08bd2253ceab29bc8a28ccd4638d57a6b222f2705cafd8db50602e30af3105
Aug 01 00:12:26.354331 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:26.354310    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-754fb5455-wt65r"
Aug 01 00:12:26.484764 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:26.484714    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-754fb5455-wt65r"]
Aug 01 00:12:26.487445 ip-10-0-130-148 kubenswrapper[2559]: W0801 00:12:26.487410    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51358fba_bbd5_4131_94ee_e99e1dc28ed7.slice/crio-8c064d441c06719e088c678777d989c65008c1a153581817a38bdcfc53cb3713 WatchSource:0}: Error finding container 8c064d441c06719e088c678777d989c65008c1a153581817a38bdcfc53cb3713: Status 404 returned error can't find the container with id 8c064d441c06719e088c678777d989c65008c1a153581817a38bdcfc53cb3713
Aug 01 00:12:27.075220 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:27.075183    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-754fb5455-wt65r" event={"ID":"51358fba-bbd5-4131-94ee-e99e1dc28ed7","Type":"ContainerStarted","Data":"54ae96a22ced13231db0c134448b8bbb379699e0d1d7bea5bb8408b5c4471c80"}
Aug 01 00:12:27.075689 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:27.075249    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-754fb5455-wt65r" event={"ID":"51358fba-bbd5-4131-94ee-e99e1dc28ed7","Type":"ContainerStarted","Data":"8c064d441c06719e088c678777d989c65008c1a153581817a38bdcfc53cb3713"}
Aug 01 00:12:27.076776 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:27.076753    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-nhzcl" event={"ID":"29a5095a-8dfd-4f90-84d6-9332ffb8c188","Type":"ContainerStarted","Data":"04b99efd2a5ae5632de820b3162834a8e25fefd0feab995871106c0fb14e1fce"}
Aug 01 00:12:27.076863 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:27.076792    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-nhzcl" event={"ID":"29a5095a-8dfd-4f90-84d6-9332ffb8c188","Type":"ContainerStarted","Data":"4f08bd2253ceab29bc8a28ccd4638d57a6b222f2705cafd8db50602e30af3105"}
Aug 01 00:12:29.088831 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:29.088798    2559 generic.go:356] "Generic (PLEG): container finished" podID="51358fba-bbd5-4131-94ee-e99e1dc28ed7" containerID="54ae96a22ced13231db0c134448b8bbb379699e0d1d7bea5bb8408b5c4471c80" exitCode=0
Aug 01 00:12:29.089289 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:29.088879    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-754fb5455-wt65r" event={"ID":"51358fba-bbd5-4131-94ee-e99e1dc28ed7","Type":"ContainerDied","Data":"54ae96a22ced13231db0c134448b8bbb379699e0d1d7bea5bb8408b5c4471c80"}
Aug 01 00:12:29.090472 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:29.090447    2559 generic.go:356] "Generic (PLEG): container finished" podID="29a5095a-8dfd-4f90-84d6-9332ffb8c188" containerID="04b99efd2a5ae5632de820b3162834a8e25fefd0feab995871106c0fb14e1fce" exitCode=0
Aug 01 00:12:29.090586 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:29.090489    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-nhzcl" event={"ID":"29a5095a-8dfd-4f90-84d6-9332ffb8c188","Type":"ContainerDied","Data":"04b99efd2a5ae5632de820b3162834a8e25fefd0feab995871106c0fb14e1fce"}
Aug 01 00:12:30.097041 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:30.096956    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-nhzcl" event={"ID":"29a5095a-8dfd-4f90-84d6-9332ffb8c188","Type":"ContainerStarted","Data":"1c0b36b8fcdbe4439c073dfe3ceb741e14414e99fe1fa0cbb7e0f13e5b25c2ac"}
Aug 01 00:12:30.098924 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:30.098899    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-754fb5455-wt65r" event={"ID":"51358fba-bbd5-4131-94ee-e99e1dc28ed7","Type":"ContainerStarted","Data":"a638a40b4d1d0c4e7c6f11ec7be8553a9e1ea45e027753dec1ba6ed2e870c5fe"}
Aug 01 00:12:30.099123 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:30.099107    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-754fb5455-wt65r"
Aug 01 00:12:30.102604 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:30.102569    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-nhzcl"
Aug 01 00:12:30.136736 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:30.136685    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-nhzcl" podStartSLOduration=2.838317656 podStartE2EDuration="5.136668406s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.29835075s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:12:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:12:30.134624945 +0000 UTC m=+1612.854424790" watchObservedRunningTime="2026-08-01 00:12:30.136668406 +0000 UTC m=+1612.856468252"
Aug 01 00:12:30.158200 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:30.158149    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-754fb5455-wt65r" podStartSLOduration=1.875435849 podStartE2EDuration="4.158134004s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.282698155s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:12:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:12:30.156555403 +0000 UTC m=+1612.876355248" watchObservedRunningTime="2026-08-01 00:12:30.158134004 +0000 UTC m=+1612.877933875"
Aug 01 00:12:34.613277 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:34.613233    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hplv2"]
Aug 01 00:12:36.102777 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:36.102721    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-nhzcl"
Aug 01 00:12:36.116039 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:36.116014    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-nhzcl"
Aug 01 00:12:36.139622 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:36.139595    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-nhzcl"
Aug 01 00:12:49.632427 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:49.632375    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hplv2" podUID="007f751b-4f41-46f1-90a3-007e369556bf" containerName="main" containerID="cri-o://89972181fc15575238b5167d49aa73cf8c7e7bcf338092de967292570ad6c8c1" gracePeriod=45
Aug 01 00:12:49.882875 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:49.882808    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hplv2"
Aug 01 00:12:49.928373 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:49.928344    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/007f751b-4f41-46f1-90a3-007e369556bf-model-cache\") pod \"007f751b-4f41-46f1-90a3-007e369556bf\" (UID: \"007f751b-4f41-46f1-90a3-007e369556bf\") "
Aug 01 00:12:49.928550 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:49.928390    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/007f751b-4f41-46f1-90a3-007e369556bf-tmp-dir\") pod \"007f751b-4f41-46f1-90a3-007e369556bf\" (UID: \"007f751b-4f41-46f1-90a3-007e369556bf\") "
Aug 01 00:12:49.928550 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:49.928417    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrx9l\" (UniqueName: \"kubernetes.io/projected/007f751b-4f41-46f1-90a3-007e369556bf-kube-api-access-hrx9l\") pod \"007f751b-4f41-46f1-90a3-007e369556bf\" (UID: \"007f751b-4f41-46f1-90a3-007e369556bf\") "
Aug 01 00:12:49.928550 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:49.928450    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/007f751b-4f41-46f1-90a3-007e369556bf-home\") pod \"007f751b-4f41-46f1-90a3-007e369556bf\" (UID: \"007f751b-4f41-46f1-90a3-007e369556bf\") "
Aug 01 00:12:49.928550 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:49.928507    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/007f751b-4f41-46f1-90a3-007e369556bf-tls-certs\") pod \"007f751b-4f41-46f1-90a3-007e369556bf\" (UID: \"007f751b-4f41-46f1-90a3-007e369556bf\") "
Aug 01 00:12:49.928550 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:49.928540    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/007f751b-4f41-46f1-90a3-007e369556bf-dshm\") pod \"007f751b-4f41-46f1-90a3-007e369556bf\" (UID: \"007f751b-4f41-46f1-90a3-007e369556bf\") "
Aug 01 00:12:49.928839 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:49.928658    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/007f751b-4f41-46f1-90a3-007e369556bf-model-cache" (OuterVolumeSpecName: "model-cache") pod "007f751b-4f41-46f1-90a3-007e369556bf" (UID: "007f751b-4f41-46f1-90a3-007e369556bf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:12:49.928839 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:49.928666    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/007f751b-4f41-46f1-90a3-007e369556bf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "007f751b-4f41-46f1-90a3-007e369556bf" (UID: "007f751b-4f41-46f1-90a3-007e369556bf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:12:49.928839 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:49.928788    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/007f751b-4f41-46f1-90a3-007e369556bf-home" (OuterVolumeSpecName: "home") pod "007f751b-4f41-46f1-90a3-007e369556bf" (UID: "007f751b-4f41-46f1-90a3-007e369556bf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:12:49.928996 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:49.928899    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/007f751b-4f41-46f1-90a3-007e369556bf-model-cache\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:12:49.928996 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:49.928917    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/007f751b-4f41-46f1-90a3-007e369556bf-tmp-dir\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:12:49.928996 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:49.928930    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/007f751b-4f41-46f1-90a3-007e369556bf-home\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:12:49.930801 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:49.930764    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/007f751b-4f41-46f1-90a3-007e369556bf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "007f751b-4f41-46f1-90a3-007e369556bf" (UID: "007f751b-4f41-46f1-90a3-007e369556bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:12:49.931106 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:49.931084    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/007f751b-4f41-46f1-90a3-007e369556bf-dshm" (OuterVolumeSpecName: "dshm") pod "007f751b-4f41-46f1-90a3-007e369556bf" (UID: "007f751b-4f41-46f1-90a3-007e369556bf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:12:49.931189 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:49.931126    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/007f751b-4f41-46f1-90a3-007e369556bf-kube-api-access-hrx9l" (OuterVolumeSpecName: "kube-api-access-hrx9l") pod "007f751b-4f41-46f1-90a3-007e369556bf" (UID: "007f751b-4f41-46f1-90a3-007e369556bf"). InnerVolumeSpecName "kube-api-access-hrx9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:12:50.030059 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:50.030022    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/007f751b-4f41-46f1-90a3-007e369556bf-tls-certs\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:12:50.030059 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:50.030052    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/007f751b-4f41-46f1-90a3-007e369556bf-dshm\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:12:50.030059 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:50.030064    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrx9l\" (UniqueName: \"kubernetes.io/projected/007f751b-4f41-46f1-90a3-007e369556bf-kube-api-access-hrx9l\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:12:50.195562 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:50.195529    2559 generic.go:356] "Generic (PLEG): container finished" podID="007f751b-4f41-46f1-90a3-007e369556bf" containerID="89972181fc15575238b5167d49aa73cf8c7e7bcf338092de967292570ad6c8c1" exitCode=0
Aug 01 00:12:50.195729 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:50.195589    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hplv2"
Aug 01 00:12:50.195729 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:50.195620    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hplv2" event={"ID":"007f751b-4f41-46f1-90a3-007e369556bf","Type":"ContainerDied","Data":"89972181fc15575238b5167d49aa73cf8c7e7bcf338092de967292570ad6c8c1"}
Aug 01 00:12:50.195729 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:50.195674    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hplv2" event={"ID":"007f751b-4f41-46f1-90a3-007e369556bf","Type":"ContainerDied","Data":"f8067e4f5ec9f9c1665368df0467855c927c7790c157e3ca6bb58c2ef442048a"}
Aug 01 00:12:50.195729 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:50.195695    2559 scope.go:117] "RemoveContainer" containerID="89972181fc15575238b5167d49aa73cf8c7e7bcf338092de967292570ad6c8c1"
Aug 01 00:12:50.204375 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:50.204355    2559 scope.go:117] "RemoveContainer" containerID="89972181fc15575238b5167d49aa73cf8c7e7bcf338092de967292570ad6c8c1"
Aug 01 00:12:50.204623 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:12:50.204603    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89972181fc15575238b5167d49aa73cf8c7e7bcf338092de967292570ad6c8c1\": container with ID starting with 89972181fc15575238b5167d49aa73cf8c7e7bcf338092de967292570ad6c8c1 not found: ID does not exist" containerID="89972181fc15575238b5167d49aa73cf8c7e7bcf338092de967292570ad6c8c1"
Aug 01 00:12:50.204696 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:50.204632    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89972181fc15575238b5167d49aa73cf8c7e7bcf338092de967292570ad6c8c1"} err="failed to get container status \"89972181fc15575238b5167d49aa73cf8c7e7bcf338092de967292570ad6c8c1\": rpc error: code = NotFound desc = could not find container \"89972181fc15575238b5167d49aa73cf8c7e7bcf338092de967292570ad6c8c1\": container with ID starting with 89972181fc15575238b5167d49aa73cf8c7e7bcf338092de967292570ad6c8c1 not found: ID does not exist"
Aug 01 00:12:50.221864 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:50.221839    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hplv2"]
Aug 01 00:12:50.225765 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:50.225730    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hplv2"]
Aug 01 00:12:51.787032 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:12:51.786996    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="007f751b-4f41-46f1-90a3-007e369556bf" path="/var/lib/kubelet/pods/007f751b-4f41-46f1-90a3-007e369556bf/volumes"
Aug 01 00:13:01.117420 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:01.117385    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-754fb5455-wt65r"
Aug 01 00:13:02.779055 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:02.779013    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-0eb24714_tokenizer-template-args-test-tokenizer-754fb5455-wt65r_51358fba-bbd5-4131-94ee-e99e1dc28ed7/main/0.log"
Aug 01 00:13:24.401440 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:24.401403    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-754fb5455-wt65r"]
Aug 01 00:13:24.402025 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:24.401670    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-754fb5455-wt65r" podUID="51358fba-bbd5-4131-94ee-e99e1dc28ed7" containerName="main" containerID="cri-o://a638a40b4d1d0c4e7c6f11ec7be8553a9e1ea45e027753dec1ba6ed2e870c5fe" gracePeriod=30
Aug 01 00:13:24.413470 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:24.413440    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-nhzcl"]
Aug 01 00:13:24.423511 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:13:24.423479    2559 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 01 00:13:24.423511 ip-10-0-130-148 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:13:24.423511 ip-10-0-130-148 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Aug 01 00:13:24.423511 ip-10-0-130-148 kubenswrapper[2559]:  > containerID="1c0b36b8fcdbe4439c073dfe3ceb741e14414e99fe1fa0cbb7e0f13e5b25c2ac" cmd=["/bin/sleep","15"]
Aug 01 00:13:24.423735 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:24.423529    2559 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-nhzcl" message="" err=<
Aug 01 00:13:24.423735 ip-10-0-130-148 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:13:24.423735 ip-10-0-130-148 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Aug 01 00:13:24.423735 ip-10-0-130-148 kubenswrapper[2559]:  >
Aug 01 00:13:24.423735 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:13:24.423542    2559 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 01 00:13:24.423735 ip-10-0-130-148 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:13:24.423735 ip-10-0-130-148 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Aug 01 00:13:24.423735 ip-10-0-130-148 kubenswrapper[2559]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-nhzcl" podUID="29a5095a-8dfd-4f90-84d6-9332ffb8c188" containerName="main" containerID="cri-o://1c0b36b8fcdbe4439c073dfe3ceb741e14414e99fe1fa0cbb7e0f13e5b25c2ac"
Aug 01 00:13:24.423735 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:24.423576    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-nhzcl" podUID="29a5095a-8dfd-4f90-84d6-9332ffb8c188" containerName="main" containerID="cri-o://1c0b36b8fcdbe4439c073dfe3ceb741e14414e99fe1fa0cbb7e0f13e5b25c2ac" gracePeriod=60
Aug 01 00:13:24.783546 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:24.783522    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-nhzcl"
Aug 01 00:13:24.932477 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:24.932448    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g8klb\" (UniqueName: \"kubernetes.io/projected/29a5095a-8dfd-4f90-84d6-9332ffb8c188-kube-api-access-g8klb\") pod \"29a5095a-8dfd-4f90-84d6-9332ffb8c188\" (UID: \"29a5095a-8dfd-4f90-84d6-9332ffb8c188\") "
Aug 01 00:13:24.932647 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:24.932522    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29a5095a-8dfd-4f90-84d6-9332ffb8c188-kserve-provision-location\") pod \"29a5095a-8dfd-4f90-84d6-9332ffb8c188\" (UID: \"29a5095a-8dfd-4f90-84d6-9332ffb8c188\") "
Aug 01 00:13:24.932647 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:24.932549    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29a5095a-8dfd-4f90-84d6-9332ffb8c188-dshm\") pod \"29a5095a-8dfd-4f90-84d6-9332ffb8c188\" (UID: \"29a5095a-8dfd-4f90-84d6-9332ffb8c188\") "
Aug 01 00:13:24.932647 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:24.932574    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29a5095a-8dfd-4f90-84d6-9332ffb8c188-tmp-dir\") pod \"29a5095a-8dfd-4f90-84d6-9332ffb8c188\" (UID: \"29a5095a-8dfd-4f90-84d6-9332ffb8c188\") "
Aug 01 00:13:24.932647 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:24.932623    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29a5095a-8dfd-4f90-84d6-9332ffb8c188-cabundle-cert\") pod \"29a5095a-8dfd-4f90-84d6-9332ffb8c188\" (UID: \"29a5095a-8dfd-4f90-84d6-9332ffb8c188\") "
Aug 01 00:13:24.932892 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:24.932667    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29a5095a-8dfd-4f90-84d6-9332ffb8c188-model-cache\") pod \"29a5095a-8dfd-4f90-84d6-9332ffb8c188\" (UID: \"29a5095a-8dfd-4f90-84d6-9332ffb8c188\") "
Aug 01 00:13:24.932961 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:24.932908    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29a5095a-8dfd-4f90-84d6-9332ffb8c188-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "29a5095a-8dfd-4f90-84d6-9332ffb8c188" (UID: "29a5095a-8dfd-4f90-84d6-9332ffb8c188"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:13:24.933179 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:24.933154    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29a5095a-8dfd-4f90-84d6-9332ffb8c188-model-cache" (OuterVolumeSpecName: "model-cache") pod "29a5095a-8dfd-4f90-84d6-9332ffb8c188" (UID: "29a5095a-8dfd-4f90-84d6-9332ffb8c188"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:13:24.933271 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:24.933244    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29a5095a-8dfd-4f90-84d6-9332ffb8c188-home\") pod \"29a5095a-8dfd-4f90-84d6-9332ffb8c188\" (UID: \"29a5095a-8dfd-4f90-84d6-9332ffb8c188\") "
Aug 01 00:13:24.933326 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:24.933281    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29a5095a-8dfd-4f90-84d6-9332ffb8c188-tls-certs\") pod \"29a5095a-8dfd-4f90-84d6-9332ffb8c188\" (UID: \"29a5095a-8dfd-4f90-84d6-9332ffb8c188\") "
Aug 01 00:13:24.933386 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:24.933321    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29a5095a-8dfd-4f90-84d6-9332ffb8c188-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "29a5095a-8dfd-4f90-84d6-9332ffb8c188" (UID: "29a5095a-8dfd-4f90-84d6-9332ffb8c188"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:13:24.933495 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:24.933472    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29a5095a-8dfd-4f90-84d6-9332ffb8c188-home" (OuterVolumeSpecName: "home") pod "29a5095a-8dfd-4f90-84d6-9332ffb8c188" (UID: "29a5095a-8dfd-4f90-84d6-9332ffb8c188"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:13:24.933622 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:24.933604    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29a5095a-8dfd-4f90-84d6-9332ffb8c188-tmp-dir\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:13:24.933682 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:24.933626    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29a5095a-8dfd-4f90-84d6-9332ffb8c188-cabundle-cert\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:13:24.933682 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:24.933640    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29a5095a-8dfd-4f90-84d6-9332ffb8c188-model-cache\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:13:24.933682 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:24.933652    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29a5095a-8dfd-4f90-84d6-9332ffb8c188-home\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:13:24.935007 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:24.934982    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29a5095a-8dfd-4f90-84d6-9332ffb8c188-kube-api-access-g8klb" (OuterVolumeSpecName: "kube-api-access-g8klb") pod "29a5095a-8dfd-4f90-84d6-9332ffb8c188" (UID: "29a5095a-8dfd-4f90-84d6-9332ffb8c188"). InnerVolumeSpecName "kube-api-access-g8klb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:13:24.935176 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:24.935155    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29a5095a-8dfd-4f90-84d6-9332ffb8c188-dshm" (OuterVolumeSpecName: "dshm") pod "29a5095a-8dfd-4f90-84d6-9332ffb8c188" (UID: "29a5095a-8dfd-4f90-84d6-9332ffb8c188"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:13:24.935388 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:24.935361    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29a5095a-8dfd-4f90-84d6-9332ffb8c188-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "29a5095a-8dfd-4f90-84d6-9332ffb8c188" (UID: "29a5095a-8dfd-4f90-84d6-9332ffb8c188"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:13:24.960543 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:24.960512    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29a5095a-8dfd-4f90-84d6-9332ffb8c188-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "29a5095a-8dfd-4f90-84d6-9332ffb8c188" (UID: "29a5095a-8dfd-4f90-84d6-9332ffb8c188"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:13:25.034120 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:25.034088    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29a5095a-8dfd-4f90-84d6-9332ffb8c188-tls-certs\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:13:25.034120 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:25.034117    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8klb\" (UniqueName: \"kubernetes.io/projected/29a5095a-8dfd-4f90-84d6-9332ffb8c188-kube-api-access-g8klb\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:13:25.034320 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:25.034129    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29a5095a-8dfd-4f90-84d6-9332ffb8c188-kserve-provision-location\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:13:25.034320 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:25.034139    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29a5095a-8dfd-4f90-84d6-9332ffb8c188-dshm\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:13:25.046411 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:25.046385    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-754fb5455-wt65r"
Aug 01 00:13:25.135167 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:25.135128    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51358fba-bbd5-4131-94ee-e99e1dc28ed7-kserve-provision-location\") pod \"51358fba-bbd5-4131-94ee-e99e1dc28ed7\" (UID: \"51358fba-bbd5-4131-94ee-e99e1dc28ed7\") "
Aug 01 00:13:25.135362 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:25.135192    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51358fba-bbd5-4131-94ee-e99e1dc28ed7-tls-certs\") pod \"51358fba-bbd5-4131-94ee-e99e1dc28ed7\" (UID: \"51358fba-bbd5-4131-94ee-e99e1dc28ed7\") "
Aug 01 00:13:25.135362 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:25.135234    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51358fba-bbd5-4131-94ee-e99e1dc28ed7-cabundle-cert\") pod \"51358fba-bbd5-4131-94ee-e99e1dc28ed7\" (UID: \"51358fba-bbd5-4131-94ee-e99e1dc28ed7\") "
Aug 01 00:13:25.135362 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:25.135284    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51358fba-bbd5-4131-94ee-e99e1dc28ed7-tmp-dir\") pod \"51358fba-bbd5-4131-94ee-e99e1dc28ed7\" (UID: \"51358fba-bbd5-4131-94ee-e99e1dc28ed7\") "
Aug 01 00:13:25.135591 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:25.135554    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51358fba-bbd5-4131-94ee-e99e1dc28ed7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "51358fba-bbd5-4131-94ee-e99e1dc28ed7" (UID: "51358fba-bbd5-4131-94ee-e99e1dc28ed7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:13:25.135662 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:25.135600    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51358fba-bbd5-4131-94ee-e99e1dc28ed7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "51358fba-bbd5-4131-94ee-e99e1dc28ed7" (UID: "51358fba-bbd5-4131-94ee-e99e1dc28ed7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:13:25.137317 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:25.137299    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51358fba-bbd5-4131-94ee-e99e1dc28ed7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "51358fba-bbd5-4131-94ee-e99e1dc28ed7" (UID: "51358fba-bbd5-4131-94ee-e99e1dc28ed7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:13:25.157295 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:25.157266    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51358fba-bbd5-4131-94ee-e99e1dc28ed7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "51358fba-bbd5-4131-94ee-e99e1dc28ed7" (UID: "51358fba-bbd5-4131-94ee-e99e1dc28ed7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:13:25.236220 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:25.236188    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51358fba-bbd5-4131-94ee-e99e1dc28ed7-tmp-dir\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:13:25.236220 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:25.236214    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51358fba-bbd5-4131-94ee-e99e1dc28ed7-kserve-provision-location\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:13:25.236220 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:25.236224    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51358fba-bbd5-4131-94ee-e99e1dc28ed7-tls-certs\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:13:25.236497 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:25.236233    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51358fba-bbd5-4131-94ee-e99e1dc28ed7-cabundle-cert\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:13:25.354653 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:25.354613    2559 generic.go:356] "Generic (PLEG): container finished" podID="51358fba-bbd5-4131-94ee-e99e1dc28ed7" containerID="a638a40b4d1d0c4e7c6f11ec7be8553a9e1ea45e027753dec1ba6ed2e870c5fe" exitCode=0
Aug 01 00:13:25.354841 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:25.354670    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-754fb5455-wt65r"
Aug 01 00:13:25.354841 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:25.354696    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-754fb5455-wt65r" event={"ID":"51358fba-bbd5-4131-94ee-e99e1dc28ed7","Type":"ContainerDied","Data":"a638a40b4d1d0c4e7c6f11ec7be8553a9e1ea45e027753dec1ba6ed2e870c5fe"}
Aug 01 00:13:25.354841 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:25.354760    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-754fb5455-wt65r" event={"ID":"51358fba-bbd5-4131-94ee-e99e1dc28ed7","Type":"ContainerDied","Data":"8c064d441c06719e088c678777d989c65008c1a153581817a38bdcfc53cb3713"}
Aug 01 00:13:25.354841 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:25.354794    2559 scope.go:117] "RemoveContainer" containerID="a638a40b4d1d0c4e7c6f11ec7be8553a9e1ea45e027753dec1ba6ed2e870c5fe"
Aug 01 00:13:25.356470 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:25.356447    2559 generic.go:356] "Generic (PLEG): container finished" podID="29a5095a-8dfd-4f90-84d6-9332ffb8c188" containerID="1c0b36b8fcdbe4439c073dfe3ceb741e14414e99fe1fa0cbb7e0f13e5b25c2ac" exitCode=0
Aug 01 00:13:25.356589 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:25.356501    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-nhzcl" event={"ID":"29a5095a-8dfd-4f90-84d6-9332ffb8c188","Type":"ContainerDied","Data":"1c0b36b8fcdbe4439c073dfe3ceb741e14414e99fe1fa0cbb7e0f13e5b25c2ac"}
Aug 01 00:13:25.356589 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:25.356569    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-nhzcl" event={"ID":"29a5095a-8dfd-4f90-84d6-9332ffb8c188","Type":"ContainerDied","Data":"4f08bd2253ceab29bc8a28ccd4638d57a6b222f2705cafd8db50602e30af3105"}
Aug 01 00:13:25.356671 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:25.356511    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-nhzcl"
Aug 01 00:13:25.363474 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:25.363457    2559 scope.go:117] "RemoveContainer" containerID="54ae96a22ced13231db0c134448b8bbb379699e0d1d7bea5bb8408b5c4471c80"
Aug 01 00:13:25.371619 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:25.371602    2559 scope.go:117] "RemoveContainer" containerID="a638a40b4d1d0c4e7c6f11ec7be8553a9e1ea45e027753dec1ba6ed2e870c5fe"
Aug 01 00:13:25.377577 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:13:25.372342    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a638a40b4d1d0c4e7c6f11ec7be8553a9e1ea45e027753dec1ba6ed2e870c5fe\": container with ID starting with a638a40b4d1d0c4e7c6f11ec7be8553a9e1ea45e027753dec1ba6ed2e870c5fe not found: ID does not exist" containerID="a638a40b4d1d0c4e7c6f11ec7be8553a9e1ea45e027753dec1ba6ed2e870c5fe"
Aug 01 00:13:25.377577 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:25.372394    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a638a40b4d1d0c4e7c6f11ec7be8553a9e1ea45e027753dec1ba6ed2e870c5fe"} err="failed to get container status \"a638a40b4d1d0c4e7c6f11ec7be8553a9e1ea45e027753dec1ba6ed2e870c5fe\": rpc error: code = NotFound desc = could not find container \"a638a40b4d1d0c4e7c6f11ec7be8553a9e1ea45e027753dec1ba6ed2e870c5fe\": container with ID starting with a638a40b4d1d0c4e7c6f11ec7be8553a9e1ea45e027753dec1ba6ed2e870c5fe not found: ID does not exist"
Aug 01 00:13:25.377577 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:25.372419    2559 scope.go:117] "RemoveContainer" containerID="54ae96a22ced13231db0c134448b8bbb379699e0d1d7bea5bb8408b5c4471c80"
Aug 01 00:13:25.377865 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:13:25.377651    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54ae96a22ced13231db0c134448b8bbb379699e0d1d7bea5bb8408b5c4471c80\": container with ID starting with 54ae96a22ced13231db0c134448b8bbb379699e0d1d7bea5bb8408b5c4471c80 not found: ID does not exist" containerID="54ae96a22ced13231db0c134448b8bbb379699e0d1d7bea5bb8408b5c4471c80"
Aug 01 00:13:25.377910 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:25.377884    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54ae96a22ced13231db0c134448b8bbb379699e0d1d7bea5bb8408b5c4471c80"} err="failed to get container status \"54ae96a22ced13231db0c134448b8bbb379699e0d1d7bea5bb8408b5c4471c80\": rpc error: code = NotFound desc = could not find container \"54ae96a22ced13231db0c134448b8bbb379699e0d1d7bea5bb8408b5c4471c80\": container with ID starting with 54ae96a22ced13231db0c134448b8bbb379699e0d1d7bea5bb8408b5c4471c80 not found: ID does not exist"
Aug 01 00:13:25.377948 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:25.377908    2559 scope.go:117] "RemoveContainer" containerID="1c0b36b8fcdbe4439c073dfe3ceb741e14414e99fe1fa0cbb7e0f13e5b25c2ac"
Aug 01 00:13:25.382391 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:25.382368    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-754fb5455-wt65r"]
Aug 01 00:13:25.384876 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:25.384856    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-754fb5455-wt65r"]
Aug 01 00:13:25.387697 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:25.387680    2559 scope.go:117] "RemoveContainer" containerID="04b99efd2a5ae5632de820b3162834a8e25fefd0feab995871106c0fb14e1fce"
Aug 01 00:13:25.394560 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:25.394542    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-nhzcl"]
Aug 01 00:13:25.395113 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:25.395094    2559 scope.go:117] "RemoveContainer" containerID="1c0b36b8fcdbe4439c073dfe3ceb741e14414e99fe1fa0cbb7e0f13e5b25c2ac"
Aug 01 00:13:25.395355 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:13:25.395336    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c0b36b8fcdbe4439c073dfe3ceb741e14414e99fe1fa0cbb7e0f13e5b25c2ac\": container with ID starting with 1c0b36b8fcdbe4439c073dfe3ceb741e14414e99fe1fa0cbb7e0f13e5b25c2ac not found: ID does not exist" containerID="1c0b36b8fcdbe4439c073dfe3ceb741e14414e99fe1fa0cbb7e0f13e5b25c2ac"
Aug 01 00:13:25.395423 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:25.395362    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c0b36b8fcdbe4439c073dfe3ceb741e14414e99fe1fa0cbb7e0f13e5b25c2ac"} err="failed to get container status \"1c0b36b8fcdbe4439c073dfe3ceb741e14414e99fe1fa0cbb7e0f13e5b25c2ac\": rpc error: code = NotFound desc = could not find container \"1c0b36b8fcdbe4439c073dfe3ceb741e14414e99fe1fa0cbb7e0f13e5b25c2ac\": container with ID starting with 1c0b36b8fcdbe4439c073dfe3ceb741e14414e99fe1fa0cbb7e0f13e5b25c2ac not found: ID does not exist"
Aug 01 00:13:25.395423 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:25.395379    2559 scope.go:117] "RemoveContainer" containerID="04b99efd2a5ae5632de820b3162834a8e25fefd0feab995871106c0fb14e1fce"
Aug 01 00:13:25.395625 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:13:25.395607    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04b99efd2a5ae5632de820b3162834a8e25fefd0feab995871106c0fb14e1fce\": container with ID starting with 04b99efd2a5ae5632de820b3162834a8e25fefd0feab995871106c0fb14e1fce not found: ID does not exist" containerID="04b99efd2a5ae5632de820b3162834a8e25fefd0feab995871106c0fb14e1fce"
Aug 01 00:13:25.395679 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:25.395634    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04b99efd2a5ae5632de820b3162834a8e25fefd0feab995871106c0fb14e1fce"} err="failed to get container status \"04b99efd2a5ae5632de820b3162834a8e25fefd0feab995871106c0fb14e1fce\": rpc error: code = NotFound desc = could not find container \"04b99efd2a5ae5632de820b3162834a8e25fefd0feab995871106c0fb14e1fce\": container with ID starting with 04b99efd2a5ae5632de820b3162834a8e25fefd0feab995871106c0fb14e1fce not found: ID does not exist"
Aug 01 00:13:25.397928 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:25.397908    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-nhzcl"]
Aug 01 00:13:25.785205 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:25.785174    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29a5095a-8dfd-4f90-84d6-9332ffb8c188" path="/var/lib/kubelet/pods/29a5095a-8dfd-4f90-84d6-9332ffb8c188/volumes"
Aug 01 00:13:25.785627 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:25.785611    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51358fba-bbd5-4131-94ee-e99e1dc28ed7" path="/var/lib/kubelet/pods/51358fba-bbd5-4131-94ee-e99e1dc28ed7/volumes"
Aug 01 00:13:42.966135 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:42.966098    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-svc4b"]
Aug 01 00:13:42.966529 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:42.966389    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29a5095a-8dfd-4f90-84d6-9332ffb8c188" containerName="main"
Aug 01 00:13:42.966529 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:42.966399    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="29a5095a-8dfd-4f90-84d6-9332ffb8c188" containerName="main"
Aug 01 00:13:42.966529 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:42.966412    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29a5095a-8dfd-4f90-84d6-9332ffb8c188" containerName="storage-initializer"
Aug 01 00:13:42.966529 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:42.966419    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="29a5095a-8dfd-4f90-84d6-9332ffb8c188" containerName="storage-initializer"
Aug 01 00:13:42.966529 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:42.966427    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="007f751b-4f41-46f1-90a3-007e369556bf" containerName="main"
Aug 01 00:13:42.966529 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:42.966433    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="007f751b-4f41-46f1-90a3-007e369556bf" containerName="main"
Aug 01 00:13:42.966529 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:42.966441    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51358fba-bbd5-4131-94ee-e99e1dc28ed7" containerName="main"
Aug 01 00:13:42.966529 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:42.966446    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="51358fba-bbd5-4131-94ee-e99e1dc28ed7" containerName="main"
Aug 01 00:13:42.966529 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:42.966456    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51358fba-bbd5-4131-94ee-e99e1dc28ed7" containerName="storage-initializer"
Aug 01 00:13:42.966529 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:42.966461    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="51358fba-bbd5-4131-94ee-e99e1dc28ed7" containerName="storage-initializer"
Aug 01 00:13:42.966529 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:42.966522    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="007f751b-4f41-46f1-90a3-007e369556bf" containerName="main"
Aug 01 00:13:42.966529 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:42.966529    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="51358fba-bbd5-4131-94ee-e99e1dc28ed7" containerName="main"
Aug 01 00:13:42.966529 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:42.966535    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="29a5095a-8dfd-4f90-84d6-9332ffb8c188" containerName="main"
Aug 01 00:13:42.969082 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:42.969064    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-svc4b"
Aug 01 00:13:42.972975 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:42.972952    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"kube-root-ca.crt\""
Aug 01 00:13:42.973093 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:42.972953    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"default-dockercfg-nvh4p\""
Aug 01 00:13:42.973093 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:42.972957    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"openshift-service-ca.crt\""
Aug 01 00:13:42.973093 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:42.972958    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"stop-v1-kserve-self-signed-certs\""
Aug 01 00:13:42.979048 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:42.979028    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-svc4b"]
Aug 01 00:13:43.082937 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:43.082899    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e09c15e-16b6-4824-82bc-f3efd8fc3b9e-home\") pod \"stop-v1-kserve-5657b88c95-svc4b\" (UID: \"4e09c15e-16b6-4824-82bc-f3efd8fc3b9e\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-svc4b"
Aug 01 00:13:43.083144 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:43.082944    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e09c15e-16b6-4824-82bc-f3efd8fc3b9e-model-cache\") pod \"stop-v1-kserve-5657b88c95-svc4b\" (UID: \"4e09c15e-16b6-4824-82bc-f3efd8fc3b9e\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-svc4b"
Aug 01 00:13:43.083144 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:43.083049    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e09c15e-16b6-4824-82bc-f3efd8fc3b9e-dshm\") pod \"stop-v1-kserve-5657b88c95-svc4b\" (UID: \"4e09c15e-16b6-4824-82bc-f3efd8fc3b9e\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-svc4b"
Aug 01 00:13:43.083144 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:43.083106    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77mqr\" (UniqueName: \"kubernetes.io/projected/4e09c15e-16b6-4824-82bc-f3efd8fc3b9e-kube-api-access-77mqr\") pod \"stop-v1-kserve-5657b88c95-svc4b\" (UID: \"4e09c15e-16b6-4824-82bc-f3efd8fc3b9e\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-svc4b"
Aug 01 00:13:43.083310 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:43.083144    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e09c15e-16b6-4824-82bc-f3efd8fc3b9e-tmp-dir\") pod \"stop-v1-kserve-5657b88c95-svc4b\" (UID: \"4e09c15e-16b6-4824-82bc-f3efd8fc3b9e\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-svc4b"
Aug 01 00:13:43.083310 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:43.083167    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e09c15e-16b6-4824-82bc-f3efd8fc3b9e-tls-certs\") pod \"stop-v1-kserve-5657b88c95-svc4b\" (UID: \"4e09c15e-16b6-4824-82bc-f3efd8fc3b9e\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-svc4b"
Aug 01 00:13:43.184159 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:43.184125    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77mqr\" (UniqueName: \"kubernetes.io/projected/4e09c15e-16b6-4824-82bc-f3efd8fc3b9e-kube-api-access-77mqr\") pod \"stop-v1-kserve-5657b88c95-svc4b\" (UID: \"4e09c15e-16b6-4824-82bc-f3efd8fc3b9e\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-svc4b"
Aug 01 00:13:43.184316 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:43.184168    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e09c15e-16b6-4824-82bc-f3efd8fc3b9e-tmp-dir\") pod \"stop-v1-kserve-5657b88c95-svc4b\" (UID: \"4e09c15e-16b6-4824-82bc-f3efd8fc3b9e\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-svc4b"
Aug 01 00:13:43.184316 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:43.184193    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e09c15e-16b6-4824-82bc-f3efd8fc3b9e-tls-certs\") pod \"stop-v1-kserve-5657b88c95-svc4b\" (UID: \"4e09c15e-16b6-4824-82bc-f3efd8fc3b9e\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-svc4b"
Aug 01 00:13:43.184316 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:43.184238    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e09c15e-16b6-4824-82bc-f3efd8fc3b9e-home\") pod \"stop-v1-kserve-5657b88c95-svc4b\" (UID: \"4e09c15e-16b6-4824-82bc-f3efd8fc3b9e\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-svc4b"
Aug 01 00:13:43.184316 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:43.184275    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e09c15e-16b6-4824-82bc-f3efd8fc3b9e-model-cache\") pod \"stop-v1-kserve-5657b88c95-svc4b\" (UID: \"4e09c15e-16b6-4824-82bc-f3efd8fc3b9e\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-svc4b"
Aug 01 00:13:43.184539 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:43.184337    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e09c15e-16b6-4824-82bc-f3efd8fc3b9e-dshm\") pod \"stop-v1-kserve-5657b88c95-svc4b\" (UID: \"4e09c15e-16b6-4824-82bc-f3efd8fc3b9e\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-svc4b"
Aug 01 00:13:43.184599 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:43.184570    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e09c15e-16b6-4824-82bc-f3efd8fc3b9e-tmp-dir\") pod \"stop-v1-kserve-5657b88c95-svc4b\" (UID: \"4e09c15e-16b6-4824-82bc-f3efd8fc3b9e\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-svc4b"
Aug 01 00:13:43.184657 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:43.184610    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e09c15e-16b6-4824-82bc-f3efd8fc3b9e-home\") pod \"stop-v1-kserve-5657b88c95-svc4b\" (UID: \"4e09c15e-16b6-4824-82bc-f3efd8fc3b9e\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-svc4b"
Aug 01 00:13:43.184695 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:43.184673    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e09c15e-16b6-4824-82bc-f3efd8fc3b9e-model-cache\") pod \"stop-v1-kserve-5657b88c95-svc4b\" (UID: \"4e09c15e-16b6-4824-82bc-f3efd8fc3b9e\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-svc4b"
Aug 01 00:13:43.186437 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:43.186421    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e09c15e-16b6-4824-82bc-f3efd8fc3b9e-dshm\") pod \"stop-v1-kserve-5657b88c95-svc4b\" (UID: \"4e09c15e-16b6-4824-82bc-f3efd8fc3b9e\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-svc4b"
Aug 01 00:13:43.186615 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:43.186601    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e09c15e-16b6-4824-82bc-f3efd8fc3b9e-tls-certs\") pod \"stop-v1-kserve-5657b88c95-svc4b\" (UID: \"4e09c15e-16b6-4824-82bc-f3efd8fc3b9e\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-svc4b"
Aug 01 00:13:43.199301 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:43.199278    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77mqr\" (UniqueName: \"kubernetes.io/projected/4e09c15e-16b6-4824-82bc-f3efd8fc3b9e-kube-api-access-77mqr\") pod \"stop-v1-kserve-5657b88c95-svc4b\" (UID: \"4e09c15e-16b6-4824-82bc-f3efd8fc3b9e\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-svc4b"
Aug 01 00:13:43.279199 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:43.279119    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-svc4b"
Aug 01 00:13:43.407248 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:43.407223    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-svc4b"]
Aug 01 00:13:43.409824 ip-10-0-130-148 kubenswrapper[2559]: W0801 00:13:43.409797    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e09c15e_16b6_4824_82bc_f3efd8fc3b9e.slice/crio-680ce6c299c60e76f1f78511031e7846ac1af36b4c04cd6375679926c99e158d WatchSource:0}: Error finding container 680ce6c299c60e76f1f78511031e7846ac1af36b4c04cd6375679926c99e158d: Status 404 returned error can't find the container with id 680ce6c299c60e76f1f78511031e7846ac1af36b4c04cd6375679926c99e158d
Aug 01 00:13:43.436810 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:43.436769    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-svc4b" event={"ID":"4e09c15e-16b6-4824-82bc-f3efd8fc3b9e","Type":"ContainerStarted","Data":"680ce6c299c60e76f1f78511031e7846ac1af36b4c04cd6375679926c99e158d"}
Aug 01 00:13:44.443619 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:44.443584    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-svc4b" event={"ID":"4e09c15e-16b6-4824-82bc-f3efd8fc3b9e","Type":"ContainerStarted","Data":"30241a841f35cef8f7e2dbda104fb0be227d40bcb03e91b129c92058a6ab1068"}
Aug 01 00:13:44.460280 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:44.460228    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-svc4b" podStartSLOduration=2.460210956 podStartE2EDuration="2.460210956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:13:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:13:44.458705306 +0000 UTC m=+1687.178505152" watchObservedRunningTime="2026-08-01 00:13:44.460210956 +0000 UTC m=+1687.180010801"
Aug 01 00:13:45.279796 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:45.279763    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-svc4b"
Aug 01 00:13:53.279923 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:53.279872    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-svc4b"
Aug 01 00:13:53.292848 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:53.292823    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-svc4b"
Aug 01 00:13:53.492480 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:13:53.492444    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-svc4b"
Aug 01 00:14:15.056920 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:14:15.056847    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zhn7f"]
Aug 01 00:14:15.059991 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:14:15.059974    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zhn7f"
Aug 01 00:14:15.062281 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:14:15.062261    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Aug 01 00:14:15.063420 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:14:15.063398    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Aug 01 00:14:15.063522 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:14:15.063444    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Aug 01 00:14:15.063522 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:14:15.063501    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-l5g76\""
Aug 01 00:14:15.071087 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:14:15.071068    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zhn7f"]
Aug 01 00:14:15.148095 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:14:15.148062    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79b4d168-b16f-494b-88f6-ba0b8b660d6e-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zhn7f\" (UID: \"79b4d168-b16f-494b-88f6-ba0b8b660d6e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zhn7f"
Aug 01 00:14:15.148238 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:14:15.148102    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79b4d168-b16f-494b-88f6-ba0b8b660d6e-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zhn7f\" (UID: \"79b4d168-b16f-494b-88f6-ba0b8b660d6e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zhn7f"
Aug 01 00:14:15.148238 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:14:15.148123    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk77r\" (UniqueName: \"kubernetes.io/projected/79b4d168-b16f-494b-88f6-ba0b8b660d6e-kube-api-access-fk77r\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zhn7f\" (UID: \"79b4d168-b16f-494b-88f6-ba0b8b660d6e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zhn7f"
Aug 01 00:14:15.148238 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:14:15.148200    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79b4d168-b16f-494b-88f6-ba0b8b660d6e-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zhn7f\" (UID: \"79b4d168-b16f-494b-88f6-ba0b8b660d6e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zhn7f"
Aug 01 00:14:15.148413 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:14:15.148259    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79b4d168-b16f-494b-88f6-ba0b8b660d6e-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zhn7f\" (UID: \"79b4d168-b16f-494b-88f6-ba0b8b660d6e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zhn7f"
Aug 01 00:14:15.148413 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:14:15.148349    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79b4d168-b16f-494b-88f6-ba0b8b660d6e-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zhn7f\" (UID: \"79b4d168-b16f-494b-88f6-ba0b8b660d6e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zhn7f"
Aug 01 00:14:15.249688 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:14:15.249658    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79b4d168-b16f-494b-88f6-ba0b8b660d6e-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zhn7f\" (UID: \"79b4d168-b16f-494b-88f6-ba0b8b660d6e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zhn7f"
Aug 01 00:14:15.249847 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:14:15.249707    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79b4d168-b16f-494b-88f6-ba0b8b660d6e-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zhn7f\" (UID: \"79b4d168-b16f-494b-88f6-ba0b8b660d6e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zhn7f"
Aug 01 00:14:15.249847 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:14:15.249738    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79b4d168-b16f-494b-88f6-ba0b8b660d6e-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zhn7f\" (UID: \"79b4d168-b16f-494b-88f6-ba0b8b660d6e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zhn7f"
Aug 01 00:14:15.249847 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:14:15.249791    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk77r\" (UniqueName: \"kubernetes.io/projected/79b4d168-b16f-494b-88f6-ba0b8b660d6e-kube-api-access-fk77r\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zhn7f\" (UID: \"79b4d168-b16f-494b-88f6-ba0b8b660d6e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zhn7f"
Aug 01 00:14:15.249847 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:14:15.249830    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79b4d168-b16f-494b-88f6-ba0b8b660d6e-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zhn7f\" (UID: \"79b4d168-b16f-494b-88f6-ba0b8b660d6e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zhn7f"
Aug 01 00:14:15.250011 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:14:15.249853    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79b4d168-b16f-494b-88f6-ba0b8b660d6e-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zhn7f\" (UID: \"79b4d168-b16f-494b-88f6-ba0b8b660d6e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zhn7f"
Aug 01 00:14:15.250151 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:14:15.250130    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79b4d168-b16f-494b-88f6-ba0b8b660d6e-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zhn7f\" (UID: \"79b4d168-b16f-494b-88f6-ba0b8b660d6e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zhn7f"
Aug 01 00:14:15.250191 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:14:15.250161    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79b4d168-b16f-494b-88f6-ba0b8b660d6e-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zhn7f\" (UID: \"79b4d168-b16f-494b-88f6-ba0b8b660d6e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zhn7f"
Aug 01 00:14:15.250238 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:14:15.250217    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79b4d168-b16f-494b-88f6-ba0b8b660d6e-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zhn7f\" (UID: \"79b4d168-b16f-494b-88f6-ba0b8b660d6e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zhn7f"
Aug 01 00:14:15.252108 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:14:15.252087    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79b4d168-b16f-494b-88f6-ba0b8b660d6e-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zhn7f\" (UID: \"79b4d168-b16f-494b-88f6-ba0b8b660d6e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zhn7f"
Aug 01 00:14:15.252370 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:14:15.252351    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79b4d168-b16f-494b-88f6-ba0b8b660d6e-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zhn7f\" (UID: \"79b4d168-b16f-494b-88f6-ba0b8b660d6e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zhn7f"
Aug 01 00:14:15.257351 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:14:15.257328    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk77r\" (UniqueName: \"kubernetes.io/projected/79b4d168-b16f-494b-88f6-ba0b8b660d6e-kube-api-access-fk77r\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zhn7f\" (UID: \"79b4d168-b16f-494b-88f6-ba0b8b660d6e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zhn7f"
Aug 01 00:14:15.371085 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:14:15.371015    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zhn7f"
Aug 01 00:14:15.495480 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:14:15.495452    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zhn7f"]
Aug 01 00:14:15.498063 ip-10-0-130-148 kubenswrapper[2559]: W0801 00:14:15.498037    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79b4d168_b16f_494b_88f6_ba0b8b660d6e.slice/crio-d209cb46390f527b3f6971a3ffa951eb9df017237bbe8328ea94c17590906cf4 WatchSource:0}: Error finding container d209cb46390f527b3f6971a3ffa951eb9df017237bbe8328ea94c17590906cf4: Status 404 returned error can't find the container with id d209cb46390f527b3f6971a3ffa951eb9df017237bbe8328ea94c17590906cf4
Aug 01 00:14:15.574803 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:14:15.574775    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zhn7f" event={"ID":"79b4d168-b16f-494b-88f6-ba0b8b660d6e","Type":"ContainerStarted","Data":"c26605948b953aefdb57e0e51e122bb54b0231e6596ac82a38cdf5cf3948abe9"}
Aug 01 00:14:15.574937 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:14:15.574831    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zhn7f" event={"ID":"79b4d168-b16f-494b-88f6-ba0b8b660d6e","Type":"ContainerStarted","Data":"d209cb46390f527b3f6971a3ffa951eb9df017237bbe8328ea94c17590906cf4"}
Aug 01 00:14:15.591018 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:14:15.590969    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zhn7f" podStartSLOduration=0.590953431 podStartE2EDuration="590.953431ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:14:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:14:15.589804784 +0000 UTC m=+1718.309604631" watchObservedRunningTime="2026-08-01 00:14:15.590953431 +0000 UTC m=+1718.310753276"
Aug 01 00:14:16.371789 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:14:16.371720    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zhn7f"
Aug 01 00:14:25.371709 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:14:25.371672    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zhn7f"
Aug 01 00:14:25.384103 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:14:25.384076    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zhn7f"
Aug 01 00:14:25.626943 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:14:25.626857    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zhn7f"
Aug 01 00:15:08.091793 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:08.091762    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-svc4b"]
Aug 01 00:15:13.196059 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:13.196026    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zhn7f"]
Aug 01 00:15:13.205250 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:15:13.205219    2559 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 01 00:15:13.205250 ip-10-0-130-148 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:15:13.205250 ip-10-0-130-148 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Aug 01 00:15:13.205250 ip-10-0-130-148 kubenswrapper[2559]:  > containerID="c26605948b953aefdb57e0e51e122bb54b0231e6596ac82a38cdf5cf3948abe9" cmd=["/bin/sleep","15"]
Aug 01 00:15:13.205486 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:13.205267    2559 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zhn7f" message="" err=<
Aug 01 00:15:13.205486 ip-10-0-130-148 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:15:13.205486 ip-10-0-130-148 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Aug 01 00:15:13.205486 ip-10-0-130-148 kubenswrapper[2559]:  >
Aug 01 00:15:13.205486 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:15:13.205281    2559 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 01 00:15:13.205486 ip-10-0-130-148 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:15:13.205486 ip-10-0-130-148 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Aug 01 00:15:13.205486 ip-10-0-130-148 kubenswrapper[2559]:  > pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zhn7f" podUID="79b4d168-b16f-494b-88f6-ba0b8b660d6e" containerName="main" containerID="cri-o://c26605948b953aefdb57e0e51e122bb54b0231e6596ac82a38cdf5cf3948abe9"
Aug 01 00:15:13.205486 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:13.205315    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zhn7f" podUID="79b4d168-b16f-494b-88f6-ba0b8b660d6e" containerName="main" containerID="cri-o://c26605948b953aefdb57e0e51e122bb54b0231e6596ac82a38cdf5cf3948abe9" gracePeriod=60
Aug 01 00:15:13.549204 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:13.549180    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zhn7f"
Aug 01 00:15:13.615449 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:13.615420    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79b4d168-b16f-494b-88f6-ba0b8b660d6e-tmp-dir\") pod \"79b4d168-b16f-494b-88f6-ba0b8b660d6e\" (UID: \"79b4d168-b16f-494b-88f6-ba0b8b660d6e\") "
Aug 01 00:15:13.615449 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:13.615451    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79b4d168-b16f-494b-88f6-ba0b8b660d6e-tls-certs\") pod \"79b4d168-b16f-494b-88f6-ba0b8b660d6e\" (UID: \"79b4d168-b16f-494b-88f6-ba0b8b660d6e\") "
Aug 01 00:15:13.615633 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:13.615484    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fk77r\" (UniqueName: \"kubernetes.io/projected/79b4d168-b16f-494b-88f6-ba0b8b660d6e-kube-api-access-fk77r\") pod \"79b4d168-b16f-494b-88f6-ba0b8b660d6e\" (UID: \"79b4d168-b16f-494b-88f6-ba0b8b660d6e\") "
Aug 01 00:15:13.615633 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:13.615612    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79b4d168-b16f-494b-88f6-ba0b8b660d6e-model-cache\") pod \"79b4d168-b16f-494b-88f6-ba0b8b660d6e\" (UID: \"79b4d168-b16f-494b-88f6-ba0b8b660d6e\") "
Aug 01 00:15:13.615734 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:13.615651    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79b4d168-b16f-494b-88f6-ba0b8b660d6e-dshm\") pod \"79b4d168-b16f-494b-88f6-ba0b8b660d6e\" (UID: \"79b4d168-b16f-494b-88f6-ba0b8b660d6e\") "
Aug 01 00:15:13.615734 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:13.615732    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79b4d168-b16f-494b-88f6-ba0b8b660d6e-home\") pod \"79b4d168-b16f-494b-88f6-ba0b8b660d6e\" (UID: \"79b4d168-b16f-494b-88f6-ba0b8b660d6e\") "
Aug 01 00:15:13.615892 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:13.615730    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79b4d168-b16f-494b-88f6-ba0b8b660d6e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "79b4d168-b16f-494b-88f6-ba0b8b660d6e" (UID: "79b4d168-b16f-494b-88f6-ba0b8b660d6e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:15:13.615948 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:13.615905    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79b4d168-b16f-494b-88f6-ba0b8b660d6e-model-cache" (OuterVolumeSpecName: "model-cache") pod "79b4d168-b16f-494b-88f6-ba0b8b660d6e" (UID: "79b4d168-b16f-494b-88f6-ba0b8b660d6e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:15:13.616070 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:13.616039    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79b4d168-b16f-494b-88f6-ba0b8b660d6e-home" (OuterVolumeSpecName: "home") pod "79b4d168-b16f-494b-88f6-ba0b8b660d6e" (UID: "79b4d168-b16f-494b-88f6-ba0b8b660d6e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:15:13.616194 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:13.616072    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79b4d168-b16f-494b-88f6-ba0b8b660d6e-tmp-dir\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:15:13.616194 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:13.616100    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79b4d168-b16f-494b-88f6-ba0b8b660d6e-model-cache\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:15:13.617816 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:13.617788    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79b4d168-b16f-494b-88f6-ba0b8b660d6e-dshm" (OuterVolumeSpecName: "dshm") pod "79b4d168-b16f-494b-88f6-ba0b8b660d6e" (UID: "79b4d168-b16f-494b-88f6-ba0b8b660d6e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:15:13.617903 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:13.617875    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79b4d168-b16f-494b-88f6-ba0b8b660d6e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "79b4d168-b16f-494b-88f6-ba0b8b660d6e" (UID: "79b4d168-b16f-494b-88f6-ba0b8b660d6e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:15:13.617903 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:13.617874    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79b4d168-b16f-494b-88f6-ba0b8b660d6e-kube-api-access-fk77r" (OuterVolumeSpecName: "kube-api-access-fk77r") pod "79b4d168-b16f-494b-88f6-ba0b8b660d6e" (UID: "79b4d168-b16f-494b-88f6-ba0b8b660d6e"). InnerVolumeSpecName "kube-api-access-fk77r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:15:13.716829 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:13.716791    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79b4d168-b16f-494b-88f6-ba0b8b660d6e-tls-certs\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:15:13.716829 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:13.716826    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk77r\" (UniqueName: \"kubernetes.io/projected/79b4d168-b16f-494b-88f6-ba0b8b660d6e-kube-api-access-fk77r\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:15:13.717014 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:13.716839    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79b4d168-b16f-494b-88f6-ba0b8b660d6e-dshm\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:15:13.717014 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:13.716850    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79b4d168-b16f-494b-88f6-ba0b8b660d6e-home\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:15:13.822950 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:13.822922    2559 generic.go:356] "Generic (PLEG): container finished" podID="79b4d168-b16f-494b-88f6-ba0b8b660d6e" containerID="c26605948b953aefdb57e0e51e122bb54b0231e6596ac82a38cdf5cf3948abe9" exitCode=0
Aug 01 00:15:13.823075 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:13.822966    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zhn7f"
Aug 01 00:15:13.823075 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:13.823003    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zhn7f" event={"ID":"79b4d168-b16f-494b-88f6-ba0b8b660d6e","Type":"ContainerDied","Data":"c26605948b953aefdb57e0e51e122bb54b0231e6596ac82a38cdf5cf3948abe9"}
Aug 01 00:15:13.823075 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:13.823056    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zhn7f" event={"ID":"79b4d168-b16f-494b-88f6-ba0b8b660d6e","Type":"ContainerDied","Data":"d209cb46390f527b3f6971a3ffa951eb9df017237bbe8328ea94c17590906cf4"}
Aug 01 00:15:13.823189 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:13.823084    2559 scope.go:117] "RemoveContainer" containerID="c26605948b953aefdb57e0e51e122bb54b0231e6596ac82a38cdf5cf3948abe9"
Aug 01 00:15:13.833263 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:13.833224    2559 scope.go:117] "RemoveContainer" containerID="c26605948b953aefdb57e0e51e122bb54b0231e6596ac82a38cdf5cf3948abe9"
Aug 01 00:15:13.833649 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:15:13.833627    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c26605948b953aefdb57e0e51e122bb54b0231e6596ac82a38cdf5cf3948abe9\": container with ID starting with c26605948b953aefdb57e0e51e122bb54b0231e6596ac82a38cdf5cf3948abe9 not found: ID does not exist" containerID="c26605948b953aefdb57e0e51e122bb54b0231e6596ac82a38cdf5cf3948abe9"
Aug 01 00:15:13.833721 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:13.833661    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c26605948b953aefdb57e0e51e122bb54b0231e6596ac82a38cdf5cf3948abe9"} err="failed to get container status \"c26605948b953aefdb57e0e51e122bb54b0231e6596ac82a38cdf5cf3948abe9\": rpc error: code = NotFound desc = could not find container \"c26605948b953aefdb57e0e51e122bb54b0231e6596ac82a38cdf5cf3948abe9\": container with ID starting with c26605948b953aefdb57e0e51e122bb54b0231e6596ac82a38cdf5cf3948abe9 not found: ID does not exist"
Aug 01 00:15:13.839616 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:13.839594    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zhn7f"]
Aug 01 00:15:13.843488 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:13.843470    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zhn7f"]
Aug 01 00:15:15.786056 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:15.786016    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79b4d168-b16f-494b-88f6-ba0b8b660d6e" path="/var/lib/kubelet/pods/79b4d168-b16f-494b-88f6-ba0b8b660d6e/volumes"
Aug 01 00:15:23.110228 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:23.110167    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-svc4b" podUID="4e09c15e-16b6-4824-82bc-f3efd8fc3b9e" containerName="main" containerID="cri-o://30241a841f35cef8f7e2dbda104fb0be227d40bcb03e91b129c92058a6ab1068" gracePeriod=45
Aug 01 00:15:23.352397 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:23.352373    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-svc4b"
Aug 01 00:15:23.491117 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:23.491090    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77mqr\" (UniqueName: \"kubernetes.io/projected/4e09c15e-16b6-4824-82bc-f3efd8fc3b9e-kube-api-access-77mqr\") pod \"4e09c15e-16b6-4824-82bc-f3efd8fc3b9e\" (UID: \"4e09c15e-16b6-4824-82bc-f3efd8fc3b9e\") "
Aug 01 00:15:23.491259 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:23.491137    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e09c15e-16b6-4824-82bc-f3efd8fc3b9e-tmp-dir\") pod \"4e09c15e-16b6-4824-82bc-f3efd8fc3b9e\" (UID: \"4e09c15e-16b6-4824-82bc-f3efd8fc3b9e\") "
Aug 01 00:15:23.491259 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:23.491161    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e09c15e-16b6-4824-82bc-f3efd8fc3b9e-home\") pod \"4e09c15e-16b6-4824-82bc-f3efd8fc3b9e\" (UID: \"4e09c15e-16b6-4824-82bc-f3efd8fc3b9e\") "
Aug 01 00:15:23.491259 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:23.491198    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e09c15e-16b6-4824-82bc-f3efd8fc3b9e-model-cache\") pod \"4e09c15e-16b6-4824-82bc-f3efd8fc3b9e\" (UID: \"4e09c15e-16b6-4824-82bc-f3efd8fc3b9e\") "
Aug 01 00:15:23.491259 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:23.491215    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e09c15e-16b6-4824-82bc-f3efd8fc3b9e-dshm\") pod \"4e09c15e-16b6-4824-82bc-f3efd8fc3b9e\" (UID: \"4e09c15e-16b6-4824-82bc-f3efd8fc3b9e\") "
Aug 01 00:15:23.491259 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:23.491235    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e09c15e-16b6-4824-82bc-f3efd8fc3b9e-tls-certs\") pod \"4e09c15e-16b6-4824-82bc-f3efd8fc3b9e\" (UID: \"4e09c15e-16b6-4824-82bc-f3efd8fc3b9e\") "
Aug 01 00:15:23.491528 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:23.491401    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e09c15e-16b6-4824-82bc-f3efd8fc3b9e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4e09c15e-16b6-4824-82bc-f3efd8fc3b9e" (UID: "4e09c15e-16b6-4824-82bc-f3efd8fc3b9e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:15:23.491528 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:23.491481    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e09c15e-16b6-4824-82bc-f3efd8fc3b9e-model-cache" (OuterVolumeSpecName: "model-cache") pod "4e09c15e-16b6-4824-82bc-f3efd8fc3b9e" (UID: "4e09c15e-16b6-4824-82bc-f3efd8fc3b9e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:15:23.491528 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:23.491493    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e09c15e-16b6-4824-82bc-f3efd8fc3b9e-home" (OuterVolumeSpecName: "home") pod "4e09c15e-16b6-4824-82bc-f3efd8fc3b9e" (UID: "4e09c15e-16b6-4824-82bc-f3efd8fc3b9e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:15:23.493375 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:23.493353    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e09c15e-16b6-4824-82bc-f3efd8fc3b9e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4e09c15e-16b6-4824-82bc-f3efd8fc3b9e" (UID: "4e09c15e-16b6-4824-82bc-f3efd8fc3b9e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:15:23.493462 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:23.493353    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e09c15e-16b6-4824-82bc-f3efd8fc3b9e-kube-api-access-77mqr" (OuterVolumeSpecName: "kube-api-access-77mqr") pod "4e09c15e-16b6-4824-82bc-f3efd8fc3b9e" (UID: "4e09c15e-16b6-4824-82bc-f3efd8fc3b9e"). InnerVolumeSpecName "kube-api-access-77mqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:15:23.493462 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:23.493401    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e09c15e-16b6-4824-82bc-f3efd8fc3b9e-dshm" (OuterVolumeSpecName: "dshm") pod "4e09c15e-16b6-4824-82bc-f3efd8fc3b9e" (UID: "4e09c15e-16b6-4824-82bc-f3efd8fc3b9e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:15:23.592376 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:23.592347    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77mqr\" (UniqueName: \"kubernetes.io/projected/4e09c15e-16b6-4824-82bc-f3efd8fc3b9e-kube-api-access-77mqr\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:15:23.592489 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:23.592392    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e09c15e-16b6-4824-82bc-f3efd8fc3b9e-tmp-dir\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:15:23.592489 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:23.592402    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e09c15e-16b6-4824-82bc-f3efd8fc3b9e-home\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:15:23.592489 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:23.592410    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e09c15e-16b6-4824-82bc-f3efd8fc3b9e-model-cache\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:15:23.592489 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:23.592418    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e09c15e-16b6-4824-82bc-f3efd8fc3b9e-dshm\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:15:23.592489 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:23.592425    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e09c15e-16b6-4824-82bc-f3efd8fc3b9e-tls-certs\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:15:23.878958 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:23.878930    2559 generic.go:356] "Generic (PLEG): container finished" podID="4e09c15e-16b6-4824-82bc-f3efd8fc3b9e" containerID="30241a841f35cef8f7e2dbda104fb0be227d40bcb03e91b129c92058a6ab1068" exitCode=0
Aug 01 00:15:23.879112 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:23.878971    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-svc4b"
Aug 01 00:15:23.879112 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:23.878990    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-svc4b" event={"ID":"4e09c15e-16b6-4824-82bc-f3efd8fc3b9e","Type":"ContainerDied","Data":"30241a841f35cef8f7e2dbda104fb0be227d40bcb03e91b129c92058a6ab1068"}
Aug 01 00:15:23.879112 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:23.879030    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-svc4b" event={"ID":"4e09c15e-16b6-4824-82bc-f3efd8fc3b9e","Type":"ContainerDied","Data":"680ce6c299c60e76f1f78511031e7846ac1af36b4c04cd6375679926c99e158d"}
Aug 01 00:15:23.879112 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:23.879057    2559 scope.go:117] "RemoveContainer" containerID="30241a841f35cef8f7e2dbda104fb0be227d40bcb03e91b129c92058a6ab1068"
Aug 01 00:15:23.887101 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:23.887084    2559 scope.go:117] "RemoveContainer" containerID="30241a841f35cef8f7e2dbda104fb0be227d40bcb03e91b129c92058a6ab1068"
Aug 01 00:15:23.887367 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:15:23.887344    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30241a841f35cef8f7e2dbda104fb0be227d40bcb03e91b129c92058a6ab1068\": container with ID starting with 30241a841f35cef8f7e2dbda104fb0be227d40bcb03e91b129c92058a6ab1068 not found: ID does not exist" containerID="30241a841f35cef8f7e2dbda104fb0be227d40bcb03e91b129c92058a6ab1068"
Aug 01 00:15:23.887474 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:23.887370    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30241a841f35cef8f7e2dbda104fb0be227d40bcb03e91b129c92058a6ab1068"} err="failed to get container status \"30241a841f35cef8f7e2dbda104fb0be227d40bcb03e91b129c92058a6ab1068\": rpc error: code = NotFound desc = could not find container \"30241a841f35cef8f7e2dbda104fb0be227d40bcb03e91b129c92058a6ab1068\": container with ID starting with 30241a841f35cef8f7e2dbda104fb0be227d40bcb03e91b129c92058a6ab1068 not found: ID does not exist"
Aug 01 00:15:23.895448 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:23.895423    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-svc4b"]
Aug 01 00:15:23.901259 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:23.901239    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-svc4b"]
Aug 01 00:15:25.356790 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:25.356757    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-26gzp"]
Aug 01 00:15:25.357165 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:25.357068    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79b4d168-b16f-494b-88f6-ba0b8b660d6e" containerName="main"
Aug 01 00:15:25.357165 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:25.357080    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="79b4d168-b16f-494b-88f6-ba0b8b660d6e" containerName="main"
Aug 01 00:15:25.357165 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:25.357093    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e09c15e-16b6-4824-82bc-f3efd8fc3b9e" containerName="main"
Aug 01 00:15:25.357165 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:25.357098    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e09c15e-16b6-4824-82bc-f3efd8fc3b9e" containerName="main"
Aug 01 00:15:25.357165 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:25.357151    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="79b4d168-b16f-494b-88f6-ba0b8b660d6e" containerName="main"
Aug 01 00:15:25.357165 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:25.357165    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e09c15e-16b6-4824-82bc-f3efd8fc3b9e" containerName="main"
Aug 01 00:15:25.361437 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:25.361420    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-26gzp"
Aug 01 00:15:25.363893 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:25.363865    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"default-dockercfg-mdm6t\""
Aug 01 00:15:25.363893 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:25.363885    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"openshift-service-ca.crt\""
Aug 01 00:15:25.365452 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:25.365431    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"decom-v1-kserve-self-signed-certs\""
Aug 01 00:15:25.365659 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:25.365460    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"kube-root-ca.crt\""
Aug 01 00:15:25.367977 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:25.367960    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-26gzp"]
Aug 01 00:15:25.504799 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:25.504772    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24af24fe-5787-4b11-9113-b6e40ad086a0-model-cache\") pod \"decom-v1-kserve-6cb797fb75-26gzp\" (UID: \"24af24fe-5787-4b11-9113-b6e40ad086a0\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-26gzp"
Aug 01 00:15:25.504996 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:25.504814    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24af24fe-5787-4b11-9113-b6e40ad086a0-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-26gzp\" (UID: \"24af24fe-5787-4b11-9113-b6e40ad086a0\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-26gzp"
Aug 01 00:15:25.504996 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:25.504870    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dz7c\" (UniqueName: \"kubernetes.io/projected/24af24fe-5787-4b11-9113-b6e40ad086a0-kube-api-access-7dz7c\") pod \"decom-v1-kserve-6cb797fb75-26gzp\" (UID: \"24af24fe-5787-4b11-9113-b6e40ad086a0\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-26gzp"
Aug 01 00:15:25.504996 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:25.504972    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24af24fe-5787-4b11-9113-b6e40ad086a0-home\") pod \"decom-v1-kserve-6cb797fb75-26gzp\" (UID: \"24af24fe-5787-4b11-9113-b6e40ad086a0\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-26gzp"
Aug 01 00:15:25.505145 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:25.504998    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24af24fe-5787-4b11-9113-b6e40ad086a0-dshm\") pod \"decom-v1-kserve-6cb797fb75-26gzp\" (UID: \"24af24fe-5787-4b11-9113-b6e40ad086a0\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-26gzp"
Aug 01 00:15:25.505145 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:25.505018    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24af24fe-5787-4b11-9113-b6e40ad086a0-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-26gzp\" (UID: \"24af24fe-5787-4b11-9113-b6e40ad086a0\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-26gzp"
Aug 01 00:15:25.605546 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:25.605512    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24af24fe-5787-4b11-9113-b6e40ad086a0-home\") pod \"decom-v1-kserve-6cb797fb75-26gzp\" (UID: \"24af24fe-5787-4b11-9113-b6e40ad086a0\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-26gzp"
Aug 01 00:15:25.605546 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:25.605544    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24af24fe-5787-4b11-9113-b6e40ad086a0-dshm\") pod \"decom-v1-kserve-6cb797fb75-26gzp\" (UID: \"24af24fe-5787-4b11-9113-b6e40ad086a0\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-26gzp"
Aug 01 00:15:25.605813 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:25.605562    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24af24fe-5787-4b11-9113-b6e40ad086a0-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-26gzp\" (UID: \"24af24fe-5787-4b11-9113-b6e40ad086a0\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-26gzp"
Aug 01 00:15:25.605813 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:25.605583    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24af24fe-5787-4b11-9113-b6e40ad086a0-model-cache\") pod \"decom-v1-kserve-6cb797fb75-26gzp\" (UID: \"24af24fe-5787-4b11-9113-b6e40ad086a0\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-26gzp"
Aug 01 00:15:25.605813 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:25.605624    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24af24fe-5787-4b11-9113-b6e40ad086a0-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-26gzp\" (UID: \"24af24fe-5787-4b11-9113-b6e40ad086a0\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-26gzp"
Aug 01 00:15:25.605813 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:25.605648    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dz7c\" (UniqueName: \"kubernetes.io/projected/24af24fe-5787-4b11-9113-b6e40ad086a0-kube-api-access-7dz7c\") pod \"decom-v1-kserve-6cb797fb75-26gzp\" (UID: \"24af24fe-5787-4b11-9113-b6e40ad086a0\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-26gzp"
Aug 01 00:15:25.606076 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:25.605983    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24af24fe-5787-4b11-9113-b6e40ad086a0-home\") pod \"decom-v1-kserve-6cb797fb75-26gzp\" (UID: \"24af24fe-5787-4b11-9113-b6e40ad086a0\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-26gzp"
Aug 01 00:15:25.606076 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:25.606034    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24af24fe-5787-4b11-9113-b6e40ad086a0-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-26gzp\" (UID: \"24af24fe-5787-4b11-9113-b6e40ad086a0\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-26gzp"
Aug 01 00:15:25.606149 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:25.606081    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24af24fe-5787-4b11-9113-b6e40ad086a0-model-cache\") pod \"decom-v1-kserve-6cb797fb75-26gzp\" (UID: \"24af24fe-5787-4b11-9113-b6e40ad086a0\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-26gzp"
Aug 01 00:15:25.608013 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:25.607955    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24af24fe-5787-4b11-9113-b6e40ad086a0-dshm\") pod \"decom-v1-kserve-6cb797fb75-26gzp\" (UID: \"24af24fe-5787-4b11-9113-b6e40ad086a0\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-26gzp"
Aug 01 00:15:25.608177 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:25.608159    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24af24fe-5787-4b11-9113-b6e40ad086a0-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-26gzp\" (UID: \"24af24fe-5787-4b11-9113-b6e40ad086a0\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-26gzp"
Aug 01 00:15:25.613122 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:25.613100    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dz7c\" (UniqueName: \"kubernetes.io/projected/24af24fe-5787-4b11-9113-b6e40ad086a0-kube-api-access-7dz7c\") pod \"decom-v1-kserve-6cb797fb75-26gzp\" (UID: \"24af24fe-5787-4b11-9113-b6e40ad086a0\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-26gzp"
Aug 01 00:15:25.672409 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:25.672375    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-26gzp"
Aug 01 00:15:25.785795 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:25.785763    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e09c15e-16b6-4824-82bc-f3efd8fc3b9e" path="/var/lib/kubelet/pods/4e09c15e-16b6-4824-82bc-f3efd8fc3b9e/volumes"
Aug 01 00:15:25.802357 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:25.802276    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-26gzp"]
Aug 01 00:15:25.804545 ip-10-0-130-148 kubenswrapper[2559]: W0801 00:15:25.804519    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24af24fe_5787_4b11_9113_b6e40ad086a0.slice/crio-66b69b5e4ac11dc3ed0df4bcdd886dd8c3a2eda4860f646ec65d382029f90b6b WatchSource:0}: Error finding container 66b69b5e4ac11dc3ed0df4bcdd886dd8c3a2eda4860f646ec65d382029f90b6b: Status 404 returned error can't find the container with id 66b69b5e4ac11dc3ed0df4bcdd886dd8c3a2eda4860f646ec65d382029f90b6b
Aug 01 00:15:25.889918 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:25.889852    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-26gzp" event={"ID":"24af24fe-5787-4b11-9113-b6e40ad086a0","Type":"ContainerStarted","Data":"bf26befcdcda96bf545d9d4c8a3c49d1c1e12b89a37c9c0517e7430d4ea1b6fe"}
Aug 01 00:15:25.889918 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:25.889903    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-26gzp" event={"ID":"24af24fe-5787-4b11-9113-b6e40ad086a0","Type":"ContainerStarted","Data":"66b69b5e4ac11dc3ed0df4bcdd886dd8c3a2eda4860f646ec65d382029f90b6b"}
Aug 01 00:15:25.911102 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:25.911053    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-26gzp" podStartSLOduration=0.911038367 podStartE2EDuration="911.038367ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:15:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:15:25.909139324 +0000 UTC m=+1788.628939184" watchObservedRunningTime="2026-08-01 00:15:25.911038367 +0000 UTC m=+1788.630838212"
Aug 01 00:15:26.672653 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:26.672625    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-26gzp"
Aug 01 00:15:35.673176 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:35.673138    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-26gzp"
Aug 01 00:15:35.685192 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:35.685167    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-26gzp"
Aug 01 00:15:35.942382 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:35.942292    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-26gzp"
Aug 01 00:15:37.831098 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:37.831072    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-spblg_c2e50946-aea4-4886-82ec-0bd5a31196cf/console-operator/1.log"
Aug 01 00:15:37.835428 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:37.835408    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-spblg_c2e50946-aea4-4886-82ec-0bd5a31196cf/console-operator/1.log"
Aug 01 00:15:59.259696 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:59.259660    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4n86"]
Aug 01 00:15:59.261848 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:59.261832    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4n86"
Aug 01 00:15:59.265919 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:59.265898    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Aug 01 00:15:59.266079 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:59.265898    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-t5l4c\""
Aug 01 00:15:59.266079 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:59.266045    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Aug 01 00:15:59.266217 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:59.266154    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Aug 01 00:15:59.273191 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:59.273168    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4n86"]
Aug 01 00:15:59.278519 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:59.278496    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/442245a5-ef18-4c25-bdbe-ae2f52a33cb5-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4n86\" (UID: \"442245a5-ef18-4c25-bdbe-ae2f52a33cb5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4n86"
Aug 01 00:15:59.278636 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:59.278525    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/442245a5-ef18-4c25-bdbe-ae2f52a33cb5-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4n86\" (UID: \"442245a5-ef18-4c25-bdbe-ae2f52a33cb5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4n86"
Aug 01 00:15:59.278636 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:59.278560    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csr9z\" (UniqueName: \"kubernetes.io/projected/442245a5-ef18-4c25-bdbe-ae2f52a33cb5-kube-api-access-csr9z\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4n86\" (UID: \"442245a5-ef18-4c25-bdbe-ae2f52a33cb5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4n86"
Aug 01 00:15:59.278789 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:59.278643    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/442245a5-ef18-4c25-bdbe-ae2f52a33cb5-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4n86\" (UID: \"442245a5-ef18-4c25-bdbe-ae2f52a33cb5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4n86"
Aug 01 00:15:59.278789 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:59.278683    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/442245a5-ef18-4c25-bdbe-ae2f52a33cb5-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4n86\" (UID: \"442245a5-ef18-4c25-bdbe-ae2f52a33cb5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4n86"
Aug 01 00:15:59.278789 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:59.278736    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/442245a5-ef18-4c25-bdbe-ae2f52a33cb5-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4n86\" (UID: \"442245a5-ef18-4c25-bdbe-ae2f52a33cb5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4n86"
Aug 01 00:15:59.379989 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:59.379955    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csr9z\" (UniqueName: \"kubernetes.io/projected/442245a5-ef18-4c25-bdbe-ae2f52a33cb5-kube-api-access-csr9z\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4n86\" (UID: \"442245a5-ef18-4c25-bdbe-ae2f52a33cb5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4n86"
Aug 01 00:15:59.380153 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:59.379996    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/442245a5-ef18-4c25-bdbe-ae2f52a33cb5-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4n86\" (UID: \"442245a5-ef18-4c25-bdbe-ae2f52a33cb5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4n86"
Aug 01 00:15:59.380153 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:59.380016    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/442245a5-ef18-4c25-bdbe-ae2f52a33cb5-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4n86\" (UID: \"442245a5-ef18-4c25-bdbe-ae2f52a33cb5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4n86"
Aug 01 00:15:59.380153 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:59.380038    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/442245a5-ef18-4c25-bdbe-ae2f52a33cb5-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4n86\" (UID: \"442245a5-ef18-4c25-bdbe-ae2f52a33cb5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4n86"
Aug 01 00:15:59.380153 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:59.380075    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/442245a5-ef18-4c25-bdbe-ae2f52a33cb5-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4n86\" (UID: \"442245a5-ef18-4c25-bdbe-ae2f52a33cb5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4n86"
Aug 01 00:15:59.380153 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:59.380093    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/442245a5-ef18-4c25-bdbe-ae2f52a33cb5-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4n86\" (UID: \"442245a5-ef18-4c25-bdbe-ae2f52a33cb5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4n86"
Aug 01 00:15:59.380459 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:59.380430    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/442245a5-ef18-4c25-bdbe-ae2f52a33cb5-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4n86\" (UID: \"442245a5-ef18-4c25-bdbe-ae2f52a33cb5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4n86"
Aug 01 00:15:59.380521 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:59.380487    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/442245a5-ef18-4c25-bdbe-ae2f52a33cb5-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4n86\" (UID: \"442245a5-ef18-4c25-bdbe-ae2f52a33cb5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4n86"
Aug 01 00:15:59.380521 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:59.380507    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/442245a5-ef18-4c25-bdbe-ae2f52a33cb5-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4n86\" (UID: \"442245a5-ef18-4c25-bdbe-ae2f52a33cb5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4n86"
Aug 01 00:15:59.382362 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:59.382338    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/442245a5-ef18-4c25-bdbe-ae2f52a33cb5-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4n86\" (UID: \"442245a5-ef18-4c25-bdbe-ae2f52a33cb5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4n86"
Aug 01 00:15:59.382613 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:59.382595    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/442245a5-ef18-4c25-bdbe-ae2f52a33cb5-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4n86\" (UID: \"442245a5-ef18-4c25-bdbe-ae2f52a33cb5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4n86"
Aug 01 00:15:59.387680 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:59.387657    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csr9z\" (UniqueName: \"kubernetes.io/projected/442245a5-ef18-4c25-bdbe-ae2f52a33cb5-kube-api-access-csr9z\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4n86\" (UID: \"442245a5-ef18-4c25-bdbe-ae2f52a33cb5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4n86"
Aug 01 00:15:59.572834 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:59.572728    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4n86"
Aug 01 00:15:59.701929 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:15:59.701905    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4n86"]
Aug 01 00:15:59.704051 ip-10-0-130-148 kubenswrapper[2559]: W0801 00:15:59.704022    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod442245a5_ef18_4c25_bdbe_ae2f52a33cb5.slice/crio-77d27208b1b36bc3f434ad569d40f1366c2631260ea75d1afa65851a73800c90 WatchSource:0}: Error finding container 77d27208b1b36bc3f434ad569d40f1366c2631260ea75d1afa65851a73800c90: Status 404 returned error can't find the container with id 77d27208b1b36bc3f434ad569d40f1366c2631260ea75d1afa65851a73800c90
Aug 01 00:16:00.038023 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:00.037987    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4n86" event={"ID":"442245a5-ef18-4c25-bdbe-ae2f52a33cb5","Type":"ContainerStarted","Data":"27879120114ae62e4618362fb6085a1cb6db6aaaa80d47c91b2b2780b757fb30"}
Aug 01 00:16:00.038234 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:00.038042    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4n86" event={"ID":"442245a5-ef18-4c25-bdbe-ae2f52a33cb5","Type":"ContainerStarted","Data":"77d27208b1b36bc3f434ad569d40f1366c2631260ea75d1afa65851a73800c90"}
Aug 01 00:16:00.054370 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:00.054320    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4n86" podStartSLOduration=1.054305602 podStartE2EDuration="1.054305602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:15:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:16:00.053535324 +0000 UTC m=+1822.773335169" watchObservedRunningTime="2026-08-01 00:16:00.054305602 +0000 UTC m=+1822.774105449"
Aug 01 00:16:00.573511 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:00.573468    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4n86"
Aug 01 00:16:09.573583 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:09.573542    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4n86"
Aug 01 00:16:09.586043 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:09.586017    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4n86"
Aug 01 00:16:10.094071 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:10.094040    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4n86"
Aug 01 00:16:40.533715 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:40.533680    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-26gzp"]
Aug 01 00:16:55.552919 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:55.552872    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-26gzp" podUID="24af24fe-5787-4b11-9113-b6e40ad086a0" containerName="main" containerID="cri-o://bf26befcdcda96bf545d9d4c8a3c49d1c1e12b89a37c9c0517e7430d4ea1b6fe" gracePeriod=45
Aug 01 00:16:55.810191 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:55.810130    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-26gzp"
Aug 01 00:16:55.860306 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:55.860276    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24af24fe-5787-4b11-9113-b6e40ad086a0-home\") pod \"24af24fe-5787-4b11-9113-b6e40ad086a0\" (UID: \"24af24fe-5787-4b11-9113-b6e40ad086a0\") "
Aug 01 00:16:55.860306 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:55.860310    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24af24fe-5787-4b11-9113-b6e40ad086a0-dshm\") pod \"24af24fe-5787-4b11-9113-b6e40ad086a0\" (UID: \"24af24fe-5787-4b11-9113-b6e40ad086a0\") "
Aug 01 00:16:55.860537 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:55.860349    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7dz7c\" (UniqueName: \"kubernetes.io/projected/24af24fe-5787-4b11-9113-b6e40ad086a0-kube-api-access-7dz7c\") pod \"24af24fe-5787-4b11-9113-b6e40ad086a0\" (UID: \"24af24fe-5787-4b11-9113-b6e40ad086a0\") "
Aug 01 00:16:55.860537 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:55.860384    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24af24fe-5787-4b11-9113-b6e40ad086a0-tmp-dir\") pod \"24af24fe-5787-4b11-9113-b6e40ad086a0\" (UID: \"24af24fe-5787-4b11-9113-b6e40ad086a0\") "
Aug 01 00:16:55.860537 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:55.860404    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24af24fe-5787-4b11-9113-b6e40ad086a0-model-cache\") pod \"24af24fe-5787-4b11-9113-b6e40ad086a0\" (UID: \"24af24fe-5787-4b11-9113-b6e40ad086a0\") "
Aug 01 00:16:55.860537 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:55.860457    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24af24fe-5787-4b11-9113-b6e40ad086a0-tls-certs\") pod \"24af24fe-5787-4b11-9113-b6e40ad086a0\" (UID: \"24af24fe-5787-4b11-9113-b6e40ad086a0\") "
Aug 01 00:16:55.860763 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:55.860588    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24af24fe-5787-4b11-9113-b6e40ad086a0-home" (OuterVolumeSpecName: "home") pod "24af24fe-5787-4b11-9113-b6e40ad086a0" (UID: "24af24fe-5787-4b11-9113-b6e40ad086a0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:16:55.860763 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:55.860646    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24af24fe-5787-4b11-9113-b6e40ad086a0-model-cache" (OuterVolumeSpecName: "model-cache") pod "24af24fe-5787-4b11-9113-b6e40ad086a0" (UID: "24af24fe-5787-4b11-9113-b6e40ad086a0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:16:55.860763 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:55.860661    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24af24fe-5787-4b11-9113-b6e40ad086a0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "24af24fe-5787-4b11-9113-b6e40ad086a0" (UID: "24af24fe-5787-4b11-9113-b6e40ad086a0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:16:55.860930 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:55.860731    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24af24fe-5787-4b11-9113-b6e40ad086a0-tmp-dir\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:16:55.860930 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:55.860781    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24af24fe-5787-4b11-9113-b6e40ad086a0-model-cache\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:16:55.860930 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:55.860795    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24af24fe-5787-4b11-9113-b6e40ad086a0-home\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:16:55.862626 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:55.862598    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24af24fe-5787-4b11-9113-b6e40ad086a0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "24af24fe-5787-4b11-9113-b6e40ad086a0" (UID: "24af24fe-5787-4b11-9113-b6e40ad086a0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:16:55.862732 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:55.862640    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24af24fe-5787-4b11-9113-b6e40ad086a0-kube-api-access-7dz7c" (OuterVolumeSpecName: "kube-api-access-7dz7c") pod "24af24fe-5787-4b11-9113-b6e40ad086a0" (UID: "24af24fe-5787-4b11-9113-b6e40ad086a0"). InnerVolumeSpecName "kube-api-access-7dz7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:16:55.862732 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:55.862670    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24af24fe-5787-4b11-9113-b6e40ad086a0-dshm" (OuterVolumeSpecName: "dshm") pod "24af24fe-5787-4b11-9113-b6e40ad086a0" (UID: "24af24fe-5787-4b11-9113-b6e40ad086a0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:16:55.961939 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:55.961906    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24af24fe-5787-4b11-9113-b6e40ad086a0-dshm\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:16:55.961939 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:55.961934    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dz7c\" (UniqueName: \"kubernetes.io/projected/24af24fe-5787-4b11-9113-b6e40ad086a0-kube-api-access-7dz7c\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:16:55.961939 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:55.961944    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24af24fe-5787-4b11-9113-b6e40ad086a0-tls-certs\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:16:56.288350 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:56.288322    2559 generic.go:356] "Generic (PLEG): container finished" podID="24af24fe-5787-4b11-9113-b6e40ad086a0" containerID="bf26befcdcda96bf545d9d4c8a3c49d1c1e12b89a37c9c0517e7430d4ea1b6fe" exitCode=0
Aug 01 00:16:56.288527 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:56.288363    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-26gzp" event={"ID":"24af24fe-5787-4b11-9113-b6e40ad086a0","Type":"ContainerDied","Data":"bf26befcdcda96bf545d9d4c8a3c49d1c1e12b89a37c9c0517e7430d4ea1b6fe"}
Aug 01 00:16:56.288527 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:56.288369    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-26gzp"
Aug 01 00:16:56.288527 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:56.288405    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-26gzp" event={"ID":"24af24fe-5787-4b11-9113-b6e40ad086a0","Type":"ContainerDied","Data":"66b69b5e4ac11dc3ed0df4bcdd886dd8c3a2eda4860f646ec65d382029f90b6b"}
Aug 01 00:16:56.288527 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:56.288429    2559 scope.go:117] "RemoveContainer" containerID="bf26befcdcda96bf545d9d4c8a3c49d1c1e12b89a37c9c0517e7430d4ea1b6fe"
Aug 01 00:16:56.297694 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:56.297683    2559 scope.go:117] "RemoveContainer" containerID="bf26befcdcda96bf545d9d4c8a3c49d1c1e12b89a37c9c0517e7430d4ea1b6fe"
Aug 01 00:16:56.297983 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:16:56.297965    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf26befcdcda96bf545d9d4c8a3c49d1c1e12b89a37c9c0517e7430d4ea1b6fe\": container with ID starting with bf26befcdcda96bf545d9d4c8a3c49d1c1e12b89a37c9c0517e7430d4ea1b6fe not found: ID does not exist" containerID="bf26befcdcda96bf545d9d4c8a3c49d1c1e12b89a37c9c0517e7430d4ea1b6fe"
Aug 01 00:16:56.298035 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:56.297991    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf26befcdcda96bf545d9d4c8a3c49d1c1e12b89a37c9c0517e7430d4ea1b6fe"} err="failed to get container status \"bf26befcdcda96bf545d9d4c8a3c49d1c1e12b89a37c9c0517e7430d4ea1b6fe\": rpc error: code = NotFound desc = could not find container \"bf26befcdcda96bf545d9d4c8a3c49d1c1e12b89a37c9c0517e7430d4ea1b6fe\": container with ID starting with bf26befcdcda96bf545d9d4c8a3c49d1c1e12b89a37c9c0517e7430d4ea1b6fe not found: ID does not exist"
Aug 01 00:16:56.309940 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:56.309917    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-26gzp"]
Aug 01 00:16:56.315348 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:56.315321    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-26gzp"]
Aug 01 00:16:57.241632 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:57.241599    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vnzqd"]
Aug 01 00:16:57.242237 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:57.241942    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24af24fe-5787-4b11-9113-b6e40ad086a0" containerName="main"
Aug 01 00:16:57.242237 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:57.241954    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="24af24fe-5787-4b11-9113-b6e40ad086a0" containerName="main"
Aug 01 00:16:57.242237 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:57.242004    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="24af24fe-5787-4b11-9113-b6e40ad086a0" containerName="main"
Aug 01 00:16:57.246231 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:57.246212    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vnzqd"
Aug 01 00:16:57.249960 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:57.249935    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"default-dockercfg-82rwb\""
Aug 01 00:16:57.250105 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:57.249989    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"openshift-service-ca.crt\""
Aug 01 00:16:57.250105 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:57.250037    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"leave-v1-kserve-self-signed-certs\""
Aug 01 00:16:57.250105 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:57.249938    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"kube-root-ca.crt\""
Aug 01 00:16:57.255004 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:57.254979    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vnzqd"]
Aug 01 00:16:57.272449 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:57.272424    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcec521b-1b0a-48a4-84f5-4ecdb6c7b797-dshm\") pod \"leave-v1-kserve-65684d4f5-vnzqd\" (UID: \"fcec521b-1b0a-48a4-84f5-4ecdb6c7b797\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vnzqd"
Aug 01 00:16:57.272563 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:57.272454    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcec521b-1b0a-48a4-84f5-4ecdb6c7b797-home\") pod \"leave-v1-kserve-65684d4f5-vnzqd\" (UID: \"fcec521b-1b0a-48a4-84f5-4ecdb6c7b797\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vnzqd"
Aug 01 00:16:57.272563 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:57.272512    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nb5w\" (UniqueName: \"kubernetes.io/projected/fcec521b-1b0a-48a4-84f5-4ecdb6c7b797-kube-api-access-4nb5w\") pod \"leave-v1-kserve-65684d4f5-vnzqd\" (UID: \"fcec521b-1b0a-48a4-84f5-4ecdb6c7b797\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vnzqd"
Aug 01 00:16:57.272563 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:57.272546    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcec521b-1b0a-48a4-84f5-4ecdb6c7b797-model-cache\") pod \"leave-v1-kserve-65684d4f5-vnzqd\" (UID: \"fcec521b-1b0a-48a4-84f5-4ecdb6c7b797\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vnzqd"
Aug 01 00:16:57.272678 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:57.272581    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcec521b-1b0a-48a4-84f5-4ecdb6c7b797-tls-certs\") pod \"leave-v1-kserve-65684d4f5-vnzqd\" (UID: \"fcec521b-1b0a-48a4-84f5-4ecdb6c7b797\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vnzqd"
Aug 01 00:16:57.272678 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:57.272606    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcec521b-1b0a-48a4-84f5-4ecdb6c7b797-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-vnzqd\" (UID: \"fcec521b-1b0a-48a4-84f5-4ecdb6c7b797\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vnzqd"
Aug 01 00:16:57.373703 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:57.373670    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcec521b-1b0a-48a4-84f5-4ecdb6c7b797-dshm\") pod \"leave-v1-kserve-65684d4f5-vnzqd\" (UID: \"fcec521b-1b0a-48a4-84f5-4ecdb6c7b797\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vnzqd"
Aug 01 00:16:57.373703 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:57.373705    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcec521b-1b0a-48a4-84f5-4ecdb6c7b797-home\") pod \"leave-v1-kserve-65684d4f5-vnzqd\" (UID: \"fcec521b-1b0a-48a4-84f5-4ecdb6c7b797\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vnzqd"
Aug 01 00:16:57.373958 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:57.373791    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nb5w\" (UniqueName: \"kubernetes.io/projected/fcec521b-1b0a-48a4-84f5-4ecdb6c7b797-kube-api-access-4nb5w\") pod \"leave-v1-kserve-65684d4f5-vnzqd\" (UID: \"fcec521b-1b0a-48a4-84f5-4ecdb6c7b797\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vnzqd"
Aug 01 00:16:57.373958 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:57.373833    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcec521b-1b0a-48a4-84f5-4ecdb6c7b797-model-cache\") pod \"leave-v1-kserve-65684d4f5-vnzqd\" (UID: \"fcec521b-1b0a-48a4-84f5-4ecdb6c7b797\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vnzqd"
Aug 01 00:16:57.373958 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:57.373873    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcec521b-1b0a-48a4-84f5-4ecdb6c7b797-tls-certs\") pod \"leave-v1-kserve-65684d4f5-vnzqd\" (UID: \"fcec521b-1b0a-48a4-84f5-4ecdb6c7b797\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vnzqd"
Aug 01 00:16:57.373958 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:57.373897    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcec521b-1b0a-48a4-84f5-4ecdb6c7b797-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-vnzqd\" (UID: \"fcec521b-1b0a-48a4-84f5-4ecdb6c7b797\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vnzqd"
Aug 01 00:16:57.374196 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:57.374058    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcec521b-1b0a-48a4-84f5-4ecdb6c7b797-home\") pod \"leave-v1-kserve-65684d4f5-vnzqd\" (UID: \"fcec521b-1b0a-48a4-84f5-4ecdb6c7b797\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vnzqd"
Aug 01 00:16:57.374196 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:57.374180    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcec521b-1b0a-48a4-84f5-4ecdb6c7b797-model-cache\") pod \"leave-v1-kserve-65684d4f5-vnzqd\" (UID: \"fcec521b-1b0a-48a4-84f5-4ecdb6c7b797\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vnzqd"
Aug 01 00:16:57.374297 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:57.374221    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcec521b-1b0a-48a4-84f5-4ecdb6c7b797-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-vnzqd\" (UID: \"fcec521b-1b0a-48a4-84f5-4ecdb6c7b797\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vnzqd"
Aug 01 00:16:57.376063 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:57.376043    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcec521b-1b0a-48a4-84f5-4ecdb6c7b797-dshm\") pod \"leave-v1-kserve-65684d4f5-vnzqd\" (UID: \"fcec521b-1b0a-48a4-84f5-4ecdb6c7b797\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vnzqd"
Aug 01 00:16:57.376330 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:57.376314    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcec521b-1b0a-48a4-84f5-4ecdb6c7b797-tls-certs\") pod \"leave-v1-kserve-65684d4f5-vnzqd\" (UID: \"fcec521b-1b0a-48a4-84f5-4ecdb6c7b797\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vnzqd"
Aug 01 00:16:57.381051 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:57.381032    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nb5w\" (UniqueName: \"kubernetes.io/projected/fcec521b-1b0a-48a4-84f5-4ecdb6c7b797-kube-api-access-4nb5w\") pod \"leave-v1-kserve-65684d4f5-vnzqd\" (UID: \"fcec521b-1b0a-48a4-84f5-4ecdb6c7b797\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vnzqd"
Aug 01 00:16:57.557847 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:57.557759    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vnzqd"
Aug 01 00:16:57.689224 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:57.689200    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vnzqd"]
Aug 01 00:16:57.691301 ip-10-0-130-148 kubenswrapper[2559]: W0801 00:16:57.691270    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcec521b_1b0a_48a4_84f5_4ecdb6c7b797.slice/crio-088702c92baeab1f26be1f84f8ea5385b0c24bfeec3e9b1826bf9b4449e156cc WatchSource:0}: Error finding container 088702c92baeab1f26be1f84f8ea5385b0c24bfeec3e9b1826bf9b4449e156cc: Status 404 returned error can't find the container with id 088702c92baeab1f26be1f84f8ea5385b0c24bfeec3e9b1826bf9b4449e156cc
Aug 01 00:16:57.786164 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:57.786132    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24af24fe-5787-4b11-9113-b6e40ad086a0" path="/var/lib/kubelet/pods/24af24fe-5787-4b11-9113-b6e40ad086a0/volumes"
Aug 01 00:16:58.298928 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:58.298894    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vnzqd" event={"ID":"fcec521b-1b0a-48a4-84f5-4ecdb6c7b797","Type":"ContainerStarted","Data":"5764f8bf9f9db55900c51dc2250024a4bad499bc96a177e5be4a2afbc325d5ce"}
Aug 01 00:16:58.299323 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:58.298970    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vnzqd" event={"ID":"fcec521b-1b0a-48a4-84f5-4ecdb6c7b797","Type":"ContainerStarted","Data":"088702c92baeab1f26be1f84f8ea5385b0c24bfeec3e9b1826bf9b4449e156cc"}
Aug 01 00:16:58.321858 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:58.321790    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vnzqd" podStartSLOduration=1.321776604 podStartE2EDuration="1.321776604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:16:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:16:58.320675784 +0000 UTC m=+1881.040475629" watchObservedRunningTime="2026-08-01 00:16:58.321776604 +0000 UTC m=+1881.041576446"
Aug 01 00:16:58.558886 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:16:58.558802    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vnzqd"
Aug 01 00:17:07.357795 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:07.357707    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4ctbx"]
Aug 01 00:17:07.361466 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:07.361445    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4ctbx"
Aug 01 00:17:07.364659 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:07.364635    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"leave-v2-kserve-self-signed-certs\""
Aug 01 00:17:07.372364 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:07.372343    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4ctbx"]
Aug 01 00:17:07.462658 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:07.462624    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d6c4767-863e-4435-96e6-a421405dbac7-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-4ctbx\" (UID: \"5d6c4767-863e-4435-96e6-a421405dbac7\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4ctbx"
Aug 01 00:17:07.462847 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:07.462665    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d6c4767-863e-4435-96e6-a421405dbac7-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-4ctbx\" (UID: \"5d6c4767-863e-4435-96e6-a421405dbac7\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4ctbx"
Aug 01 00:17:07.462847 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:07.462731    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d6c4767-863e-4435-96e6-a421405dbac7-model-cache\") pod \"leave-v2-kserve-556b4dddf9-4ctbx\" (UID: \"5d6c4767-863e-4435-96e6-a421405dbac7\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4ctbx"
Aug 01 00:17:07.462847 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:07.462799    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d6c4767-863e-4435-96e6-a421405dbac7-dshm\") pod \"leave-v2-kserve-556b4dddf9-4ctbx\" (UID: \"5d6c4767-863e-4435-96e6-a421405dbac7\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4ctbx"
Aug 01 00:17:07.462847 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:07.462826    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d6c4767-863e-4435-96e6-a421405dbac7-home\") pod \"leave-v2-kserve-556b4dddf9-4ctbx\" (UID: \"5d6c4767-863e-4435-96e6-a421405dbac7\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4ctbx"
Aug 01 00:17:07.463044 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:07.462862    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bkjr\" (UniqueName: \"kubernetes.io/projected/5d6c4767-863e-4435-96e6-a421405dbac7-kube-api-access-2bkjr\") pod \"leave-v2-kserve-556b4dddf9-4ctbx\" (UID: \"5d6c4767-863e-4435-96e6-a421405dbac7\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4ctbx"
Aug 01 00:17:07.558845 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:07.558812    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vnzqd"
Aug 01 00:17:07.563690 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:07.563667    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d6c4767-863e-4435-96e6-a421405dbac7-model-cache\") pod \"leave-v2-kserve-556b4dddf9-4ctbx\" (UID: \"5d6c4767-863e-4435-96e6-a421405dbac7\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4ctbx"
Aug 01 00:17:07.563825 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:07.563702    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d6c4767-863e-4435-96e6-a421405dbac7-dshm\") pod \"leave-v2-kserve-556b4dddf9-4ctbx\" (UID: \"5d6c4767-863e-4435-96e6-a421405dbac7\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4ctbx"
Aug 01 00:17:07.563825 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:07.563728    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d6c4767-863e-4435-96e6-a421405dbac7-home\") pod \"leave-v2-kserve-556b4dddf9-4ctbx\" (UID: \"5d6c4767-863e-4435-96e6-a421405dbac7\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4ctbx"
Aug 01 00:17:07.563825 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:07.563778    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bkjr\" (UniqueName: \"kubernetes.io/projected/5d6c4767-863e-4435-96e6-a421405dbac7-kube-api-access-2bkjr\") pod \"leave-v2-kserve-556b4dddf9-4ctbx\" (UID: \"5d6c4767-863e-4435-96e6-a421405dbac7\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4ctbx"
Aug 01 00:17:07.563984 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:07.563824    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d6c4767-863e-4435-96e6-a421405dbac7-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-4ctbx\" (UID: \"5d6c4767-863e-4435-96e6-a421405dbac7\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4ctbx"
Aug 01 00:17:07.563984 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:07.563866    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d6c4767-863e-4435-96e6-a421405dbac7-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-4ctbx\" (UID: \"5d6c4767-863e-4435-96e6-a421405dbac7\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4ctbx"
Aug 01 00:17:07.564398 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:07.564136    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d6c4767-863e-4435-96e6-a421405dbac7-model-cache\") pod \"leave-v2-kserve-556b4dddf9-4ctbx\" (UID: \"5d6c4767-863e-4435-96e6-a421405dbac7\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4ctbx"
Aug 01 00:17:07.564398 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:07.564175    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d6c4767-863e-4435-96e6-a421405dbac7-home\") pod \"leave-v2-kserve-556b4dddf9-4ctbx\" (UID: \"5d6c4767-863e-4435-96e6-a421405dbac7\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4ctbx"
Aug 01 00:17:07.564398 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:07.564311    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d6c4767-863e-4435-96e6-a421405dbac7-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-4ctbx\" (UID: \"5d6c4767-863e-4435-96e6-a421405dbac7\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4ctbx"
Aug 01 00:17:07.566020 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:07.565998    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d6c4767-863e-4435-96e6-a421405dbac7-dshm\") pod \"leave-v2-kserve-556b4dddf9-4ctbx\" (UID: \"5d6c4767-863e-4435-96e6-a421405dbac7\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4ctbx"
Aug 01 00:17:07.566447 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:07.566428    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d6c4767-863e-4435-96e6-a421405dbac7-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-4ctbx\" (UID: \"5d6c4767-863e-4435-96e6-a421405dbac7\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4ctbx"
Aug 01 00:17:07.571623 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:07.571592    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bkjr\" (UniqueName: \"kubernetes.io/projected/5d6c4767-863e-4435-96e6-a421405dbac7-kube-api-access-2bkjr\") pod \"leave-v2-kserve-556b4dddf9-4ctbx\" (UID: \"5d6c4767-863e-4435-96e6-a421405dbac7\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4ctbx"
Aug 01 00:17:07.571938 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:07.571923    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vnzqd"
Aug 01 00:17:07.674641 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:07.674558    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4ctbx"
Aug 01 00:17:07.805552 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:07.805530    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4ctbx"]
Aug 01 00:17:07.808010 ip-10-0-130-148 kubenswrapper[2559]: W0801 00:17:07.807982    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d6c4767_863e_4435_96e6_a421405dbac7.slice/crio-14318f78af26cfd8e4ee9dc207bdc9782d1923778e8e1976bb3f8229f8e7aff8 WatchSource:0}: Error finding container 14318f78af26cfd8e4ee9dc207bdc9782d1923778e8e1976bb3f8229f8e7aff8: Status 404 returned error can't find the container with id 14318f78af26cfd8e4ee9dc207bdc9782d1923778e8e1976bb3f8229f8e7aff8
Aug 01 00:17:08.342842 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:08.342807    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4ctbx" event={"ID":"5d6c4767-863e-4435-96e6-a421405dbac7","Type":"ContainerStarted","Data":"4ded0907976676e052bae6d8dd11ba23198759c27ac3453ef7763e69f4dbd2c9"}
Aug 01 00:17:08.343016 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:08.342864    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4ctbx" event={"ID":"5d6c4767-863e-4435-96e6-a421405dbac7","Type":"ContainerStarted","Data":"14318f78af26cfd8e4ee9dc207bdc9782d1923778e8e1976bb3f8229f8e7aff8"}
Aug 01 00:17:08.354338 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:08.354312    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vnzqd"
Aug 01 00:17:08.359078 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:08.359038    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4ctbx" podStartSLOduration=1.359024325 podStartE2EDuration="1.359024325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:17:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:17:08.358295078 +0000 UTC m=+1891.078094947" watchObservedRunningTime="2026-08-01 00:17:08.359024325 +0000 UTC m=+1891.078824170"
Aug 01 00:17:08.675609 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:08.675527    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4ctbx"
Aug 01 00:17:17.676052 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:17.676015    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4ctbx"
Aug 01 00:17:17.688157 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:17.688135    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4ctbx"
Aug 01 00:17:18.397653 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:18.397626    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4ctbx"
Aug 01 00:17:42.739473 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:42.739441    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4ctbx"]
Aug 01 00:17:45.500630 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:45.500598    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vnzqd"]
Aug 01 00:17:57.758525 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:57.758482    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4ctbx" podUID="5d6c4767-863e-4435-96e6-a421405dbac7" containerName="main" containerID="cri-o://4ded0907976676e052bae6d8dd11ba23198759c27ac3453ef7763e69f4dbd2c9" gracePeriod=45
Aug 01 00:17:58.000350 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:58.000324    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4ctbx"
Aug 01 00:17:58.101353 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:58.101262    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d6c4767-863e-4435-96e6-a421405dbac7-model-cache\") pod \"5d6c4767-863e-4435-96e6-a421405dbac7\" (UID: \"5d6c4767-863e-4435-96e6-a421405dbac7\") "
Aug 01 00:17:58.101353 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:58.101302    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d6c4767-863e-4435-96e6-a421405dbac7-tls-certs\") pod \"5d6c4767-863e-4435-96e6-a421405dbac7\" (UID: \"5d6c4767-863e-4435-96e6-a421405dbac7\") "
Aug 01 00:17:58.101353 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:58.101342    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d6c4767-863e-4435-96e6-a421405dbac7-tmp-dir\") pod \"5d6c4767-863e-4435-96e6-a421405dbac7\" (UID: \"5d6c4767-863e-4435-96e6-a421405dbac7\") "
Aug 01 00:17:58.101647 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:58.101365    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d6c4767-863e-4435-96e6-a421405dbac7-dshm\") pod \"5d6c4767-863e-4435-96e6-a421405dbac7\" (UID: \"5d6c4767-863e-4435-96e6-a421405dbac7\") "
Aug 01 00:17:58.101647 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:58.101427    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d6c4767-863e-4435-96e6-a421405dbac7-home\") pod \"5d6c4767-863e-4435-96e6-a421405dbac7\" (UID: \"5d6c4767-863e-4435-96e6-a421405dbac7\") "
Aug 01 00:17:58.101647 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:58.101451    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2bkjr\" (UniqueName: \"kubernetes.io/projected/5d6c4767-863e-4435-96e6-a421405dbac7-kube-api-access-2bkjr\") pod \"5d6c4767-863e-4435-96e6-a421405dbac7\" (UID: \"5d6c4767-863e-4435-96e6-a421405dbac7\") "
Aug 01 00:17:58.101647 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:58.101603    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d6c4767-863e-4435-96e6-a421405dbac7-model-cache" (OuterVolumeSpecName: "model-cache") pod "5d6c4767-863e-4435-96e6-a421405dbac7" (UID: "5d6c4767-863e-4435-96e6-a421405dbac7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:17:58.101835 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:58.101687    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d6c4767-863e-4435-96e6-a421405dbac7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5d6c4767-863e-4435-96e6-a421405dbac7" (UID: "5d6c4767-863e-4435-96e6-a421405dbac7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:17:58.101835 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:58.101781    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d6c4767-863e-4435-96e6-a421405dbac7-home" (OuterVolumeSpecName: "home") pod "5d6c4767-863e-4435-96e6-a421405dbac7" (UID: "5d6c4767-863e-4435-96e6-a421405dbac7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:17:58.103711 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:58.103683    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d6c4767-863e-4435-96e6-a421405dbac7-kube-api-access-2bkjr" (OuterVolumeSpecName: "kube-api-access-2bkjr") pod "5d6c4767-863e-4435-96e6-a421405dbac7" (UID: "5d6c4767-863e-4435-96e6-a421405dbac7"). InnerVolumeSpecName "kube-api-access-2bkjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:17:58.103839 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:58.103702    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d6c4767-863e-4435-96e6-a421405dbac7-dshm" (OuterVolumeSpecName: "dshm") pod "5d6c4767-863e-4435-96e6-a421405dbac7" (UID: "5d6c4767-863e-4435-96e6-a421405dbac7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:17:58.103839 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:58.103730    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d6c4767-863e-4435-96e6-a421405dbac7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5d6c4767-863e-4435-96e6-a421405dbac7" (UID: "5d6c4767-863e-4435-96e6-a421405dbac7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:17:58.202196 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:58.202155    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d6c4767-863e-4435-96e6-a421405dbac7-tls-certs\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:17:58.202196 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:58.202187    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d6c4767-863e-4435-96e6-a421405dbac7-tmp-dir\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:17:58.202196 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:58.202195    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d6c4767-863e-4435-96e6-a421405dbac7-dshm\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:17:58.202196 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:58.202204    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d6c4767-863e-4435-96e6-a421405dbac7-home\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:17:58.202469 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:58.202212    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bkjr\" (UniqueName: \"kubernetes.io/projected/5d6c4767-863e-4435-96e6-a421405dbac7-kube-api-access-2bkjr\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:17:58.202469 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:58.202221    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d6c4767-863e-4435-96e6-a421405dbac7-model-cache\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:17:58.562335 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:58.562296    2559 generic.go:356] "Generic (PLEG): container finished" podID="5d6c4767-863e-4435-96e6-a421405dbac7" containerID="4ded0907976676e052bae6d8dd11ba23198759c27ac3453ef7763e69f4dbd2c9" exitCode=0
Aug 01 00:17:58.562507 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:58.562345    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4ctbx"
Aug 01 00:17:58.562507 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:58.562344    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4ctbx" event={"ID":"5d6c4767-863e-4435-96e6-a421405dbac7","Type":"ContainerDied","Data":"4ded0907976676e052bae6d8dd11ba23198759c27ac3453ef7763e69f4dbd2c9"}
Aug 01 00:17:58.562507 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:58.562457    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4ctbx" event={"ID":"5d6c4767-863e-4435-96e6-a421405dbac7","Type":"ContainerDied","Data":"14318f78af26cfd8e4ee9dc207bdc9782d1923778e8e1976bb3f8229f8e7aff8"}
Aug 01 00:17:58.562507 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:58.562484    2559 scope.go:117] "RemoveContainer" containerID="4ded0907976676e052bae6d8dd11ba23198759c27ac3453ef7763e69f4dbd2c9"
Aug 01 00:17:58.571671 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:58.571653    2559 scope.go:117] "RemoveContainer" containerID="4ded0907976676e052bae6d8dd11ba23198759c27ac3453ef7763e69f4dbd2c9"
Aug 01 00:17:58.571960 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:17:58.571941    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ded0907976676e052bae6d8dd11ba23198759c27ac3453ef7763e69f4dbd2c9\": container with ID starting with 4ded0907976676e052bae6d8dd11ba23198759c27ac3453ef7763e69f4dbd2c9 not found: ID does not exist" containerID="4ded0907976676e052bae6d8dd11ba23198759c27ac3453ef7763e69f4dbd2c9"
Aug 01 00:17:58.572008 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:58.571969    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ded0907976676e052bae6d8dd11ba23198759c27ac3453ef7763e69f4dbd2c9"} err="failed to get container status \"4ded0907976676e052bae6d8dd11ba23198759c27ac3453ef7763e69f4dbd2c9\": rpc error: code = NotFound desc = could not find container \"4ded0907976676e052bae6d8dd11ba23198759c27ac3453ef7763e69f4dbd2c9\": container with ID starting with 4ded0907976676e052bae6d8dd11ba23198759c27ac3453ef7763e69f4dbd2c9 not found: ID does not exist"
Aug 01 00:17:58.583638 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:58.583612    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4ctbx"]
Aug 01 00:17:58.587975 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:58.587952    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4ctbx"]
Aug 01 00:17:59.786241 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:17:59.786199    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d6c4767-863e-4435-96e6-a421405dbac7" path="/var/lib/kubelet/pods/5d6c4767-863e-4435-96e6-a421405dbac7/volumes"
Aug 01 00:18:00.519649 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:00.519608    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vnzqd" podUID="fcec521b-1b0a-48a4-84f5-4ecdb6c7b797" containerName="main" containerID="cri-o://5764f8bf9f9db55900c51dc2250024a4bad499bc96a177e5be4a2afbc325d5ce" gracePeriod=45
Aug 01 00:18:00.772604 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:00.772540    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vnzqd"
Aug 01 00:18:00.829304 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:00.829266    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcec521b-1b0a-48a4-84f5-4ecdb6c7b797-dshm\") pod \"fcec521b-1b0a-48a4-84f5-4ecdb6c7b797\" (UID: \"fcec521b-1b0a-48a4-84f5-4ecdb6c7b797\") "
Aug 01 00:18:00.829807 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:00.829317    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcec521b-1b0a-48a4-84f5-4ecdb6c7b797-home\") pod \"fcec521b-1b0a-48a4-84f5-4ecdb6c7b797\" (UID: \"fcec521b-1b0a-48a4-84f5-4ecdb6c7b797\") "
Aug 01 00:18:00.829807 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:00.829350    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4nb5w\" (UniqueName: \"kubernetes.io/projected/fcec521b-1b0a-48a4-84f5-4ecdb6c7b797-kube-api-access-4nb5w\") pod \"fcec521b-1b0a-48a4-84f5-4ecdb6c7b797\" (UID: \"fcec521b-1b0a-48a4-84f5-4ecdb6c7b797\") "
Aug 01 00:18:00.829807 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:00.829380    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcec521b-1b0a-48a4-84f5-4ecdb6c7b797-tmp-dir\") pod \"fcec521b-1b0a-48a4-84f5-4ecdb6c7b797\" (UID: \"fcec521b-1b0a-48a4-84f5-4ecdb6c7b797\") "
Aug 01 00:18:00.829807 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:00.829415    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcec521b-1b0a-48a4-84f5-4ecdb6c7b797-tls-certs\") pod \"fcec521b-1b0a-48a4-84f5-4ecdb6c7b797\" (UID: \"fcec521b-1b0a-48a4-84f5-4ecdb6c7b797\") "
Aug 01 00:18:00.829807 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:00.829633    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcec521b-1b0a-48a4-84f5-4ecdb6c7b797-home" (OuterVolumeSpecName: "home") pod "fcec521b-1b0a-48a4-84f5-4ecdb6c7b797" (UID: "fcec521b-1b0a-48a4-84f5-4ecdb6c7b797"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:18:00.829807 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:00.829656    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcec521b-1b0a-48a4-84f5-4ecdb6c7b797-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fcec521b-1b0a-48a4-84f5-4ecdb6c7b797" (UID: "fcec521b-1b0a-48a4-84f5-4ecdb6c7b797"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:18:00.829807 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:00.829672    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcec521b-1b0a-48a4-84f5-4ecdb6c7b797-model-cache\") pod \"fcec521b-1b0a-48a4-84f5-4ecdb6c7b797\" (UID: \"fcec521b-1b0a-48a4-84f5-4ecdb6c7b797\") "
Aug 01 00:18:00.830107 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:00.829895    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcec521b-1b0a-48a4-84f5-4ecdb6c7b797-model-cache" (OuterVolumeSpecName: "model-cache") pod "fcec521b-1b0a-48a4-84f5-4ecdb6c7b797" (UID: "fcec521b-1b0a-48a4-84f5-4ecdb6c7b797"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:18:00.830107 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:00.829975    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcec521b-1b0a-48a4-84f5-4ecdb6c7b797-home\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:18:00.830107 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:00.829994    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcec521b-1b0a-48a4-84f5-4ecdb6c7b797-tmp-dir\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:18:00.830107 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:00.830009    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcec521b-1b0a-48a4-84f5-4ecdb6c7b797-model-cache\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:18:00.831601 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:00.831574    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcec521b-1b0a-48a4-84f5-4ecdb6c7b797-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fcec521b-1b0a-48a4-84f5-4ecdb6c7b797" (UID: "fcec521b-1b0a-48a4-84f5-4ecdb6c7b797"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:18:00.831784 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:00.831762    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcec521b-1b0a-48a4-84f5-4ecdb6c7b797-kube-api-access-4nb5w" (OuterVolumeSpecName: "kube-api-access-4nb5w") pod "fcec521b-1b0a-48a4-84f5-4ecdb6c7b797" (UID: "fcec521b-1b0a-48a4-84f5-4ecdb6c7b797"). InnerVolumeSpecName "kube-api-access-4nb5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:18:00.831863 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:00.831826    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcec521b-1b0a-48a4-84f5-4ecdb6c7b797-dshm" (OuterVolumeSpecName: "dshm") pod "fcec521b-1b0a-48a4-84f5-4ecdb6c7b797" (UID: "fcec521b-1b0a-48a4-84f5-4ecdb6c7b797"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:18:00.931279 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:00.931236    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcec521b-1b0a-48a4-84f5-4ecdb6c7b797-dshm\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:18:00.931279 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:00.931274    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nb5w\" (UniqueName: \"kubernetes.io/projected/fcec521b-1b0a-48a4-84f5-4ecdb6c7b797-kube-api-access-4nb5w\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:18:00.931279 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:00.931288    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcec521b-1b0a-48a4-84f5-4ecdb6c7b797-tls-certs\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:18:01.576683 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:01.576652    2559 generic.go:356] "Generic (PLEG): container finished" podID="fcec521b-1b0a-48a4-84f5-4ecdb6c7b797" containerID="5764f8bf9f9db55900c51dc2250024a4bad499bc96a177e5be4a2afbc325d5ce" exitCode=0
Aug 01 00:18:01.576885 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:01.576701    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vnzqd"
Aug 01 00:18:01.576885 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:01.576757    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vnzqd" event={"ID":"fcec521b-1b0a-48a4-84f5-4ecdb6c7b797","Type":"ContainerDied","Data":"5764f8bf9f9db55900c51dc2250024a4bad499bc96a177e5be4a2afbc325d5ce"}
Aug 01 00:18:01.576885 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:01.576817    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vnzqd" event={"ID":"fcec521b-1b0a-48a4-84f5-4ecdb6c7b797","Type":"ContainerDied","Data":"088702c92baeab1f26be1f84f8ea5385b0c24bfeec3e9b1826bf9b4449e156cc"}
Aug 01 00:18:01.576885 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:01.576847    2559 scope.go:117] "RemoveContainer" containerID="5764f8bf9f9db55900c51dc2250024a4bad499bc96a177e5be4a2afbc325d5ce"
Aug 01 00:18:01.585496 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:01.585477    2559 scope.go:117] "RemoveContainer" containerID="5764f8bf9f9db55900c51dc2250024a4bad499bc96a177e5be4a2afbc325d5ce"
Aug 01 00:18:01.585782 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:18:01.585762    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5764f8bf9f9db55900c51dc2250024a4bad499bc96a177e5be4a2afbc325d5ce\": container with ID starting with 5764f8bf9f9db55900c51dc2250024a4bad499bc96a177e5be4a2afbc325d5ce not found: ID does not exist" containerID="5764f8bf9f9db55900c51dc2250024a4bad499bc96a177e5be4a2afbc325d5ce"
Aug 01 00:18:01.585849 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:01.585792    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5764f8bf9f9db55900c51dc2250024a4bad499bc96a177e5be4a2afbc325d5ce"} err="failed to get container status \"5764f8bf9f9db55900c51dc2250024a4bad499bc96a177e5be4a2afbc325d5ce\": rpc error: code = NotFound desc = could not find container \"5764f8bf9f9db55900c51dc2250024a4bad499bc96a177e5be4a2afbc325d5ce\": container with ID starting with 5764f8bf9f9db55900c51dc2250024a4bad499bc96a177e5be4a2afbc325d5ce not found: ID does not exist"
Aug 01 00:18:01.599373 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:01.599348    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vnzqd"]
Aug 01 00:18:01.602299 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:01.602277    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vnzqd"]
Aug 01 00:18:01.785959 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:01.785925    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcec521b-1b0a-48a4-84f5-4ecdb6c7b797" path="/var/lib/kubelet/pods/fcec521b-1b0a-48a4-84f5-4ecdb6c7b797/volumes"
Aug 01 00:18:32.651590 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:32.651496    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-95qgt"]
Aug 01 00:18:32.652172 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:32.651869    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d6c4767-863e-4435-96e6-a421405dbac7" containerName="main"
Aug 01 00:18:32.652172 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:32.651882    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d6c4767-863e-4435-96e6-a421405dbac7" containerName="main"
Aug 01 00:18:32.652172 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:32.651905    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcec521b-1b0a-48a4-84f5-4ecdb6c7b797" containerName="main"
Aug 01 00:18:32.652172 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:32.651911    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcec521b-1b0a-48a4-84f5-4ecdb6c7b797" containerName="main"
Aug 01 00:18:32.652172 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:32.651976    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d6c4767-863e-4435-96e6-a421405dbac7" containerName="main"
Aug 01 00:18:32.652172 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:32.651986    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcec521b-1b0a-48a4-84f5-4ecdb6c7b797" containerName="main"
Aug 01 00:18:32.655216 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:32.655195    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-95qgt"
Aug 01 00:18:32.657964 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:32.657942    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"kube-root-ca.crt\""
Aug 01 00:18:32.658084 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:32.657945    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"openshift-service-ca.crt\""
Aug 01 00:18:32.659112 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:32.659092    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v3-kserve-self-signed-certs\""
Aug 01 00:18:32.659112 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:32.659110    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"default-dockercfg-zcnn2\""
Aug 01 00:18:32.665070 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:32.665047    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-95qgt"]
Aug 01 00:18:32.702062 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:32.702029    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0408995f-3c69-4cc9-9bae-440aa4dd1ef9-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-95qgt\" (UID: \"0408995f-3c69-4cc9-9bae-440aa4dd1ef9\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-95qgt"
Aug 01 00:18:32.702268 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:32.702082    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0408995f-3c69-4cc9-9bae-440aa4dd1ef9-tls-certs\") pod \"tri-v3-kserve-798c88c66c-95qgt\" (UID: \"0408995f-3c69-4cc9-9bae-440aa4dd1ef9\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-95qgt"
Aug 01 00:18:32.702268 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:32.702130    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0408995f-3c69-4cc9-9bae-440aa4dd1ef9-home\") pod \"tri-v3-kserve-798c88c66c-95qgt\" (UID: \"0408995f-3c69-4cc9-9bae-440aa4dd1ef9\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-95qgt"
Aug 01 00:18:32.702268 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:32.702147    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhbp2\" (UniqueName: \"kubernetes.io/projected/0408995f-3c69-4cc9-9bae-440aa4dd1ef9-kube-api-access-hhbp2\") pod \"tri-v3-kserve-798c88c66c-95qgt\" (UID: \"0408995f-3c69-4cc9-9bae-440aa4dd1ef9\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-95qgt"
Aug 01 00:18:32.702268 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:32.702169    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0408995f-3c69-4cc9-9bae-440aa4dd1ef9-dshm\") pod \"tri-v3-kserve-798c88c66c-95qgt\" (UID: \"0408995f-3c69-4cc9-9bae-440aa4dd1ef9\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-95qgt"
Aug 01 00:18:32.702268 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:32.702215    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0408995f-3c69-4cc9-9bae-440aa4dd1ef9-model-cache\") pod \"tri-v3-kserve-798c88c66c-95qgt\" (UID: \"0408995f-3c69-4cc9-9bae-440aa4dd1ef9\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-95qgt"
Aug 01 00:18:32.807764 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:32.807697    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0408995f-3c69-4cc9-9bae-440aa4dd1ef9-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-95qgt\" (UID: \"0408995f-3c69-4cc9-9bae-440aa4dd1ef9\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-95qgt"
Aug 01 00:18:32.807964 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:32.807876    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0408995f-3c69-4cc9-9bae-440aa4dd1ef9-tls-certs\") pod \"tri-v3-kserve-798c88c66c-95qgt\" (UID: \"0408995f-3c69-4cc9-9bae-440aa4dd1ef9\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-95qgt"
Aug 01 00:18:32.807964 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:32.807924    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0408995f-3c69-4cc9-9bae-440aa4dd1ef9-home\") pod \"tri-v3-kserve-798c88c66c-95qgt\" (UID: \"0408995f-3c69-4cc9-9bae-440aa4dd1ef9\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-95qgt"
Aug 01 00:18:32.807964 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:32.807954    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhbp2\" (UniqueName: \"kubernetes.io/projected/0408995f-3c69-4cc9-9bae-440aa4dd1ef9-kube-api-access-hhbp2\") pod \"tri-v3-kserve-798c88c66c-95qgt\" (UID: \"0408995f-3c69-4cc9-9bae-440aa4dd1ef9\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-95qgt"
Aug 01 00:18:32.808147 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:32.807991    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0408995f-3c69-4cc9-9bae-440aa4dd1ef9-dshm\") pod \"tri-v3-kserve-798c88c66c-95qgt\" (UID: \"0408995f-3c69-4cc9-9bae-440aa4dd1ef9\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-95qgt"
Aug 01 00:18:32.808147 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:32.808019    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0408995f-3c69-4cc9-9bae-440aa4dd1ef9-model-cache\") pod \"tri-v3-kserve-798c88c66c-95qgt\" (UID: \"0408995f-3c69-4cc9-9bae-440aa4dd1ef9\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-95qgt"
Aug 01 00:18:32.808261 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:32.808177    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0408995f-3c69-4cc9-9bae-440aa4dd1ef9-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-95qgt\" (UID: \"0408995f-3c69-4cc9-9bae-440aa4dd1ef9\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-95qgt"
Aug 01 00:18:32.808387 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:32.808366    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0408995f-3c69-4cc9-9bae-440aa4dd1ef9-model-cache\") pod \"tri-v3-kserve-798c88c66c-95qgt\" (UID: \"0408995f-3c69-4cc9-9bae-440aa4dd1ef9\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-95qgt"
Aug 01 00:18:32.808939 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:32.808915    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0408995f-3c69-4cc9-9bae-440aa4dd1ef9-home\") pod \"tri-v3-kserve-798c88c66c-95qgt\" (UID: \"0408995f-3c69-4cc9-9bae-440aa4dd1ef9\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-95qgt"
Aug 01 00:18:32.810638 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:32.810611    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0408995f-3c69-4cc9-9bae-440aa4dd1ef9-tls-certs\") pod \"tri-v3-kserve-798c88c66c-95qgt\" (UID: \"0408995f-3c69-4cc9-9bae-440aa4dd1ef9\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-95qgt"
Aug 01 00:18:32.810804 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:32.810787    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0408995f-3c69-4cc9-9bae-440aa4dd1ef9-dshm\") pod \"tri-v3-kserve-798c88c66c-95qgt\" (UID: \"0408995f-3c69-4cc9-9bae-440aa4dd1ef9\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-95qgt"
Aug 01 00:18:32.816628 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:32.816607    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhbp2\" (UniqueName: \"kubernetes.io/projected/0408995f-3c69-4cc9-9bae-440aa4dd1ef9-kube-api-access-hhbp2\") pod \"tri-v3-kserve-798c88c66c-95qgt\" (UID: \"0408995f-3c69-4cc9-9bae-440aa4dd1ef9\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-95qgt"
Aug 01 00:18:32.965726 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:32.965692    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-95qgt"
Aug 01 00:18:33.098041 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:33.098012    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-95qgt"]
Aug 01 00:18:33.100583 ip-10-0-130-148 kubenswrapper[2559]: W0801 00:18:33.100559    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0408995f_3c69_4cc9_9bae_440aa4dd1ef9.slice/crio-ddc681d9b103456a9bfc40e11d801bd3c4b047641101e1df934492055eb57e30 WatchSource:0}: Error finding container ddc681d9b103456a9bfc40e11d801bd3c4b047641101e1df934492055eb57e30: Status 404 returned error can't find the container with id ddc681d9b103456a9bfc40e11d801bd3c4b047641101e1df934492055eb57e30
Aug 01 00:18:33.720978 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:33.720936    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-95qgt" event={"ID":"0408995f-3c69-4cc9-9bae-440aa4dd1ef9","Type":"ContainerStarted","Data":"7b32a8bb52735c5b6469165dcb8d970b59cef7eca7d04d1d990dc40815067a9e"}
Aug 01 00:18:33.721367 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:33.721005    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-95qgt" event={"ID":"0408995f-3c69-4cc9-9bae-440aa4dd1ef9","Type":"ContainerStarted","Data":"ddc681d9b103456a9bfc40e11d801bd3c4b047641101e1df934492055eb57e30"}
Aug 01 00:18:33.737574 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:33.737525    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-95qgt" podStartSLOduration=1.737511255 podStartE2EDuration="1.737511255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:18:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:18:33.736237581 +0000 UTC m=+1976.456037426" watchObservedRunningTime="2026-08-01 00:18:33.737511255 +0000 UTC m=+1976.457311100"
Aug 01 00:18:33.966084 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:33.966046    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-95qgt"
Aug 01 00:18:34.973056 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:34.973025    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4n86"]
Aug 01 00:18:34.981812 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:18:34.981775    2559 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 01 00:18:34.981812 ip-10-0-130-148 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:18:34.981812 ip-10-0-130-148 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Aug 01 00:18:34.981812 ip-10-0-130-148 kubenswrapper[2559]:  > containerID="27879120114ae62e4618362fb6085a1cb6db6aaaa80d47c91b2b2780b757fb30" cmd=["/bin/sleep","15"]
Aug 01 00:18:34.982115 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:34.981832    2559 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4n86" message="" err=<
Aug 01 00:18:34.982115 ip-10-0-130-148 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:18:34.982115 ip-10-0-130-148 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Aug 01 00:18:34.982115 ip-10-0-130-148 kubenswrapper[2559]:  >
Aug 01 00:18:34.982115 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:18:34.981847    2559 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 01 00:18:34.982115 ip-10-0-130-148 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:18:34.982115 ip-10-0-130-148 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Aug 01 00:18:34.982115 ip-10-0-130-148 kubenswrapper[2559]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4n86" podUID="442245a5-ef18-4c25-bdbe-ae2f52a33cb5" containerName="main" containerID="cri-o://27879120114ae62e4618362fb6085a1cb6db6aaaa80d47c91b2b2780b757fb30"
Aug 01 00:18:34.982115 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:34.981894    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4n86" podUID="442245a5-ef18-4c25-bdbe-ae2f52a33cb5" containerName="main" containerID="cri-o://27879120114ae62e4618362fb6085a1cb6db6aaaa80d47c91b2b2780b757fb30" gracePeriod=60
Aug 01 00:18:35.083890 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:35.083843    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4n86" podUID="442245a5-ef18-4c25-bdbe-ae2f52a33cb5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Aug 01 00:18:35.331629 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:35.331603    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4n86"
Aug 01 00:18:35.428073 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:35.428025    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-csr9z\" (UniqueName: \"kubernetes.io/projected/442245a5-ef18-4c25-bdbe-ae2f52a33cb5-kube-api-access-csr9z\") pod \"442245a5-ef18-4c25-bdbe-ae2f52a33cb5\" (UID: \"442245a5-ef18-4c25-bdbe-ae2f52a33cb5\") "
Aug 01 00:18:35.428254 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:35.428098    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/442245a5-ef18-4c25-bdbe-ae2f52a33cb5-tls-certs\") pod \"442245a5-ef18-4c25-bdbe-ae2f52a33cb5\" (UID: \"442245a5-ef18-4c25-bdbe-ae2f52a33cb5\") "
Aug 01 00:18:35.428254 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:35.428136    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/442245a5-ef18-4c25-bdbe-ae2f52a33cb5-dshm\") pod \"442245a5-ef18-4c25-bdbe-ae2f52a33cb5\" (UID: \"442245a5-ef18-4c25-bdbe-ae2f52a33cb5\") "
Aug 01 00:18:35.428254 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:35.428168    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/442245a5-ef18-4c25-bdbe-ae2f52a33cb5-tmp-dir\") pod \"442245a5-ef18-4c25-bdbe-ae2f52a33cb5\" (UID: \"442245a5-ef18-4c25-bdbe-ae2f52a33cb5\") "
Aug 01 00:18:35.428254 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:35.428195    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/442245a5-ef18-4c25-bdbe-ae2f52a33cb5-home\") pod \"442245a5-ef18-4c25-bdbe-ae2f52a33cb5\" (UID: \"442245a5-ef18-4c25-bdbe-ae2f52a33cb5\") "
Aug 01 00:18:35.428487 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:35.428262    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/442245a5-ef18-4c25-bdbe-ae2f52a33cb5-model-cache\") pod \"442245a5-ef18-4c25-bdbe-ae2f52a33cb5\" (UID: \"442245a5-ef18-4c25-bdbe-ae2f52a33cb5\") "
Aug 01 00:18:35.428799 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:35.428571    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/442245a5-ef18-4c25-bdbe-ae2f52a33cb5-home" (OuterVolumeSpecName: "home") pod "442245a5-ef18-4c25-bdbe-ae2f52a33cb5" (UID: "442245a5-ef18-4c25-bdbe-ae2f52a33cb5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:18:35.428799 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:35.428657    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/442245a5-ef18-4c25-bdbe-ae2f52a33cb5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "442245a5-ef18-4c25-bdbe-ae2f52a33cb5" (UID: "442245a5-ef18-4c25-bdbe-ae2f52a33cb5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:18:35.428799 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:35.428671    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/442245a5-ef18-4c25-bdbe-ae2f52a33cb5-model-cache" (OuterVolumeSpecName: "model-cache") pod "442245a5-ef18-4c25-bdbe-ae2f52a33cb5" (UID: "442245a5-ef18-4c25-bdbe-ae2f52a33cb5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:18:35.430437 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:35.430328    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/442245a5-ef18-4c25-bdbe-ae2f52a33cb5-dshm" (OuterVolumeSpecName: "dshm") pod "442245a5-ef18-4c25-bdbe-ae2f52a33cb5" (UID: "442245a5-ef18-4c25-bdbe-ae2f52a33cb5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:18:35.430437 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:35.430405    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/442245a5-ef18-4c25-bdbe-ae2f52a33cb5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "442245a5-ef18-4c25-bdbe-ae2f52a33cb5" (UID: "442245a5-ef18-4c25-bdbe-ae2f52a33cb5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:18:35.430567 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:35.430496    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/442245a5-ef18-4c25-bdbe-ae2f52a33cb5-kube-api-access-csr9z" (OuterVolumeSpecName: "kube-api-access-csr9z") pod "442245a5-ef18-4c25-bdbe-ae2f52a33cb5" (UID: "442245a5-ef18-4c25-bdbe-ae2f52a33cb5"). InnerVolumeSpecName "kube-api-access-csr9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:18:35.530003 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:35.529912    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csr9z\" (UniqueName: \"kubernetes.io/projected/442245a5-ef18-4c25-bdbe-ae2f52a33cb5-kube-api-access-csr9z\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:18:35.530003 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:35.529942    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/442245a5-ef18-4c25-bdbe-ae2f52a33cb5-tls-certs\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:18:35.530003 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:35.529954    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/442245a5-ef18-4c25-bdbe-ae2f52a33cb5-dshm\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:18:35.530003 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:35.529962    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/442245a5-ef18-4c25-bdbe-ae2f52a33cb5-tmp-dir\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:18:35.530003 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:35.529971    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/442245a5-ef18-4c25-bdbe-ae2f52a33cb5-home\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:18:35.530003 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:35.529979    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/442245a5-ef18-4c25-bdbe-ae2f52a33cb5-model-cache\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:18:35.731182 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:35.731145    2559 generic.go:356] "Generic (PLEG): container finished" podID="442245a5-ef18-4c25-bdbe-ae2f52a33cb5" containerID="27879120114ae62e4618362fb6085a1cb6db6aaaa80d47c91b2b2780b757fb30" exitCode=0
Aug 01 00:18:35.731354 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:35.731192    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4n86"
Aug 01 00:18:35.731354 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:35.731230    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4n86" event={"ID":"442245a5-ef18-4c25-bdbe-ae2f52a33cb5","Type":"ContainerDied","Data":"27879120114ae62e4618362fb6085a1cb6db6aaaa80d47c91b2b2780b757fb30"}
Aug 01 00:18:35.731354 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:35.731292    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4n86" event={"ID":"442245a5-ef18-4c25-bdbe-ae2f52a33cb5","Type":"ContainerDied","Data":"77d27208b1b36bc3f434ad569d40f1366c2631260ea75d1afa65851a73800c90"}
Aug 01 00:18:35.731354 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:35.731315    2559 scope.go:117] "RemoveContainer" containerID="27879120114ae62e4618362fb6085a1cb6db6aaaa80d47c91b2b2780b757fb30"
Aug 01 00:18:35.747340 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:35.747320    2559 scope.go:117] "RemoveContainer" containerID="27879120114ae62e4618362fb6085a1cb6db6aaaa80d47c91b2b2780b757fb30"
Aug 01 00:18:35.747605 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:18:35.747584    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27879120114ae62e4618362fb6085a1cb6db6aaaa80d47c91b2b2780b757fb30\": container with ID starting with 27879120114ae62e4618362fb6085a1cb6db6aaaa80d47c91b2b2780b757fb30 not found: ID does not exist" containerID="27879120114ae62e4618362fb6085a1cb6db6aaaa80d47c91b2b2780b757fb30"
Aug 01 00:18:35.747669 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:35.747617    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27879120114ae62e4618362fb6085a1cb6db6aaaa80d47c91b2b2780b757fb30"} err="failed to get container status \"27879120114ae62e4618362fb6085a1cb6db6aaaa80d47c91b2b2780b757fb30\": rpc error: code = NotFound desc = could not find container \"27879120114ae62e4618362fb6085a1cb6db6aaaa80d47c91b2b2780b757fb30\": container with ID starting with 27879120114ae62e4618362fb6085a1cb6db6aaaa80d47c91b2b2780b757fb30 not found: ID does not exist"
Aug 01 00:18:35.756604 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:35.756579    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4n86"]
Aug 01 00:18:35.760227 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:35.760204    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4n86"]
Aug 01 00:18:35.786767 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:35.786685    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="442245a5-ef18-4c25-bdbe-ae2f52a33cb5" path="/var/lib/kubelet/pods/442245a5-ef18-4c25-bdbe-ae2f52a33cb5/volumes"
Aug 01 00:18:42.966123 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:42.966084    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-95qgt"
Aug 01 00:18:42.978364 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:42.978334    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-95qgt"
Aug 01 00:18:43.786100 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:18:43.786070    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-95qgt"
Aug 01 00:19:39.061387 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:19:39.061353    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lcf4l"]
Aug 01 00:19:39.061884 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:19:39.061695    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="442245a5-ef18-4c25-bdbe-ae2f52a33cb5" containerName="main"
Aug 01 00:19:39.061884 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:19:39.061708    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="442245a5-ef18-4c25-bdbe-ae2f52a33cb5" containerName="main"
Aug 01 00:19:39.061884 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:19:39.061794    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="442245a5-ef18-4c25-bdbe-ae2f52a33cb5" containerName="main"
Aug 01 00:19:39.064638 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:19:39.064617    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lcf4l"
Aug 01 00:19:39.067268 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:19:39.067249    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Aug 01 00:19:39.068436 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:19:39.068408    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Aug 01 00:19:39.068436 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:19:39.068422    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-76kns\""
Aug 01 00:19:39.068604 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:19:39.068412    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Aug 01 00:19:39.075738 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:19:39.075711    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lcf4l"]
Aug 01 00:19:39.148357 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:19:39.148322    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7tt9\" (UniqueName: \"kubernetes.io/projected/83abd782-637f-481b-9296-3bbfed12f803-kube-api-access-f7tt9\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lcf4l\" (UID: \"83abd782-637f-481b-9296-3bbfed12f803\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lcf4l"
Aug 01 00:19:39.148520 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:19:39.148366    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83abd782-637f-481b-9296-3bbfed12f803-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lcf4l\" (UID: \"83abd782-637f-481b-9296-3bbfed12f803\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lcf4l"
Aug 01 00:19:39.148520 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:19:39.148387    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83abd782-637f-481b-9296-3bbfed12f803-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lcf4l\" (UID: \"83abd782-637f-481b-9296-3bbfed12f803\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lcf4l"
Aug 01 00:19:39.148520 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:19:39.148409    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83abd782-637f-481b-9296-3bbfed12f803-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lcf4l\" (UID: \"83abd782-637f-481b-9296-3bbfed12f803\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lcf4l"
Aug 01 00:19:39.148520 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:19:39.148430    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83abd782-637f-481b-9296-3bbfed12f803-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lcf4l\" (UID: \"83abd782-637f-481b-9296-3bbfed12f803\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lcf4l"
Aug 01 00:19:39.148664 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:19:39.148520    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83abd782-637f-481b-9296-3bbfed12f803-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lcf4l\" (UID: \"83abd782-637f-481b-9296-3bbfed12f803\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lcf4l"
Aug 01 00:19:39.249854 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:19:39.249813    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83abd782-637f-481b-9296-3bbfed12f803-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lcf4l\" (UID: \"83abd782-637f-481b-9296-3bbfed12f803\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lcf4l"
Aug 01 00:19:39.250028 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:19:39.249868    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7tt9\" (UniqueName: \"kubernetes.io/projected/83abd782-637f-481b-9296-3bbfed12f803-kube-api-access-f7tt9\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lcf4l\" (UID: \"83abd782-637f-481b-9296-3bbfed12f803\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lcf4l"
Aug 01 00:19:39.250028 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:19:39.249891    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83abd782-637f-481b-9296-3bbfed12f803-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lcf4l\" (UID: \"83abd782-637f-481b-9296-3bbfed12f803\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lcf4l"
Aug 01 00:19:39.250028 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:19:39.249909    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83abd782-637f-481b-9296-3bbfed12f803-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lcf4l\" (UID: \"83abd782-637f-481b-9296-3bbfed12f803\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lcf4l"
Aug 01 00:19:39.250028 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:19:39.249933    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83abd782-637f-481b-9296-3bbfed12f803-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lcf4l\" (UID: \"83abd782-637f-481b-9296-3bbfed12f803\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lcf4l"
Aug 01 00:19:39.250028 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:19:39.249956    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83abd782-637f-481b-9296-3bbfed12f803-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lcf4l\" (UID: \"83abd782-637f-481b-9296-3bbfed12f803\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lcf4l"
Aug 01 00:19:39.250378 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:19:39.250350    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83abd782-637f-481b-9296-3bbfed12f803-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lcf4l\" (UID: \"83abd782-637f-481b-9296-3bbfed12f803\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lcf4l"
Aug 01 00:19:39.250496 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:19:39.250378    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83abd782-637f-481b-9296-3bbfed12f803-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lcf4l\" (UID: \"83abd782-637f-481b-9296-3bbfed12f803\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lcf4l"
Aug 01 00:19:39.250496 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:19:39.250408    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83abd782-637f-481b-9296-3bbfed12f803-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lcf4l\" (UID: \"83abd782-637f-481b-9296-3bbfed12f803\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lcf4l"
Aug 01 00:19:39.252146 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:19:39.252131    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83abd782-637f-481b-9296-3bbfed12f803-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lcf4l\" (UID: \"83abd782-637f-481b-9296-3bbfed12f803\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lcf4l"
Aug 01 00:19:39.252502 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:19:39.252483    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83abd782-637f-481b-9296-3bbfed12f803-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lcf4l\" (UID: \"83abd782-637f-481b-9296-3bbfed12f803\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lcf4l"
Aug 01 00:19:39.267012 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:19:39.266982    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7tt9\" (UniqueName: \"kubernetes.io/projected/83abd782-637f-481b-9296-3bbfed12f803-kube-api-access-f7tt9\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lcf4l\" (UID: \"83abd782-637f-481b-9296-3bbfed12f803\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lcf4l"
Aug 01 00:19:39.374910 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:19:39.374804    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lcf4l"
Aug 01 00:19:39.507707 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:19:39.507680    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lcf4l"]
Aug 01 00:19:39.509444 ip-10-0-130-148 kubenswrapper[2559]: W0801 00:19:39.509415    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83abd782_637f_481b_9296_3bbfed12f803.slice/crio-b37045d50c5522106b3aab0859b27179e714090092e09cb1145971ee4999152f WatchSource:0}: Error finding container b37045d50c5522106b3aab0859b27179e714090092e09cb1145971ee4999152f: Status 404 returned error can't find the container with id b37045d50c5522106b3aab0859b27179e714090092e09cb1145971ee4999152f
Aug 01 00:19:40.004205 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:19:40.004170    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lcf4l" event={"ID":"83abd782-637f-481b-9296-3bbfed12f803","Type":"ContainerStarted","Data":"b06900a60f69b0b8cc70b947778a2e4453972adfe3cdf6861ed0a1a85cde6557"}
Aug 01 00:19:40.004382 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:19:40.004223    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lcf4l" event={"ID":"83abd782-637f-481b-9296-3bbfed12f803","Type":"ContainerStarted","Data":"b37045d50c5522106b3aab0859b27179e714090092e09cb1145971ee4999152f"}
Aug 01 00:19:40.023055 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:19:40.023006    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lcf4l" podStartSLOduration=1.022991733 podStartE2EDuration="1.022991733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:19:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:19:40.02135676 +0000 UTC m=+2042.741156625" watchObservedRunningTime="2026-08-01 00:19:40.022991733 +0000 UTC m=+2042.742791579"
Aug 01 00:19:40.375309 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:19:40.375228    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lcf4l"
Aug 01 00:19:49.375689 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:19:49.375654    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lcf4l"
Aug 01 00:19:49.388481 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:19:49.388456    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lcf4l"
Aug 01 00:19:50.058189 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:19:50.058160    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lcf4l"
Aug 01 00:20:23.052613 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:20:23.052581    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-95qgt"]
Aug 01 00:20:37.857615 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:20:37.857585    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-spblg_c2e50946-aea4-4886-82ec-0bd5a31196cf/console-operator/1.log"
Aug 01 00:20:37.862354 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:20:37.862336    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-spblg_c2e50946-aea4-4886-82ec-0bd5a31196cf/console-operator/1.log"
Aug 01 00:20:38.071317 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:20:38.071259    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-95qgt" podUID="0408995f-3c69-4cc9-9bae-440aa4dd1ef9" containerName="main" containerID="cri-o://7b32a8bb52735c5b6469165dcb8d970b59cef7eca7d04d1d990dc40815067a9e" gracePeriod=45
Aug 01 00:20:38.261532 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:20:38.261497    2559 generic.go:356] "Generic (PLEG): container finished" podID="0408995f-3c69-4cc9-9bae-440aa4dd1ef9" containerID="7b32a8bb52735c5b6469165dcb8d970b59cef7eca7d04d1d990dc40815067a9e" exitCode=0
Aug 01 00:20:38.261693 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:20:38.261576    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-95qgt" event={"ID":"0408995f-3c69-4cc9-9bae-440aa4dd1ef9","Type":"ContainerDied","Data":"7b32a8bb52735c5b6469165dcb8d970b59cef7eca7d04d1d990dc40815067a9e"}
Aug 01 00:20:38.318509 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:20:38.318485    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-95qgt"
Aug 01 00:20:38.419135 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:20:38.419105    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0408995f-3c69-4cc9-9bae-440aa4dd1ef9-home\") pod \"0408995f-3c69-4cc9-9bae-440aa4dd1ef9\" (UID: \"0408995f-3c69-4cc9-9bae-440aa4dd1ef9\") "
Aug 01 00:20:38.419285 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:20:38.419149    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0408995f-3c69-4cc9-9bae-440aa4dd1ef9-model-cache\") pod \"0408995f-3c69-4cc9-9bae-440aa4dd1ef9\" (UID: \"0408995f-3c69-4cc9-9bae-440aa4dd1ef9\") "
Aug 01 00:20:38.419285 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:20:38.419200    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0408995f-3c69-4cc9-9bae-440aa4dd1ef9-tmp-dir\") pod \"0408995f-3c69-4cc9-9bae-440aa4dd1ef9\" (UID: \"0408995f-3c69-4cc9-9bae-440aa4dd1ef9\") "
Aug 01 00:20:38.419285 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:20:38.419217    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0408995f-3c69-4cc9-9bae-440aa4dd1ef9-tls-certs\") pod \"0408995f-3c69-4cc9-9bae-440aa4dd1ef9\" (UID: \"0408995f-3c69-4cc9-9bae-440aa4dd1ef9\") "
Aug 01 00:20:38.419285 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:20:38.419234    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0408995f-3c69-4cc9-9bae-440aa4dd1ef9-dshm\") pod \"0408995f-3c69-4cc9-9bae-440aa4dd1ef9\" (UID: \"0408995f-3c69-4cc9-9bae-440aa4dd1ef9\") "
Aug 01 00:20:38.419285 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:20:38.419261    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hhbp2\" (UniqueName: \"kubernetes.io/projected/0408995f-3c69-4cc9-9bae-440aa4dd1ef9-kube-api-access-hhbp2\") pod \"0408995f-3c69-4cc9-9bae-440aa4dd1ef9\" (UID: \"0408995f-3c69-4cc9-9bae-440aa4dd1ef9\") "
Aug 01 00:20:38.419531 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:20:38.419444    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0408995f-3c69-4cc9-9bae-440aa4dd1ef9-home" (OuterVolumeSpecName: "home") pod "0408995f-3c69-4cc9-9bae-440aa4dd1ef9" (UID: "0408995f-3c69-4cc9-9bae-440aa4dd1ef9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:20:38.419531 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:20:38.419484    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0408995f-3c69-4cc9-9bae-440aa4dd1ef9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0408995f-3c69-4cc9-9bae-440aa4dd1ef9" (UID: "0408995f-3c69-4cc9-9bae-440aa4dd1ef9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:20:38.419531 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:20:38.419497    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0408995f-3c69-4cc9-9bae-440aa4dd1ef9-model-cache" (OuterVolumeSpecName: "model-cache") pod "0408995f-3c69-4cc9-9bae-440aa4dd1ef9" (UID: "0408995f-3c69-4cc9-9bae-440aa4dd1ef9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:20:38.421591 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:20:38.421565    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0408995f-3c69-4cc9-9bae-440aa4dd1ef9-kube-api-access-hhbp2" (OuterVolumeSpecName: "kube-api-access-hhbp2") pod "0408995f-3c69-4cc9-9bae-440aa4dd1ef9" (UID: "0408995f-3c69-4cc9-9bae-440aa4dd1ef9"). InnerVolumeSpecName "kube-api-access-hhbp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:20:38.421703 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:20:38.421607    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0408995f-3c69-4cc9-9bae-440aa4dd1ef9-dshm" (OuterVolumeSpecName: "dshm") pod "0408995f-3c69-4cc9-9bae-440aa4dd1ef9" (UID: "0408995f-3c69-4cc9-9bae-440aa4dd1ef9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:20:38.421703 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:20:38.421613    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0408995f-3c69-4cc9-9bae-440aa4dd1ef9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0408995f-3c69-4cc9-9bae-440aa4dd1ef9" (UID: "0408995f-3c69-4cc9-9bae-440aa4dd1ef9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:20:38.520782 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:20:38.520736    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0408995f-3c69-4cc9-9bae-440aa4dd1ef9-model-cache\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:20:38.520918 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:20:38.520806    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0408995f-3c69-4cc9-9bae-440aa4dd1ef9-tmp-dir\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:20:38.520918 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:20:38.520818    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0408995f-3c69-4cc9-9bae-440aa4dd1ef9-tls-certs\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:20:38.520918 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:20:38.520829    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0408995f-3c69-4cc9-9bae-440aa4dd1ef9-dshm\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:20:38.520918 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:20:38.520841    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhbp2\" (UniqueName: \"kubernetes.io/projected/0408995f-3c69-4cc9-9bae-440aa4dd1ef9-kube-api-access-hhbp2\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:20:38.520918 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:20:38.520854    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0408995f-3c69-4cc9-9bae-440aa4dd1ef9-home\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:20:39.268264 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:20:39.268236    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-95qgt"
Aug 01 00:20:39.268727 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:20:39.268234    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-95qgt" event={"ID":"0408995f-3c69-4cc9-9bae-440aa4dd1ef9","Type":"ContainerDied","Data":"ddc681d9b103456a9bfc40e11d801bd3c4b047641101e1df934492055eb57e30"}
Aug 01 00:20:39.268727 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:20:39.268369    2559 scope.go:117] "RemoveContainer" containerID="7b32a8bb52735c5b6469165dcb8d970b59cef7eca7d04d1d990dc40815067a9e"
Aug 01 00:20:39.290838 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:20:39.290817    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-95qgt"]
Aug 01 00:20:39.294089 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:20:39.294064    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-95qgt"]
Aug 01 00:20:39.786641 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:20:39.786609    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0408995f-3c69-4cc9-9bae-440aa4dd1ef9" path="/var/lib/kubelet/pods/0408995f-3c69-4cc9-9bae-440aa4dd1ef9/volumes"
Aug 01 00:21:04.060315 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:04.060282    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qv8w7"]
Aug 01 00:21:04.060800 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:04.060628    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0408995f-3c69-4cc9-9bae-440aa4dd1ef9" containerName="main"
Aug 01 00:21:04.060800 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:04.060639    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0408995f-3c69-4cc9-9bae-440aa4dd1ef9" containerName="main"
Aug 01 00:21:04.060800 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:04.060692    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0408995f-3c69-4cc9-9bae-440aa4dd1ef9" containerName="main"
Aug 01 00:21:04.063681 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:04.063661    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qv8w7"
Aug 01 00:21:04.066410 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:04.066380    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"openshift-service-ca.crt\""
Aug 01 00:21:04.066543 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:04.066525    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"default-dockercfg-4nw7b\""
Aug 01 00:21:04.067629 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:04.067610    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"late-v1-kserve-self-signed-certs\""
Aug 01 00:21:04.067735 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:04.067659    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"kube-root-ca.crt\""
Aug 01 00:21:04.073765 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:04.073725    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qv8w7"]
Aug 01 00:21:04.232074 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:04.232040    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a33756b8-7000-455e-ac32-c7ad9fb1b64b-tls-certs\") pod \"late-v1-kserve-56cbc8456c-qv8w7\" (UID: \"a33756b8-7000-455e-ac32-c7ad9fb1b64b\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qv8w7"
Aug 01 00:21:04.232273 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:04.232097    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a33756b8-7000-455e-ac32-c7ad9fb1b64b-dshm\") pod \"late-v1-kserve-56cbc8456c-qv8w7\" (UID: \"a33756b8-7000-455e-ac32-c7ad9fb1b64b\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qv8w7"
Aug 01 00:21:04.232273 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:04.232121    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a33756b8-7000-455e-ac32-c7ad9fb1b64b-model-cache\") pod \"late-v1-kserve-56cbc8456c-qv8w7\" (UID: \"a33756b8-7000-455e-ac32-c7ad9fb1b64b\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qv8w7"
Aug 01 00:21:04.232273 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:04.232168    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a33756b8-7000-455e-ac32-c7ad9fb1b64b-home\") pod \"late-v1-kserve-56cbc8456c-qv8w7\" (UID: \"a33756b8-7000-455e-ac32-c7ad9fb1b64b\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qv8w7"
Aug 01 00:21:04.232273 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:04.232184    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2c9n\" (UniqueName: \"kubernetes.io/projected/a33756b8-7000-455e-ac32-c7ad9fb1b64b-kube-api-access-c2c9n\") pod \"late-v1-kserve-56cbc8456c-qv8w7\" (UID: \"a33756b8-7000-455e-ac32-c7ad9fb1b64b\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qv8w7"
Aug 01 00:21:04.232273 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:04.232256    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a33756b8-7000-455e-ac32-c7ad9fb1b64b-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-qv8w7\" (UID: \"a33756b8-7000-455e-ac32-c7ad9fb1b64b\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qv8w7"
Aug 01 00:21:04.333416 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:04.333328    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a33756b8-7000-455e-ac32-c7ad9fb1b64b-tls-certs\") pod \"late-v1-kserve-56cbc8456c-qv8w7\" (UID: \"a33756b8-7000-455e-ac32-c7ad9fb1b64b\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qv8w7"
Aug 01 00:21:04.333416 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:04.333384    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a33756b8-7000-455e-ac32-c7ad9fb1b64b-dshm\") pod \"late-v1-kserve-56cbc8456c-qv8w7\" (UID: \"a33756b8-7000-455e-ac32-c7ad9fb1b64b\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qv8w7"
Aug 01 00:21:04.333416 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:04.333416    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a33756b8-7000-455e-ac32-c7ad9fb1b64b-model-cache\") pod \"late-v1-kserve-56cbc8456c-qv8w7\" (UID: \"a33756b8-7000-455e-ac32-c7ad9fb1b64b\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qv8w7"
Aug 01 00:21:04.333680 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:04.333473    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a33756b8-7000-455e-ac32-c7ad9fb1b64b-home\") pod \"late-v1-kserve-56cbc8456c-qv8w7\" (UID: \"a33756b8-7000-455e-ac32-c7ad9fb1b64b\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qv8w7"
Aug 01 00:21:04.333680 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:04.333497    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2c9n\" (UniqueName: \"kubernetes.io/projected/a33756b8-7000-455e-ac32-c7ad9fb1b64b-kube-api-access-c2c9n\") pod \"late-v1-kserve-56cbc8456c-qv8w7\" (UID: \"a33756b8-7000-455e-ac32-c7ad9fb1b64b\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qv8w7"
Aug 01 00:21:04.333680 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:04.333524    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a33756b8-7000-455e-ac32-c7ad9fb1b64b-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-qv8w7\" (UID: \"a33756b8-7000-455e-ac32-c7ad9fb1b64b\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qv8w7"
Aug 01 00:21:04.333899 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:04.333878    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a33756b8-7000-455e-ac32-c7ad9fb1b64b-home\") pod \"late-v1-kserve-56cbc8456c-qv8w7\" (UID: \"a33756b8-7000-455e-ac32-c7ad9fb1b64b\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qv8w7"
Aug 01 00:21:04.333967 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:04.333921    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a33756b8-7000-455e-ac32-c7ad9fb1b64b-model-cache\") pod \"late-v1-kserve-56cbc8456c-qv8w7\" (UID: \"a33756b8-7000-455e-ac32-c7ad9fb1b64b\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qv8w7"
Aug 01 00:21:04.333967 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:04.333948    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a33756b8-7000-455e-ac32-c7ad9fb1b64b-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-qv8w7\" (UID: \"a33756b8-7000-455e-ac32-c7ad9fb1b64b\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qv8w7"
Aug 01 00:21:04.336179 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:04.336159    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a33756b8-7000-455e-ac32-c7ad9fb1b64b-dshm\") pod \"late-v1-kserve-56cbc8456c-qv8w7\" (UID: \"a33756b8-7000-455e-ac32-c7ad9fb1b64b\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qv8w7"
Aug 01 00:21:04.336377 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:04.336361    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a33756b8-7000-455e-ac32-c7ad9fb1b64b-tls-certs\") pod \"late-v1-kserve-56cbc8456c-qv8w7\" (UID: \"a33756b8-7000-455e-ac32-c7ad9fb1b64b\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qv8w7"
Aug 01 00:21:04.342924 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:04.342904    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2c9n\" (UniqueName: \"kubernetes.io/projected/a33756b8-7000-455e-ac32-c7ad9fb1b64b-kube-api-access-c2c9n\") pod \"late-v1-kserve-56cbc8456c-qv8w7\" (UID: \"a33756b8-7000-455e-ac32-c7ad9fb1b64b\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qv8w7"
Aug 01 00:21:04.375045 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:04.375025    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qv8w7"
Aug 01 00:21:04.504258 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:04.504231    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qv8w7"]
Aug 01 00:21:04.506239 ip-10-0-130-148 kubenswrapper[2559]: W0801 00:21:04.506212    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda33756b8_7000_455e_ac32_c7ad9fb1b64b.slice/crio-c1c9207b35b64ca305faf81812b365a9ae0bb264cda569cc606a3b56d5e182da WatchSource:0}: Error finding container c1c9207b35b64ca305faf81812b365a9ae0bb264cda569cc606a3b56d5e182da: Status 404 returned error can't find the container with id c1c9207b35b64ca305faf81812b365a9ae0bb264cda569cc606a3b56d5e182da
Aug 01 00:21:05.376329 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:05.376284    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qv8w7" event={"ID":"a33756b8-7000-455e-ac32-c7ad9fb1b64b","Type":"ContainerStarted","Data":"cdbfab136770c2b8b7434d402168411a0993714cab8368d58b62d1f3801e6761"}
Aug 01 00:21:05.376760 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:05.376350    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qv8w7" event={"ID":"a33756b8-7000-455e-ac32-c7ad9fb1b64b","Type":"ContainerStarted","Data":"c1c9207b35b64ca305faf81812b365a9ae0bb264cda569cc606a3b56d5e182da"}
Aug 01 00:21:05.393939 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:05.393881    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qv8w7" podStartSLOduration=1.393859343 podStartE2EDuration="1.393859343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:21:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:21:05.391951399 +0000 UTC m=+2128.111751246" watchObservedRunningTime="2026-08-01 00:21:05.393859343 +0000 UTC m=+2128.113659190"
Aug 01 00:21:06.375208 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:06.375168    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qv8w7"
Aug 01 00:21:14.375641 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:14.375597    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qv8w7"
Aug 01 00:21:14.387938 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:14.387909    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qv8w7"
Aug 01 00:21:14.425571 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:14.425529    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qv8w7"
Aug 01 00:21:51.520440 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:51.520409    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lcf4l"]
Aug 01 00:21:51.529452 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:21:51.529417    2559 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 01 00:21:51.529452 ip-10-0-130-148 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:21:51.529452 ip-10-0-130-148 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Aug 01 00:21:51.529452 ip-10-0-130-148 kubenswrapper[2559]:  > containerID="b06900a60f69b0b8cc70b947778a2e4453972adfe3cdf6861ed0a1a85cde6557" cmd=["/bin/sleep","15"]
Aug 01 00:21:51.529725 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:51.529471    2559 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lcf4l" message="" err=<
Aug 01 00:21:51.529725 ip-10-0-130-148 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:21:51.529725 ip-10-0-130-148 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Aug 01 00:21:51.529725 ip-10-0-130-148 kubenswrapper[2559]:  >
Aug 01 00:21:51.529725 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:21:51.529484    2559 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 01 00:21:51.529725 ip-10-0-130-148 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:21:51.529725 ip-10-0-130-148 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Aug 01 00:21:51.529725 ip-10-0-130-148 kubenswrapper[2559]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lcf4l" podUID="83abd782-637f-481b-9296-3bbfed12f803" containerName="main" containerID="cri-o://b06900a60f69b0b8cc70b947778a2e4453972adfe3cdf6861ed0a1a85cde6557"
Aug 01 00:21:51.529725 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:51.529526    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lcf4l" podUID="83abd782-637f-481b-9296-3bbfed12f803" containerName="main" containerID="cri-o://b06900a60f69b0b8cc70b947778a2e4453972adfe3cdf6861ed0a1a85cde6557" gracePeriod=60
Aug 01 00:21:51.875376 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:51.875352    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lcf4l"
Aug 01 00:21:51.922580 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:51.922540    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83abd782-637f-481b-9296-3bbfed12f803-tmp-dir\") pod \"83abd782-637f-481b-9296-3bbfed12f803\" (UID: \"83abd782-637f-481b-9296-3bbfed12f803\") "
Aug 01 00:21:51.922580 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:51.922577    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7tt9\" (UniqueName: \"kubernetes.io/projected/83abd782-637f-481b-9296-3bbfed12f803-kube-api-access-f7tt9\") pod \"83abd782-637f-481b-9296-3bbfed12f803\" (UID: \"83abd782-637f-481b-9296-3bbfed12f803\") "
Aug 01 00:21:51.922816 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:51.922613    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83abd782-637f-481b-9296-3bbfed12f803-home\") pod \"83abd782-637f-481b-9296-3bbfed12f803\" (UID: \"83abd782-637f-481b-9296-3bbfed12f803\") "
Aug 01 00:21:51.922816 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:51.922635    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83abd782-637f-481b-9296-3bbfed12f803-model-cache\") pod \"83abd782-637f-481b-9296-3bbfed12f803\" (UID: \"83abd782-637f-481b-9296-3bbfed12f803\") "
Aug 01 00:21:51.922816 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:51.922663    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83abd782-637f-481b-9296-3bbfed12f803-dshm\") pod \"83abd782-637f-481b-9296-3bbfed12f803\" (UID: \"83abd782-637f-481b-9296-3bbfed12f803\") "
Aug 01 00:21:51.922816 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:51.922690    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83abd782-637f-481b-9296-3bbfed12f803-tls-certs\") pod \"83abd782-637f-481b-9296-3bbfed12f803\" (UID: \"83abd782-637f-481b-9296-3bbfed12f803\") "
Aug 01 00:21:51.923086 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:51.922832    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83abd782-637f-481b-9296-3bbfed12f803-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "83abd782-637f-481b-9296-3bbfed12f803" (UID: "83abd782-637f-481b-9296-3bbfed12f803"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:21:51.923086 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:51.922914    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83abd782-637f-481b-9296-3bbfed12f803-home" (OuterVolumeSpecName: "home") pod "83abd782-637f-481b-9296-3bbfed12f803" (UID: "83abd782-637f-481b-9296-3bbfed12f803"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:21:51.923086 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:51.922930    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83abd782-637f-481b-9296-3bbfed12f803-model-cache" (OuterVolumeSpecName: "model-cache") pod "83abd782-637f-481b-9296-3bbfed12f803" (UID: "83abd782-637f-481b-9296-3bbfed12f803"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:21:51.923086 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:51.923005    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83abd782-637f-481b-9296-3bbfed12f803-home\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:21:51.923086 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:51.923023    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83abd782-637f-481b-9296-3bbfed12f803-model-cache\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:21:51.923086 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:51.923040    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83abd782-637f-481b-9296-3bbfed12f803-tmp-dir\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:21:51.925015 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:51.924985    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83abd782-637f-481b-9296-3bbfed12f803-dshm" (OuterVolumeSpecName: "dshm") pod "83abd782-637f-481b-9296-3bbfed12f803" (UID: "83abd782-637f-481b-9296-3bbfed12f803"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:21:51.925126 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:51.925028    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83abd782-637f-481b-9296-3bbfed12f803-kube-api-access-f7tt9" (OuterVolumeSpecName: "kube-api-access-f7tt9") pod "83abd782-637f-481b-9296-3bbfed12f803" (UID: "83abd782-637f-481b-9296-3bbfed12f803"). InnerVolumeSpecName "kube-api-access-f7tt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:21:51.925126 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:51.925064    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83abd782-637f-481b-9296-3bbfed12f803-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "83abd782-637f-481b-9296-3bbfed12f803" (UID: "83abd782-637f-481b-9296-3bbfed12f803"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:21:52.023706 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:52.023683    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83abd782-637f-481b-9296-3bbfed12f803-dshm\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:21:52.023706 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:52.023705    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83abd782-637f-481b-9296-3bbfed12f803-tls-certs\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:21:52.023880 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:52.023717    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7tt9\" (UniqueName: \"kubernetes.io/projected/83abd782-637f-481b-9296-3bbfed12f803-kube-api-access-f7tt9\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:21:52.583761 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:52.583714    2559 generic.go:356] "Generic (PLEG): container finished" podID="83abd782-637f-481b-9296-3bbfed12f803" containerID="b06900a60f69b0b8cc70b947778a2e4453972adfe3cdf6861ed0a1a85cde6557" exitCode=0
Aug 01 00:21:52.584250 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:52.583776    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lcf4l"
Aug 01 00:21:52.584250 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:52.583780    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lcf4l" event={"ID":"83abd782-637f-481b-9296-3bbfed12f803","Type":"ContainerDied","Data":"b06900a60f69b0b8cc70b947778a2e4453972adfe3cdf6861ed0a1a85cde6557"}
Aug 01 00:21:52.584250 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:52.583835    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lcf4l" event={"ID":"83abd782-637f-481b-9296-3bbfed12f803","Type":"ContainerDied","Data":"b37045d50c5522106b3aab0859b27179e714090092e09cb1145971ee4999152f"}
Aug 01 00:21:52.584250 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:52.583864    2559 scope.go:117] "RemoveContainer" containerID="b06900a60f69b0b8cc70b947778a2e4453972adfe3cdf6861ed0a1a85cde6557"
Aug 01 00:21:52.592877 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:52.592858    2559 scope.go:117] "RemoveContainer" containerID="b06900a60f69b0b8cc70b947778a2e4453972adfe3cdf6861ed0a1a85cde6557"
Aug 01 00:21:52.593096 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:21:52.593078    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b06900a60f69b0b8cc70b947778a2e4453972adfe3cdf6861ed0a1a85cde6557\": container with ID starting with b06900a60f69b0b8cc70b947778a2e4453972adfe3cdf6861ed0a1a85cde6557 not found: ID does not exist" containerID="b06900a60f69b0b8cc70b947778a2e4453972adfe3cdf6861ed0a1a85cde6557"
Aug 01 00:21:52.593153 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:52.593105    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b06900a60f69b0b8cc70b947778a2e4453972adfe3cdf6861ed0a1a85cde6557"} err="failed to get container status \"b06900a60f69b0b8cc70b947778a2e4453972adfe3cdf6861ed0a1a85cde6557\": rpc error: code = NotFound desc = could not find container \"b06900a60f69b0b8cc70b947778a2e4453972adfe3cdf6861ed0a1a85cde6557\": container with ID starting with b06900a60f69b0b8cc70b947778a2e4453972adfe3cdf6861ed0a1a85cde6557 not found: ID does not exist"
Aug 01 00:21:52.605317 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:52.605289    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lcf4l"]
Aug 01 00:21:52.609605 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:52.609585    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lcf4l"]
Aug 01 00:21:53.785133 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:21:53.785103    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83abd782-637f-481b-9296-3bbfed12f803" path="/var/lib/kubelet/pods/83abd782-637f-481b-9296-3bbfed12f803/volumes"
Aug 01 00:22:19.901813 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:19.901774    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24"]
Aug 01 00:22:19.902251 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:19.902089    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83abd782-637f-481b-9296-3bbfed12f803" containerName="main"
Aug 01 00:22:19.902251 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:19.902100    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="83abd782-637f-481b-9296-3bbfed12f803" containerName="main"
Aug 01 00:22:19.902251 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:19.902175    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="83abd782-637f-481b-9296-3bbfed12f803" containerName="main"
Aug 01 00:22:19.905044 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:19.905027    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24"
Aug 01 00:22:19.907731 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:19.907704    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Aug 01 00:22:19.907731 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:19.907714    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"seaweedfs-s3-creds\""
Aug 01 00:22:19.908056 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:19.907731    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"odh-kserve-custom-ca-bundle\""
Aug 01 00:22:19.908056 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:19.907803    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-ndrmn\""
Aug 01 00:22:19.908056 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:19.907842    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Aug 01 00:22:19.908195 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:19.908128    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Aug 01 00:22:19.915387 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:19.915363    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24"]
Aug 01 00:22:20.052728 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:20.052683    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/103bd86c-3f7e-4eb0-988f-4639d27f7e32-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24\" (UID: \"103bd86c-3f7e-4eb0-988f-4639d27f7e32\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24"
Aug 01 00:22:20.052933 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:20.052766    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/103bd86c-3f7e-4eb0-988f-4639d27f7e32-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24\" (UID: \"103bd86c-3f7e-4eb0-988f-4639d27f7e32\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24"
Aug 01 00:22:20.052933 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:20.052796    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/103bd86c-3f7e-4eb0-988f-4639d27f7e32-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24\" (UID: \"103bd86c-3f7e-4eb0-988f-4639d27f7e32\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24"
Aug 01 00:22:20.052933 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:20.052828    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/103bd86c-3f7e-4eb0-988f-4639d27f7e32-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24\" (UID: \"103bd86c-3f7e-4eb0-988f-4639d27f7e32\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24"
Aug 01 00:22:20.052933 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:20.052904    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/103bd86c-3f7e-4eb0-988f-4639d27f7e32-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24\" (UID: \"103bd86c-3f7e-4eb0-988f-4639d27f7e32\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24"
Aug 01 00:22:20.053079 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:20.052938    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/103bd86c-3f7e-4eb0-988f-4639d27f7e32-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24\" (UID: \"103bd86c-3f7e-4eb0-988f-4639d27f7e32\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24"
Aug 01 00:22:20.053079 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:20.052959    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hf6q\" (UniqueName: \"kubernetes.io/projected/103bd86c-3f7e-4eb0-988f-4639d27f7e32-kube-api-access-5hf6q\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24\" (UID: \"103bd86c-3f7e-4eb0-988f-4639d27f7e32\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24"
Aug 01 00:22:20.053079 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:20.053020    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/103bd86c-3f7e-4eb0-988f-4639d27f7e32-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24\" (UID: \"103bd86c-3f7e-4eb0-988f-4639d27f7e32\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24"
Aug 01 00:22:20.153946 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:20.153866    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/103bd86c-3f7e-4eb0-988f-4639d27f7e32-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24\" (UID: \"103bd86c-3f7e-4eb0-988f-4639d27f7e32\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24"
Aug 01 00:22:20.153946 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:20.153907    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/103bd86c-3f7e-4eb0-988f-4639d27f7e32-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24\" (UID: \"103bd86c-3f7e-4eb0-988f-4639d27f7e32\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24"
Aug 01 00:22:20.153946 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:20.153929    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/103bd86c-3f7e-4eb0-988f-4639d27f7e32-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24\" (UID: \"103bd86c-3f7e-4eb0-988f-4639d27f7e32\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24"
Aug 01 00:22:20.154218 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:20.153952    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hf6q\" (UniqueName: \"kubernetes.io/projected/103bd86c-3f7e-4eb0-988f-4639d27f7e32-kube-api-access-5hf6q\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24\" (UID: \"103bd86c-3f7e-4eb0-988f-4639d27f7e32\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24"
Aug 01 00:22:20.154218 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:20.154009    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/103bd86c-3f7e-4eb0-988f-4639d27f7e32-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24\" (UID: \"103bd86c-3f7e-4eb0-988f-4639d27f7e32\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24"
Aug 01 00:22:20.154218 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:20.154038    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/103bd86c-3f7e-4eb0-988f-4639d27f7e32-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24\" (UID: \"103bd86c-3f7e-4eb0-988f-4639d27f7e32\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24"
Aug 01 00:22:20.154218 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:20.154075    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/103bd86c-3f7e-4eb0-988f-4639d27f7e32-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24\" (UID: \"103bd86c-3f7e-4eb0-988f-4639d27f7e32\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24"
Aug 01 00:22:20.154218 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:20.154108    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/103bd86c-3f7e-4eb0-988f-4639d27f7e32-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24\" (UID: \"103bd86c-3f7e-4eb0-988f-4639d27f7e32\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24"
Aug 01 00:22:20.154500 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:20.154325    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/103bd86c-3f7e-4eb0-988f-4639d27f7e32-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24\" (UID: \"103bd86c-3f7e-4eb0-988f-4639d27f7e32\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24"
Aug 01 00:22:20.154500 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:20.154375    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/103bd86c-3f7e-4eb0-988f-4639d27f7e32-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24\" (UID: \"103bd86c-3f7e-4eb0-988f-4639d27f7e32\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24"
Aug 01 00:22:20.154500 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:20.154414    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/103bd86c-3f7e-4eb0-988f-4639d27f7e32-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24\" (UID: \"103bd86c-3f7e-4eb0-988f-4639d27f7e32\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24"
Aug 01 00:22:20.154500 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:20.154463    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/103bd86c-3f7e-4eb0-988f-4639d27f7e32-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24\" (UID: \"103bd86c-3f7e-4eb0-988f-4639d27f7e32\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24"
Aug 01 00:22:20.155008 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:20.154987    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/103bd86c-3f7e-4eb0-988f-4639d27f7e32-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24\" (UID: \"103bd86c-3f7e-4eb0-988f-4639d27f7e32\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24"
Aug 01 00:22:20.156379 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:20.156356    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/103bd86c-3f7e-4eb0-988f-4639d27f7e32-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24\" (UID: \"103bd86c-3f7e-4eb0-988f-4639d27f7e32\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24"
Aug 01 00:22:20.156534 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:20.156517    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/103bd86c-3f7e-4eb0-988f-4639d27f7e32-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24\" (UID: \"103bd86c-3f7e-4eb0-988f-4639d27f7e32\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24"
Aug 01 00:22:20.162033 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:20.162016    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hf6q\" (UniqueName: \"kubernetes.io/projected/103bd86c-3f7e-4eb0-988f-4639d27f7e32-kube-api-access-5hf6q\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24\" (UID: \"103bd86c-3f7e-4eb0-988f-4639d27f7e32\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24"
Aug 01 00:22:20.215998 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:20.215969    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24"
Aug 01 00:22:20.350512 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:20.350486    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24"]
Aug 01 00:22:20.352668 ip-10-0-130-148 kubenswrapper[2559]: W0801 00:22:20.352635    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod103bd86c_3f7e_4eb0_988f_4639d27f7e32.slice/crio-d9cde9bede74d8dd5a0bdbd885435afa7435523a949019388d39ad3cc8022156 WatchSource:0}: Error finding container d9cde9bede74d8dd5a0bdbd885435afa7435523a949019388d39ad3cc8022156: Status 404 returned error can't find the container with id d9cde9bede74d8dd5a0bdbd885435afa7435523a949019388d39ad3cc8022156
Aug 01 00:22:20.711781 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:20.711730    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24" event={"ID":"103bd86c-3f7e-4eb0-988f-4639d27f7e32","Type":"ContainerStarted","Data":"7b94054c608127be878c1b75aad9eef3259f9420573dab44b3757fc97d25802e"}
Aug 01 00:22:20.711963 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:20.711799    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24" event={"ID":"103bd86c-3f7e-4eb0-988f-4639d27f7e32","Type":"ContainerStarted","Data":"d9cde9bede74d8dd5a0bdbd885435afa7435523a949019388d39ad3cc8022156"}
Aug 01 00:22:23.727228 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:23.727196    2559 generic.go:356] "Generic (PLEG): container finished" podID="103bd86c-3f7e-4eb0-988f-4639d27f7e32" containerID="7b94054c608127be878c1b75aad9eef3259f9420573dab44b3757fc97d25802e" exitCode=0
Aug 01 00:22:23.727600 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:23.727269    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24" event={"ID":"103bd86c-3f7e-4eb0-988f-4639d27f7e32","Type":"ContainerDied","Data":"7b94054c608127be878c1b75aad9eef3259f9420573dab44b3757fc97d25802e"}
Aug 01 00:22:24.733575 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:24.733535    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24" event={"ID":"103bd86c-3f7e-4eb0-988f-4639d27f7e32","Type":"ContainerStarted","Data":"d58ca0b8ecae1916ed5f04cf89fa6b601ec520df6bbbbfe4a765b0dc89eaf65e"}
Aug 01 00:22:24.753583 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:24.753533    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24" podStartSLOduration=3.165763783 podStartE2EDuration="5.753497561s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.587733778s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:22:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:22:24.752308788 +0000 UTC m=+2207.472108634" watchObservedRunningTime="2026-08-01 00:22:24.753497561 +0000 UTC m=+2207.473297407"
Aug 01 00:22:25.216801 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:25.216770    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24"
Aug 01 00:22:30.216934 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:30.216899    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24"
Aug 01 00:22:30.218567 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:30.218540    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24" podUID="103bd86c-3f7e-4eb0-988f-4639d27f7e32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Aug 01 00:22:36.245184 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:36.245148    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qv8w7"]
Aug 01 00:22:40.216492 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:40.216441    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24" podUID="103bd86c-3f7e-4eb0-988f-4639d27f7e32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Aug 01 00:22:50.217263 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:50.217215    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24" podUID="103bd86c-3f7e-4eb0-988f-4639d27f7e32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Aug 01 00:22:51.263970 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:51.263931    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qv8w7" podUID="a33756b8-7000-455e-ac32-c7ad9fb1b64b" containerName="main" containerID="cri-o://cdbfab136770c2b8b7434d402168411a0993714cab8368d58b62d1f3801e6761" gracePeriod=45
Aug 01 00:22:51.415388 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:51.415340    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qv8w7" podUID="a33756b8-7000-455e-ac32-c7ad9fb1b64b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Aug 01 00:22:51.514283 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:51.514219    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qv8w7"
Aug 01 00:22:51.617065 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:51.617034    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a33756b8-7000-455e-ac32-c7ad9fb1b64b-model-cache\") pod \"a33756b8-7000-455e-ac32-c7ad9fb1b64b\" (UID: \"a33756b8-7000-455e-ac32-c7ad9fb1b64b\") "
Aug 01 00:22:51.617272 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:51.617087    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c2c9n\" (UniqueName: \"kubernetes.io/projected/a33756b8-7000-455e-ac32-c7ad9fb1b64b-kube-api-access-c2c9n\") pod \"a33756b8-7000-455e-ac32-c7ad9fb1b64b\" (UID: \"a33756b8-7000-455e-ac32-c7ad9fb1b64b\") "
Aug 01 00:22:51.617272 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:51.617142    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a33756b8-7000-455e-ac32-c7ad9fb1b64b-dshm\") pod \"a33756b8-7000-455e-ac32-c7ad9fb1b64b\" (UID: \"a33756b8-7000-455e-ac32-c7ad9fb1b64b\") "
Aug 01 00:22:51.617272 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:51.617172    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a33756b8-7000-455e-ac32-c7ad9fb1b64b-tmp-dir\") pod \"a33756b8-7000-455e-ac32-c7ad9fb1b64b\" (UID: \"a33756b8-7000-455e-ac32-c7ad9fb1b64b\") "
Aug 01 00:22:51.617272 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:51.617200    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a33756b8-7000-455e-ac32-c7ad9fb1b64b-home\") pod \"a33756b8-7000-455e-ac32-c7ad9fb1b64b\" (UID: \"a33756b8-7000-455e-ac32-c7ad9fb1b64b\") "
Aug 01 00:22:51.617272 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:51.617261    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a33756b8-7000-455e-ac32-c7ad9fb1b64b-tls-certs\") pod \"a33756b8-7000-455e-ac32-c7ad9fb1b64b\" (UID: \"a33756b8-7000-455e-ac32-c7ad9fb1b64b\") "
Aug 01 00:22:51.617570 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:51.617340    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a33756b8-7000-455e-ac32-c7ad9fb1b64b-model-cache" (OuterVolumeSpecName: "model-cache") pod "a33756b8-7000-455e-ac32-c7ad9fb1b64b" (UID: "a33756b8-7000-455e-ac32-c7ad9fb1b64b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:22:51.617570 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:51.617440    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a33756b8-7000-455e-ac32-c7ad9fb1b64b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a33756b8-7000-455e-ac32-c7ad9fb1b64b" (UID: "a33756b8-7000-455e-ac32-c7ad9fb1b64b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:22:51.617570 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:51.617454    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a33756b8-7000-455e-ac32-c7ad9fb1b64b-home" (OuterVolumeSpecName: "home") pod "a33756b8-7000-455e-ac32-c7ad9fb1b64b" (UID: "a33756b8-7000-455e-ac32-c7ad9fb1b64b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:22:51.617570 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:51.617530    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a33756b8-7000-455e-ac32-c7ad9fb1b64b-model-cache\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:22:51.617570 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:51.617549    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a33756b8-7000-455e-ac32-c7ad9fb1b64b-tmp-dir\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:22:51.617570 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:51.617567    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a33756b8-7000-455e-ac32-c7ad9fb1b64b-home\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:22:51.619468 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:51.619435    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a33756b8-7000-455e-ac32-c7ad9fb1b64b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a33756b8-7000-455e-ac32-c7ad9fb1b64b" (UID: "a33756b8-7000-455e-ac32-c7ad9fb1b64b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:22:51.619572 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:51.619478    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a33756b8-7000-455e-ac32-c7ad9fb1b64b-dshm" (OuterVolumeSpecName: "dshm") pod "a33756b8-7000-455e-ac32-c7ad9fb1b64b" (UID: "a33756b8-7000-455e-ac32-c7ad9fb1b64b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:22:51.619572 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:51.619496    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a33756b8-7000-455e-ac32-c7ad9fb1b64b-kube-api-access-c2c9n" (OuterVolumeSpecName: "kube-api-access-c2c9n") pod "a33756b8-7000-455e-ac32-c7ad9fb1b64b" (UID: "a33756b8-7000-455e-ac32-c7ad9fb1b64b"). InnerVolumeSpecName "kube-api-access-c2c9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:22:51.718948 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:51.718916    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a33756b8-7000-455e-ac32-c7ad9fb1b64b-tls-certs\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:22:51.718948 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:51.718944    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2c9n\" (UniqueName: \"kubernetes.io/projected/a33756b8-7000-455e-ac32-c7ad9fb1b64b-kube-api-access-c2c9n\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:22:51.718948 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:51.718954    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a33756b8-7000-455e-ac32-c7ad9fb1b64b-dshm\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:22:51.862956 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:51.862918    2559 generic.go:356] "Generic (PLEG): container finished" podID="a33756b8-7000-455e-ac32-c7ad9fb1b64b" containerID="cdbfab136770c2b8b7434d402168411a0993714cab8368d58b62d1f3801e6761" exitCode=0
Aug 01 00:22:51.863147 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:51.862971    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qv8w7"
Aug 01 00:22:51.863147 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:51.862992    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qv8w7" event={"ID":"a33756b8-7000-455e-ac32-c7ad9fb1b64b","Type":"ContainerDied","Data":"cdbfab136770c2b8b7434d402168411a0993714cab8368d58b62d1f3801e6761"}
Aug 01 00:22:51.863147 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:51.863035    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qv8w7" event={"ID":"a33756b8-7000-455e-ac32-c7ad9fb1b64b","Type":"ContainerDied","Data":"c1c9207b35b64ca305faf81812b365a9ae0bb264cda569cc606a3b56d5e182da"}
Aug 01 00:22:51.863147 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:51.863055    2559 scope.go:117] "RemoveContainer" containerID="cdbfab136770c2b8b7434d402168411a0993714cab8368d58b62d1f3801e6761"
Aug 01 00:22:51.873587 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:51.873521    2559 scope.go:117] "RemoveContainer" containerID="cdbfab136770c2b8b7434d402168411a0993714cab8368d58b62d1f3801e6761"
Aug 01 00:22:51.874065 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:22:51.874029    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdbfab136770c2b8b7434d402168411a0993714cab8368d58b62d1f3801e6761\": container with ID starting with cdbfab136770c2b8b7434d402168411a0993714cab8368d58b62d1f3801e6761 not found: ID does not exist" containerID="cdbfab136770c2b8b7434d402168411a0993714cab8368d58b62d1f3801e6761"
Aug 01 00:22:51.874187 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:51.874077    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdbfab136770c2b8b7434d402168411a0993714cab8368d58b62d1f3801e6761"} err="failed to get container status \"cdbfab136770c2b8b7434d402168411a0993714cab8368d58b62d1f3801e6761\": rpc error: code = NotFound desc = could not find container \"cdbfab136770c2b8b7434d402168411a0993714cab8368d58b62d1f3801e6761\": container with ID starting with cdbfab136770c2b8b7434d402168411a0993714cab8368d58b62d1f3801e6761 not found: ID does not exist"
Aug 01 00:22:51.878391 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:51.878367    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qv8w7"]
Aug 01 00:22:51.880631 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:51.880610    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qv8w7"]
Aug 01 00:22:53.785642 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:22:53.785607    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a33756b8-7000-455e-ac32-c7ad9fb1b64b" path="/var/lib/kubelet/pods/a33756b8-7000-455e-ac32-c7ad9fb1b64b/volumes"
Aug 01 00:23:00.216704 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:23:00.216611    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24" podUID="103bd86c-3f7e-4eb0-988f-4639d27f7e32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Aug 01 00:23:10.216756 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:23:10.216697    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24" podUID="103bd86c-3f7e-4eb0-988f-4639d27f7e32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Aug 01 00:23:20.216581 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:23:20.216533    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24" podUID="103bd86c-3f7e-4eb0-988f-4639d27f7e32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Aug 01 00:23:30.216632 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:23:30.216578    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24" podUID="103bd86c-3f7e-4eb0-988f-4639d27f7e32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Aug 01 00:23:40.226498 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:23:40.226466    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24"
Aug 01 00:23:40.234389 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:23:40.234364    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24"
Aug 01 00:24:23.303678 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:23.303645    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24"]
Aug 01 00:24:35.559531 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:35.559495    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-nkqrz"]
Aug 01 00:24:35.559933 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:35.559838    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a33756b8-7000-455e-ac32-c7ad9fb1b64b" containerName="main"
Aug 01 00:24:35.559933 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:35.559850    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a33756b8-7000-455e-ac32-c7ad9fb1b64b" containerName="main"
Aug 01 00:24:35.559933 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:35.559912    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a33756b8-7000-455e-ac32-c7ad9fb1b64b" containerName="main"
Aug 01 00:24:35.562624 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:35.562605    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-nkqrz"
Aug 01 00:24:35.566571 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:35.566546    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"kube-root-ca.crt\""
Aug 01 00:24:35.566712 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:35.566587    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"default-dockercfg-w467j\""
Aug 01 00:24:35.566712 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:35.566636    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"roll-v1-kserve-self-signed-certs\""
Aug 01 00:24:35.566712 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:35.566672    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"openshift-service-ca.crt\""
Aug 01 00:24:35.572790 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:35.572766    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-nkqrz"]
Aug 01 00:24:35.595137 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:35.595100    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfbkb\" (UniqueName: \"kubernetes.io/projected/572567b0-9795-4702-94ad-70598bd196cb-kube-api-access-lfbkb\") pod \"roll-v1-kserve-69cf8bcfc-nkqrz\" (UID: \"572567b0-9795-4702-94ad-70598bd196cb\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-nkqrz"
Aug 01 00:24:35.595346 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:35.595142    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/572567b0-9795-4702-94ad-70598bd196cb-dshm\") pod \"roll-v1-kserve-69cf8bcfc-nkqrz\" (UID: \"572567b0-9795-4702-94ad-70598bd196cb\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-nkqrz"
Aug 01 00:24:35.595346 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:35.595164    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/572567b0-9795-4702-94ad-70598bd196cb-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-nkqrz\" (UID: \"572567b0-9795-4702-94ad-70598bd196cb\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-nkqrz"
Aug 01 00:24:35.595346 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:35.595284    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/572567b0-9795-4702-94ad-70598bd196cb-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-nkqrz\" (UID: \"572567b0-9795-4702-94ad-70598bd196cb\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-nkqrz"
Aug 01 00:24:35.595346 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:35.595335    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/572567b0-9795-4702-94ad-70598bd196cb-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-nkqrz\" (UID: \"572567b0-9795-4702-94ad-70598bd196cb\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-nkqrz"
Aug 01 00:24:35.595521 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:35.595362    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/572567b0-9795-4702-94ad-70598bd196cb-home\") pod \"roll-v1-kserve-69cf8bcfc-nkqrz\" (UID: \"572567b0-9795-4702-94ad-70598bd196cb\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-nkqrz"
Aug 01 00:24:35.696257 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:35.696218    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/572567b0-9795-4702-94ad-70598bd196cb-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-nkqrz\" (UID: \"572567b0-9795-4702-94ad-70598bd196cb\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-nkqrz"
Aug 01 00:24:35.696257 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:35.696262    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/572567b0-9795-4702-94ad-70598bd196cb-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-nkqrz\" (UID: \"572567b0-9795-4702-94ad-70598bd196cb\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-nkqrz"
Aug 01 00:24:35.696502 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:35.696282    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/572567b0-9795-4702-94ad-70598bd196cb-home\") pod \"roll-v1-kserve-69cf8bcfc-nkqrz\" (UID: \"572567b0-9795-4702-94ad-70598bd196cb\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-nkqrz"
Aug 01 00:24:35.696502 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:35.696310    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfbkb\" (UniqueName: \"kubernetes.io/projected/572567b0-9795-4702-94ad-70598bd196cb-kube-api-access-lfbkb\") pod \"roll-v1-kserve-69cf8bcfc-nkqrz\" (UID: \"572567b0-9795-4702-94ad-70598bd196cb\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-nkqrz"
Aug 01 00:24:35.696502 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:35.696333    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/572567b0-9795-4702-94ad-70598bd196cb-dshm\") pod \"roll-v1-kserve-69cf8bcfc-nkqrz\" (UID: \"572567b0-9795-4702-94ad-70598bd196cb\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-nkqrz"
Aug 01 00:24:35.696502 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:35.696347    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/572567b0-9795-4702-94ad-70598bd196cb-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-nkqrz\" (UID: \"572567b0-9795-4702-94ad-70598bd196cb\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-nkqrz"
Aug 01 00:24:35.696721 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:35.696693    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/572567b0-9795-4702-94ad-70598bd196cb-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-nkqrz\" (UID: \"572567b0-9795-4702-94ad-70598bd196cb\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-nkqrz"
Aug 01 00:24:35.696833 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:35.696786    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/572567b0-9795-4702-94ad-70598bd196cb-home\") pod \"roll-v1-kserve-69cf8bcfc-nkqrz\" (UID: \"572567b0-9795-4702-94ad-70598bd196cb\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-nkqrz"
Aug 01 00:24:35.696891 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:35.696866    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/572567b0-9795-4702-94ad-70598bd196cb-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-nkqrz\" (UID: \"572567b0-9795-4702-94ad-70598bd196cb\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-nkqrz"
Aug 01 00:24:35.698707 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:35.698684    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/572567b0-9795-4702-94ad-70598bd196cb-dshm\") pod \"roll-v1-kserve-69cf8bcfc-nkqrz\" (UID: \"572567b0-9795-4702-94ad-70598bd196cb\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-nkqrz"
Aug 01 00:24:35.698867 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:35.698848    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/572567b0-9795-4702-94ad-70598bd196cb-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-nkqrz\" (UID: \"572567b0-9795-4702-94ad-70598bd196cb\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-nkqrz"
Aug 01 00:24:35.705343 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:35.705307    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfbkb\" (UniqueName: \"kubernetes.io/projected/572567b0-9795-4702-94ad-70598bd196cb-kube-api-access-lfbkb\") pod \"roll-v1-kserve-69cf8bcfc-nkqrz\" (UID: \"572567b0-9795-4702-94ad-70598bd196cb\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-nkqrz"
Aug 01 00:24:35.873455 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:35.873353    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-nkqrz"
Aug 01 00:24:36.002322 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:36.002292    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-nkqrz"]
Aug 01 00:24:36.004542 ip-10-0-130-148 kubenswrapper[2559]: W0801 00:24:36.004512    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod572567b0_9795_4702_94ad_70598bd196cb.slice/crio-bf4d8b45d36d822ecdbad0bbf0de30836c522a7148d7fe6dc029f2eb11214bd1 WatchSource:0}: Error finding container bf4d8b45d36d822ecdbad0bbf0de30836c522a7148d7fe6dc029f2eb11214bd1: Status 404 returned error can't find the container with id bf4d8b45d36d822ecdbad0bbf0de30836c522a7148d7fe6dc029f2eb11214bd1
Aug 01 00:24:36.357647 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:36.357606    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-nkqrz" event={"ID":"572567b0-9795-4702-94ad-70598bd196cb","Type":"ContainerStarted","Data":"25289ca7c7b0b3cf554026d7b7e8e239bfc3cbcb3d96db81076e9354b44f7699"}
Aug 01 00:24:36.357849 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:36.357673    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-nkqrz" event={"ID":"572567b0-9795-4702-94ad-70598bd196cb","Type":"ContainerStarted","Data":"bf4d8b45d36d822ecdbad0bbf0de30836c522a7148d7fe6dc029f2eb11214bd1"}
Aug 01 00:24:36.377021 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:36.376971    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-nkqrz" podStartSLOduration=1.376955139 podStartE2EDuration="1.376955139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:24:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:24:36.374963847 +0000 UTC m=+2339.094763706" watchObservedRunningTime="2026-08-01 00:24:36.376955139 +0000 UTC m=+2339.096754985"
Aug 01 00:24:36.874336 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:36.874296    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-nkqrz"
Aug 01 00:24:38.326250 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:38.326198    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24" podUID="103bd86c-3f7e-4eb0-988f-4639d27f7e32" containerName="main" containerID="cri-o://d58ca0b8ecae1916ed5f04cf89fa6b601ec520df6bbbbfe4a765b0dc89eaf65e" gracePeriod=45
Aug 01 00:24:38.981708 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:38.981682    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24"
Aug 01 00:24:39.023254 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:39.023159    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/103bd86c-3f7e-4eb0-988f-4639d27f7e32-kserve-provision-location\") pod \"103bd86c-3f7e-4eb0-988f-4639d27f7e32\" (UID: \"103bd86c-3f7e-4eb0-988f-4639d27f7e32\") "
Aug 01 00:24:39.023254 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:39.023203    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/103bd86c-3f7e-4eb0-988f-4639d27f7e32-dshm\") pod \"103bd86c-3f7e-4eb0-988f-4639d27f7e32\" (UID: \"103bd86c-3f7e-4eb0-988f-4639d27f7e32\") "
Aug 01 00:24:39.023254 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:39.023226    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/103bd86c-3f7e-4eb0-988f-4639d27f7e32-cabundle-cert\") pod \"103bd86c-3f7e-4eb0-988f-4639d27f7e32\" (UID: \"103bd86c-3f7e-4eb0-988f-4639d27f7e32\") "
Aug 01 00:24:39.023254 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:39.023243    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5hf6q\" (UniqueName: \"kubernetes.io/projected/103bd86c-3f7e-4eb0-988f-4639d27f7e32-kube-api-access-5hf6q\") pod \"103bd86c-3f7e-4eb0-988f-4639d27f7e32\" (UID: \"103bd86c-3f7e-4eb0-988f-4639d27f7e32\") "
Aug 01 00:24:39.023853 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:39.023287    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/103bd86c-3f7e-4eb0-988f-4639d27f7e32-tmp-dir\") pod \"103bd86c-3f7e-4eb0-988f-4639d27f7e32\" (UID: \"103bd86c-3f7e-4eb0-988f-4639d27f7e32\") "
Aug 01 00:24:39.023853 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:39.023311    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/103bd86c-3f7e-4eb0-988f-4639d27f7e32-home\") pod \"103bd86c-3f7e-4eb0-988f-4639d27f7e32\" (UID: \"103bd86c-3f7e-4eb0-988f-4639d27f7e32\") "
Aug 01 00:24:39.023853 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:39.023347    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/103bd86c-3f7e-4eb0-988f-4639d27f7e32-tls-certs\") pod \"103bd86c-3f7e-4eb0-988f-4639d27f7e32\" (UID: \"103bd86c-3f7e-4eb0-988f-4639d27f7e32\") "
Aug 01 00:24:39.023853 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:39.023546    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/103bd86c-3f7e-4eb0-988f-4639d27f7e32-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "103bd86c-3f7e-4eb0-988f-4639d27f7e32" (UID: "103bd86c-3f7e-4eb0-988f-4639d27f7e32"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:24:39.023853 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:39.023604    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/103bd86c-3f7e-4eb0-988f-4639d27f7e32-model-cache\") pod \"103bd86c-3f7e-4eb0-988f-4639d27f7e32\" (UID: \"103bd86c-3f7e-4eb0-988f-4639d27f7e32\") "
Aug 01 00:24:39.024117 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:39.023863    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103bd86c-3f7e-4eb0-988f-4639d27f7e32-home" (OuterVolumeSpecName: "home") pod "103bd86c-3f7e-4eb0-988f-4639d27f7e32" (UID: "103bd86c-3f7e-4eb0-988f-4639d27f7e32"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:24:39.024117 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:39.023963    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/103bd86c-3f7e-4eb0-988f-4639d27f7e32-home\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:24:39.024117 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:39.023982    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/103bd86c-3f7e-4eb0-988f-4639d27f7e32-cabundle-cert\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:24:39.024117 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:39.024059    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103bd86c-3f7e-4eb0-988f-4639d27f7e32-model-cache" (OuterVolumeSpecName: "model-cache") pod "103bd86c-3f7e-4eb0-988f-4639d27f7e32" (UID: "103bd86c-3f7e-4eb0-988f-4639d27f7e32"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:24:39.026276 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:39.026215    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/103bd86c-3f7e-4eb0-988f-4639d27f7e32-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "103bd86c-3f7e-4eb0-988f-4639d27f7e32" (UID: "103bd86c-3f7e-4eb0-988f-4639d27f7e32"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:24:39.026411 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:39.026337    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/103bd86c-3f7e-4eb0-988f-4639d27f7e32-kube-api-access-5hf6q" (OuterVolumeSpecName: "kube-api-access-5hf6q") pod "103bd86c-3f7e-4eb0-988f-4639d27f7e32" (UID: "103bd86c-3f7e-4eb0-988f-4639d27f7e32"). InnerVolumeSpecName "kube-api-access-5hf6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:24:39.027456 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:39.027436    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103bd86c-3f7e-4eb0-988f-4639d27f7e32-dshm" (OuterVolumeSpecName: "dshm") pod "103bd86c-3f7e-4eb0-988f-4639d27f7e32" (UID: "103bd86c-3f7e-4eb0-988f-4639d27f7e32"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:24:39.041260 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:39.041221    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103bd86c-3f7e-4eb0-988f-4639d27f7e32-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "103bd86c-3f7e-4eb0-988f-4639d27f7e32" (UID: "103bd86c-3f7e-4eb0-988f-4639d27f7e32"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:24:39.053120 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:39.053083    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103bd86c-3f7e-4eb0-988f-4639d27f7e32-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "103bd86c-3f7e-4eb0-988f-4639d27f7e32" (UID: "103bd86c-3f7e-4eb0-988f-4639d27f7e32"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:24:39.124827 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:39.124790    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/103bd86c-3f7e-4eb0-988f-4639d27f7e32-kserve-provision-location\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:24:39.124827 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:39.124823    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/103bd86c-3f7e-4eb0-988f-4639d27f7e32-dshm\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:24:39.125025 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:39.124837    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hf6q\" (UniqueName: \"kubernetes.io/projected/103bd86c-3f7e-4eb0-988f-4639d27f7e32-kube-api-access-5hf6q\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:24:39.125025 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:39.124849    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/103bd86c-3f7e-4eb0-988f-4639d27f7e32-tmp-dir\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:24:39.125025 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:39.124858    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/103bd86c-3f7e-4eb0-988f-4639d27f7e32-tls-certs\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:24:39.125025 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:39.124866    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/103bd86c-3f7e-4eb0-988f-4639d27f7e32-model-cache\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:24:39.371212 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:39.371124    2559 generic.go:356] "Generic (PLEG): container finished" podID="103bd86c-3f7e-4eb0-988f-4639d27f7e32" containerID="d58ca0b8ecae1916ed5f04cf89fa6b601ec520df6bbbbfe4a765b0dc89eaf65e" exitCode=0
Aug 01 00:24:39.371212 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:39.371176    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24" event={"ID":"103bd86c-3f7e-4eb0-988f-4639d27f7e32","Type":"ContainerDied","Data":"d58ca0b8ecae1916ed5f04cf89fa6b601ec520df6bbbbfe4a765b0dc89eaf65e"}
Aug 01 00:24:39.371708 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:39.371215    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24" event={"ID":"103bd86c-3f7e-4eb0-988f-4639d27f7e32","Type":"ContainerDied","Data":"d9cde9bede74d8dd5a0bdbd885435afa7435523a949019388d39ad3cc8022156"}
Aug 01 00:24:39.371708 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:39.371236    2559 scope.go:117] "RemoveContainer" containerID="d58ca0b8ecae1916ed5f04cf89fa6b601ec520df6bbbbfe4a765b0dc89eaf65e"
Aug 01 00:24:39.371708 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:39.371178    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24"
Aug 01 00:24:39.380248 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:39.380227    2559 scope.go:117] "RemoveContainer" containerID="7b94054c608127be878c1b75aad9eef3259f9420573dab44b3757fc97d25802e"
Aug 01 00:24:39.388657 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:39.388636    2559 scope.go:117] "RemoveContainer" containerID="d58ca0b8ecae1916ed5f04cf89fa6b601ec520df6bbbbfe4a765b0dc89eaf65e"
Aug 01 00:24:39.389007 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:24:39.388986    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d58ca0b8ecae1916ed5f04cf89fa6b601ec520df6bbbbfe4a765b0dc89eaf65e\": container with ID starting with d58ca0b8ecae1916ed5f04cf89fa6b601ec520df6bbbbfe4a765b0dc89eaf65e not found: ID does not exist" containerID="d58ca0b8ecae1916ed5f04cf89fa6b601ec520df6bbbbfe4a765b0dc89eaf65e"
Aug 01 00:24:39.389085 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:39.389017    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d58ca0b8ecae1916ed5f04cf89fa6b601ec520df6bbbbfe4a765b0dc89eaf65e"} err="failed to get container status \"d58ca0b8ecae1916ed5f04cf89fa6b601ec520df6bbbbfe4a765b0dc89eaf65e\": rpc error: code = NotFound desc = could not find container \"d58ca0b8ecae1916ed5f04cf89fa6b601ec520df6bbbbfe4a765b0dc89eaf65e\": container with ID starting with d58ca0b8ecae1916ed5f04cf89fa6b601ec520df6bbbbfe4a765b0dc89eaf65e not found: ID does not exist"
Aug 01 00:24:39.389085 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:39.389037    2559 scope.go:117] "RemoveContainer" containerID="7b94054c608127be878c1b75aad9eef3259f9420573dab44b3757fc97d25802e"
Aug 01 00:24:39.389334 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:24:39.389314    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b94054c608127be878c1b75aad9eef3259f9420573dab44b3757fc97d25802e\": container with ID starting with 7b94054c608127be878c1b75aad9eef3259f9420573dab44b3757fc97d25802e not found: ID does not exist" containerID="7b94054c608127be878c1b75aad9eef3259f9420573dab44b3757fc97d25802e"
Aug 01 00:24:39.389382 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:39.389340    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b94054c608127be878c1b75aad9eef3259f9420573dab44b3757fc97d25802e"} err="failed to get container status \"7b94054c608127be878c1b75aad9eef3259f9420573dab44b3757fc97d25802e\": rpc error: code = NotFound desc = could not find container \"7b94054c608127be878c1b75aad9eef3259f9420573dab44b3757fc97d25802e\": container with ID starting with 7b94054c608127be878c1b75aad9eef3259f9420573dab44b3757fc97d25802e not found: ID does not exist"
Aug 01 00:24:39.392965 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:39.392937    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24"]
Aug 01 00:24:39.398277 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:39.398254    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-765c4cdcdf-qqg24"]
Aug 01 00:24:39.785440 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:39.785408    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="103bd86c-3f7e-4eb0-988f-4639d27f7e32" path="/var/lib/kubelet/pods/103bd86c-3f7e-4eb0-988f-4639d27f7e32/volumes"
Aug 01 00:24:45.874475 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:45.874433    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-nkqrz"
Aug 01 00:24:45.886851 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:45.886825    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-nkqrz"
Aug 01 00:24:46.411790 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:24:46.411735    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-nkqrz"
Aug 01 00:25:37.882936 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:25:37.882905    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-spblg_c2e50946-aea4-4886-82ec-0bd5a31196cf/console-operator/1.log"
Aug 01 00:25:37.889167 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:25:37.889144    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-spblg_c2e50946-aea4-4886-82ec-0bd5a31196cf/console-operator/1.log"
Aug 01 00:25:48.249207 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:25:48.249170    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-rollback-b8fdb441_roll-v1-kserve-69cf8bcfc-nkqrz_572567b0-9795-4702-94ad-70598bd196cb/main/0.log"
Aug 01 00:26:01.152621 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:01.152542    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-2qsht"]
Aug 01 00:26:01.153156 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:01.152940    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="103bd86c-3f7e-4eb0-988f-4639d27f7e32" containerName="main"
Aug 01 00:26:01.153156 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:01.152958    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="103bd86c-3f7e-4eb0-988f-4639d27f7e32" containerName="main"
Aug 01 00:26:01.153156 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:01.152979    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="103bd86c-3f7e-4eb0-988f-4639d27f7e32" containerName="storage-initializer"
Aug 01 00:26:01.153156 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:01.152987    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="103bd86c-3f7e-4eb0-988f-4639d27f7e32" containerName="storage-initializer"
Aug 01 00:26:01.153357 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:01.153184    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="103bd86c-3f7e-4eb0-988f-4639d27f7e32" containerName="main"
Aug 01 00:26:01.155173 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:01.155156    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-2qsht"
Aug 01 00:26:01.157719 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:01.157699    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"kube-root-ca.crt\""
Aug 01 00:26:01.158932 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:01.158910    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"openshift-service-ca.crt\""
Aug 01 00:26:01.159038 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:01.159002    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"default-dockercfg-gj5gn\""
Aug 01 00:26:01.159038 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:01.159021    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"owner-v1-kserve-self-signed-certs\""
Aug 01 00:26:01.163958 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:01.163936    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-2qsht"]
Aug 01 00:26:01.316374 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:01.316335    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhfgh\" (UniqueName: \"kubernetes.io/projected/eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd-kube-api-access-jhfgh\") pod \"owner-v1-kserve-75579ff5bc-2qsht\" (UID: \"eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-2qsht"
Aug 01 00:26:01.316374 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:01.316377    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd-dshm\") pod \"owner-v1-kserve-75579ff5bc-2qsht\" (UID: \"eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-2qsht"
Aug 01 00:26:01.316613 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:01.316406    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd-home\") pod \"owner-v1-kserve-75579ff5bc-2qsht\" (UID: \"eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-2qsht"
Aug 01 00:26:01.316613 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:01.316426    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-2qsht\" (UID: \"eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-2qsht"
Aug 01 00:26:01.316613 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:01.316472    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-2qsht\" (UID: \"eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-2qsht"
Aug 01 00:26:01.316613 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:01.316524    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd-model-cache\") pod \"owner-v1-kserve-75579ff5bc-2qsht\" (UID: \"eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-2qsht"
Aug 01 00:26:01.417238 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:01.417157    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhfgh\" (UniqueName: \"kubernetes.io/projected/eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd-kube-api-access-jhfgh\") pod \"owner-v1-kserve-75579ff5bc-2qsht\" (UID: \"eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-2qsht"
Aug 01 00:26:01.417238 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:01.417195    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd-dshm\") pod \"owner-v1-kserve-75579ff5bc-2qsht\" (UID: \"eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-2qsht"
Aug 01 00:26:01.417432 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:01.417359    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd-home\") pod \"owner-v1-kserve-75579ff5bc-2qsht\" (UID: \"eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-2qsht"
Aug 01 00:26:01.417432 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:01.417393    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-2qsht\" (UID: \"eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-2qsht"
Aug 01 00:26:01.417432 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:01.417416    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-2qsht\" (UID: \"eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-2qsht"
Aug 01 00:26:01.417580 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:01.417468    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd-model-cache\") pod \"owner-v1-kserve-75579ff5bc-2qsht\" (UID: \"eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-2qsht"
Aug 01 00:26:01.417685 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:01.417666    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd-home\") pod \"owner-v1-kserve-75579ff5bc-2qsht\" (UID: \"eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-2qsht"
Aug 01 00:26:01.417827 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:01.417808    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-2qsht\" (UID: \"eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-2qsht"
Aug 01 00:26:01.417899 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:01.417848    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd-model-cache\") pod \"owner-v1-kserve-75579ff5bc-2qsht\" (UID: \"eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-2qsht"
Aug 01 00:26:01.419467 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:01.419439    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd-dshm\") pod \"owner-v1-kserve-75579ff5bc-2qsht\" (UID: \"eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-2qsht"
Aug 01 00:26:01.419819 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:01.419799    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-2qsht\" (UID: \"eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-2qsht"
Aug 01 00:26:01.424531 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:01.424507    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhfgh\" (UniqueName: \"kubernetes.io/projected/eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd-kube-api-access-jhfgh\") pod \"owner-v1-kserve-75579ff5bc-2qsht\" (UID: \"eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-2qsht"
Aug 01 00:26:01.466683 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:01.466651    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-2qsht"
Aug 01 00:26:01.595649 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:01.595623    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-2qsht"]
Aug 01 00:26:01.597972 ip-10-0-130-148 kubenswrapper[2559]: W0801 00:26:01.597939    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb3e92a1_fa8f_4cfd_a503_4dff3476a1bd.slice/crio-97920c9795b63680cd488a22e53947065e22e671c11d808ac65d155fd9052a80 WatchSource:0}: Error finding container 97920c9795b63680cd488a22e53947065e22e671c11d808ac65d155fd9052a80: Status 404 returned error can't find the container with id 97920c9795b63680cd488a22e53947065e22e671c11d808ac65d155fd9052a80
Aug 01 00:26:01.724092 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:01.724056    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-2qsht" event={"ID":"eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd","Type":"ContainerStarted","Data":"c15bdc672101b1a8311edc4e123adc2f1055522dd90594723b811a3f19e974f5"}
Aug 01 00:26:01.724298 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:01.724114    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-2qsht" event={"ID":"eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd","Type":"ContainerStarted","Data":"97920c9795b63680cd488a22e53947065e22e671c11d808ac65d155fd9052a80"}
Aug 01 00:26:01.741536 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:01.741484    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-2qsht" podStartSLOduration=0.741470633 podStartE2EDuration="741.470633ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:26:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:26:01.739618713 +0000 UTC m=+2424.459418556" watchObservedRunningTime="2026-08-01 00:26:01.741470633 +0000 UTC m=+2424.461270478"
Aug 01 00:26:02.466888 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:02.466857    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-2qsht"
Aug 01 00:26:06.262374 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:06.262339    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-kv6f2"]
Aug 01 00:26:06.265046 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:06.265027    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-kv6f2"
Aug 01 00:26:06.267455 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:06.267433    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"owner-v2-kserve-self-signed-certs\""
Aug 01 00:26:06.273628 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:06.273604    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-kv6f2"]
Aug 01 00:26:06.459984 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:06.459956    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/092c7db9-757f-4dc5-bb97-fa79039243c9-dshm\") pod \"owner-v2-kserve-5cf8959f95-kv6f2\" (UID: \"092c7db9-757f-4dc5-bb97-fa79039243c9\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-kv6f2"
Aug 01 00:26:06.460152 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:06.459989    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/092c7db9-757f-4dc5-bb97-fa79039243c9-home\") pod \"owner-v2-kserve-5cf8959f95-kv6f2\" (UID: \"092c7db9-757f-4dc5-bb97-fa79039243c9\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-kv6f2"
Aug 01 00:26:06.460152 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:06.460015    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/092c7db9-757f-4dc5-bb97-fa79039243c9-tmp-dir\") pod \"owner-v2-kserve-5cf8959f95-kv6f2\" (UID: \"092c7db9-757f-4dc5-bb97-fa79039243c9\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-kv6f2"
Aug 01 00:26:06.460152 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:06.460071    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/092c7db9-757f-4dc5-bb97-fa79039243c9-tls-certs\") pod \"owner-v2-kserve-5cf8959f95-kv6f2\" (UID: \"092c7db9-757f-4dc5-bb97-fa79039243c9\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-kv6f2"
Aug 01 00:26:06.460152 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:06.460109    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/092c7db9-757f-4dc5-bb97-fa79039243c9-model-cache\") pod \"owner-v2-kserve-5cf8959f95-kv6f2\" (UID: \"092c7db9-757f-4dc5-bb97-fa79039243c9\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-kv6f2"
Aug 01 00:26:06.460152 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:06.460143    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pfkq\" (UniqueName: \"kubernetes.io/projected/092c7db9-757f-4dc5-bb97-fa79039243c9-kube-api-access-4pfkq\") pod \"owner-v2-kserve-5cf8959f95-kv6f2\" (UID: \"092c7db9-757f-4dc5-bb97-fa79039243c9\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-kv6f2"
Aug 01 00:26:06.560857 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:06.560769    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/092c7db9-757f-4dc5-bb97-fa79039243c9-dshm\") pod \"owner-v2-kserve-5cf8959f95-kv6f2\" (UID: \"092c7db9-757f-4dc5-bb97-fa79039243c9\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-kv6f2"
Aug 01 00:26:06.560857 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:06.560812    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/092c7db9-757f-4dc5-bb97-fa79039243c9-home\") pod \"owner-v2-kserve-5cf8959f95-kv6f2\" (UID: \"092c7db9-757f-4dc5-bb97-fa79039243c9\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-kv6f2"
Aug 01 00:26:06.560857 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:06.560844    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/092c7db9-757f-4dc5-bb97-fa79039243c9-tmp-dir\") pod \"owner-v2-kserve-5cf8959f95-kv6f2\" (UID: \"092c7db9-757f-4dc5-bb97-fa79039243c9\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-kv6f2"
Aug 01 00:26:06.561104 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:06.560872    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/092c7db9-757f-4dc5-bb97-fa79039243c9-tls-certs\") pod \"owner-v2-kserve-5cf8959f95-kv6f2\" (UID: \"092c7db9-757f-4dc5-bb97-fa79039243c9\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-kv6f2"
Aug 01 00:26:06.561104 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:06.560897    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/092c7db9-757f-4dc5-bb97-fa79039243c9-model-cache\") pod \"owner-v2-kserve-5cf8959f95-kv6f2\" (UID: \"092c7db9-757f-4dc5-bb97-fa79039243c9\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-kv6f2"
Aug 01 00:26:06.561104 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:06.561054    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pfkq\" (UniqueName: \"kubernetes.io/projected/092c7db9-757f-4dc5-bb97-fa79039243c9-kube-api-access-4pfkq\") pod \"owner-v2-kserve-5cf8959f95-kv6f2\" (UID: \"092c7db9-757f-4dc5-bb97-fa79039243c9\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-kv6f2"
Aug 01 00:26:06.561288 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:06.561266    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/092c7db9-757f-4dc5-bb97-fa79039243c9-tmp-dir\") pod \"owner-v2-kserve-5cf8959f95-kv6f2\" (UID: \"092c7db9-757f-4dc5-bb97-fa79039243c9\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-kv6f2"
Aug 01 00:26:06.561353 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:06.561289    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/092c7db9-757f-4dc5-bb97-fa79039243c9-home\") pod \"owner-v2-kserve-5cf8959f95-kv6f2\" (UID: \"092c7db9-757f-4dc5-bb97-fa79039243c9\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-kv6f2"
Aug 01 00:26:06.561353 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:06.561331    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/092c7db9-757f-4dc5-bb97-fa79039243c9-model-cache\") pod \"owner-v2-kserve-5cf8959f95-kv6f2\" (UID: \"092c7db9-757f-4dc5-bb97-fa79039243c9\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-kv6f2"
Aug 01 00:26:06.562991 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:06.562971    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/092c7db9-757f-4dc5-bb97-fa79039243c9-dshm\") pod \"owner-v2-kserve-5cf8959f95-kv6f2\" (UID: \"092c7db9-757f-4dc5-bb97-fa79039243c9\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-kv6f2"
Aug 01 00:26:06.563300 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:06.563281    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/092c7db9-757f-4dc5-bb97-fa79039243c9-tls-certs\") pod \"owner-v2-kserve-5cf8959f95-kv6f2\" (UID: \"092c7db9-757f-4dc5-bb97-fa79039243c9\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-kv6f2"
Aug 01 00:26:06.568551 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:06.568528    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pfkq\" (UniqueName: \"kubernetes.io/projected/092c7db9-757f-4dc5-bb97-fa79039243c9-kube-api-access-4pfkq\") pod \"owner-v2-kserve-5cf8959f95-kv6f2\" (UID: \"092c7db9-757f-4dc5-bb97-fa79039243c9\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-kv6f2"
Aug 01 00:26:06.575402 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:06.575381    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-kv6f2"
Aug 01 00:26:06.699886 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:06.699857    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-kv6f2"]
Aug 01 00:26:06.701915 ip-10-0-130-148 kubenswrapper[2559]: W0801 00:26:06.701887    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod092c7db9_757f_4dc5_bb97_fa79039243c9.slice/crio-50e3f8b1e47398359ffa3dd02502ed4b57b5811c07e83058e58f42d6bae9bc66 WatchSource:0}: Error finding container 50e3f8b1e47398359ffa3dd02502ed4b57b5811c07e83058e58f42d6bae9bc66: Status 404 returned error can't find the container with id 50e3f8b1e47398359ffa3dd02502ed4b57b5811c07e83058e58f42d6bae9bc66
Aug 01 00:26:06.747573 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:06.747548    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-kv6f2" event={"ID":"092c7db9-757f-4dc5-bb97-fa79039243c9","Type":"ContainerStarted","Data":"50e3f8b1e47398359ffa3dd02502ed4b57b5811c07e83058e58f42d6bae9bc66"}
Aug 01 00:26:07.753640 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:07.753609    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-kv6f2" event={"ID":"092c7db9-757f-4dc5-bb97-fa79039243c9","Type":"ContainerStarted","Data":"91b9a1e31bc57679e367dd77b81f7114da71dd112fe8b19f448587d4dbc963a7"}
Aug 01 00:26:07.769584 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:07.769538    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-kv6f2" podStartSLOduration=1.76952298 podStartE2EDuration="1.76952298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:26:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:26:07.76820364 +0000 UTC m=+2430.488003498" watchObservedRunningTime="2026-08-01 00:26:07.76952298 +0000 UTC m=+2430.489322875"
Aug 01 00:26:08.576428 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:08.576387    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-kv6f2"
Aug 01 00:26:11.467334 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:11.467303    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-2qsht"
Aug 01 00:26:11.479376 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:11.479349    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-2qsht"
Aug 01 00:26:11.786369 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:11.786287    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-2qsht"
Aug 01 00:26:16.575711 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:16.575672    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-kv6f2"
Aug 01 00:26:16.588192 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:16.588166    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-kv6f2"
Aug 01 00:26:16.803456 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:16.803426    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-kv6f2"
Aug 01 00:26:40.933885 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:40.933848    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-2qsht"]
Aug 01 00:26:55.953101 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:55.953053    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-2qsht" podUID="eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd" containerName="main" containerID="cri-o://c15bdc672101b1a8311edc4e123adc2f1055522dd90594723b811a3f19e974f5" gracePeriod=45
Aug 01 00:26:56.188450 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:56.188424    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-2qsht"
Aug 01 00:26:56.273134 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:56.273038    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd-home\") pod \"eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd\" (UID: \"eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd\") "
Aug 01 00:26:56.273134 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:56.273076    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd-tls-certs\") pod \"eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd\" (UID: \"eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd\") "
Aug 01 00:26:56.273134 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:56.273100    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd-dshm\") pod \"eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd\" (UID: \"eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd\") "
Aug 01 00:26:56.273134 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:56.273118    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd-tmp-dir\") pod \"eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd\" (UID: \"eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd\") "
Aug 01 00:26:56.273476 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:56.273171    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd-model-cache\") pod \"eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd\" (UID: \"eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd\") "
Aug 01 00:26:56.273476 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:56.273210    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhfgh\" (UniqueName: \"kubernetes.io/projected/eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd-kube-api-access-jhfgh\") pod \"eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd\" (UID: \"eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd\") "
Aug 01 00:26:56.273476 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:56.273396    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd-home" (OuterVolumeSpecName: "home") pod "eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd" (UID: "eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:26:56.273476 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:56.273459    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd" (UID: "eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:26:56.273674 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:56.273480    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd-model-cache" (OuterVolumeSpecName: "model-cache") pod "eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd" (UID: "eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:26:56.275443 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:56.275412    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd-dshm" (OuterVolumeSpecName: "dshm") pod "eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd" (UID: "eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:26:56.275551 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:56.275500    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd" (UID: "eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:26:56.275551 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:56.275512    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd-kube-api-access-jhfgh" (OuterVolumeSpecName: "kube-api-access-jhfgh") pod "eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd" (UID: "eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd"). InnerVolumeSpecName "kube-api-access-jhfgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:26:56.373889 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:56.373850    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhfgh\" (UniqueName: \"kubernetes.io/projected/eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd-kube-api-access-jhfgh\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:26:56.373889 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:56.373880    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd-home\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:26:56.373889 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:56.373889    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd-tls-certs\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:26:56.373889 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:56.373898    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd-dshm\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:26:56.374160 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:56.373907    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd-tmp-dir\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:26:56.374160 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:56.373915    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd-model-cache\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:26:56.968122 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:56.968087    2559 generic.go:356] "Generic (PLEG): container finished" podID="eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd" containerID="c15bdc672101b1a8311edc4e123adc2f1055522dd90594723b811a3f19e974f5" exitCode=0
Aug 01 00:26:56.968536 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:56.968130    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-2qsht"
Aug 01 00:26:56.968536 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:56.968163    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-2qsht" event={"ID":"eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd","Type":"ContainerDied","Data":"c15bdc672101b1a8311edc4e123adc2f1055522dd90594723b811a3f19e974f5"}
Aug 01 00:26:56.968536 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:56.968217    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-2qsht" event={"ID":"eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd","Type":"ContainerDied","Data":"97920c9795b63680cd488a22e53947065e22e671c11d808ac65d155fd9052a80"}
Aug 01 00:26:56.968536 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:56.968247    2559 scope.go:117] "RemoveContainer" containerID="c15bdc672101b1a8311edc4e123adc2f1055522dd90594723b811a3f19e974f5"
Aug 01 00:26:56.977421 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:56.977403    2559 scope.go:117] "RemoveContainer" containerID="c15bdc672101b1a8311edc4e123adc2f1055522dd90594723b811a3f19e974f5"
Aug 01 00:26:56.977727 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:26:56.977707    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c15bdc672101b1a8311edc4e123adc2f1055522dd90594723b811a3f19e974f5\": container with ID starting with c15bdc672101b1a8311edc4e123adc2f1055522dd90594723b811a3f19e974f5 not found: ID does not exist" containerID="c15bdc672101b1a8311edc4e123adc2f1055522dd90594723b811a3f19e974f5"
Aug 01 00:26:56.977794 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:56.977764    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c15bdc672101b1a8311edc4e123adc2f1055522dd90594723b811a3f19e974f5"} err="failed to get container status \"c15bdc672101b1a8311edc4e123adc2f1055522dd90594723b811a3f19e974f5\": rpc error: code = NotFound desc = could not find container \"c15bdc672101b1a8311edc4e123adc2f1055522dd90594723b811a3f19e974f5\": container with ID starting with c15bdc672101b1a8311edc4e123adc2f1055522dd90594723b811a3f19e974f5 not found: ID does not exist"
Aug 01 00:26:56.989081 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:56.989058    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-2qsht"]
Aug 01 00:26:56.992718 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:56.992697    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-2qsht"]
Aug 01 00:26:57.785701 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:26:57.785665    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd" path="/var/lib/kubelet/pods/eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd/volumes"
Aug 01 00:27:07.244451 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:07.244412    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-kv6f2"]
Aug 01 00:27:13.849119 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:13.849081    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997x87b7"]
Aug 01 00:27:13.849563 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:13.849401    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd" containerName="main"
Aug 01 00:27:13.849563 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:13.849412    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd" containerName="main"
Aug 01 00:27:13.849563 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:13.849477    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb3e92a1-fa8f-4cfd-a503-4dff3476a1bd" containerName="main"
Aug 01 00:27:13.852538 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:13.852517    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997x87b7"
Aug 01 00:27:13.856338 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:13.856307    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Aug 01 00:27:13.856338 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:13.856332    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"default-dockercfg-j2d8t\""
Aug 01 00:27:13.856520 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:13.856346    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs\""
Aug 01 00:27:13.856520 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:13.856319    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Aug 01 00:27:13.861073 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:13.861049    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997x87b7"]
Aug 01 00:27:13.915861 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:13.915817    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9adb282a-b6ab-4107-81de-852078b60c4f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997x87b7\" (UID: \"9adb282a-b6ab-4107-81de-852078b60c4f\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997x87b7"
Aug 01 00:27:13.916057 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:13.915883    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9adb282a-b6ab-4107-81de-852078b60c4f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997x87b7\" (UID: \"9adb282a-b6ab-4107-81de-852078b60c4f\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997x87b7"
Aug 01 00:27:13.916057 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:13.915966    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kkkq\" (UniqueName: \"kubernetes.io/projected/9adb282a-b6ab-4107-81de-852078b60c4f-kube-api-access-5kkkq\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997x87b7\" (UID: \"9adb282a-b6ab-4107-81de-852078b60c4f\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997x87b7"
Aug 01 00:27:13.916057 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:13.916000    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9adb282a-b6ab-4107-81de-852078b60c4f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997x87b7\" (UID: \"9adb282a-b6ab-4107-81de-852078b60c4f\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997x87b7"
Aug 01 00:27:13.916057 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:13.916022    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9adb282a-b6ab-4107-81de-852078b60c4f-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997x87b7\" (UID: \"9adb282a-b6ab-4107-81de-852078b60c4f\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997x87b7"
Aug 01 00:27:13.916240 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:13.916110    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9adb282a-b6ab-4107-81de-852078b60c4f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997x87b7\" (UID: \"9adb282a-b6ab-4107-81de-852078b60c4f\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997x87b7"
Aug 01 00:27:14.017407 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:14.017371    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9adb282a-b6ab-4107-81de-852078b60c4f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997x87b7\" (UID: \"9adb282a-b6ab-4107-81de-852078b60c4f\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997x87b7"
Aug 01 00:27:14.017606 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:14.017418    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9adb282a-b6ab-4107-81de-852078b60c4f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997x87b7\" (UID: \"9adb282a-b6ab-4107-81de-852078b60c4f\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997x87b7"
Aug 01 00:27:14.017606 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:14.017452    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9adb282a-b6ab-4107-81de-852078b60c4f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997x87b7\" (UID: \"9adb282a-b6ab-4107-81de-852078b60c4f\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997x87b7"
Aug 01 00:27:14.017606 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:14.017488    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kkkq\" (UniqueName: \"kubernetes.io/projected/9adb282a-b6ab-4107-81de-852078b60c4f-kube-api-access-5kkkq\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997x87b7\" (UID: \"9adb282a-b6ab-4107-81de-852078b60c4f\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997x87b7"
Aug 01 00:27:14.017606 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:14.017520    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9adb282a-b6ab-4107-81de-852078b60c4f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997x87b7\" (UID: \"9adb282a-b6ab-4107-81de-852078b60c4f\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997x87b7"
Aug 01 00:27:14.017606 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:14.017548    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9adb282a-b6ab-4107-81de-852078b60c4f-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997x87b7\" (UID: \"9adb282a-b6ab-4107-81de-852078b60c4f\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997x87b7"
Aug 01 00:27:14.018017 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:14.017985    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9adb282a-b6ab-4107-81de-852078b60c4f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997x87b7\" (UID: \"9adb282a-b6ab-4107-81de-852078b60c4f\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997x87b7"
Aug 01 00:27:14.018143 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:14.018022    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9adb282a-b6ab-4107-81de-852078b60c4f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997x87b7\" (UID: \"9adb282a-b6ab-4107-81de-852078b60c4f\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997x87b7"
Aug 01 00:27:14.018143 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:14.018050    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9adb282a-b6ab-4107-81de-852078b60c4f-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997x87b7\" (UID: \"9adb282a-b6ab-4107-81de-852078b60c4f\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997x87b7"
Aug 01 00:27:14.019846 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:14.019830    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9adb282a-b6ab-4107-81de-852078b60c4f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997x87b7\" (UID: \"9adb282a-b6ab-4107-81de-852078b60c4f\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997x87b7"
Aug 01 00:27:14.020011 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:14.019993    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9adb282a-b6ab-4107-81de-852078b60c4f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997x87b7\" (UID: \"9adb282a-b6ab-4107-81de-852078b60c4f\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997x87b7"
Aug 01 00:27:14.025318 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:14.025294    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kkkq\" (UniqueName: \"kubernetes.io/projected/9adb282a-b6ab-4107-81de-852078b60c4f-kube-api-access-5kkkq\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997x87b7\" (UID: \"9adb282a-b6ab-4107-81de-852078b60c4f\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997x87b7"
Aug 01 00:27:14.164040 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:14.163932    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997x87b7"
Aug 01 00:27:14.300620 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:14.300590    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997x87b7"]
Aug 01 00:27:14.303380 ip-10-0-130-148 kubenswrapper[2559]: W0801 00:27:14.303347    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9adb282a_b6ab_4107_81de_852078b60c4f.slice/crio-112e31bdd22bb3a2d49dd8e7c17ba296d6d8f8d5eb5187cc38491806f9d57620 WatchSource:0}: Error finding container 112e31bdd22bb3a2d49dd8e7c17ba296d6d8f8d5eb5187cc38491806f9d57620: Status 404 returned error can't find the container with id 112e31bdd22bb3a2d49dd8e7c17ba296d6d8f8d5eb5187cc38491806f9d57620
Aug 01 00:27:15.050262 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:15.050229    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997x87b7" event={"ID":"9adb282a-b6ab-4107-81de-852078b60c4f","Type":"ContainerStarted","Data":"8c1f41ab6ec94c72b9fc3c1bff190f60cd9a1fadc1422b5c9f899dbf15863cc1"}
Aug 01 00:27:15.050655 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:15.050276    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997x87b7" event={"ID":"9adb282a-b6ab-4107-81de-852078b60c4f","Type":"ContainerStarted","Data":"112e31bdd22bb3a2d49dd8e7c17ba296d6d8f8d5eb5187cc38491806f9d57620"}
Aug 01 00:27:15.068009 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:15.067961    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997x87b7" podStartSLOduration=2.067944709 podStartE2EDuration="2.067944709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:27:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:27:15.065885526 +0000 UTC m=+2497.785685371" watchObservedRunningTime="2026-08-01 00:27:15.067944709 +0000 UTC m=+2497.787744554"
Aug 01 00:27:15.164133 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:15.164097    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997x87b7"
Aug 01 00:27:22.262968 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:22.262929    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-kv6f2" podUID="092c7db9-757f-4dc5-bb97-fa79039243c9" containerName="main" containerID="cri-o://91b9a1e31bc57679e367dd77b81f7114da71dd112fe8b19f448587d4dbc963a7" gracePeriod=45
Aug 01 00:27:22.499494 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:22.499467    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-kv6f2"
Aug 01 00:27:22.588732 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:22.588650    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/092c7db9-757f-4dc5-bb97-fa79039243c9-model-cache\") pod \"092c7db9-757f-4dc5-bb97-fa79039243c9\" (UID: \"092c7db9-757f-4dc5-bb97-fa79039243c9\") "
Aug 01 00:27:22.588732 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:22.588710    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4pfkq\" (UniqueName: \"kubernetes.io/projected/092c7db9-757f-4dc5-bb97-fa79039243c9-kube-api-access-4pfkq\") pod \"092c7db9-757f-4dc5-bb97-fa79039243c9\" (UID: \"092c7db9-757f-4dc5-bb97-fa79039243c9\") "
Aug 01 00:27:22.588929 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:22.588759    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/092c7db9-757f-4dc5-bb97-fa79039243c9-home\") pod \"092c7db9-757f-4dc5-bb97-fa79039243c9\" (UID: \"092c7db9-757f-4dc5-bb97-fa79039243c9\") "
Aug 01 00:27:22.588929 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:22.588794    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/092c7db9-757f-4dc5-bb97-fa79039243c9-tls-certs\") pod \"092c7db9-757f-4dc5-bb97-fa79039243c9\" (UID: \"092c7db9-757f-4dc5-bb97-fa79039243c9\") "
Aug 01 00:27:22.588929 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:22.588852    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/092c7db9-757f-4dc5-bb97-fa79039243c9-dshm\") pod \"092c7db9-757f-4dc5-bb97-fa79039243c9\" (UID: \"092c7db9-757f-4dc5-bb97-fa79039243c9\") "
Aug 01 00:27:22.588929 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:22.588879    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/092c7db9-757f-4dc5-bb97-fa79039243c9-tmp-dir\") pod \"092c7db9-757f-4dc5-bb97-fa79039243c9\" (UID: \"092c7db9-757f-4dc5-bb97-fa79039243c9\") "
Aug 01 00:27:22.589120 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:22.588994    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092c7db9-757f-4dc5-bb97-fa79039243c9-model-cache" (OuterVolumeSpecName: "model-cache") pod "092c7db9-757f-4dc5-bb97-fa79039243c9" (UID: "092c7db9-757f-4dc5-bb97-fa79039243c9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:27:22.589120 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:22.589040    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092c7db9-757f-4dc5-bb97-fa79039243c9-home" (OuterVolumeSpecName: "home") pod "092c7db9-757f-4dc5-bb97-fa79039243c9" (UID: "092c7db9-757f-4dc5-bb97-fa79039243c9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:27:22.589120 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:22.589117    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/092c7db9-757f-4dc5-bb97-fa79039243c9-model-cache\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:27:22.589265 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:22.589134    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/092c7db9-757f-4dc5-bb97-fa79039243c9-home\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:27:22.589265 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:22.589211    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092c7db9-757f-4dc5-bb97-fa79039243c9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "092c7db9-757f-4dc5-bb97-fa79039243c9" (UID: "092c7db9-757f-4dc5-bb97-fa79039243c9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:27:22.590957 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:22.590938    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092c7db9-757f-4dc5-bb97-fa79039243c9-dshm" (OuterVolumeSpecName: "dshm") pod "092c7db9-757f-4dc5-bb97-fa79039243c9" (UID: "092c7db9-757f-4dc5-bb97-fa79039243c9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:27:22.591018 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:22.591004    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/092c7db9-757f-4dc5-bb97-fa79039243c9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "092c7db9-757f-4dc5-bb97-fa79039243c9" (UID: "092c7db9-757f-4dc5-bb97-fa79039243c9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:27:22.591059 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:22.591038    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/092c7db9-757f-4dc5-bb97-fa79039243c9-kube-api-access-4pfkq" (OuterVolumeSpecName: "kube-api-access-4pfkq") pod "092c7db9-757f-4dc5-bb97-fa79039243c9" (UID: "092c7db9-757f-4dc5-bb97-fa79039243c9"). InnerVolumeSpecName "kube-api-access-4pfkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:27:22.689615 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:22.689590    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/092c7db9-757f-4dc5-bb97-fa79039243c9-tls-certs\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:27:22.689615 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:22.689611    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/092c7db9-757f-4dc5-bb97-fa79039243c9-dshm\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:27:22.689615 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:22.689620    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/092c7db9-757f-4dc5-bb97-fa79039243c9-tmp-dir\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:27:22.689889 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:22.689628    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pfkq\" (UniqueName: \"kubernetes.io/projected/092c7db9-757f-4dc5-bb97-fa79039243c9-kube-api-access-4pfkq\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:27:23.092234 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:23.092197    2559 generic.go:356] "Generic (PLEG): container finished" podID="092c7db9-757f-4dc5-bb97-fa79039243c9" containerID="91b9a1e31bc57679e367dd77b81f7114da71dd112fe8b19f448587d4dbc963a7" exitCode=0
Aug 01 00:27:23.092234 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:23.092239    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-kv6f2" event={"ID":"092c7db9-757f-4dc5-bb97-fa79039243c9","Type":"ContainerDied","Data":"91b9a1e31bc57679e367dd77b81f7114da71dd112fe8b19f448587d4dbc963a7"}
Aug 01 00:27:23.092490 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:23.092247    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-kv6f2"
Aug 01 00:27:23.092490 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:23.092278    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-kv6f2" event={"ID":"092c7db9-757f-4dc5-bb97-fa79039243c9","Type":"ContainerDied","Data":"50e3f8b1e47398359ffa3dd02502ed4b57b5811c07e83058e58f42d6bae9bc66"}
Aug 01 00:27:23.092490 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:23.092299    2559 scope.go:117] "RemoveContainer" containerID="91b9a1e31bc57679e367dd77b81f7114da71dd112fe8b19f448587d4dbc963a7"
Aug 01 00:27:23.107016 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:23.106995    2559 scope.go:117] "RemoveContainer" containerID="91b9a1e31bc57679e367dd77b81f7114da71dd112fe8b19f448587d4dbc963a7"
Aug 01 00:27:23.107287 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:27:23.107269    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91b9a1e31bc57679e367dd77b81f7114da71dd112fe8b19f448587d4dbc963a7\": container with ID starting with 91b9a1e31bc57679e367dd77b81f7114da71dd112fe8b19f448587d4dbc963a7 not found: ID does not exist" containerID="91b9a1e31bc57679e367dd77b81f7114da71dd112fe8b19f448587d4dbc963a7"
Aug 01 00:27:23.107341 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:23.107294    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91b9a1e31bc57679e367dd77b81f7114da71dd112fe8b19f448587d4dbc963a7"} err="failed to get container status \"91b9a1e31bc57679e367dd77b81f7114da71dd112fe8b19f448587d4dbc963a7\": rpc error: code = NotFound desc = could not find container \"91b9a1e31bc57679e367dd77b81f7114da71dd112fe8b19f448587d4dbc963a7\": container with ID starting with 91b9a1e31bc57679e367dd77b81f7114da71dd112fe8b19f448587d4dbc963a7 not found: ID does not exist"
Aug 01 00:27:23.116761 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:23.116725    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-kv6f2"]
Aug 01 00:27:23.122059 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:23.122039    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-kv6f2"]
Aug 01 00:27:23.786171 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:23.786138    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="092c7db9-757f-4dc5-bb97-fa79039243c9" path="/var/lib/kubelet/pods/092c7db9-757f-4dc5-bb97-fa79039243c9/volumes"
Aug 01 00:27:24.164235 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:24.164152    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997x87b7"
Aug 01 00:27:24.176783 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:24.176755    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997x87b7"
Aug 01 00:27:25.113786 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:27:25.113759    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997x87b7"
Aug 01 00:29:55.070600 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:29:55.070562    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997x87b7"]
Aug 01 00:29:55.079246 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:29:55.079212    2559 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 01 00:29:55.079246 ip-10-0-130-148 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:29:55.079246 ip-10-0-130-148 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Aug 01 00:29:55.079246 ip-10-0-130-148 kubenswrapper[2559]:  > containerID="8c1f41ab6ec94c72b9fc3c1bff190f60cd9a1fadc1422b5c9f899dbf15863cc1" cmd=["/bin/sleep","15"]
Aug 01 00:29:55.079481 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:29:55.079263    2559 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997x87b7" message="" err=<
Aug 01 00:29:55.079481 ip-10-0-130-148 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:29:55.079481 ip-10-0-130-148 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Aug 01 00:29:55.079481 ip-10-0-130-148 kubenswrapper[2559]:  >
Aug 01 00:29:55.079481 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:29:55.079275    2559 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 01 00:29:55.079481 ip-10-0-130-148 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:29:55.079481 ip-10-0-130-148 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Aug 01 00:29:55.079481 ip-10-0-130-148 kubenswrapper[2559]:  > pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997x87b7" podUID="9adb282a-b6ab-4107-81de-852078b60c4f" containerName="main" containerID="cri-o://8c1f41ab6ec94c72b9fc3c1bff190f60cd9a1fadc1422b5c9f899dbf15863cc1"
Aug 01 00:29:55.079481 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:29:55.079312    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997x87b7" podUID="9adb282a-b6ab-4107-81de-852078b60c4f" containerName="main" containerID="cri-o://8c1f41ab6ec94c72b9fc3c1bff190f60cd9a1fadc1422b5c9f899dbf15863cc1" gracePeriod=60
Aug 01 00:29:55.103148 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:29:55.103105    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997x87b7" podUID="9adb282a-b6ab-4107-81de-852078b60c4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Aug 01 00:29:55.420263 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:29:55.420236    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997x87b7"
Aug 01 00:29:55.557957 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:29:55.557925    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9adb282a-b6ab-4107-81de-852078b60c4f-model-cache\") pod \"9adb282a-b6ab-4107-81de-852078b60c4f\" (UID: \"9adb282a-b6ab-4107-81de-852078b60c4f\") "
Aug 01 00:29:55.557957 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:29:55.557964    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9adb282a-b6ab-4107-81de-852078b60c4f-home\") pod \"9adb282a-b6ab-4107-81de-852078b60c4f\" (UID: \"9adb282a-b6ab-4107-81de-852078b60c4f\") "
Aug 01 00:29:55.558219 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:29:55.557994    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9adb282a-b6ab-4107-81de-852078b60c4f-dshm\") pod \"9adb282a-b6ab-4107-81de-852078b60c4f\" (UID: \"9adb282a-b6ab-4107-81de-852078b60c4f\") "
Aug 01 00:29:55.558219 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:29:55.558017    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9adb282a-b6ab-4107-81de-852078b60c4f-tls-certs\") pod \"9adb282a-b6ab-4107-81de-852078b60c4f\" (UID: \"9adb282a-b6ab-4107-81de-852078b60c4f\") "
Aug 01 00:29:55.558219 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:29:55.558145    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9adb282a-b6ab-4107-81de-852078b60c4f-tmp-dir\") pod \"9adb282a-b6ab-4107-81de-852078b60c4f\" (UID: \"9adb282a-b6ab-4107-81de-852078b60c4f\") "
Aug 01 00:29:55.558219 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:29:55.558177    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5kkkq\" (UniqueName: \"kubernetes.io/projected/9adb282a-b6ab-4107-81de-852078b60c4f-kube-api-access-5kkkq\") pod \"9adb282a-b6ab-4107-81de-852078b60c4f\" (UID: \"9adb282a-b6ab-4107-81de-852078b60c4f\") "
Aug 01 00:29:55.558418 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:29:55.558275    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9adb282a-b6ab-4107-81de-852078b60c4f-home" (OuterVolumeSpecName: "home") pod "9adb282a-b6ab-4107-81de-852078b60c4f" (UID: "9adb282a-b6ab-4107-81de-852078b60c4f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:29:55.558418 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:29:55.558285    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9adb282a-b6ab-4107-81de-852078b60c4f-model-cache" (OuterVolumeSpecName: "model-cache") pod "9adb282a-b6ab-4107-81de-852078b60c4f" (UID: "9adb282a-b6ab-4107-81de-852078b60c4f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:29:55.558418 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:29:55.558396    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9adb282a-b6ab-4107-81de-852078b60c4f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9adb282a-b6ab-4107-81de-852078b60c4f" (UID: "9adb282a-b6ab-4107-81de-852078b60c4f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:29:55.558576 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:29:55.558414    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9adb282a-b6ab-4107-81de-852078b60c4f-model-cache\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:29:55.558576 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:29:55.558438    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9adb282a-b6ab-4107-81de-852078b60c4f-home\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:29:55.560347 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:29:55.560313    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9adb282a-b6ab-4107-81de-852078b60c4f-dshm" (OuterVolumeSpecName: "dshm") pod "9adb282a-b6ab-4107-81de-852078b60c4f" (UID: "9adb282a-b6ab-4107-81de-852078b60c4f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:29:55.560449 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:29:55.560349    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9adb282a-b6ab-4107-81de-852078b60c4f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9adb282a-b6ab-4107-81de-852078b60c4f" (UID: "9adb282a-b6ab-4107-81de-852078b60c4f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:29:55.560449 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:29:55.560384    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9adb282a-b6ab-4107-81de-852078b60c4f-kube-api-access-5kkkq" (OuterVolumeSpecName: "kube-api-access-5kkkq") pod "9adb282a-b6ab-4107-81de-852078b60c4f" (UID: "9adb282a-b6ab-4107-81de-852078b60c4f"). InnerVolumeSpecName "kube-api-access-5kkkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:29:55.659309 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:29:55.659233    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9adb282a-b6ab-4107-81de-852078b60c4f-dshm\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:29:55.659309 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:29:55.659259    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9adb282a-b6ab-4107-81de-852078b60c4f-tls-certs\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:29:55.659309 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:29:55.659271    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9adb282a-b6ab-4107-81de-852078b60c4f-tmp-dir\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:29:55.659309 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:29:55.659280    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kkkq\" (UniqueName: \"kubernetes.io/projected/9adb282a-b6ab-4107-81de-852078b60c4f-kube-api-access-5kkkq\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:29:55.777327 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:29:55.777295    2559 generic.go:356] "Generic (PLEG): container finished" podID="9adb282a-b6ab-4107-81de-852078b60c4f" containerID="8c1f41ab6ec94c72b9fc3c1bff190f60cd9a1fadc1422b5c9f899dbf15863cc1" exitCode=0
Aug 01 00:29:55.777507 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:29:55.777337    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997x87b7"
Aug 01 00:29:55.777507 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:29:55.777366    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997x87b7" event={"ID":"9adb282a-b6ab-4107-81de-852078b60c4f","Type":"ContainerDied","Data":"8c1f41ab6ec94c72b9fc3c1bff190f60cd9a1fadc1422b5c9f899dbf15863cc1"}
Aug 01 00:29:55.777507 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:29:55.777409    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997x87b7" event={"ID":"9adb282a-b6ab-4107-81de-852078b60c4f","Type":"ContainerDied","Data":"112e31bdd22bb3a2d49dd8e7c17ba296d6d8f8d5eb5187cc38491806f9d57620"}
Aug 01 00:29:55.777507 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:29:55.777431    2559 scope.go:117] "RemoveContainer" containerID="8c1f41ab6ec94c72b9fc3c1bff190f60cd9a1fadc1422b5c9f899dbf15863cc1"
Aug 01 00:29:55.787783 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:29:55.787759    2559 scope.go:117] "RemoveContainer" containerID="8c1f41ab6ec94c72b9fc3c1bff190f60cd9a1fadc1422b5c9f899dbf15863cc1"
Aug 01 00:29:55.788045 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:29:55.788024    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c1f41ab6ec94c72b9fc3c1bff190f60cd9a1fadc1422b5c9f899dbf15863cc1\": container with ID starting with 8c1f41ab6ec94c72b9fc3c1bff190f60cd9a1fadc1422b5c9f899dbf15863cc1 not found: ID does not exist" containerID="8c1f41ab6ec94c72b9fc3c1bff190f60cd9a1fadc1422b5c9f899dbf15863cc1"
Aug 01 00:29:55.788175 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:29:55.788057    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c1f41ab6ec94c72b9fc3c1bff190f60cd9a1fadc1422b5c9f899dbf15863cc1"} err="failed to get container status \"8c1f41ab6ec94c72b9fc3c1bff190f60cd9a1fadc1422b5c9f899dbf15863cc1\": rpc error: code = NotFound desc = could not find container \"8c1f41ab6ec94c72b9fc3c1bff190f60cd9a1fadc1422b5c9f899dbf15863cc1\": container with ID starting with 8c1f41ab6ec94c72b9fc3c1bff190f60cd9a1fadc1422b5c9f899dbf15863cc1 not found: ID does not exist"
Aug 01 00:29:55.801251 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:29:55.801226    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997x87b7"]
Aug 01 00:29:55.804304 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:29:55.804284    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997x87b7"]
Aug 01 00:29:57.791608 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:29:57.791573    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9adb282a-b6ab-4107-81de-852078b60c4f" path="/var/lib/kubelet/pods/9adb282a-b6ab-4107-81de-852078b60c4f/volumes"
Aug 01 00:30:10.578471 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:30:10.578387    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz"]
Aug 01 00:30:10.578866 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:30:10.578695    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9adb282a-b6ab-4107-81de-852078b60c4f" containerName="main"
Aug 01 00:30:10.578866 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:30:10.578706    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9adb282a-b6ab-4107-81de-852078b60c4f" containerName="main"
Aug 01 00:30:10.578866 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:30:10.578722    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="092c7db9-757f-4dc5-bb97-fa79039243c9" containerName="main"
Aug 01 00:30:10.578866 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:30:10.578729    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="092c7db9-757f-4dc5-bb97-fa79039243c9" containerName="main"
Aug 01 00:30:10.578866 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:30:10.578808    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9adb282a-b6ab-4107-81de-852078b60c4f" containerName="main"
Aug 01 00:30:10.578866 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:30:10.578817    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="092c7db9-757f-4dc5-bb97-fa79039243c9" containerName="main"
Aug 01 00:30:10.582939 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:30:10.582909    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz"
Aug 01 00:30:10.585866 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:30:10.585833    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"odh-kserve-custom-ca-bundle\""
Aug 01 00:30:10.586012 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:30:10.585934    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"seaweedfs-s3-creds\""
Aug 01 00:30:10.586012 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:30:10.585945    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-fgvz5\""
Aug 01 00:30:10.586012 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:30:10.585980    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Aug 01 00:30:10.586196 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:30:10.586058    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Aug 01 00:30:10.586196 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:30:10.586118    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Aug 01 00:30:10.592115 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:30:10.592093    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz"]
Aug 01 00:30:10.683696 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:30:10.683655    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66643e38-edf1-43c4-a3f7-199f17b5b955-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz\" (UID: \"66643e38-edf1-43c4-a3f7-199f17b5b955\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz"
Aug 01 00:30:10.683696 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:30:10.683701    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66643e38-edf1-43c4-a3f7-199f17b5b955-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz\" (UID: \"66643e38-edf1-43c4-a3f7-199f17b5b955\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz"
Aug 01 00:30:10.683975 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:30:10.683783    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66643e38-edf1-43c4-a3f7-199f17b5b955-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz\" (UID: \"66643e38-edf1-43c4-a3f7-199f17b5b955\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz"
Aug 01 00:30:10.683975 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:30:10.683816    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66643e38-edf1-43c4-a3f7-199f17b5b955-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz\" (UID: \"66643e38-edf1-43c4-a3f7-199f17b5b955\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz"
Aug 01 00:30:10.683975 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:30:10.683850    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-669v4\" (UniqueName: \"kubernetes.io/projected/66643e38-edf1-43c4-a3f7-199f17b5b955-kube-api-access-669v4\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz\" (UID: \"66643e38-edf1-43c4-a3f7-199f17b5b955\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz"
Aug 01 00:30:10.683975 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:30:10.683908    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66643e38-edf1-43c4-a3f7-199f17b5b955-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz\" (UID: \"66643e38-edf1-43c4-a3f7-199f17b5b955\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz"
Aug 01 00:30:10.683975 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:30:10.683932    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66643e38-edf1-43c4-a3f7-199f17b5b955-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz\" (UID: \"66643e38-edf1-43c4-a3f7-199f17b5b955\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz"
Aug 01 00:30:10.683975 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:30:10.683957    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66643e38-edf1-43c4-a3f7-199f17b5b955-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz\" (UID: \"66643e38-edf1-43c4-a3f7-199f17b5b955\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz"
Aug 01 00:30:10.784421 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:30:10.784377    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66643e38-edf1-43c4-a3f7-199f17b5b955-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz\" (UID: \"66643e38-edf1-43c4-a3f7-199f17b5b955\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz"
Aug 01 00:30:10.784598 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:30:10.784518    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66643e38-edf1-43c4-a3f7-199f17b5b955-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz\" (UID: \"66643e38-edf1-43c4-a3f7-199f17b5b955\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz"
Aug 01 00:30:10.784598 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:30:10.784546    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66643e38-edf1-43c4-a3f7-199f17b5b955-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz\" (UID: \"66643e38-edf1-43c4-a3f7-199f17b5b955\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz"
Aug 01 00:30:10.784598 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:30:10.784579    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66643e38-edf1-43c4-a3f7-199f17b5b955-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz\" (UID: \"66643e38-edf1-43c4-a3f7-199f17b5b955\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz"
Aug 01 00:30:10.784598 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:30:10.784596    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66643e38-edf1-43c4-a3f7-199f17b5b955-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz\" (UID: \"66643e38-edf1-43c4-a3f7-199f17b5b955\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz"
Aug 01 00:30:10.784862 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:30:10.784614    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-669v4\" (UniqueName: \"kubernetes.io/projected/66643e38-edf1-43c4-a3f7-199f17b5b955-kube-api-access-669v4\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz\" (UID: \"66643e38-edf1-43c4-a3f7-199f17b5b955\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz"
Aug 01 00:30:10.784862 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:30:10.784670    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66643e38-edf1-43c4-a3f7-199f17b5b955-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz\" (UID: \"66643e38-edf1-43c4-a3f7-199f17b5b955\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz"
Aug 01 00:30:10.785048 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:30:10.785023    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66643e38-edf1-43c4-a3f7-199f17b5b955-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz\" (UID: \"66643e38-edf1-43c4-a3f7-199f17b5b955\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz"
Aug 01 00:30:10.785132 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:30:10.785080    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66643e38-edf1-43c4-a3f7-199f17b5b955-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz\" (UID: \"66643e38-edf1-43c4-a3f7-199f17b5b955\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz"
Aug 01 00:30:10.785132 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:30:10.785118    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66643e38-edf1-43c4-a3f7-199f17b5b955-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz\" (UID: \"66643e38-edf1-43c4-a3f7-199f17b5b955\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz"
Aug 01 00:30:10.785239 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:30:10.784699    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66643e38-edf1-43c4-a3f7-199f17b5b955-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz\" (UID: \"66643e38-edf1-43c4-a3f7-199f17b5b955\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz"
Aug 01 00:30:10.785239 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:30:10.785229    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66643e38-edf1-43c4-a3f7-199f17b5b955-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz\" (UID: \"66643e38-edf1-43c4-a3f7-199f17b5b955\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz"
Aug 01 00:30:10.785319 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:30:10.785232    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66643e38-edf1-43c4-a3f7-199f17b5b955-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz\" (UID: \"66643e38-edf1-43c4-a3f7-199f17b5b955\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz"
Aug 01 00:30:10.787115 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:30:10.787095    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66643e38-edf1-43c4-a3f7-199f17b5b955-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz\" (UID: \"66643e38-edf1-43c4-a3f7-199f17b5b955\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz"
Aug 01 00:30:10.787282 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:30:10.787265    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66643e38-edf1-43c4-a3f7-199f17b5b955-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz\" (UID: \"66643e38-edf1-43c4-a3f7-199f17b5b955\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz"
Aug 01 00:30:10.792244 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:30:10.792221    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-669v4\" (UniqueName: \"kubernetes.io/projected/66643e38-edf1-43c4-a3f7-199f17b5b955-kube-api-access-669v4\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz\" (UID: \"66643e38-edf1-43c4-a3f7-199f17b5b955\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz"
Aug 01 00:30:10.893580 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:30:10.893486    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz"
Aug 01 00:30:11.029324 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:30:11.029298    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz"]
Aug 01 00:30:11.031562 ip-10-0-130-148 kubenswrapper[2559]: W0801 00:30:11.031536    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66643e38_edf1_43c4_a3f7_199f17b5b955.slice/crio-b1422339cfe3b3eff3735b429a29e7c823e5b5377c7e3aa6fa220a8c25531a2c WatchSource:0}: Error finding container b1422339cfe3b3eff3735b429a29e7c823e5b5377c7e3aa6fa220a8c25531a2c: Status 404 returned error can't find the container with id b1422339cfe3b3eff3735b429a29e7c823e5b5377c7e3aa6fa220a8c25531a2c
Aug 01 00:30:11.854451 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:30:11.854410    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz" event={"ID":"66643e38-edf1-43c4-a3f7-199f17b5b955","Type":"ContainerStarted","Data":"4a2e82782397aaf8afd47eaa68a8b9794fe242cec3c090d6876b0ce8262bf699"}
Aug 01 00:30:11.854860 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:30:11.854481    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz" event={"ID":"66643e38-edf1-43c4-a3f7-199f17b5b955","Type":"ContainerStarted","Data":"b1422339cfe3b3eff3735b429a29e7c823e5b5377c7e3aa6fa220a8c25531a2c"}
Aug 01 00:30:13.865510 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:30:13.865474    2559 generic.go:356] "Generic (PLEG): container finished" podID="66643e38-edf1-43c4-a3f7-199f17b5b955" containerID="4a2e82782397aaf8afd47eaa68a8b9794fe242cec3c090d6876b0ce8262bf699" exitCode=0
Aug 01 00:30:13.865912 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:30:13.865543    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz" event={"ID":"66643e38-edf1-43c4-a3f7-199f17b5b955","Type":"ContainerDied","Data":"4a2e82782397aaf8afd47eaa68a8b9794fe242cec3c090d6876b0ce8262bf699"}
Aug 01 00:30:14.872694 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:30:14.872656    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz" event={"ID":"66643e38-edf1-43c4-a3f7-199f17b5b955","Type":"ContainerStarted","Data":"3879434386d5d109692b44f0a8b478e3edb0a7671342a42f1d6fdd1579791a6a"}
Aug 01 00:30:14.894051 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:30:14.894018    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz"
Aug 01 00:30:14.894471 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:30:14.894427    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz" podStartSLOduration=2.89168188 podStartE2EDuration="4.894412862s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.002730982s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:30:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:30:14.892066515 +0000 UTC m=+2677.611866361" watchObservedRunningTime="2026-08-01 00:30:14.894412862 +0000 UTC m=+2677.614212708"
Aug 01 00:30:20.894581 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:30:20.894545    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz"
Aug 01 00:30:20.896423 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:30:20.896386    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz" podUID="66643e38-edf1-43c4-a3f7-199f17b5b955" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Aug 01 00:30:30.894645 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:30:30.894545    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz" podUID="66643e38-edf1-43c4-a3f7-199f17b5b955" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Aug 01 00:30:37.910934 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:30:37.910886    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-spblg_c2e50946-aea4-4886-82ec-0bd5a31196cf/console-operator/1.log"
Aug 01 00:30:37.920463 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:30:37.920440    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-spblg_c2e50946-aea4-4886-82ec-0bd5a31196cf/console-operator/1.log"
Aug 01 00:30:40.894779 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:30:40.894716    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz" podUID="66643e38-edf1-43c4-a3f7-199f17b5b955" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Aug 01 00:30:50.893937 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:30:50.893895    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz" podUID="66643e38-edf1-43c4-a3f7-199f17b5b955" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Aug 01 00:31:00.894356 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:31:00.894310    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz" podUID="66643e38-edf1-43c4-a3f7-199f17b5b955" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Aug 01 00:31:10.894185 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:31:10.894137    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz" podUID="66643e38-edf1-43c4-a3f7-199f17b5b955" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Aug 01 00:31:20.893998 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:31:20.893955    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz" podUID="66643e38-edf1-43c4-a3f7-199f17b5b955" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Aug 01 00:31:30.894573 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:31:30.894525    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz" podUID="66643e38-edf1-43c4-a3f7-199f17b5b955" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Aug 01 00:31:40.894258 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:31:40.894202    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz" podUID="66643e38-edf1-43c4-a3f7-199f17b5b955" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Aug 01 00:31:50.894103 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:31:50.894050    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz" podUID="66643e38-edf1-43c4-a3f7-199f17b5b955" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Aug 01 00:32:00.895051 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:00.894951    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz" podUID="66643e38-edf1-43c4-a3f7-199f17b5b955" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Aug 01 00:32:10.903897 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:10.903865    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz"
Aug 01 00:32:10.911669 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:10.911639    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz"
Aug 01 00:32:15.289044 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:15.289013    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz"]
Aug 01 00:32:30.312494 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:30.312429    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz" podUID="66643e38-edf1-43c4-a3f7-199f17b5b955" containerName="main" containerID="cri-o://3879434386d5d109692b44f0a8b478e3edb0a7671342a42f1d6fdd1579791a6a" gracePeriod=45
Aug 01 00:32:30.905099 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:30.905050    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz" podUID="66643e38-edf1-43c4-a3f7-199f17b5b955" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Aug 01 00:32:31.054891 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:31.054865    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz"
Aug 01 00:32:31.159371 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:31.159257    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66643e38-edf1-43c4-a3f7-199f17b5b955-home\") pod \"66643e38-edf1-43c4-a3f7-199f17b5b955\" (UID: \"66643e38-edf1-43c4-a3f7-199f17b5b955\") "
Aug 01 00:32:31.159371 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:31.159343    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66643e38-edf1-43c4-a3f7-199f17b5b955-kserve-provision-location\") pod \"66643e38-edf1-43c4-a3f7-199f17b5b955\" (UID: \"66643e38-edf1-43c4-a3f7-199f17b5b955\") "
Aug 01 00:32:31.159371 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:31.159367    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-669v4\" (UniqueName: \"kubernetes.io/projected/66643e38-edf1-43c4-a3f7-199f17b5b955-kube-api-access-669v4\") pod \"66643e38-edf1-43c4-a3f7-199f17b5b955\" (UID: \"66643e38-edf1-43c4-a3f7-199f17b5b955\") "
Aug 01 00:32:31.159678 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:31.159391    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66643e38-edf1-43c4-a3f7-199f17b5b955-tls-certs\") pod \"66643e38-edf1-43c4-a3f7-199f17b5b955\" (UID: \"66643e38-edf1-43c4-a3f7-199f17b5b955\") "
Aug 01 00:32:31.159678 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:31.159413    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66643e38-edf1-43c4-a3f7-199f17b5b955-tmp-dir\") pod \"66643e38-edf1-43c4-a3f7-199f17b5b955\" (UID: \"66643e38-edf1-43c4-a3f7-199f17b5b955\") "
Aug 01 00:32:31.159678 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:31.159428    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66643e38-edf1-43c4-a3f7-199f17b5b955-model-cache\") pod \"66643e38-edf1-43c4-a3f7-199f17b5b955\" (UID: \"66643e38-edf1-43c4-a3f7-199f17b5b955\") "
Aug 01 00:32:31.159678 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:31.159474    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66643e38-edf1-43c4-a3f7-199f17b5b955-cabundle-cert\") pod \"66643e38-edf1-43c4-a3f7-199f17b5b955\" (UID: \"66643e38-edf1-43c4-a3f7-199f17b5b955\") "
Aug 01 00:32:31.159678 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:31.159513    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66643e38-edf1-43c4-a3f7-199f17b5b955-dshm\") pod \"66643e38-edf1-43c4-a3f7-199f17b5b955\" (UID: \"66643e38-edf1-43c4-a3f7-199f17b5b955\") "
Aug 01 00:32:31.159977 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:31.159711    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66643e38-edf1-43c4-a3f7-199f17b5b955-model-cache" (OuterVolumeSpecName: "model-cache") pod "66643e38-edf1-43c4-a3f7-199f17b5b955" (UID: "66643e38-edf1-43c4-a3f7-199f17b5b955"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:32:31.160148 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:31.160113    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66643e38-edf1-43c4-a3f7-199f17b5b955-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "66643e38-edf1-43c4-a3f7-199f17b5b955" (UID: "66643e38-edf1-43c4-a3f7-199f17b5b955"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:32:31.160333 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:31.160158    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66643e38-edf1-43c4-a3f7-199f17b5b955-home" (OuterVolumeSpecName: "home") pod "66643e38-edf1-43c4-a3f7-199f17b5b955" (UID: "66643e38-edf1-43c4-a3f7-199f17b5b955"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:32:31.161953 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:31.161926    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66643e38-edf1-43c4-a3f7-199f17b5b955-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "66643e38-edf1-43c4-a3f7-199f17b5b955" (UID: "66643e38-edf1-43c4-a3f7-199f17b5b955"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:32:31.161953 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:31.161939    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66643e38-edf1-43c4-a3f7-199f17b5b955-kube-api-access-669v4" (OuterVolumeSpecName: "kube-api-access-669v4") pod "66643e38-edf1-43c4-a3f7-199f17b5b955" (UID: "66643e38-edf1-43c4-a3f7-199f17b5b955"). InnerVolumeSpecName "kube-api-access-669v4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:32:31.162111 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:31.162080    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66643e38-edf1-43c4-a3f7-199f17b5b955-dshm" (OuterVolumeSpecName: "dshm") pod "66643e38-edf1-43c4-a3f7-199f17b5b955" (UID: "66643e38-edf1-43c4-a3f7-199f17b5b955"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:32:31.173784 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:31.173729    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66643e38-edf1-43c4-a3f7-199f17b5b955-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "66643e38-edf1-43c4-a3f7-199f17b5b955" (UID: "66643e38-edf1-43c4-a3f7-199f17b5b955"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:32:31.184808 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:31.184783    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66643e38-edf1-43c4-a3f7-199f17b5b955-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "66643e38-edf1-43c4-a3f7-199f17b5b955" (UID: "66643e38-edf1-43c4-a3f7-199f17b5b955"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:32:31.260957 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:31.260927    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66643e38-edf1-43c4-a3f7-199f17b5b955-dshm\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:32:31.260957 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:31.260952    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66643e38-edf1-43c4-a3f7-199f17b5b955-home\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:32:31.260957 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:31.260963    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66643e38-edf1-43c4-a3f7-199f17b5b955-kserve-provision-location\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:32:31.261183 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:31.260973    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-669v4\" (UniqueName: \"kubernetes.io/projected/66643e38-edf1-43c4-a3f7-199f17b5b955-kube-api-access-669v4\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:32:31.261183 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:31.260982    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66643e38-edf1-43c4-a3f7-199f17b5b955-tls-certs\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:32:31.261183 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:31.261010    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66643e38-edf1-43c4-a3f7-199f17b5b955-tmp-dir\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:32:31.261183 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:31.261018    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66643e38-edf1-43c4-a3f7-199f17b5b955-model-cache\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:32:31.261183 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:31.261026    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66643e38-edf1-43c4-a3f7-199f17b5b955-cabundle-cert\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:32:31.510607 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:31.510571    2559 generic.go:356] "Generic (PLEG): container finished" podID="66643e38-edf1-43c4-a3f7-199f17b5b955" containerID="3879434386d5d109692b44f0a8b478e3edb0a7671342a42f1d6fdd1579791a6a" exitCode=0
Aug 01 00:32:31.511096 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:31.510643    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz" event={"ID":"66643e38-edf1-43c4-a3f7-199f17b5b955","Type":"ContainerDied","Data":"3879434386d5d109692b44f0a8b478e3edb0a7671342a42f1d6fdd1579791a6a"}
Aug 01 00:32:31.511096 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:31.510706    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz" event={"ID":"66643e38-edf1-43c4-a3f7-199f17b5b955","Type":"ContainerDied","Data":"b1422339cfe3b3eff3735b429a29e7c823e5b5377c7e3aa6fa220a8c25531a2c"}
Aug 01 00:32:31.511096 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:31.510645    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz"
Aug 01 00:32:31.511096 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:31.510727    2559 scope.go:117] "RemoveContainer" containerID="3879434386d5d109692b44f0a8b478e3edb0a7671342a42f1d6fdd1579791a6a"
Aug 01 00:32:31.520391 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:31.520370    2559 scope.go:117] "RemoveContainer" containerID="4a2e82782397aaf8afd47eaa68a8b9794fe242cec3c090d6876b0ce8262bf699"
Aug 01 00:32:31.528598 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:31.528578    2559 scope.go:117] "RemoveContainer" containerID="3879434386d5d109692b44f0a8b478e3edb0a7671342a42f1d6fdd1579791a6a"
Aug 01 00:32:31.528884 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:32:31.528862    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3879434386d5d109692b44f0a8b478e3edb0a7671342a42f1d6fdd1579791a6a\": container with ID starting with 3879434386d5d109692b44f0a8b478e3edb0a7671342a42f1d6fdd1579791a6a not found: ID does not exist" containerID="3879434386d5d109692b44f0a8b478e3edb0a7671342a42f1d6fdd1579791a6a"
Aug 01 00:32:31.528969 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:31.528897    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3879434386d5d109692b44f0a8b478e3edb0a7671342a42f1d6fdd1579791a6a"} err="failed to get container status \"3879434386d5d109692b44f0a8b478e3edb0a7671342a42f1d6fdd1579791a6a\": rpc error: code = NotFound desc = could not find container \"3879434386d5d109692b44f0a8b478e3edb0a7671342a42f1d6fdd1579791a6a\": container with ID starting with 3879434386d5d109692b44f0a8b478e3edb0a7671342a42f1d6fdd1579791a6a not found: ID does not exist"
Aug 01 00:32:31.528969 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:31.528921    2559 scope.go:117] "RemoveContainer" containerID="4a2e82782397aaf8afd47eaa68a8b9794fe242cec3c090d6876b0ce8262bf699"
Aug 01 00:32:31.529180 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:32:31.529159    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a2e82782397aaf8afd47eaa68a8b9794fe242cec3c090d6876b0ce8262bf699\": container with ID starting with 4a2e82782397aaf8afd47eaa68a8b9794fe242cec3c090d6876b0ce8262bf699 not found: ID does not exist" containerID="4a2e82782397aaf8afd47eaa68a8b9794fe242cec3c090d6876b0ce8262bf699"
Aug 01 00:32:31.529224 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:31.529186    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a2e82782397aaf8afd47eaa68a8b9794fe242cec3c090d6876b0ce8262bf699"} err="failed to get container status \"4a2e82782397aaf8afd47eaa68a8b9794fe242cec3c090d6876b0ce8262bf699\": rpc error: code = NotFound desc = could not find container \"4a2e82782397aaf8afd47eaa68a8b9794fe242cec3c090d6876b0ce8262bf699\": container with ID starting with 4a2e82782397aaf8afd47eaa68a8b9794fe242cec3c090d6876b0ce8262bf699 not found: ID does not exist"
Aug 01 00:32:31.533473 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:31.533450    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz"]
Aug 01 00:32:31.536777 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:31.536756    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-58484ddfc-5rhfz"]
Aug 01 00:32:31.787023 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:31.786944    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66643e38-edf1-43c4-a3f7-199f17b5b955" path="/var/lib/kubelet/pods/66643e38-edf1-43c4-a3f7-199f17b5b955/volumes"
Aug 01 00:32:45.566690 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:45.566652    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5496985dd4-586db"]
Aug 01 00:32:45.567159 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:45.567106    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66643e38-edf1-43c4-a3f7-199f17b5b955" containerName="storage-initializer"
Aug 01 00:32:45.567159 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:45.567121    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="66643e38-edf1-43c4-a3f7-199f17b5b955" containerName="storage-initializer"
Aug 01 00:32:45.567159 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:45.567134    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66643e38-edf1-43c4-a3f7-199f17b5b955" containerName="main"
Aug 01 00:32:45.567159 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:45.567141    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="66643e38-edf1-43c4-a3f7-199f17b5b955" containerName="main"
Aug 01 00:32:45.567292 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:45.567209    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="66643e38-edf1-43c4-a3f7-199f17b5b955" containerName="main"
Aug 01 00:32:45.570543 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:45.570522    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5496985dd4-586db"
Aug 01 00:32:45.573193 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:45.573169    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Aug 01 00:32:45.573304 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:45.573277    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Aug 01 00:32:45.573508 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:45.573493    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"odh-kserve-custom-ca-bundle\""
Aug 01 00:32:45.573651 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:45.573629    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Aug 01 00:32:45.573797 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:45.573668    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-dnj7v\""
Aug 01 00:32:45.574178 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:45.574159    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"seaweedfs-s3-creds\""
Aug 01 00:32:45.585927 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:45.585901    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5496985dd4-586db"]
Aug 01 00:32:45.680868 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:45.680826    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec227919-d402-4a8d-bbcb-2a490983b334-home\") pod \"custom-route-timeout-test-kserve-5496985dd4-586db\" (UID: \"ec227919-d402-4a8d-bbcb-2a490983b334\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5496985dd4-586db"
Aug 01 00:32:45.681049 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:45.680884    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec227919-d402-4a8d-bbcb-2a490983b334-tmp-dir\") pod \"custom-route-timeout-test-kserve-5496985dd4-586db\" (UID: \"ec227919-d402-4a8d-bbcb-2a490983b334\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5496985dd4-586db"
Aug 01 00:32:45.681049 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:45.680904    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec227919-d402-4a8d-bbcb-2a490983b334-tls-certs\") pod \"custom-route-timeout-test-kserve-5496985dd4-586db\" (UID: \"ec227919-d402-4a8d-bbcb-2a490983b334\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5496985dd4-586db"
Aug 01 00:32:45.681049 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:45.680954    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec227919-d402-4a8d-bbcb-2a490983b334-dshm\") pod \"custom-route-timeout-test-kserve-5496985dd4-586db\" (UID: \"ec227919-d402-4a8d-bbcb-2a490983b334\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5496985dd4-586db"
Aug 01 00:32:45.681049 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:45.680982    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec227919-d402-4a8d-bbcb-2a490983b334-cabundle-cert\") pod \"custom-route-timeout-test-kserve-5496985dd4-586db\" (UID: \"ec227919-d402-4a8d-bbcb-2a490983b334\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5496985dd4-586db"
Aug 01 00:32:45.681049 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:45.681012    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec227919-d402-4a8d-bbcb-2a490983b334-model-cache\") pod \"custom-route-timeout-test-kserve-5496985dd4-586db\" (UID: \"ec227919-d402-4a8d-bbcb-2a490983b334\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5496985dd4-586db"
Aug 01 00:32:45.681049 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:45.681030    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c8pz\" (UniqueName: \"kubernetes.io/projected/ec227919-d402-4a8d-bbcb-2a490983b334-kube-api-access-8c8pz\") pod \"custom-route-timeout-test-kserve-5496985dd4-586db\" (UID: \"ec227919-d402-4a8d-bbcb-2a490983b334\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5496985dd4-586db"
Aug 01 00:32:45.681049 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:45.681049    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec227919-d402-4a8d-bbcb-2a490983b334-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-5496985dd4-586db\" (UID: \"ec227919-d402-4a8d-bbcb-2a490983b334\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5496985dd4-586db"
Aug 01 00:32:45.781863 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:45.781827    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec227919-d402-4a8d-bbcb-2a490983b334-model-cache\") pod \"custom-route-timeout-test-kserve-5496985dd4-586db\" (UID: \"ec227919-d402-4a8d-bbcb-2a490983b334\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5496985dd4-586db"
Aug 01 00:32:45.781863 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:45.781870    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c8pz\" (UniqueName: \"kubernetes.io/projected/ec227919-d402-4a8d-bbcb-2a490983b334-kube-api-access-8c8pz\") pod \"custom-route-timeout-test-kserve-5496985dd4-586db\" (UID: \"ec227919-d402-4a8d-bbcb-2a490983b334\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5496985dd4-586db"
Aug 01 00:32:45.782133 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:45.781897    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec227919-d402-4a8d-bbcb-2a490983b334-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-5496985dd4-586db\" (UID: \"ec227919-d402-4a8d-bbcb-2a490983b334\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5496985dd4-586db"
Aug 01 00:32:45.782133 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:45.781961    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec227919-d402-4a8d-bbcb-2a490983b334-home\") pod \"custom-route-timeout-test-kserve-5496985dd4-586db\" (UID: \"ec227919-d402-4a8d-bbcb-2a490983b334\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5496985dd4-586db"
Aug 01 00:32:45.782133 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:45.782007    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec227919-d402-4a8d-bbcb-2a490983b334-tmp-dir\") pod \"custom-route-timeout-test-kserve-5496985dd4-586db\" (UID: \"ec227919-d402-4a8d-bbcb-2a490983b334\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5496985dd4-586db"
Aug 01 00:32:45.782133 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:45.782028    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec227919-d402-4a8d-bbcb-2a490983b334-tls-certs\") pod \"custom-route-timeout-test-kserve-5496985dd4-586db\" (UID: \"ec227919-d402-4a8d-bbcb-2a490983b334\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5496985dd4-586db"
Aug 01 00:32:45.782133 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:45.782049    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec227919-d402-4a8d-bbcb-2a490983b334-dshm\") pod \"custom-route-timeout-test-kserve-5496985dd4-586db\" (UID: \"ec227919-d402-4a8d-bbcb-2a490983b334\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5496985dd4-586db"
Aug 01 00:32:45.782133 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:45.782070    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec227919-d402-4a8d-bbcb-2a490983b334-cabundle-cert\") pod \"custom-route-timeout-test-kserve-5496985dd4-586db\" (UID: \"ec227919-d402-4a8d-bbcb-2a490983b334\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5496985dd4-586db"
Aug 01 00:32:45.782440 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:45.782272    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec227919-d402-4a8d-bbcb-2a490983b334-model-cache\") pod \"custom-route-timeout-test-kserve-5496985dd4-586db\" (UID: \"ec227919-d402-4a8d-bbcb-2a490983b334\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5496985dd4-586db"
Aug 01 00:32:45.782499 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:45.782471    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec227919-d402-4a8d-bbcb-2a490983b334-tmp-dir\") pod \"custom-route-timeout-test-kserve-5496985dd4-586db\" (UID: \"ec227919-d402-4a8d-bbcb-2a490983b334\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5496985dd4-586db"
Aug 01 00:32:45.782900 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:45.782876    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec227919-d402-4a8d-bbcb-2a490983b334-cabundle-cert\") pod \"custom-route-timeout-test-kserve-5496985dd4-586db\" (UID: \"ec227919-d402-4a8d-bbcb-2a490983b334\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5496985dd4-586db"
Aug 01 00:32:45.783151 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:45.783126    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec227919-d402-4a8d-bbcb-2a490983b334-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-5496985dd4-586db\" (UID: \"ec227919-d402-4a8d-bbcb-2a490983b334\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5496985dd4-586db"
Aug 01 00:32:45.783248 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:45.783221    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec227919-d402-4a8d-bbcb-2a490983b334-home\") pod \"custom-route-timeout-test-kserve-5496985dd4-586db\" (UID: \"ec227919-d402-4a8d-bbcb-2a490983b334\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5496985dd4-586db"
Aug 01 00:32:45.785634 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:45.785603    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec227919-d402-4a8d-bbcb-2a490983b334-dshm\") pod \"custom-route-timeout-test-kserve-5496985dd4-586db\" (UID: \"ec227919-d402-4a8d-bbcb-2a490983b334\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5496985dd4-586db"
Aug 01 00:32:45.786067 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:45.786041    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec227919-d402-4a8d-bbcb-2a490983b334-tls-certs\") pod \"custom-route-timeout-test-kserve-5496985dd4-586db\" (UID: \"ec227919-d402-4a8d-bbcb-2a490983b334\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5496985dd4-586db"
Aug 01 00:32:45.795201 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:45.795175    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c8pz\" (UniqueName: \"kubernetes.io/projected/ec227919-d402-4a8d-bbcb-2a490983b334-kube-api-access-8c8pz\") pod \"custom-route-timeout-test-kserve-5496985dd4-586db\" (UID: \"ec227919-d402-4a8d-bbcb-2a490983b334\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5496985dd4-586db"
Aug 01 00:32:45.880911 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:45.880815    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5496985dd4-586db"
Aug 01 00:32:46.024506 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:46.024482    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5496985dd4-586db"]
Aug 01 00:32:46.026624 ip-10-0-130-148 kubenswrapper[2559]: W0801 00:32:46.026599    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec227919_d402_4a8d_bbcb_2a490983b334.slice/crio-ce403f0ac1b930efa407bafb821647fe547b56fce9b8cb15a29ca3786426d08b WatchSource:0}: Error finding container ce403f0ac1b930efa407bafb821647fe547b56fce9b8cb15a29ca3786426d08b: Status 404 returned error can't find the container with id ce403f0ac1b930efa407bafb821647fe547b56fce9b8cb15a29ca3786426d08b
Aug 01 00:32:46.580188 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:46.580144    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5496985dd4-586db" event={"ID":"ec227919-d402-4a8d-bbcb-2a490983b334","Type":"ContainerStarted","Data":"9074c49837811ba3e46e7f02b7519cc31674ce480122f820a8db6059b2c587ef"}
Aug 01 00:32:46.580664 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:46.580226    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5496985dd4-586db" event={"ID":"ec227919-d402-4a8d-bbcb-2a490983b334","Type":"ContainerStarted","Data":"ce403f0ac1b930efa407bafb821647fe547b56fce9b8cb15a29ca3786426d08b"}
Aug 01 00:32:48.591736 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:48.591699    2559 generic.go:356] "Generic (PLEG): container finished" podID="ec227919-d402-4a8d-bbcb-2a490983b334" containerID="9074c49837811ba3e46e7f02b7519cc31674ce480122f820a8db6059b2c587ef" exitCode=0
Aug 01 00:32:48.592196 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:48.591774    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5496985dd4-586db" event={"ID":"ec227919-d402-4a8d-bbcb-2a490983b334","Type":"ContainerDied","Data":"9074c49837811ba3e46e7f02b7519cc31674ce480122f820a8db6059b2c587ef"}
Aug 01 00:32:49.605669 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:49.605630    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5496985dd4-586db" event={"ID":"ec227919-d402-4a8d-bbcb-2a490983b334","Type":"ContainerStarted","Data":"ae2a1d37d5ff5f2ca9cab27713c212796a45b585a458dff442f313b2d05dedab"}
Aug 01 00:32:49.628483 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:49.628419    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5496985dd4-586db" podStartSLOduration=2.6127483419999997 podStartE2EDuration="4.628397977s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.015649635s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:32:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:32:49.626186357 +0000 UTC m=+2832.345986203" watchObservedRunningTime="2026-08-01 00:32:49.628397977 +0000 UTC m=+2832.348197826"
Aug 01 00:32:49.881760 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:49.881657    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5496985dd4-586db"
Aug 01 00:32:55.881892 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:55.881854    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5496985dd4-586db"
Aug 01 00:32:55.883587 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:32:55.883552    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5496985dd4-586db" podUID="ec227919-d402-4a8d-bbcb-2a490983b334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Aug 01 00:33:05.881441 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:33:05.881398    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5496985dd4-586db" podUID="ec227919-d402-4a8d-bbcb-2a490983b334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Aug 01 00:33:15.881251 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:33:15.881203    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5496985dd4-586db" podUID="ec227919-d402-4a8d-bbcb-2a490983b334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Aug 01 00:33:25.882075 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:33:25.882031    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5496985dd4-586db" podUID="ec227919-d402-4a8d-bbcb-2a490983b334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Aug 01 00:33:35.881946 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:33:35.881891    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5496985dd4-586db" podUID="ec227919-d402-4a8d-bbcb-2a490983b334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Aug 01 00:33:45.881630 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:33:45.881578    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5496985dd4-586db" podUID="ec227919-d402-4a8d-bbcb-2a490983b334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Aug 01 00:33:55.881275 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:33:55.881227    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5496985dd4-586db" podUID="ec227919-d402-4a8d-bbcb-2a490983b334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Aug 01 00:34:05.881520 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:34:05.881478    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5496985dd4-586db" podUID="ec227919-d402-4a8d-bbcb-2a490983b334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Aug 01 00:34:15.881816 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:34:15.881769    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5496985dd4-586db" podUID="ec227919-d402-4a8d-bbcb-2a490983b334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Aug 01 00:34:25.881619 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:34:25.881575    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5496985dd4-586db" podUID="ec227919-d402-4a8d-bbcb-2a490983b334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Aug 01 00:34:35.881424 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:34:35.881385    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5496985dd4-586db" podUID="ec227919-d402-4a8d-bbcb-2a490983b334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Aug 01 00:34:45.891503 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:34:45.891473    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5496985dd4-586db"
Aug 01 00:34:45.898941 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:34:45.898917    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5496985dd4-586db"
Aug 01 00:34:53.037760 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:34:53.037714    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5496985dd4-586db"]
Aug 01 00:35:08.059574 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:08.059470    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5496985dd4-586db" podUID="ec227919-d402-4a8d-bbcb-2a490983b334" containerName="main" containerID="cri-o://ae2a1d37d5ff5f2ca9cab27713c212796a45b585a458dff442f313b2d05dedab" gracePeriod=45
Aug 01 00:35:08.803543 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:08.803516    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5496985dd4-586db"
Aug 01 00:35:08.888295 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:08.888223    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec227919-d402-4a8d-bbcb-2a490983b334-kserve-provision-location\") pod \"ec227919-d402-4a8d-bbcb-2a490983b334\" (UID: \"ec227919-d402-4a8d-bbcb-2a490983b334\") "
Aug 01 00:35:08.888295 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:08.888257    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec227919-d402-4a8d-bbcb-2a490983b334-cabundle-cert\") pod \"ec227919-d402-4a8d-bbcb-2a490983b334\" (UID: \"ec227919-d402-4a8d-bbcb-2a490983b334\") "
Aug 01 00:35:08.888295 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:08.888282    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8c8pz\" (UniqueName: \"kubernetes.io/projected/ec227919-d402-4a8d-bbcb-2a490983b334-kube-api-access-8c8pz\") pod \"ec227919-d402-4a8d-bbcb-2a490983b334\" (UID: \"ec227919-d402-4a8d-bbcb-2a490983b334\") "
Aug 01 00:35:08.888581 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:08.888327    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec227919-d402-4a8d-bbcb-2a490983b334-tmp-dir\") pod \"ec227919-d402-4a8d-bbcb-2a490983b334\" (UID: \"ec227919-d402-4a8d-bbcb-2a490983b334\") "
Aug 01 00:35:08.888581 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:08.888348    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec227919-d402-4a8d-bbcb-2a490983b334-home\") pod \"ec227919-d402-4a8d-bbcb-2a490983b334\" (UID: \"ec227919-d402-4a8d-bbcb-2a490983b334\") "
Aug 01 00:35:08.888581 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:08.888514    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec227919-d402-4a8d-bbcb-2a490983b334-tls-certs\") pod \"ec227919-d402-4a8d-bbcb-2a490983b334\" (UID: \"ec227919-d402-4a8d-bbcb-2a490983b334\") "
Aug 01 00:35:08.888581 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:08.888570    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec227919-d402-4a8d-bbcb-2a490983b334-dshm\") pod \"ec227919-d402-4a8d-bbcb-2a490983b334\" (UID: \"ec227919-d402-4a8d-bbcb-2a490983b334\") "
Aug 01 00:35:08.888827 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:08.888612    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec227919-d402-4a8d-bbcb-2a490983b334-model-cache\") pod \"ec227919-d402-4a8d-bbcb-2a490983b334\" (UID: \"ec227919-d402-4a8d-bbcb-2a490983b334\") "
Aug 01 00:35:08.888827 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:08.888676    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec227919-d402-4a8d-bbcb-2a490983b334-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ec227919-d402-4a8d-bbcb-2a490983b334" (UID: "ec227919-d402-4a8d-bbcb-2a490983b334"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:35:08.888946 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:08.888889    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec227919-d402-4a8d-bbcb-2a490983b334-cabundle-cert\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:35:08.889168 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:08.889135    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec227919-d402-4a8d-bbcb-2a490983b334-home" (OuterVolumeSpecName: "home") pod "ec227919-d402-4a8d-bbcb-2a490983b334" (UID: "ec227919-d402-4a8d-bbcb-2a490983b334"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:35:08.889249 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:08.889170    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec227919-d402-4a8d-bbcb-2a490983b334-model-cache" (OuterVolumeSpecName: "model-cache") pod "ec227919-d402-4a8d-bbcb-2a490983b334" (UID: "ec227919-d402-4a8d-bbcb-2a490983b334"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:35:08.890969 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:08.890945    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec227919-d402-4a8d-bbcb-2a490983b334-dshm" (OuterVolumeSpecName: "dshm") pod "ec227919-d402-4a8d-bbcb-2a490983b334" (UID: "ec227919-d402-4a8d-bbcb-2a490983b334"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:35:08.891145 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:08.891120    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec227919-d402-4a8d-bbcb-2a490983b334-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ec227919-d402-4a8d-bbcb-2a490983b334" (UID: "ec227919-d402-4a8d-bbcb-2a490983b334"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:35:08.891199 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:08.891135    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec227919-d402-4a8d-bbcb-2a490983b334-kube-api-access-8c8pz" (OuterVolumeSpecName: "kube-api-access-8c8pz") pod "ec227919-d402-4a8d-bbcb-2a490983b334" (UID: "ec227919-d402-4a8d-bbcb-2a490983b334"). InnerVolumeSpecName "kube-api-access-8c8pz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:35:08.900813 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:08.900789    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec227919-d402-4a8d-bbcb-2a490983b334-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ec227919-d402-4a8d-bbcb-2a490983b334" (UID: "ec227919-d402-4a8d-bbcb-2a490983b334"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:35:08.920033 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:08.920007    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec227919-d402-4a8d-bbcb-2a490983b334-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ec227919-d402-4a8d-bbcb-2a490983b334" (UID: "ec227919-d402-4a8d-bbcb-2a490983b334"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:35:08.990059 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:08.990032    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec227919-d402-4a8d-bbcb-2a490983b334-tls-certs\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:35:08.990059 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:08.990059    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec227919-d402-4a8d-bbcb-2a490983b334-dshm\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:35:08.990210 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:08.990067    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec227919-d402-4a8d-bbcb-2a490983b334-model-cache\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:35:08.990210 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:08.990076    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec227919-d402-4a8d-bbcb-2a490983b334-kserve-provision-location\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:35:08.990210 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:08.990086    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c8pz\" (UniqueName: \"kubernetes.io/projected/ec227919-d402-4a8d-bbcb-2a490983b334-kube-api-access-8c8pz\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:35:08.990210 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:08.990094    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec227919-d402-4a8d-bbcb-2a490983b334-tmp-dir\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:35:08.990210 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:08.990104    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec227919-d402-4a8d-bbcb-2a490983b334-home\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:35:09.250204 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:09.250167    2559 generic.go:356] "Generic (PLEG): container finished" podID="ec227919-d402-4a8d-bbcb-2a490983b334" containerID="ae2a1d37d5ff5f2ca9cab27713c212796a45b585a458dff442f313b2d05dedab" exitCode=0
Aug 01 00:35:09.250611 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:09.250227    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5496985dd4-586db"
Aug 01 00:35:09.250611 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:09.250250    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5496985dd4-586db" event={"ID":"ec227919-d402-4a8d-bbcb-2a490983b334","Type":"ContainerDied","Data":"ae2a1d37d5ff5f2ca9cab27713c212796a45b585a458dff442f313b2d05dedab"}
Aug 01 00:35:09.250611 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:09.250328    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5496985dd4-586db" event={"ID":"ec227919-d402-4a8d-bbcb-2a490983b334","Type":"ContainerDied","Data":"ce403f0ac1b930efa407bafb821647fe547b56fce9b8cb15a29ca3786426d08b"}
Aug 01 00:35:09.250611 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:09.250351    2559 scope.go:117] "RemoveContainer" containerID="ae2a1d37d5ff5f2ca9cab27713c212796a45b585a458dff442f313b2d05dedab"
Aug 01 00:35:09.259612 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:09.259598    2559 scope.go:117] "RemoveContainer" containerID="9074c49837811ba3e46e7f02b7519cc31674ce480122f820a8db6059b2c587ef"
Aug 01 00:35:09.266856 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:09.266836    2559 scope.go:117] "RemoveContainer" containerID="ae2a1d37d5ff5f2ca9cab27713c212796a45b585a458dff442f313b2d05dedab"
Aug 01 00:35:09.267115 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:35:09.267096    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae2a1d37d5ff5f2ca9cab27713c212796a45b585a458dff442f313b2d05dedab\": container with ID starting with ae2a1d37d5ff5f2ca9cab27713c212796a45b585a458dff442f313b2d05dedab not found: ID does not exist" containerID="ae2a1d37d5ff5f2ca9cab27713c212796a45b585a458dff442f313b2d05dedab"
Aug 01 00:35:09.267165 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:09.267122    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae2a1d37d5ff5f2ca9cab27713c212796a45b585a458dff442f313b2d05dedab"} err="failed to get container status \"ae2a1d37d5ff5f2ca9cab27713c212796a45b585a458dff442f313b2d05dedab\": rpc error: code = NotFound desc = could not find container \"ae2a1d37d5ff5f2ca9cab27713c212796a45b585a458dff442f313b2d05dedab\": container with ID starting with ae2a1d37d5ff5f2ca9cab27713c212796a45b585a458dff442f313b2d05dedab not found: ID does not exist"
Aug 01 00:35:09.267165 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:09.267140    2559 scope.go:117] "RemoveContainer" containerID="9074c49837811ba3e46e7f02b7519cc31674ce480122f820a8db6059b2c587ef"
Aug 01 00:35:09.267358 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:35:09.267344    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9074c49837811ba3e46e7f02b7519cc31674ce480122f820a8db6059b2c587ef\": container with ID starting with 9074c49837811ba3e46e7f02b7519cc31674ce480122f820a8db6059b2c587ef not found: ID does not exist" containerID="9074c49837811ba3e46e7f02b7519cc31674ce480122f820a8db6059b2c587ef"
Aug 01 00:35:09.267402 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:09.267360    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9074c49837811ba3e46e7f02b7519cc31674ce480122f820a8db6059b2c587ef"} err="failed to get container status \"9074c49837811ba3e46e7f02b7519cc31674ce480122f820a8db6059b2c587ef\": rpc error: code = NotFound desc = could not find container \"9074c49837811ba3e46e7f02b7519cc31674ce480122f820a8db6059b2c587ef\": container with ID starting with 9074c49837811ba3e46e7f02b7519cc31674ce480122f820a8db6059b2c587ef not found: ID does not exist"
Aug 01 00:35:09.272316 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:09.272296    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5496985dd4-586db"]
Aug 01 00:35:09.277958 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:09.277936    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5496985dd4-586db"]
Aug 01 00:35:09.786095 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:09.786063    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec227919-d402-4a8d-bbcb-2a490983b334" path="/var/lib/kubelet/pods/ec227919-d402-4a8d-bbcb-2a490983b334/volumes"
Aug 01 00:35:37.578260 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:37.578224    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5f8bf7dc88-gwn4k"]
Aug 01 00:35:37.578732 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:37.578711    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec227919-d402-4a8d-bbcb-2a490983b334" containerName="storage-initializer"
Aug 01 00:35:37.578828 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:37.578736    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec227919-d402-4a8d-bbcb-2a490983b334" containerName="storage-initializer"
Aug 01 00:35:37.578828 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:37.578785    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec227919-d402-4a8d-bbcb-2a490983b334" containerName="main"
Aug 01 00:35:37.578828 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:37.578795    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec227919-d402-4a8d-bbcb-2a490983b334" containerName="main"
Aug 01 00:35:37.578995 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:37.578888    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec227919-d402-4a8d-bbcb-2a490983b334" containerName="main"
Aug 01 00:35:37.582082 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:37.582064    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5f8bf7dc88-gwn4k"
Aug 01 00:35:37.584876 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:37.584838    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-b4k48\""
Aug 01 00:35:37.584987 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:37.584952    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Aug 01 00:35:37.585069 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:37.585002    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"seaweedfs-s3-creds\""
Aug 01 00:35:37.585069 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:37.585011    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Aug 01 00:35:37.585174 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:37.585103    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"odh-kserve-custom-ca-bundle\""
Aug 01 00:35:37.585369 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:37.585353    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Aug 01 00:35:37.593220 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:37.593195    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5f8bf7dc88-gwn4k"]
Aug 01 00:35:37.728585 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:37.728547    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a65aa9a-6963-41ca-92a9-0907a9ff0d91-tls-certs\") pod \"router-with-refs-test-kserve-5f8bf7dc88-gwn4k\" (UID: \"4a65aa9a-6963-41ca-92a9-0907a9ff0d91\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5f8bf7dc88-gwn4k"
Aug 01 00:35:37.728585 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:37.728585    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqkvf\" (UniqueName: \"kubernetes.io/projected/4a65aa9a-6963-41ca-92a9-0907a9ff0d91-kube-api-access-xqkvf\") pod \"router-with-refs-test-kserve-5f8bf7dc88-gwn4k\" (UID: \"4a65aa9a-6963-41ca-92a9-0907a9ff0d91\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5f8bf7dc88-gwn4k"
Aug 01 00:35:37.728852 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:37.728614    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a65aa9a-6963-41ca-92a9-0907a9ff0d91-dshm\") pod \"router-with-refs-test-kserve-5f8bf7dc88-gwn4k\" (UID: \"4a65aa9a-6963-41ca-92a9-0907a9ff0d91\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5f8bf7dc88-gwn4k"
Aug 01 00:35:37.728852 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:37.728650    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a65aa9a-6963-41ca-92a9-0907a9ff0d91-home\") pod \"router-with-refs-test-kserve-5f8bf7dc88-gwn4k\" (UID: \"4a65aa9a-6963-41ca-92a9-0907a9ff0d91\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5f8bf7dc88-gwn4k"
Aug 01 00:35:37.728852 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:37.728670    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a65aa9a-6963-41ca-92a9-0907a9ff0d91-tmp-dir\") pod \"router-with-refs-test-kserve-5f8bf7dc88-gwn4k\" (UID: \"4a65aa9a-6963-41ca-92a9-0907a9ff0d91\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5f8bf7dc88-gwn4k"
Aug 01 00:35:37.728852 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:37.728720    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a65aa9a-6963-41ca-92a9-0907a9ff0d91-cabundle-cert\") pod \"router-with-refs-test-kserve-5f8bf7dc88-gwn4k\" (UID: \"4a65aa9a-6963-41ca-92a9-0907a9ff0d91\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5f8bf7dc88-gwn4k"
Aug 01 00:35:37.728852 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:37.728766    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a65aa9a-6963-41ca-92a9-0907a9ff0d91-kserve-provision-location\") pod \"router-with-refs-test-kserve-5f8bf7dc88-gwn4k\" (UID: \"4a65aa9a-6963-41ca-92a9-0907a9ff0d91\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5f8bf7dc88-gwn4k"
Aug 01 00:35:37.728852 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:37.728788    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a65aa9a-6963-41ca-92a9-0907a9ff0d91-model-cache\") pod \"router-with-refs-test-kserve-5f8bf7dc88-gwn4k\" (UID: \"4a65aa9a-6963-41ca-92a9-0907a9ff0d91\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5f8bf7dc88-gwn4k"
Aug 01 00:35:37.829816 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:37.829710    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a65aa9a-6963-41ca-92a9-0907a9ff0d91-tls-certs\") pod \"router-with-refs-test-kserve-5f8bf7dc88-gwn4k\" (UID: \"4a65aa9a-6963-41ca-92a9-0907a9ff0d91\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5f8bf7dc88-gwn4k"
Aug 01 00:35:37.829816 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:37.829772    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqkvf\" (UniqueName: \"kubernetes.io/projected/4a65aa9a-6963-41ca-92a9-0907a9ff0d91-kube-api-access-xqkvf\") pod \"router-with-refs-test-kserve-5f8bf7dc88-gwn4k\" (UID: \"4a65aa9a-6963-41ca-92a9-0907a9ff0d91\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5f8bf7dc88-gwn4k"
Aug 01 00:35:37.829816 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:37.829799    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a65aa9a-6963-41ca-92a9-0907a9ff0d91-dshm\") pod \"router-with-refs-test-kserve-5f8bf7dc88-gwn4k\" (UID: \"4a65aa9a-6963-41ca-92a9-0907a9ff0d91\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5f8bf7dc88-gwn4k"
Aug 01 00:35:37.830107 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:37.829907    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a65aa9a-6963-41ca-92a9-0907a9ff0d91-home\") pod \"router-with-refs-test-kserve-5f8bf7dc88-gwn4k\" (UID: \"4a65aa9a-6963-41ca-92a9-0907a9ff0d91\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5f8bf7dc88-gwn4k"
Aug 01 00:35:37.830107 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:37.829942    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a65aa9a-6963-41ca-92a9-0907a9ff0d91-tmp-dir\") pod \"router-with-refs-test-kserve-5f8bf7dc88-gwn4k\" (UID: \"4a65aa9a-6963-41ca-92a9-0907a9ff0d91\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5f8bf7dc88-gwn4k"
Aug 01 00:35:37.830107 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:37.829994    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a65aa9a-6963-41ca-92a9-0907a9ff0d91-cabundle-cert\") pod \"router-with-refs-test-kserve-5f8bf7dc88-gwn4k\" (UID: \"4a65aa9a-6963-41ca-92a9-0907a9ff0d91\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5f8bf7dc88-gwn4k"
Aug 01 00:35:37.830107 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:37.830071    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a65aa9a-6963-41ca-92a9-0907a9ff0d91-kserve-provision-location\") pod \"router-with-refs-test-kserve-5f8bf7dc88-gwn4k\" (UID: \"4a65aa9a-6963-41ca-92a9-0907a9ff0d91\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5f8bf7dc88-gwn4k"
Aug 01 00:35:37.830334 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:37.830142    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a65aa9a-6963-41ca-92a9-0907a9ff0d91-model-cache\") pod \"router-with-refs-test-kserve-5f8bf7dc88-gwn4k\" (UID: \"4a65aa9a-6963-41ca-92a9-0907a9ff0d91\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5f8bf7dc88-gwn4k"
Aug 01 00:35:37.830563 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:37.830538    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a65aa9a-6963-41ca-92a9-0907a9ff0d91-tmp-dir\") pod \"router-with-refs-test-kserve-5f8bf7dc88-gwn4k\" (UID: \"4a65aa9a-6963-41ca-92a9-0907a9ff0d91\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5f8bf7dc88-gwn4k"
Aug 01 00:35:37.830668 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:37.830584    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a65aa9a-6963-41ca-92a9-0907a9ff0d91-kserve-provision-location\") pod \"router-with-refs-test-kserve-5f8bf7dc88-gwn4k\" (UID: \"4a65aa9a-6963-41ca-92a9-0907a9ff0d91\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5f8bf7dc88-gwn4k"
Aug 01 00:35:37.830668 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:37.830604    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a65aa9a-6963-41ca-92a9-0907a9ff0d91-home\") pod \"router-with-refs-test-kserve-5f8bf7dc88-gwn4k\" (UID: \"4a65aa9a-6963-41ca-92a9-0907a9ff0d91\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5f8bf7dc88-gwn4k"
Aug 01 00:35:37.830825 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:37.830687    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a65aa9a-6963-41ca-92a9-0907a9ff0d91-model-cache\") pod \"router-with-refs-test-kserve-5f8bf7dc88-gwn4k\" (UID: \"4a65aa9a-6963-41ca-92a9-0907a9ff0d91\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5f8bf7dc88-gwn4k"
Aug 01 00:35:37.832121 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:37.832101    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a65aa9a-6963-41ca-92a9-0907a9ff0d91-dshm\") pod \"router-with-refs-test-kserve-5f8bf7dc88-gwn4k\" (UID: \"4a65aa9a-6963-41ca-92a9-0907a9ff0d91\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5f8bf7dc88-gwn4k"
Aug 01 00:35:37.832530 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:37.832513    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"odh-kserve-custom-ca-bundle\""
Aug 01 00:35:37.832611 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:37.832559    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Aug 01 00:35:37.840726 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:37.840695    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a65aa9a-6963-41ca-92a9-0907a9ff0d91-cabundle-cert\") pod \"router-with-refs-test-kserve-5f8bf7dc88-gwn4k\" (UID: \"4a65aa9a-6963-41ca-92a9-0907a9ff0d91\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5f8bf7dc88-gwn4k"
Aug 01 00:35:37.842363 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:37.842342    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a65aa9a-6963-41ca-92a9-0907a9ff0d91-tls-certs\") pod \"router-with-refs-test-kserve-5f8bf7dc88-gwn4k\" (UID: \"4a65aa9a-6963-41ca-92a9-0907a9ff0d91\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5f8bf7dc88-gwn4k"
Aug 01 00:35:37.842702 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:37.842684    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Aug 01 00:35:37.852506 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:37.852487    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Aug 01 00:35:37.862848 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:37.862826    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqkvf\" (UniqueName: \"kubernetes.io/projected/4a65aa9a-6963-41ca-92a9-0907a9ff0d91-kube-api-access-xqkvf\") pod \"router-with-refs-test-kserve-5f8bf7dc88-gwn4k\" (UID: \"4a65aa9a-6963-41ca-92a9-0907a9ff0d91\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5f8bf7dc88-gwn4k"
Aug 01 00:35:37.895644 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:37.895609    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-b4k48\""
Aug 01 00:35:37.903703 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:37.903680    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5f8bf7dc88-gwn4k"
Aug 01 00:35:37.940724 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:37.940694    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-spblg_c2e50946-aea4-4886-82ec-0bd5a31196cf/console-operator/1.log"
Aug 01 00:35:37.951642 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:37.951615    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-spblg_c2e50946-aea4-4886-82ec-0bd5a31196cf/console-operator/1.log"
Aug 01 00:35:38.052114 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:38.052089    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5f8bf7dc88-gwn4k"]
Aug 01 00:35:38.054193 ip-10-0-130-148 kubenswrapper[2559]: W0801 00:35:38.054168    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a65aa9a_6963_41ca_92a9_0907a9ff0d91.slice/crio-911bff2e62554668cbf43d59ec5aa14980f416b19b6614664e125d5dcf097850 WatchSource:0}: Error finding container 911bff2e62554668cbf43d59ec5aa14980f416b19b6614664e125d5dcf097850: Status 404 returned error can't find the container with id 911bff2e62554668cbf43d59ec5aa14980f416b19b6614664e125d5dcf097850
Aug 01 00:35:38.058558 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:38.058541    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"seaweedfs-s3-creds\""
Aug 01 00:35:38.384195 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:38.384106    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5f8bf7dc88-gwn4k" event={"ID":"4a65aa9a-6963-41ca-92a9-0907a9ff0d91","Type":"ContainerStarted","Data":"e02e76e253ad741efd50a5e66ae1e2d1bcd63412b63364561202bbe6d9f768e3"}
Aug 01 00:35:38.384195 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:38.384158    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5f8bf7dc88-gwn4k" event={"ID":"4a65aa9a-6963-41ca-92a9-0907a9ff0d91","Type":"ContainerStarted","Data":"911bff2e62554668cbf43d59ec5aa14980f416b19b6614664e125d5dcf097850"}
Aug 01 00:35:40.394676 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:40.394644    2559 generic.go:356] "Generic (PLEG): container finished" podID="4a65aa9a-6963-41ca-92a9-0907a9ff0d91" containerID="e02e76e253ad741efd50a5e66ae1e2d1bcd63412b63364561202bbe6d9f768e3" exitCode=0
Aug 01 00:35:40.395063 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:40.394698    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5f8bf7dc88-gwn4k" event={"ID":"4a65aa9a-6963-41ca-92a9-0907a9ff0d91","Type":"ContainerDied","Data":"e02e76e253ad741efd50a5e66ae1e2d1bcd63412b63364561202bbe6d9f768e3"}
Aug 01 00:35:41.401638 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:41.401594    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5f8bf7dc88-gwn4k" event={"ID":"4a65aa9a-6963-41ca-92a9-0907a9ff0d91","Type":"ContainerStarted","Data":"dcf98d8e913b07bdff9e2bfe929503513203761c90b3a56d39280642915ab0ca"}
Aug 01 00:35:41.425134 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:41.425080    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5f8bf7dc88-gwn4k" podStartSLOduration=2.409232567 podStartE2EDuration="4.425062032s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.015829465s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:35:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:35:41.422436624 +0000 UTC m=+3004.142236693" watchObservedRunningTime="2026-08-01 00:35:41.425062032 +0000 UTC m=+3004.144861890"
Aug 01 00:35:41.904002 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:41.903965    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5f8bf7dc88-gwn4k"
Aug 01 00:35:47.904086 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:47.904043    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5f8bf7dc88-gwn4k"
Aug 01 00:35:47.905636 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:47.905601    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5f8bf7dc88-gwn4k" podUID="4a65aa9a-6963-41ca-92a9-0907a9ff0d91" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Aug 01 00:35:57.904944 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:35:57.904889    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5f8bf7dc88-gwn4k" podUID="4a65aa9a-6963-41ca-92a9-0907a9ff0d91" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Aug 01 00:36:07.905153 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:36:07.905108    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5f8bf7dc88-gwn4k" podUID="4a65aa9a-6963-41ca-92a9-0907a9ff0d91" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Aug 01 00:36:17.904614 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:36:17.904554    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5f8bf7dc88-gwn4k" podUID="4a65aa9a-6963-41ca-92a9-0907a9ff0d91" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Aug 01 00:36:27.904666 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:36:27.904618    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5f8bf7dc88-gwn4k" podUID="4a65aa9a-6963-41ca-92a9-0907a9ff0d91" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Aug 01 00:36:37.904555 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:36:37.904494    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5f8bf7dc88-gwn4k" podUID="4a65aa9a-6963-41ca-92a9-0907a9ff0d91" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Aug 01 00:36:47.905012 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:36:47.904958    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5f8bf7dc88-gwn4k" podUID="4a65aa9a-6963-41ca-92a9-0907a9ff0d91" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Aug 01 00:36:57.904648 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:36:57.904595    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5f8bf7dc88-gwn4k" podUID="4a65aa9a-6963-41ca-92a9-0907a9ff0d91" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Aug 01 00:37:07.904273 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:37:07.904225    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5f8bf7dc88-gwn4k" podUID="4a65aa9a-6963-41ca-92a9-0907a9ff0d91" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Aug 01 00:37:17.904979 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:37:17.904929    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5f8bf7dc88-gwn4k" podUID="4a65aa9a-6963-41ca-92a9-0907a9ff0d91" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Aug 01 00:37:27.904925 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:37:27.904881    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5f8bf7dc88-gwn4k" podUID="4a65aa9a-6963-41ca-92a9-0907a9ff0d91" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Aug 01 00:37:37.914440 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:37:37.914409    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5f8bf7dc88-gwn4k"
Aug 01 00:37:37.922667 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:37:37.922628    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5f8bf7dc88-gwn4k"
Aug 01 00:37:42.433561 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:37:42.433522    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5f8bf7dc88-gwn4k"]
Aug 01 00:37:57.456503 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:37:57.456441    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5f8bf7dc88-gwn4k" podUID="4a65aa9a-6963-41ca-92a9-0907a9ff0d91" containerName="main" containerID="cri-o://dcf98d8e913b07bdff9e2bfe929503513203761c90b3a56d39280642915ab0ca" gracePeriod=45
Aug 01 00:37:57.915731 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:37:57.915633    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5f8bf7dc88-gwn4k" podUID="4a65aa9a-6963-41ca-92a9-0907a9ff0d91" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Aug 01 00:37:58.197963 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:37:58.197937    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5f8bf7dc88-gwn4k"
Aug 01 00:37:58.343730 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:37:58.343684    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a65aa9a-6963-41ca-92a9-0907a9ff0d91-cabundle-cert\") pod \"4a65aa9a-6963-41ca-92a9-0907a9ff0d91\" (UID: \"4a65aa9a-6963-41ca-92a9-0907a9ff0d91\") "
Aug 01 00:37:58.344000 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:37:58.343796    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xqkvf\" (UniqueName: \"kubernetes.io/projected/4a65aa9a-6963-41ca-92a9-0907a9ff0d91-kube-api-access-xqkvf\") pod \"4a65aa9a-6963-41ca-92a9-0907a9ff0d91\" (UID: \"4a65aa9a-6963-41ca-92a9-0907a9ff0d91\") "
Aug 01 00:37:58.344000 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:37:58.343835    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a65aa9a-6963-41ca-92a9-0907a9ff0d91-dshm\") pod \"4a65aa9a-6963-41ca-92a9-0907a9ff0d91\" (UID: \"4a65aa9a-6963-41ca-92a9-0907a9ff0d91\") "
Aug 01 00:37:58.344000 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:37:58.343860    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a65aa9a-6963-41ca-92a9-0907a9ff0d91-kserve-provision-location\") pod \"4a65aa9a-6963-41ca-92a9-0907a9ff0d91\" (UID: \"4a65aa9a-6963-41ca-92a9-0907a9ff0d91\") "
Aug 01 00:37:58.344000 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:37:58.343897    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a65aa9a-6963-41ca-92a9-0907a9ff0d91-tmp-dir\") pod \"4a65aa9a-6963-41ca-92a9-0907a9ff0d91\" (UID: \"4a65aa9a-6963-41ca-92a9-0907a9ff0d91\") "
Aug 01 00:37:58.344000 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:37:58.343936    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a65aa9a-6963-41ca-92a9-0907a9ff0d91-tls-certs\") pod \"4a65aa9a-6963-41ca-92a9-0907a9ff0d91\" (UID: \"4a65aa9a-6963-41ca-92a9-0907a9ff0d91\") "
Aug 01 00:37:58.344000 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:37:58.343979    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a65aa9a-6963-41ca-92a9-0907a9ff0d91-model-cache\") pod \"4a65aa9a-6963-41ca-92a9-0907a9ff0d91\" (UID: \"4a65aa9a-6963-41ca-92a9-0907a9ff0d91\") "
Aug 01 00:37:58.344311 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:37:58.344026    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a65aa9a-6963-41ca-92a9-0907a9ff0d91-home\") pod \"4a65aa9a-6963-41ca-92a9-0907a9ff0d91\" (UID: \"4a65aa9a-6963-41ca-92a9-0907a9ff0d91\") "
Aug 01 00:37:58.344311 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:37:58.344178    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a65aa9a-6963-41ca-92a9-0907a9ff0d91-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4a65aa9a-6963-41ca-92a9-0907a9ff0d91" (UID: "4a65aa9a-6963-41ca-92a9-0907a9ff0d91"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:37:58.344424 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:37:58.344296    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a65aa9a-6963-41ca-92a9-0907a9ff0d91-cabundle-cert\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:37:58.344959 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:37:58.344383    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a65aa9a-6963-41ca-92a9-0907a9ff0d91-model-cache" (OuterVolumeSpecName: "model-cache") pod "4a65aa9a-6963-41ca-92a9-0907a9ff0d91" (UID: "4a65aa9a-6963-41ca-92a9-0907a9ff0d91"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:37:58.345058 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:37:58.344772    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a65aa9a-6963-41ca-92a9-0907a9ff0d91-home" (OuterVolumeSpecName: "home") pod "4a65aa9a-6963-41ca-92a9-0907a9ff0d91" (UID: "4a65aa9a-6963-41ca-92a9-0907a9ff0d91"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:37:58.346282 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:37:58.346257    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a65aa9a-6963-41ca-92a9-0907a9ff0d91-kube-api-access-xqkvf" (OuterVolumeSpecName: "kube-api-access-xqkvf") pod "4a65aa9a-6963-41ca-92a9-0907a9ff0d91" (UID: "4a65aa9a-6963-41ca-92a9-0907a9ff0d91"). InnerVolumeSpecName "kube-api-access-xqkvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:37:58.346405 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:37:58.346356    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a65aa9a-6963-41ca-92a9-0907a9ff0d91-dshm" (OuterVolumeSpecName: "dshm") pod "4a65aa9a-6963-41ca-92a9-0907a9ff0d91" (UID: "4a65aa9a-6963-41ca-92a9-0907a9ff0d91"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:37:58.346529 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:37:58.346509    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a65aa9a-6963-41ca-92a9-0907a9ff0d91-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4a65aa9a-6963-41ca-92a9-0907a9ff0d91" (UID: "4a65aa9a-6963-41ca-92a9-0907a9ff0d91"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:37:58.362760 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:37:58.362710    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a65aa9a-6963-41ca-92a9-0907a9ff0d91-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4a65aa9a-6963-41ca-92a9-0907a9ff0d91" (UID: "4a65aa9a-6963-41ca-92a9-0907a9ff0d91"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:37:58.374774 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:37:58.374735    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a65aa9a-6963-41ca-92a9-0907a9ff0d91-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4a65aa9a-6963-41ca-92a9-0907a9ff0d91" (UID: "4a65aa9a-6963-41ca-92a9-0907a9ff0d91"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:37:58.444812 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:37:58.444782    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a65aa9a-6963-41ca-92a9-0907a9ff0d91-home\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:37:58.444954 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:37:58.444814    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqkvf\" (UniqueName: \"kubernetes.io/projected/4a65aa9a-6963-41ca-92a9-0907a9ff0d91-kube-api-access-xqkvf\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:37:58.444954 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:37:58.444826    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a65aa9a-6963-41ca-92a9-0907a9ff0d91-dshm\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:37:58.444954 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:37:58.444835    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a65aa9a-6963-41ca-92a9-0907a9ff0d91-kserve-provision-location\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:37:58.444954 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:37:58.444844    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a65aa9a-6963-41ca-92a9-0907a9ff0d91-tmp-dir\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:37:58.444954 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:37:58.444852    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a65aa9a-6963-41ca-92a9-0907a9ff0d91-tls-certs\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:37:58.444954 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:37:58.444860    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a65aa9a-6963-41ca-92a9-0907a9ff0d91-model-cache\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:37:59.032231 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:37:59.032190    2559 generic.go:356] "Generic (PLEG): container finished" podID="4a65aa9a-6963-41ca-92a9-0907a9ff0d91" containerID="dcf98d8e913b07bdff9e2bfe929503513203761c90b3a56d39280642915ab0ca" exitCode=0
Aug 01 00:37:59.032835 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:37:59.032251    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5f8bf7dc88-gwn4k"
Aug 01 00:37:59.032835 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:37:59.032267    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5f8bf7dc88-gwn4k" event={"ID":"4a65aa9a-6963-41ca-92a9-0907a9ff0d91","Type":"ContainerDied","Data":"dcf98d8e913b07bdff9e2bfe929503513203761c90b3a56d39280642915ab0ca"}
Aug 01 00:37:59.032835 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:37:59.032323    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5f8bf7dc88-gwn4k" event={"ID":"4a65aa9a-6963-41ca-92a9-0907a9ff0d91","Type":"ContainerDied","Data":"911bff2e62554668cbf43d59ec5aa14980f416b19b6614664e125d5dcf097850"}
Aug 01 00:37:59.032835 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:37:59.032347    2559 scope.go:117] "RemoveContainer" containerID="dcf98d8e913b07bdff9e2bfe929503513203761c90b3a56d39280642915ab0ca"
Aug 01 00:37:59.044572 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:37:59.044550    2559 scope.go:117] "RemoveContainer" containerID="e02e76e253ad741efd50a5e66ae1e2d1bcd63412b63364561202bbe6d9f768e3"
Aug 01 00:37:59.053247 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:37:59.053231    2559 scope.go:117] "RemoveContainer" containerID="dcf98d8e913b07bdff9e2bfe929503513203761c90b3a56d39280642915ab0ca"
Aug 01 00:37:59.053480 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:37:59.053463    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcf98d8e913b07bdff9e2bfe929503513203761c90b3a56d39280642915ab0ca\": container with ID starting with dcf98d8e913b07bdff9e2bfe929503513203761c90b3a56d39280642915ab0ca not found: ID does not exist" containerID="dcf98d8e913b07bdff9e2bfe929503513203761c90b3a56d39280642915ab0ca"
Aug 01 00:37:59.053543 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:37:59.053486    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcf98d8e913b07bdff9e2bfe929503513203761c90b3a56d39280642915ab0ca"} err="failed to get container status \"dcf98d8e913b07bdff9e2bfe929503513203761c90b3a56d39280642915ab0ca\": rpc error: code = NotFound desc = could not find container \"dcf98d8e913b07bdff9e2bfe929503513203761c90b3a56d39280642915ab0ca\": container with ID starting with dcf98d8e913b07bdff9e2bfe929503513203761c90b3a56d39280642915ab0ca not found: ID does not exist"
Aug 01 00:37:59.053543 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:37:59.053502    2559 scope.go:117] "RemoveContainer" containerID="e02e76e253ad741efd50a5e66ae1e2d1bcd63412b63364561202bbe6d9f768e3"
Aug 01 00:37:59.053794 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:37:59.053770    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e02e76e253ad741efd50a5e66ae1e2d1bcd63412b63364561202bbe6d9f768e3\": container with ID starting with e02e76e253ad741efd50a5e66ae1e2d1bcd63412b63364561202bbe6d9f768e3 not found: ID does not exist" containerID="e02e76e253ad741efd50a5e66ae1e2d1bcd63412b63364561202bbe6d9f768e3"
Aug 01 00:37:59.053866 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:37:59.053802    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e02e76e253ad741efd50a5e66ae1e2d1bcd63412b63364561202bbe6d9f768e3"} err="failed to get container status \"e02e76e253ad741efd50a5e66ae1e2d1bcd63412b63364561202bbe6d9f768e3\": rpc error: code = NotFound desc = could not find container \"e02e76e253ad741efd50a5e66ae1e2d1bcd63412b63364561202bbe6d9f768e3\": container with ID starting with e02e76e253ad741efd50a5e66ae1e2d1bcd63412b63364561202bbe6d9f768e3 not found: ID does not exist"
Aug 01 00:37:59.057991 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:37:59.057967    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5f8bf7dc88-gwn4k"]
Aug 01 00:37:59.063291 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:37:59.063264    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5f8bf7dc88-gwn4k"]
Aug 01 00:37:59.787349 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:37:59.787315    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a65aa9a-6963-41ca-92a9-0907a9ff0d91" path="/var/lib/kubelet/pods/4a65aa9a-6963-41ca-92a9-0907a9ff0d91/volumes"
Aug 01 00:38:17.040050 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:38:17.040016    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg"]
Aug 01 00:38:17.040453 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:38:17.040325    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a65aa9a-6963-41ca-92a9-0907a9ff0d91" containerName="storage-initializer"
Aug 01 00:38:17.040453 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:38:17.040337    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a65aa9a-6963-41ca-92a9-0907a9ff0d91" containerName="storage-initializer"
Aug 01 00:38:17.040453 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:38:17.040343    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a65aa9a-6963-41ca-92a9-0907a9ff0d91" containerName="main"
Aug 01 00:38:17.040453 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:38:17.040349    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a65aa9a-6963-41ca-92a9-0907a9ff0d91" containerName="main"
Aug 01 00:38:17.040453 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:38:17.040411    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a65aa9a-6963-41ca-92a9-0907a9ff0d91" containerName="main"
Aug 01 00:38:17.045515 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:38:17.045497    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg"
Aug 01 00:38:17.048613 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:38:17.048588    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Aug 01 00:38:17.048886 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:38:17.048870    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-mgzxr\""
Aug 01 00:38:17.049332 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:38:17.049316    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Aug 01 00:38:17.049821 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:38:17.049801    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Aug 01 00:38:17.049889 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:38:17.049867    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Aug 01 00:38:17.049939 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:38:17.049913    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Aug 01 00:38:17.059098 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:38:17.059073    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg"]
Aug 01 00:38:17.089464 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:38:17.089428    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg\" (UID: \"3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg"
Aug 01 00:38:17.089649 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:38:17.089473    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg\" (UID: \"3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg"
Aug 01 00:38:17.089649 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:38:17.089538    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg\" (UID: \"3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg"
Aug 01 00:38:17.089649 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:38:17.089602    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg\" (UID: \"3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg"
Aug 01 00:38:17.089649 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:38:17.089623    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkhkq\" (UniqueName: \"kubernetes.io/projected/3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8-kube-api-access-kkhkq\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg\" (UID: \"3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg"
Aug 01 00:38:17.089649 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:38:17.089643    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg\" (UID: \"3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg"
Aug 01 00:38:17.089855 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:38:17.089691    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg\" (UID: \"3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg"
Aug 01 00:38:17.089855 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:38:17.089782    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg\" (UID: \"3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg"
Aug 01 00:38:17.190170 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:38:17.190132    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg\" (UID: \"3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg"
Aug 01 00:38:17.190170 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:38:17.190174    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg\" (UID: \"3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg"
Aug 01 00:38:17.190442 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:38:17.190193    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg\" (UID: \"3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg"
Aug 01 00:38:17.190442 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:38:17.190227    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg\" (UID: \"3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg"
Aug 01 00:38:17.190442 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:38:17.190376    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkhkq\" (UniqueName: \"kubernetes.io/projected/3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8-kube-api-access-kkhkq\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg\" (UID: \"3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg"
Aug 01 00:38:17.190442 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:38:17.190417    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg\" (UID: \"3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg"
Aug 01 00:38:17.190658 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:38:17.190472    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg\" (UID: \"3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg"
Aug 01 00:38:17.190658 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:38:17.190495    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg\" (UID: \"3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg"
Aug 01 00:38:17.190658 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:38:17.190602    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg\" (UID: \"3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg"
Aug 01 00:38:17.190658 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:38:17.190632    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg\" (UID: \"3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg"
Aug 01 00:38:17.190919 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:38:17.190894    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg\" (UID: \"3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg"
Aug 01 00:38:17.190979 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:38:17.190944    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg\" (UID: \"3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg"
Aug 01 00:38:17.191067 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:38:17.191044    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg\" (UID: \"3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg"
Aug 01 00:38:17.192717 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:38:17.192694    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg\" (UID: \"3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg"
Aug 01 00:38:17.192899 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:38:17.192880    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg\" (UID: \"3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg"
Aug 01 00:38:17.200842 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:38:17.200816    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkhkq\" (UniqueName: \"kubernetes.io/projected/3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8-kube-api-access-kkhkq\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg\" (UID: \"3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg"
Aug 01 00:38:17.356174 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:38:17.356065    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg"
Aug 01 00:38:17.496879 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:38:17.495295    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg"]
Aug 01 00:38:18.116970 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:38:18.116930    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg" event={"ID":"3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8","Type":"ContainerStarted","Data":"7b4046af38e625a61f6c6b50381437e64b31c2a83f9e5ee561990ed0c93cf0f6"}
Aug 01 00:38:18.117422 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:38:18.116989    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg" event={"ID":"3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8","Type":"ContainerStarted","Data":"f66c1f5f4dc2de1c05e57875a7e8e7fac52768e5cec9aa07195e76551630e964"}
Aug 01 00:38:20.127311 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:38:20.127274    2559 generic.go:356] "Generic (PLEG): container finished" podID="3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8" containerID="7b4046af38e625a61f6c6b50381437e64b31c2a83f9e5ee561990ed0c93cf0f6" exitCode=0
Aug 01 00:38:20.127863 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:38:20.127353    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg" event={"ID":"3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8","Type":"ContainerDied","Data":"7b4046af38e625a61f6c6b50381437e64b31c2a83f9e5ee561990ed0c93cf0f6"}
Aug 01 00:38:21.133618 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:38:21.133580    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg" event={"ID":"3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8","Type":"ContainerStarted","Data":"f24e6deebc988eb7e5f63851f0e7343851bfa37e3bf3bb78d350f7d4c11467f5"}
Aug 01 00:38:21.154557 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:38:21.154494    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg" podStartSLOduration=2.140191606 podStartE2EDuration="4.1544766s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.014284994s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:38:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:38:21.152721022 +0000 UTC m=+3163.872520869" watchObservedRunningTime="2026-08-01 00:38:21.1544766 +0000 UTC m=+3163.874276449"
Aug 01 00:38:27.357149 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:38:27.357112    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg"
Aug 01 00:38:27.357571 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:38:27.357211    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg"
Aug 01 00:38:27.358826 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:38:27.358800    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg" podUID="3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Aug 01 00:38:37.357311 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:38:37.357267    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg" podUID="3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Aug 01 00:38:47.357174 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:38:47.357123    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg" podUID="3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Aug 01 00:38:57.357032 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:38:57.356982    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg" podUID="3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Aug 01 00:39:07.357219 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:39:07.357157    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg" podUID="3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Aug 01 00:39:17.357310 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:39:17.357263    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg" podUID="3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Aug 01 00:39:27.356978 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:39:27.356930    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg" podUID="3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Aug 01 00:39:37.356972 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:39:37.356916    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg" podUID="3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Aug 01 00:39:47.356768 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:39:47.356715    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg" podUID="3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Aug 01 00:39:57.357582 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:39:57.357533    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg" podUID="3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Aug 01 00:40:07.356952 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:07.356897    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg" podUID="3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Aug 01 00:40:17.367034 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:17.366997    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg"
Aug 01 00:40:17.375005 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:17.374976    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg"
Aug 01 00:40:26.106005 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:26.105972    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg"]
Aug 01 00:40:37.967173 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:37.967146    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-spblg_c2e50946-aea4-4886-82ec-0bd5a31196cf/console-operator/1.log"
Aug 01 00:40:37.981524 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:37.981494    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-spblg_c2e50946-aea4-4886-82ec-0bd5a31196cf/console-operator/1.log"
Aug 01 00:40:41.129164 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:41.129097    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg" podUID="3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8" containerName="main" containerID="cri-o://f24e6deebc988eb7e5f63851f0e7343851bfa37e3bf3bb78d350f7d4c11467f5" gracePeriod=45
Aug 01 00:40:41.803258 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:41.803226    2559 generic.go:356] "Generic (PLEG): container finished" podID="3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8" containerID="f24e6deebc988eb7e5f63851f0e7343851bfa37e3bf3bb78d350f7d4c11467f5" exitCode=0
Aug 01 00:40:41.803424 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:41.803297    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg" event={"ID":"3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8","Type":"ContainerDied","Data":"f24e6deebc988eb7e5f63851f0e7343851bfa37e3bf3bb78d350f7d4c11467f5"}
Aug 01 00:40:41.876898 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:41.876868    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg"
Aug 01 00:40:41.973086 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:41.973052    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8-model-cache\") pod \"3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8\" (UID: \"3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8\") "
Aug 01 00:40:41.973086 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:41.973090    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8-dshm\") pod \"3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8\" (UID: \"3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8\") "
Aug 01 00:40:41.973336 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:41.973112    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kkhkq\" (UniqueName: \"kubernetes.io/projected/3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8-kube-api-access-kkhkq\") pod \"3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8\" (UID: \"3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8\") "
Aug 01 00:40:41.973336 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:41.973145    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8-kserve-provision-location\") pod \"3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8\" (UID: \"3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8\") "
Aug 01 00:40:41.973336 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:41.973165    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8-home\") pod \"3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8\" (UID: \"3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8\") "
Aug 01 00:40:41.973336 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:41.973211    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8-cabundle-cert\") pod \"3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8\" (UID: \"3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8\") "
Aug 01 00:40:41.973336 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:41.973252    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8-tmp-dir\") pod \"3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8\" (UID: \"3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8\") "
Aug 01 00:40:41.973336 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:41.973284    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8-tls-certs\") pod \"3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8\" (UID: \"3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8\") "
Aug 01 00:40:41.973657 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:41.973421    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8-model-cache" (OuterVolumeSpecName: "model-cache") pod "3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8" (UID: "3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:40:41.973657 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:41.973568    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8-model-cache\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:40:41.973825 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:41.973788    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8" (UID: "3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:40:41.973960 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:41.973918    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8-home" (OuterVolumeSpecName: "home") pod "3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8" (UID: "3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:40:41.976060 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:41.976034    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8-dshm" (OuterVolumeSpecName: "dshm") pod "3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8" (UID: "3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:40:41.976197 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:41.976175    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8-kube-api-access-kkhkq" (OuterVolumeSpecName: "kube-api-access-kkhkq") pod "3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8" (UID: "3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8"). InnerVolumeSpecName "kube-api-access-kkhkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:40:41.977563 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:41.977539    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8" (UID: "3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:40:41.994990 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:41.994948    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8" (UID: "3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:40:42.004952 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:42.004922    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8" (UID: "3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:40:42.074184 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:42.074145    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8-cabundle-cert\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:40:42.074184 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:42.074179    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8-tmp-dir\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:40:42.074184 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:42.074191    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8-tls-certs\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:40:42.074435 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:42.074199    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8-dshm\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:40:42.074435 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:42.074207    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkhkq\" (UniqueName: \"kubernetes.io/projected/3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8-kube-api-access-kkhkq\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:40:42.074435 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:42.074218    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8-kserve-provision-location\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:40:42.074435 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:42.074226    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8-home\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:40:42.808998 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:42.808967    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg"
Aug 01 00:40:42.809416 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:42.808998    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg" event={"ID":"3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8","Type":"ContainerDied","Data":"f66c1f5f4dc2de1c05e57875a7e8e7fac52768e5cec9aa07195e76551630e964"}
Aug 01 00:40:42.809416 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:42.809059    2559 scope.go:117] "RemoveContainer" containerID="f24e6deebc988eb7e5f63851f0e7343851bfa37e3bf3bb78d350f7d4c11467f5"
Aug 01 00:40:42.818267 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:42.818241    2559 scope.go:117] "RemoveContainer" containerID="7b4046af38e625a61f6c6b50381437e64b31c2a83f9e5ee561990ed0c93cf0f6"
Aug 01 00:40:42.831372 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:42.831351    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg"]
Aug 01 00:40:42.835513 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:42.835490    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bb5lsg"]
Aug 01 00:40:43.786186 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:43.786148    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8" path="/var/lib/kubelet/pods/3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8/volumes"
Aug 01 00:40:55.734983 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:55.734950    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k"]
Aug 01 00:40:55.735484 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:55.735356    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8" containerName="main"
Aug 01 00:40:55.735484 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:55.735376    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8" containerName="main"
Aug 01 00:40:55.735484 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:55.735413    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8" containerName="storage-initializer"
Aug 01 00:40:55.735484 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:55.735421    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8" containerName="storage-initializer"
Aug 01 00:40:55.735656 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:55.735500    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ddaa9bd-039f-4d9f-bcc8-d4144d1677e8" containerName="main"
Aug 01 00:40:55.738491 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:55.738465    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k"
Aug 01 00:40:55.740985 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:55.740948    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Aug 01 00:40:55.740985 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:55.740979    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Aug 01 00:40:55.741210 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:55.740979    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Aug 01 00:40:55.741210 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:55.741101    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Aug 01 00:40:55.741336 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:55.741211    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Aug 01 00:40:55.741893 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:55.741877    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-77tph\""
Aug 01 00:40:55.750558 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:55.750536    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k"]
Aug 01 00:40:55.885863 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:55.885830    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b075a704-16f8-4319-8174-3aa7b8a62110-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k\" (UID: \"b075a704-16f8-4319-8174-3aa7b8a62110\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k"
Aug 01 00:40:55.886064 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:55.885880    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b075a704-16f8-4319-8174-3aa7b8a62110-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k\" (UID: \"b075a704-16f8-4319-8174-3aa7b8a62110\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k"
Aug 01 00:40:55.886064 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:55.885957    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b075a704-16f8-4319-8174-3aa7b8a62110-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k\" (UID: \"b075a704-16f8-4319-8174-3aa7b8a62110\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k"
Aug 01 00:40:55.886064 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:55.885984    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpzqs\" (UniqueName: \"kubernetes.io/projected/b075a704-16f8-4319-8174-3aa7b8a62110-kube-api-access-hpzqs\") pod \"custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k\" (UID: \"b075a704-16f8-4319-8174-3aa7b8a62110\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k"
Aug 01 00:40:55.886064 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:55.886019    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b075a704-16f8-4319-8174-3aa7b8a62110-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k\" (UID: \"b075a704-16f8-4319-8174-3aa7b8a62110\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k"
Aug 01 00:40:55.886064 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:55.886052    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b075a704-16f8-4319-8174-3aa7b8a62110-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k\" (UID: \"b075a704-16f8-4319-8174-3aa7b8a62110\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k"
Aug 01 00:40:55.886293 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:55.886108    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b075a704-16f8-4319-8174-3aa7b8a62110-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k\" (UID: \"b075a704-16f8-4319-8174-3aa7b8a62110\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k"
Aug 01 00:40:55.886293 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:55.886143    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b075a704-16f8-4319-8174-3aa7b8a62110-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k\" (UID: \"b075a704-16f8-4319-8174-3aa7b8a62110\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k"
Aug 01 00:40:55.987309 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:55.987213    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b075a704-16f8-4319-8174-3aa7b8a62110-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k\" (UID: \"b075a704-16f8-4319-8174-3aa7b8a62110\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k"
Aug 01 00:40:55.987309 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:55.987256    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b075a704-16f8-4319-8174-3aa7b8a62110-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k\" (UID: \"b075a704-16f8-4319-8174-3aa7b8a62110\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k"
Aug 01 00:40:55.987309 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:55.987278    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b075a704-16f8-4319-8174-3aa7b8a62110-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k\" (UID: \"b075a704-16f8-4319-8174-3aa7b8a62110\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k"
Aug 01 00:40:55.987309 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:55.987303    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b075a704-16f8-4319-8174-3aa7b8a62110-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k\" (UID: \"b075a704-16f8-4319-8174-3aa7b8a62110\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k"
Aug 01 00:40:55.987626 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:55.987333    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b075a704-16f8-4319-8174-3aa7b8a62110-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k\" (UID: \"b075a704-16f8-4319-8174-3aa7b8a62110\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k"
Aug 01 00:40:55.987626 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:55.987350    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpzqs\" (UniqueName: \"kubernetes.io/projected/b075a704-16f8-4319-8174-3aa7b8a62110-kube-api-access-hpzqs\") pod \"custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k\" (UID: \"b075a704-16f8-4319-8174-3aa7b8a62110\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k"
Aug 01 00:40:55.987626 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:55.987374    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b075a704-16f8-4319-8174-3aa7b8a62110-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k\" (UID: \"b075a704-16f8-4319-8174-3aa7b8a62110\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k"
Aug 01 00:40:55.987626 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:55.987536    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b075a704-16f8-4319-8174-3aa7b8a62110-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k\" (UID: \"b075a704-16f8-4319-8174-3aa7b8a62110\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k"
Aug 01 00:40:55.987921 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:55.987898    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b075a704-16f8-4319-8174-3aa7b8a62110-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k\" (UID: \"b075a704-16f8-4319-8174-3aa7b8a62110\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k"
Aug 01 00:40:55.988033 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:55.988014    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b075a704-16f8-4319-8174-3aa7b8a62110-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k\" (UID: \"b075a704-16f8-4319-8174-3aa7b8a62110\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k"
Aug 01 00:40:55.988116 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:55.988101    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b075a704-16f8-4319-8174-3aa7b8a62110-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k\" (UID: \"b075a704-16f8-4319-8174-3aa7b8a62110\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k"
Aug 01 00:40:55.988174 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:55.988124    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b075a704-16f8-4319-8174-3aa7b8a62110-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k\" (UID: \"b075a704-16f8-4319-8174-3aa7b8a62110\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k"
Aug 01 00:40:55.988303 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:55.988286    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b075a704-16f8-4319-8174-3aa7b8a62110-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k\" (UID: \"b075a704-16f8-4319-8174-3aa7b8a62110\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k"
Aug 01 00:40:55.989681 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:55.989663    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b075a704-16f8-4319-8174-3aa7b8a62110-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k\" (UID: \"b075a704-16f8-4319-8174-3aa7b8a62110\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k"
Aug 01 00:40:55.989865 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:55.989848    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b075a704-16f8-4319-8174-3aa7b8a62110-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k\" (UID: \"b075a704-16f8-4319-8174-3aa7b8a62110\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k"
Aug 01 00:40:55.997009 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:55.996983    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpzqs\" (UniqueName: \"kubernetes.io/projected/b075a704-16f8-4319-8174-3aa7b8a62110-kube-api-access-hpzqs\") pod \"custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k\" (UID: \"b075a704-16f8-4319-8174-3aa7b8a62110\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k"
Aug 01 00:40:56.049809 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:56.049775    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k"
Aug 01 00:40:56.198632 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:56.198604    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k"]
Aug 01 00:40:56.200536 ip-10-0-130-148 kubenswrapper[2559]: W0801 00:40:56.200512    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb075a704_16f8_4319_8174_3aa7b8a62110.slice/crio-767f4163c81851eac96cff480b2c85d46fcc9f144de70f88e01e1e63ab8a28ca WatchSource:0}: Error finding container 767f4163c81851eac96cff480b2c85d46fcc9f144de70f88e01e1e63ab8a28ca: Status 404 returned error can't find the container with id 767f4163c81851eac96cff480b2c85d46fcc9f144de70f88e01e1e63ab8a28ca
Aug 01 00:40:56.872866 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:56.872824    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k" event={"ID":"b075a704-16f8-4319-8174-3aa7b8a62110","Type":"ContainerStarted","Data":"cea48521bb02e5dc2d0b7532d1048e047472d7e5d47c826bdb7981285ff2e68d"}
Aug 01 00:40:56.873279 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:56.872889    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k" event={"ID":"b075a704-16f8-4319-8174-3aa7b8a62110","Type":"ContainerStarted","Data":"767f4163c81851eac96cff480b2c85d46fcc9f144de70f88e01e1e63ab8a28ca"}
Aug 01 00:40:58.885190 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:58.885145    2559 generic.go:356] "Generic (PLEG): container finished" podID="b075a704-16f8-4319-8174-3aa7b8a62110" containerID="cea48521bb02e5dc2d0b7532d1048e047472d7e5d47c826bdb7981285ff2e68d" exitCode=0
Aug 01 00:40:58.885634 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:58.885190    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k" event={"ID":"b075a704-16f8-4319-8174-3aa7b8a62110","Type":"ContainerDied","Data":"cea48521bb02e5dc2d0b7532d1048e047472d7e5d47c826bdb7981285ff2e68d"}
Aug 01 00:40:59.892058 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:59.892018    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k" event={"ID":"b075a704-16f8-4319-8174-3aa7b8a62110","Type":"ContainerStarted","Data":"78eb1aa215858eca18a51e69dffbb7a8f41a36d58013451e6ba7fe918903358a"}
Aug 01 00:40:59.915660 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:40:59.915597    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k" podStartSLOduration=2.855920817 podStartE2EDuration="4.91557931s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.059658493s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:40:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:40:59.914515247 +0000 UTC m=+3322.634315093" watchObservedRunningTime="2026-08-01 00:40:59.91557931 +0000 UTC m=+3322.635379155"
Aug 01 00:41:06.050366 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:41:06.050330    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k"
Aug 01 00:41:06.050880 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:41:06.050388    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k"
Aug 01 00:41:06.051951 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:41:06.051918    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k" podUID="b075a704-16f8-4319-8174-3aa7b8a62110" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Aug 01 00:41:16.050921 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:41:16.050872    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k" podUID="b075a704-16f8-4319-8174-3aa7b8a62110" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Aug 01 00:41:26.050798 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:41:26.050718    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k" podUID="b075a704-16f8-4319-8174-3aa7b8a62110" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Aug 01 00:41:36.051087 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:41:36.051041    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k" podUID="b075a704-16f8-4319-8174-3aa7b8a62110" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Aug 01 00:41:46.050279 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:41:46.050240    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k" podUID="b075a704-16f8-4319-8174-3aa7b8a62110" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Aug 01 00:41:56.051071 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:41:56.051032    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k" podUID="b075a704-16f8-4319-8174-3aa7b8a62110" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Aug 01 00:42:06.050634 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:42:06.050593    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k" podUID="b075a704-16f8-4319-8174-3aa7b8a62110" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Aug 01 00:42:16.050330 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:42:16.050291    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k" podUID="b075a704-16f8-4319-8174-3aa7b8a62110" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Aug 01 00:42:26.050587 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:42:26.050544    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k" podUID="b075a704-16f8-4319-8174-3aa7b8a62110" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Aug 01 00:42:36.050396 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:42:36.050355    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k" podUID="b075a704-16f8-4319-8174-3aa7b8a62110" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Aug 01 00:42:46.051018 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:42:46.050976    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k" podUID="b075a704-16f8-4319-8174-3aa7b8a62110" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Aug 01 00:42:56.060936 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:42:56.060906    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k"
Aug 01 00:42:56.068981 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:42:56.068958    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k"
Aug 01 00:43:04.897025 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:04.896985    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k"]
Aug 01 00:43:19.918834 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:19.918795    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k" podUID="b075a704-16f8-4319-8174-3aa7b8a62110" containerName="main" containerID="cri-o://78eb1aa215858eca18a51e69dffbb7a8f41a36d58013451e6ba7fe918903358a" gracePeriod=45
Aug 01 00:43:20.546782 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:20.546733    2559 generic.go:356] "Generic (PLEG): container finished" podID="b075a704-16f8-4319-8174-3aa7b8a62110" containerID="78eb1aa215858eca18a51e69dffbb7a8f41a36d58013451e6ba7fe918903358a" exitCode=0
Aug 01 00:43:20.546916 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:20.546780    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k" event={"ID":"b075a704-16f8-4319-8174-3aa7b8a62110","Type":"ContainerDied","Data":"78eb1aa215858eca18a51e69dffbb7a8f41a36d58013451e6ba7fe918903358a"}
Aug 01 00:43:20.673060 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:20.673036    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k"
Aug 01 00:43:20.718891 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:20.718817    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b075a704-16f8-4319-8174-3aa7b8a62110-cabundle-cert\") pod \"b075a704-16f8-4319-8174-3aa7b8a62110\" (UID: \"b075a704-16f8-4319-8174-3aa7b8a62110\") "
Aug 01 00:43:20.718891 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:20.718860    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b075a704-16f8-4319-8174-3aa7b8a62110-home\") pod \"b075a704-16f8-4319-8174-3aa7b8a62110\" (UID: \"b075a704-16f8-4319-8174-3aa7b8a62110\") "
Aug 01 00:43:20.718891 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:20.718880    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b075a704-16f8-4319-8174-3aa7b8a62110-tls-certs\") pod \"b075a704-16f8-4319-8174-3aa7b8a62110\" (UID: \"b075a704-16f8-4319-8174-3aa7b8a62110\") "
Aug 01 00:43:20.719174 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:20.718910    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b075a704-16f8-4319-8174-3aa7b8a62110-model-cache\") pod \"b075a704-16f8-4319-8174-3aa7b8a62110\" (UID: \"b075a704-16f8-4319-8174-3aa7b8a62110\") "
Aug 01 00:43:20.719174 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:20.718972    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b075a704-16f8-4319-8174-3aa7b8a62110-kserve-provision-location\") pod \"b075a704-16f8-4319-8174-3aa7b8a62110\" (UID: \"b075a704-16f8-4319-8174-3aa7b8a62110\") "
Aug 01 00:43:20.719174 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:20.718998    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hpzqs\" (UniqueName: \"kubernetes.io/projected/b075a704-16f8-4319-8174-3aa7b8a62110-kube-api-access-hpzqs\") pod \"b075a704-16f8-4319-8174-3aa7b8a62110\" (UID: \"b075a704-16f8-4319-8174-3aa7b8a62110\") "
Aug 01 00:43:20.719174 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:20.719042    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b075a704-16f8-4319-8174-3aa7b8a62110-dshm\") pod \"b075a704-16f8-4319-8174-3aa7b8a62110\" (UID: \"b075a704-16f8-4319-8174-3aa7b8a62110\") "
Aug 01 00:43:20.719174 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:20.719105    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b075a704-16f8-4319-8174-3aa7b8a62110-tmp-dir\") pod \"b075a704-16f8-4319-8174-3aa7b8a62110\" (UID: \"b075a704-16f8-4319-8174-3aa7b8a62110\") "
Aug 01 00:43:20.719443 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:20.719241    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b075a704-16f8-4319-8174-3aa7b8a62110-model-cache" (OuterVolumeSpecName: "model-cache") pod "b075a704-16f8-4319-8174-3aa7b8a62110" (UID: "b075a704-16f8-4319-8174-3aa7b8a62110"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:43:20.719443 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:20.719282    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b075a704-16f8-4319-8174-3aa7b8a62110-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b075a704-16f8-4319-8174-3aa7b8a62110" (UID: "b075a704-16f8-4319-8174-3aa7b8a62110"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:43:20.719555 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:20.719382    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b075a704-16f8-4319-8174-3aa7b8a62110-cabundle-cert\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:43:20.719555 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:20.719483    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b075a704-16f8-4319-8174-3aa7b8a62110-model-cache\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:43:20.719641 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:20.719607    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b075a704-16f8-4319-8174-3aa7b8a62110-home" (OuterVolumeSpecName: "home") pod "b075a704-16f8-4319-8174-3aa7b8a62110" (UID: "b075a704-16f8-4319-8174-3aa7b8a62110"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:43:20.721944 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:20.721910    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b075a704-16f8-4319-8174-3aa7b8a62110-dshm" (OuterVolumeSpecName: "dshm") pod "b075a704-16f8-4319-8174-3aa7b8a62110" (UID: "b075a704-16f8-4319-8174-3aa7b8a62110"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:43:20.722063 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:20.721970    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b075a704-16f8-4319-8174-3aa7b8a62110-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b075a704-16f8-4319-8174-3aa7b8a62110" (UID: "b075a704-16f8-4319-8174-3aa7b8a62110"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:43:20.722063 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:20.721979    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b075a704-16f8-4319-8174-3aa7b8a62110-kube-api-access-hpzqs" (OuterVolumeSpecName: "kube-api-access-hpzqs") pod "b075a704-16f8-4319-8174-3aa7b8a62110" (UID: "b075a704-16f8-4319-8174-3aa7b8a62110"). InnerVolumeSpecName "kube-api-access-hpzqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:43:20.739280 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:20.739244    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b075a704-16f8-4319-8174-3aa7b8a62110-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b075a704-16f8-4319-8174-3aa7b8a62110" (UID: "b075a704-16f8-4319-8174-3aa7b8a62110"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:43:20.746209 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:20.746184    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b075a704-16f8-4319-8174-3aa7b8a62110-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b075a704-16f8-4319-8174-3aa7b8a62110" (UID: "b075a704-16f8-4319-8174-3aa7b8a62110"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:43:20.820569 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:20.820539    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b075a704-16f8-4319-8174-3aa7b8a62110-kserve-provision-location\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:43:20.820569 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:20.820565    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpzqs\" (UniqueName: \"kubernetes.io/projected/b075a704-16f8-4319-8174-3aa7b8a62110-kube-api-access-hpzqs\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:43:20.820569 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:20.820575    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b075a704-16f8-4319-8174-3aa7b8a62110-dshm\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:43:20.820799 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:20.820584    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b075a704-16f8-4319-8174-3aa7b8a62110-tmp-dir\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:43:20.820799 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:20.820593    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b075a704-16f8-4319-8174-3aa7b8a62110-home\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:43:20.820799 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:20.820600    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b075a704-16f8-4319-8174-3aa7b8a62110-tls-certs\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:43:21.553250 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:21.553214    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k" event={"ID":"b075a704-16f8-4319-8174-3aa7b8a62110","Type":"ContainerDied","Data":"767f4163c81851eac96cff480b2c85d46fcc9f144de70f88e01e1e63ab8a28ca"}
Aug 01 00:43:21.553250 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:21.553234    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k"
Aug 01 00:43:21.553780 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:21.553278    2559 scope.go:117] "RemoveContainer" containerID="78eb1aa215858eca18a51e69dffbb7a8f41a36d58013451e6ba7fe918903358a"
Aug 01 00:43:21.562204 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:21.562187    2559 scope.go:117] "RemoveContainer" containerID="cea48521bb02e5dc2d0b7532d1048e047472d7e5d47c826bdb7981285ff2e68d"
Aug 01 00:43:21.575786 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:21.575761    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k"]
Aug 01 00:43:21.579099 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:21.579066    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6d65fc59-4ff9k"]
Aug 01 00:43:21.785520 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:21.785488    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b075a704-16f8-4319-8174-3aa7b8a62110" path="/var/lib/kubelet/pods/b075a704-16f8-4319-8174-3aa7b8a62110/volumes"
Aug 01 00:43:59.004457 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:59.004412    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw"]
Aug 01 00:43:59.005106 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:59.004907    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b075a704-16f8-4319-8174-3aa7b8a62110" containerName="storage-initializer"
Aug 01 00:43:59.005106 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:59.004923    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b075a704-16f8-4319-8174-3aa7b8a62110" containerName="storage-initializer"
Aug 01 00:43:59.005106 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:59.004942    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b075a704-16f8-4319-8174-3aa7b8a62110" containerName="main"
Aug 01 00:43:59.005106 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:59.004948    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b075a704-16f8-4319-8174-3aa7b8a62110" containerName="main"
Aug 01 00:43:59.005106 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:59.005002    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b075a704-16f8-4319-8174-3aa7b8a62110" containerName="main"
Aug 01 00:43:59.007872 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:59.007849    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw"
Aug 01 00:43:59.010323 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:59.010297    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Aug 01 00:43:59.010442 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:59.010357    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Aug 01 00:43:59.010516 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:59.010454    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Aug 01 00:43:59.010714 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:59.010698    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Aug 01 00:43:59.010813 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:59.010712    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Aug 01 00:43:59.011473 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:59.011447    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-9hmc9\""
Aug 01 00:43:59.016597 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:59.016578    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw"]
Aug 01 00:43:59.147124 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:59.147084    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfea3687-e01f-4cfa-bc88-7ee55bc7bed1-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw\" (UID: \"cfea3687-e01f-4cfa-bc88-7ee55bc7bed1\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw"
Aug 01 00:43:59.147311 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:59.147130    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfea3687-e01f-4cfa-bc88-7ee55bc7bed1-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw\" (UID: \"cfea3687-e01f-4cfa-bc88-7ee55bc7bed1\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw"
Aug 01 00:43:59.147311 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:59.147157    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzb78\" (UniqueName: \"kubernetes.io/projected/cfea3687-e01f-4cfa-bc88-7ee55bc7bed1-kube-api-access-jzb78\") pod \"router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw\" (UID: \"cfea3687-e01f-4cfa-bc88-7ee55bc7bed1\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw"
Aug 01 00:43:59.147311 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:59.147178    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cfea3687-e01f-4cfa-bc88-7ee55bc7bed1-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw\" (UID: \"cfea3687-e01f-4cfa-bc88-7ee55bc7bed1\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw"
Aug 01 00:43:59.147311 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:59.147210    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfea3687-e01f-4cfa-bc88-7ee55bc7bed1-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw\" (UID: \"cfea3687-e01f-4cfa-bc88-7ee55bc7bed1\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw"
Aug 01 00:43:59.147311 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:59.147272    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cfea3687-e01f-4cfa-bc88-7ee55bc7bed1-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw\" (UID: \"cfea3687-e01f-4cfa-bc88-7ee55bc7bed1\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw"
Aug 01 00:43:59.147311 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:59.147294    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfea3687-e01f-4cfa-bc88-7ee55bc7bed1-home\") pod \"router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw\" (UID: \"cfea3687-e01f-4cfa-bc88-7ee55bc7bed1\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw"
Aug 01 00:43:59.147311 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:59.147312    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfea3687-e01f-4cfa-bc88-7ee55bc7bed1-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw\" (UID: \"cfea3687-e01f-4cfa-bc88-7ee55bc7bed1\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw"
Aug 01 00:43:59.248569 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:59.248518    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cfea3687-e01f-4cfa-bc88-7ee55bc7bed1-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw\" (UID: \"cfea3687-e01f-4cfa-bc88-7ee55bc7bed1\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw"
Aug 01 00:43:59.248569 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:59.248571    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfea3687-e01f-4cfa-bc88-7ee55bc7bed1-home\") pod \"router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw\" (UID: \"cfea3687-e01f-4cfa-bc88-7ee55bc7bed1\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw"
Aug 01 00:43:59.248768 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:59.248688    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfea3687-e01f-4cfa-bc88-7ee55bc7bed1-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw\" (UID: \"cfea3687-e01f-4cfa-bc88-7ee55bc7bed1\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw"
Aug 01 00:43:59.248821 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:59.248806    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfea3687-e01f-4cfa-bc88-7ee55bc7bed1-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw\" (UID: \"cfea3687-e01f-4cfa-bc88-7ee55bc7bed1\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw"
Aug 01 00:43:59.248857 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:59.248841    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfea3687-e01f-4cfa-bc88-7ee55bc7bed1-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw\" (UID: \"cfea3687-e01f-4cfa-bc88-7ee55bc7bed1\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw"
Aug 01 00:43:59.248902 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:59.248872    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzb78\" (UniqueName: \"kubernetes.io/projected/cfea3687-e01f-4cfa-bc88-7ee55bc7bed1-kube-api-access-jzb78\") pod \"router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw\" (UID: \"cfea3687-e01f-4cfa-bc88-7ee55bc7bed1\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw"
Aug 01 00:43:59.248969 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:59.248938    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cfea3687-e01f-4cfa-bc88-7ee55bc7bed1-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw\" (UID: \"cfea3687-e01f-4cfa-bc88-7ee55bc7bed1\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw"
Aug 01 00:43:59.249030 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:59.249007    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfea3687-e01f-4cfa-bc88-7ee55bc7bed1-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw\" (UID: \"cfea3687-e01f-4cfa-bc88-7ee55bc7bed1\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw"
Aug 01 00:43:59.249083 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:59.249044    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfea3687-e01f-4cfa-bc88-7ee55bc7bed1-home\") pod \"router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw\" (UID: \"cfea3687-e01f-4cfa-bc88-7ee55bc7bed1\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw"
Aug 01 00:43:59.249238 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:59.249184    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfea3687-e01f-4cfa-bc88-7ee55bc7bed1-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw\" (UID: \"cfea3687-e01f-4cfa-bc88-7ee55bc7bed1\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw"
Aug 01 00:43:59.249377 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:59.249322    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfea3687-e01f-4cfa-bc88-7ee55bc7bed1-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw\" (UID: \"cfea3687-e01f-4cfa-bc88-7ee55bc7bed1\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw"
Aug 01 00:43:59.249446 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:59.249382    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cfea3687-e01f-4cfa-bc88-7ee55bc7bed1-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw\" (UID: \"cfea3687-e01f-4cfa-bc88-7ee55bc7bed1\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw"
Aug 01 00:43:59.249446 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:59.249413    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cfea3687-e01f-4cfa-bc88-7ee55bc7bed1-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw\" (UID: \"cfea3687-e01f-4cfa-bc88-7ee55bc7bed1\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw"
Aug 01 00:43:59.251155 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:59.251128    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfea3687-e01f-4cfa-bc88-7ee55bc7bed1-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw\" (UID: \"cfea3687-e01f-4cfa-bc88-7ee55bc7bed1\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw"
Aug 01 00:43:59.251275 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:59.251256    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfea3687-e01f-4cfa-bc88-7ee55bc7bed1-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw\" (UID: \"cfea3687-e01f-4cfa-bc88-7ee55bc7bed1\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw"
Aug 01 00:43:59.256600 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:59.256550    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzb78\" (UniqueName: \"kubernetes.io/projected/cfea3687-e01f-4cfa-bc88-7ee55bc7bed1-kube-api-access-jzb78\") pod \"router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw\" (UID: \"cfea3687-e01f-4cfa-bc88-7ee55bc7bed1\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw"
Aug 01 00:43:59.319366 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:59.319328    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw"
Aug 01 00:43:59.455618 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:59.455591    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw"]
Aug 01 00:43:59.457944 ip-10-0-130-148 kubenswrapper[2559]: W0801 00:43:59.457915    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfea3687_e01f_4cfa_bc88_7ee55bc7bed1.slice/crio-0972b957a7bd46723358f2781265f67734dbfe73cd70a55a8c7ea43ed1e3a466 WatchSource:0}: Error finding container 0972b957a7bd46723358f2781265f67734dbfe73cd70a55a8c7ea43ed1e3a466: Status 404 returned error can't find the container with id 0972b957a7bd46723358f2781265f67734dbfe73cd70a55a8c7ea43ed1e3a466
Aug 01 00:43:59.721944 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:59.721905    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw" event={"ID":"cfea3687-e01f-4cfa-bc88-7ee55bc7bed1","Type":"ContainerStarted","Data":"190dededaabbf7fec8b7d0b722923280d410372ae6c3011c1f3c64249a09d577"}
Aug 01 00:43:59.722116 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:43:59.721965    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw" event={"ID":"cfea3687-e01f-4cfa-bc88-7ee55bc7bed1","Type":"ContainerStarted","Data":"0972b957a7bd46723358f2781265f67734dbfe73cd70a55a8c7ea43ed1e3a466"}
Aug 01 00:44:01.734793 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:44:01.734730    2559 generic.go:356] "Generic (PLEG): container finished" podID="cfea3687-e01f-4cfa-bc88-7ee55bc7bed1" containerID="190dededaabbf7fec8b7d0b722923280d410372ae6c3011c1f3c64249a09d577" exitCode=0
Aug 01 00:44:01.735219 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:44:01.734804    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw" event={"ID":"cfea3687-e01f-4cfa-bc88-7ee55bc7bed1","Type":"ContainerDied","Data":"190dededaabbf7fec8b7d0b722923280d410372ae6c3011c1f3c64249a09d577"}
Aug 01 00:44:02.741316 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:44:02.741278    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw" event={"ID":"cfea3687-e01f-4cfa-bc88-7ee55bc7bed1","Type":"ContainerStarted","Data":"b9f5f6f25570555768c52e385822d47b843ee62316dd3bb1f71bc3bf57538e48"}
Aug 01 00:44:02.761707 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:44:02.761646    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw" podStartSLOduration=2.688120232 podStartE2EDuration="4.761630755s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.073510523s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:43:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:44:02.760266683 +0000 UTC m=+3505.480066530" watchObservedRunningTime="2026-08-01 00:44:02.761630755 +0000 UTC m=+3505.481430601"
Aug 01 00:44:09.320412 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:44:09.320342    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw"
Aug 01 00:44:09.320412 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:44:09.320416    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw"
Aug 01 00:44:09.321828 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:44:09.321801    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw" podUID="cfea3687-e01f-4cfa-bc88-7ee55bc7bed1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Aug 01 00:44:13.444527 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:44:13.444493    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-cbdcr"]
Aug 01 00:44:13.447019 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:44:13.446999    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-cbdcr"
Aug 01 00:44:13.449640 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:44:13.449614    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Aug 01 00:44:13.449800 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:44:13.449644    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Aug 01 00:44:13.449800 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:44:13.449666    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Aug 01 00:44:13.449920 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:44:13.449866    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Aug 01 00:44:13.449964 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:44:13.449945    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-rkqvb\""
Aug 01 00:44:13.450013 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:44:13.449962    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Aug 01 00:44:13.458813 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:44:13.458789    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-cbdcr"]
Aug 01 00:44:13.460341 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:44:13.460317    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ca4268b-50a4-426c-a0ab-275b25147d67-dshm\") pod \"prestop-hook-test-kserve-65d966bbf4-cbdcr\" (UID: \"1ca4268b-50a4-426c-a0ab-275b25147d67\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-cbdcr"
Aug 01 00:44:13.460474 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:44:13.460351    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ca4268b-50a4-426c-a0ab-275b25147d67-tls-certs\") pod \"prestop-hook-test-kserve-65d966bbf4-cbdcr\" (UID: \"1ca4268b-50a4-426c-a0ab-275b25147d67\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-cbdcr"
Aug 01 00:44:13.460474 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:44:13.460391    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ca4268b-50a4-426c-a0ab-275b25147d67-model-cache\") pod \"prestop-hook-test-kserve-65d966bbf4-cbdcr\" (UID: \"1ca4268b-50a4-426c-a0ab-275b25147d67\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-cbdcr"
Aug 01 00:44:13.460474 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:44:13.460466    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ca4268b-50a4-426c-a0ab-275b25147d67-cabundle-cert\") pod \"prestop-hook-test-kserve-65d966bbf4-cbdcr\" (UID: \"1ca4268b-50a4-426c-a0ab-275b25147d67\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-cbdcr"
Aug 01 00:44:13.460649 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:44:13.460495    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ca4268b-50a4-426c-a0ab-275b25147d67-tmp-dir\") pod \"prestop-hook-test-kserve-65d966bbf4-cbdcr\" (UID: \"1ca4268b-50a4-426c-a0ab-275b25147d67\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-cbdcr"
Aug 01 00:44:13.460649 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:44:13.460536    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ca4268b-50a4-426c-a0ab-275b25147d67-home\") pod \"prestop-hook-test-kserve-65d966bbf4-cbdcr\" (UID: \"1ca4268b-50a4-426c-a0ab-275b25147d67\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-cbdcr"
Aug 01 00:44:13.460649 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:44:13.460596    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4t7k\" (UniqueName: \"kubernetes.io/projected/1ca4268b-50a4-426c-a0ab-275b25147d67-kube-api-access-p4t7k\") pod \"prestop-hook-test-kserve-65d966bbf4-cbdcr\" (UID: \"1ca4268b-50a4-426c-a0ab-275b25147d67\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-cbdcr"
Aug 01 00:44:13.460804 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:44:13.460652    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ca4268b-50a4-426c-a0ab-275b25147d67-kserve-provision-location\") pod \"prestop-hook-test-kserve-65d966bbf4-cbdcr\" (UID: \"1ca4268b-50a4-426c-a0ab-275b25147d67\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-cbdcr"
Aug 01 00:44:13.561404 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:44:13.561369    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ca4268b-50a4-426c-a0ab-275b25147d67-cabundle-cert\") pod \"prestop-hook-test-kserve-65d966bbf4-cbdcr\" (UID: \"1ca4268b-50a4-426c-a0ab-275b25147d67\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-cbdcr"
Aug 01 00:44:13.561404 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:44:13.561410    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ca4268b-50a4-426c-a0ab-275b25147d67-tmp-dir\") pod \"prestop-hook-test-kserve-65d966bbf4-cbdcr\" (UID: \"1ca4268b-50a4-426c-a0ab-275b25147d67\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-cbdcr"
Aug 01 00:44:13.561636 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:44:13.561533    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ca4268b-50a4-426c-a0ab-275b25147d67-home\") pod \"prestop-hook-test-kserve-65d966bbf4-cbdcr\" (UID: \"1ca4268b-50a4-426c-a0ab-275b25147d67\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-cbdcr"
Aug 01 00:44:13.561636 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:44:13.561578    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4t7k\" (UniqueName: \"kubernetes.io/projected/1ca4268b-50a4-426c-a0ab-275b25147d67-kube-api-access-p4t7k\") pod \"prestop-hook-test-kserve-65d966bbf4-cbdcr\" (UID: \"1ca4268b-50a4-426c-a0ab-275b25147d67\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-cbdcr"
Aug 01 00:44:13.561636 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:44:13.561618    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ca4268b-50a4-426c-a0ab-275b25147d67-kserve-provision-location\") pod \"prestop-hook-test-kserve-65d966bbf4-cbdcr\" (UID: \"1ca4268b-50a4-426c-a0ab-275b25147d67\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-cbdcr"
Aug 01 00:44:13.561850 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:44:13.561669    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ca4268b-50a4-426c-a0ab-275b25147d67-dshm\") pod \"prestop-hook-test-kserve-65d966bbf4-cbdcr\" (UID: \"1ca4268b-50a4-426c-a0ab-275b25147d67\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-cbdcr"
Aug 01 00:44:13.561850 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:44:13.561694    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ca4268b-50a4-426c-a0ab-275b25147d67-tls-certs\") pod \"prestop-hook-test-kserve-65d966bbf4-cbdcr\" (UID: \"1ca4268b-50a4-426c-a0ab-275b25147d67\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-cbdcr"
Aug 01 00:44:13.561850 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:44:13.561737    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ca4268b-50a4-426c-a0ab-275b25147d67-model-cache\") pod \"prestop-hook-test-kserve-65d966bbf4-cbdcr\" (UID: \"1ca4268b-50a4-426c-a0ab-275b25147d67\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-cbdcr"
Aug 01 00:44:13.561850 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:44:13.561844    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ca4268b-50a4-426c-a0ab-275b25147d67-tmp-dir\") pod \"prestop-hook-test-kserve-65d966bbf4-cbdcr\" (UID: \"1ca4268b-50a4-426c-a0ab-275b25147d67\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-cbdcr"
Aug 01 00:44:13.562047 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:44:13.561939    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ca4268b-50a4-426c-a0ab-275b25147d67-home\") pod \"prestop-hook-test-kserve-65d966bbf4-cbdcr\" (UID: \"1ca4268b-50a4-426c-a0ab-275b25147d67\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-cbdcr"
Aug 01 00:44:13.562154 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:44:13.562121    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ca4268b-50a4-426c-a0ab-275b25147d67-model-cache\") pod \"prestop-hook-test-kserve-65d966bbf4-cbdcr\" (UID: \"1ca4268b-50a4-426c-a0ab-275b25147d67\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-cbdcr"
Aug 01 00:44:13.562278 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:44:13.562200    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ca4268b-50a4-426c-a0ab-275b25147d67-cabundle-cert\") pod \"prestop-hook-test-kserve-65d966bbf4-cbdcr\" (UID: \"1ca4268b-50a4-426c-a0ab-275b25147d67\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-cbdcr"
Aug 01 00:44:13.562411 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:44:13.562387    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ca4268b-50a4-426c-a0ab-275b25147d67-kserve-provision-location\") pod \"prestop-hook-test-kserve-65d966bbf4-cbdcr\" (UID: \"1ca4268b-50a4-426c-a0ab-275b25147d67\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-cbdcr"
Aug 01 00:44:13.564062 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:44:13.564035    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ca4268b-50a4-426c-a0ab-275b25147d67-dshm\") pod \"prestop-hook-test-kserve-65d966bbf4-cbdcr\" (UID: \"1ca4268b-50a4-426c-a0ab-275b25147d67\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-cbdcr"
Aug 01 00:44:13.564221 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:44:13.564206    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ca4268b-50a4-426c-a0ab-275b25147d67-tls-certs\") pod \"prestop-hook-test-kserve-65d966bbf4-cbdcr\" (UID: \"1ca4268b-50a4-426c-a0ab-275b25147d67\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-cbdcr"
Aug 01 00:44:13.569983 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:44:13.569964    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4t7k\" (UniqueName: \"kubernetes.io/projected/1ca4268b-50a4-426c-a0ab-275b25147d67-kube-api-access-p4t7k\") pod \"prestop-hook-test-kserve-65d966bbf4-cbdcr\" (UID: \"1ca4268b-50a4-426c-a0ab-275b25147d67\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-cbdcr"
Aug 01 00:44:13.757918 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:44:13.757828    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-cbdcr"
Aug 01 00:44:13.901449 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:44:13.901423    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-cbdcr"]
Aug 01 00:44:13.903521 ip-10-0-130-148 kubenswrapper[2559]: W0801 00:44:13.903492    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ca4268b_50a4_426c_a0ab_275b25147d67.slice/crio-248dbf93612d071a183933da61179ba45b18e7591b7685f40f8f376792ac247d WatchSource:0}: Error finding container 248dbf93612d071a183933da61179ba45b18e7591b7685f40f8f376792ac247d: Status 404 returned error can't find the container with id 248dbf93612d071a183933da61179ba45b18e7591b7685f40f8f376792ac247d
Aug 01 00:44:14.803309 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:44:14.803272    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-cbdcr" event={"ID":"1ca4268b-50a4-426c-a0ab-275b25147d67","Type":"ContainerStarted","Data":"cb37c2f6f45a48a6edba0a8ed842882bcb82e3acfbde2fbfd69fa1bc5d08813e"}
Aug 01 00:44:14.803702 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:44:14.803325    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-cbdcr" event={"ID":"1ca4268b-50a4-426c-a0ab-275b25147d67","Type":"ContainerStarted","Data":"248dbf93612d071a183933da61179ba45b18e7591b7685f40f8f376792ac247d"}
Aug 01 00:44:16.814821 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:44:16.814782    2559 generic.go:356] "Generic (PLEG): container finished" podID="1ca4268b-50a4-426c-a0ab-275b25147d67" containerID="cb37c2f6f45a48a6edba0a8ed842882bcb82e3acfbde2fbfd69fa1bc5d08813e" exitCode=0
Aug 01 00:44:16.815220 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:44:16.814835    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-cbdcr" event={"ID":"1ca4268b-50a4-426c-a0ab-275b25147d67","Type":"ContainerDied","Data":"cb37c2f6f45a48a6edba0a8ed842882bcb82e3acfbde2fbfd69fa1bc5d08813e"}
Aug 01 00:44:17.821819 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:44:17.821784    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-cbdcr" event={"ID":"1ca4268b-50a4-426c-a0ab-275b25147d67","Type":"ContainerStarted","Data":"334cc11d57c7e1d2df32312034ecae245e407d5b83d1d14b400e534d6f2f19e4"}
Aug 01 00:44:17.842518 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:44:17.842460    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-cbdcr" podStartSLOduration=2.6801381429999998 podStartE2EDuration="4.842440229s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.162302086s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:44:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:44:17.840867111 +0000 UTC m=+3520.560666984" watchObservedRunningTime="2026-08-01 00:44:17.842440229 +0000 UTC m=+3520.562240076"
Aug 01 00:44:18.758427 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:44:18.758379    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-cbdcr"
Aug 01 00:44:19.320669 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:44:19.320625    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw" podUID="cfea3687-e01f-4cfa-bc88-7ee55bc7bed1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Aug 01 00:44:23.758801 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:44:23.758733    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-cbdcr"
Aug 01 00:44:23.760539 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:44:23.760506    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-cbdcr" podUID="1ca4268b-50a4-426c-a0ab-275b25147d67" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Aug 01 00:44:29.320574 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:44:29.320527    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw" podUID="cfea3687-e01f-4cfa-bc88-7ee55bc7bed1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Aug 01 00:44:32.227572 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:44:32.227534    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-cbdcr"]
Aug 01 00:44:39.320152 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:44:39.320097    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw" podUID="cfea3687-e01f-4cfa-bc88-7ee55bc7bed1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Aug 01 00:44:47.251541 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:44:47.251474    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-cbdcr" podUID="1ca4268b-50a4-426c-a0ab-275b25147d67" containerName="main" containerID="cri-o://334cc11d57c7e1d2df32312034ecae245e407d5b83d1d14b400e534d6f2f19e4" gracePeriod=45
Aug 01 00:44:49.320230 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:44:49.320178    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw" podUID="cfea3687-e01f-4cfa-bc88-7ee55bc7bed1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Aug 01 00:44:59.320303 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:44:59.320248    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw" podUID="cfea3687-e01f-4cfa-bc88-7ee55bc7bed1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Aug 01 00:45:09.320016 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:45:09.319964    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw" podUID="cfea3687-e01f-4cfa-bc88-7ee55bc7bed1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Aug 01 00:45:19.320401 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:45:19.320345    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw" podUID="cfea3687-e01f-4cfa-bc88-7ee55bc7bed1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Aug 01 00:45:29.320615 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:45:29.320561    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw" podUID="cfea3687-e01f-4cfa-bc88-7ee55bc7bed1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Aug 01 00:45:32.922119 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:45:32.922087    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-65d966bbf4-cbdcr_1ca4268b-50a4-426c-a0ab-275b25147d67/main/0.log"
Aug 01 00:45:32.924854 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:45:32.924831    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-cbdcr"
Aug 01 00:45:33.002194 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:45:33.002162    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ca4268b-50a4-426c-a0ab-275b25147d67-tmp-dir\") pod \"1ca4268b-50a4-426c-a0ab-275b25147d67\" (UID: \"1ca4268b-50a4-426c-a0ab-275b25147d67\") "
Aug 01 00:45:33.002383 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:45:33.002215    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ca4268b-50a4-426c-a0ab-275b25147d67-model-cache\") pod \"1ca4268b-50a4-426c-a0ab-275b25147d67\" (UID: \"1ca4268b-50a4-426c-a0ab-275b25147d67\") "
Aug 01 00:45:33.002383 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:45:33.002253    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ca4268b-50a4-426c-a0ab-275b25147d67-dshm\") pod \"1ca4268b-50a4-426c-a0ab-275b25147d67\" (UID: \"1ca4268b-50a4-426c-a0ab-275b25147d67\") "
Aug 01 00:45:33.002383 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:45:33.002272    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ca4268b-50a4-426c-a0ab-275b25147d67-kserve-provision-location\") pod \"1ca4268b-50a4-426c-a0ab-275b25147d67\" (UID: \"1ca4268b-50a4-426c-a0ab-275b25147d67\") "
Aug 01 00:45:33.002383 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:45:33.002341    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ca4268b-50a4-426c-a0ab-275b25147d67-cabundle-cert\") pod \"1ca4268b-50a4-426c-a0ab-275b25147d67\" (UID: \"1ca4268b-50a4-426c-a0ab-275b25147d67\") "
Aug 01 00:45:33.002383 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:45:33.002367    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ca4268b-50a4-426c-a0ab-275b25147d67-home\") pod \"1ca4268b-50a4-426c-a0ab-275b25147d67\" (UID: \"1ca4268b-50a4-426c-a0ab-275b25147d67\") "
Aug 01 00:45:33.002668 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:45:33.002404    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4t7k\" (UniqueName: \"kubernetes.io/projected/1ca4268b-50a4-426c-a0ab-275b25147d67-kube-api-access-p4t7k\") pod \"1ca4268b-50a4-426c-a0ab-275b25147d67\" (UID: \"1ca4268b-50a4-426c-a0ab-275b25147d67\") "
Aug 01 00:45:33.002668 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:45:33.002582    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ca4268b-50a4-426c-a0ab-275b25147d67-model-cache" (OuterVolumeSpecName: "model-cache") pod "1ca4268b-50a4-426c-a0ab-275b25147d67" (UID: "1ca4268b-50a4-426c-a0ab-275b25147d67"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:45:33.002668 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:45:33.002644    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ca4268b-50a4-426c-a0ab-275b25147d67-tls-certs\") pod \"1ca4268b-50a4-426c-a0ab-275b25147d67\" (UID: \"1ca4268b-50a4-426c-a0ab-275b25147d67\") "
Aug 01 00:45:33.002860 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:45:33.002790    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ca4268b-50a4-426c-a0ab-275b25147d67-home" (OuterVolumeSpecName: "home") pod "1ca4268b-50a4-426c-a0ab-275b25147d67" (UID: "1ca4268b-50a4-426c-a0ab-275b25147d67"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:45:33.002860 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:45:33.002803    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ca4268b-50a4-426c-a0ab-275b25147d67-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1ca4268b-50a4-426c-a0ab-275b25147d67" (UID: "1ca4268b-50a4-426c-a0ab-275b25147d67"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:45:33.002860 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:45:33.002812    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ca4268b-50a4-426c-a0ab-275b25147d67-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1ca4268b-50a4-426c-a0ab-275b25147d67" (UID: "1ca4268b-50a4-426c-a0ab-275b25147d67"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:45:33.003020 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:45:33.002988    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ca4268b-50a4-426c-a0ab-275b25147d67-cabundle-cert\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:45:33.003020 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:45:33.003007    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ca4268b-50a4-426c-a0ab-275b25147d67-home\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:45:33.003126 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:45:33.003019    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ca4268b-50a4-426c-a0ab-275b25147d67-tmp-dir\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:45:33.003126 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:45:33.003032    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ca4268b-50a4-426c-a0ab-275b25147d67-model-cache\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:45:33.005116 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:45:33.005088    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ca4268b-50a4-426c-a0ab-275b25147d67-kube-api-access-p4t7k" (OuterVolumeSpecName: "kube-api-access-p4t7k") pod "1ca4268b-50a4-426c-a0ab-275b25147d67" (UID: "1ca4268b-50a4-426c-a0ab-275b25147d67"). InnerVolumeSpecName "kube-api-access-p4t7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:45:33.005387 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:45:33.005287    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ca4268b-50a4-426c-a0ab-275b25147d67-dshm" (OuterVolumeSpecName: "dshm") pod "1ca4268b-50a4-426c-a0ab-275b25147d67" (UID: "1ca4268b-50a4-426c-a0ab-275b25147d67"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:45:33.005387 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:45:33.005357    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ca4268b-50a4-426c-a0ab-275b25147d67-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1ca4268b-50a4-426c-a0ab-275b25147d67" (UID: "1ca4268b-50a4-426c-a0ab-275b25147d67"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:45:33.024961 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:45:33.024877    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ca4268b-50a4-426c-a0ab-275b25147d67-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1ca4268b-50a4-426c-a0ab-275b25147d67" (UID: "1ca4268b-50a4-426c-a0ab-275b25147d67"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:45:33.104193 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:45:33.104161    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ca4268b-50a4-426c-a0ab-275b25147d67-dshm\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:45:33.104193 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:45:33.104191    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ca4268b-50a4-426c-a0ab-275b25147d67-kserve-provision-location\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:45:33.104382 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:45:33.104203    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4t7k\" (UniqueName: \"kubernetes.io/projected/1ca4268b-50a4-426c-a0ab-275b25147d67-kube-api-access-p4t7k\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:45:33.104382 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:45:33.104213    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ca4268b-50a4-426c-a0ab-275b25147d67-tls-certs\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:45:33.217513 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:45:33.217485    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-65d966bbf4-cbdcr_1ca4268b-50a4-426c-a0ab-275b25147d67/main/0.log"
Aug 01 00:45:33.217950 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:45:33.217928    2559 generic.go:356] "Generic (PLEG): container finished" podID="1ca4268b-50a4-426c-a0ab-275b25147d67" containerID="334cc11d57c7e1d2df32312034ecae245e407d5b83d1d14b400e534d6f2f19e4" exitCode=137
Aug 01 00:45:33.218027 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:45:33.217978    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-cbdcr"
Aug 01 00:45:33.218088 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:45:33.218021    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-cbdcr" event={"ID":"1ca4268b-50a4-426c-a0ab-275b25147d67","Type":"ContainerDied","Data":"334cc11d57c7e1d2df32312034ecae245e407d5b83d1d14b400e534d6f2f19e4"}
Aug 01 00:45:33.218143 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:45:33.218127    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-cbdcr" event={"ID":"1ca4268b-50a4-426c-a0ab-275b25147d67","Type":"ContainerDied","Data":"248dbf93612d071a183933da61179ba45b18e7591b7685f40f8f376792ac247d"}
Aug 01 00:45:33.218203 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:45:33.218164    2559 scope.go:117] "RemoveContainer" containerID="334cc11d57c7e1d2df32312034ecae245e407d5b83d1d14b400e534d6f2f19e4"
Aug 01 00:45:33.228339 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:45:33.228319    2559 scope.go:117] "RemoveContainer" containerID="cb37c2f6f45a48a6edba0a8ed842882bcb82e3acfbde2fbfd69fa1bc5d08813e"
Aug 01 00:45:33.237646 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:45:33.237622    2559 scope.go:117] "RemoveContainer" containerID="334cc11d57c7e1d2df32312034ecae245e407d5b83d1d14b400e534d6f2f19e4"
Aug 01 00:45:33.237983 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:45:33.237952    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"334cc11d57c7e1d2df32312034ecae245e407d5b83d1d14b400e534d6f2f19e4\": container with ID starting with 334cc11d57c7e1d2df32312034ecae245e407d5b83d1d14b400e534d6f2f19e4 not found: ID does not exist" containerID="334cc11d57c7e1d2df32312034ecae245e407d5b83d1d14b400e534d6f2f19e4"
Aug 01 00:45:33.238077 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:45:33.237984    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"334cc11d57c7e1d2df32312034ecae245e407d5b83d1d14b400e534d6f2f19e4"} err="failed to get container status \"334cc11d57c7e1d2df32312034ecae245e407d5b83d1d14b400e534d6f2f19e4\": rpc error: code = NotFound desc = could not find container \"334cc11d57c7e1d2df32312034ecae245e407d5b83d1d14b400e534d6f2f19e4\": container with ID starting with 334cc11d57c7e1d2df32312034ecae245e407d5b83d1d14b400e534d6f2f19e4 not found: ID does not exist"
Aug 01 00:45:33.238077 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:45:33.238008    2559 scope.go:117] "RemoveContainer" containerID="cb37c2f6f45a48a6edba0a8ed842882bcb82e3acfbde2fbfd69fa1bc5d08813e"
Aug 01 00:45:33.238460 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:45:33.238433    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb37c2f6f45a48a6edba0a8ed842882bcb82e3acfbde2fbfd69fa1bc5d08813e\": container with ID starting with cb37c2f6f45a48a6edba0a8ed842882bcb82e3acfbde2fbfd69fa1bc5d08813e not found: ID does not exist" containerID="cb37c2f6f45a48a6edba0a8ed842882bcb82e3acfbde2fbfd69fa1bc5d08813e"
Aug 01 00:45:33.238585 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:45:33.238467    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb37c2f6f45a48a6edba0a8ed842882bcb82e3acfbde2fbfd69fa1bc5d08813e"} err="failed to get container status \"cb37c2f6f45a48a6edba0a8ed842882bcb82e3acfbde2fbfd69fa1bc5d08813e\": rpc error: code = NotFound desc = could not find container \"cb37c2f6f45a48a6edba0a8ed842882bcb82e3acfbde2fbfd69fa1bc5d08813e\": container with ID starting with cb37c2f6f45a48a6edba0a8ed842882bcb82e3acfbde2fbfd69fa1bc5d08813e not found: ID does not exist"
Aug 01 00:45:33.240649 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:45:33.240627    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-cbdcr"]
Aug 01 00:45:33.243972 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:45:33.243952    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-65d966bbf4-cbdcr"]
Aug 01 00:45:33.786201 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:45:33.786167    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ca4268b-50a4-426c-a0ab-275b25147d67" path="/var/lib/kubelet/pods/1ca4268b-50a4-426c-a0ab-275b25147d67/volumes"
Aug 01 00:45:37.994113 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:45:37.994085    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-spblg_c2e50946-aea4-4886-82ec-0bd5a31196cf/console-operator/1.log"
Aug 01 00:45:38.010647 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:45:38.010622    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-spblg_c2e50946-aea4-4886-82ec-0bd5a31196cf/console-operator/1.log"
Aug 01 00:45:39.320538 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:45:39.320494    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw" podUID="cfea3687-e01f-4cfa-bc88-7ee55bc7bed1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Aug 01 00:45:49.320092 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:45:49.320049    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw" podUID="cfea3687-e01f-4cfa-bc88-7ee55bc7bed1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Aug 01 00:45:59.330434 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:45:59.330387    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw"
Aug 01 00:45:59.338715 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:45:59.338681    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw"
Aug 01 00:46:08.083103 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:08.082445    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw"]
Aug 01 00:46:18.458573 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:18.458539    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-52mqj"]
Aug 01 00:46:18.459063 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:18.458873    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ca4268b-50a4-426c-a0ab-275b25147d67" containerName="storage-initializer"
Aug 01 00:46:18.459063 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:18.458887    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ca4268b-50a4-426c-a0ab-275b25147d67" containerName="storage-initializer"
Aug 01 00:46:18.459063 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:18.458904    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ca4268b-50a4-426c-a0ab-275b25147d67" containerName="main"
Aug 01 00:46:18.459063 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:18.458909    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ca4268b-50a4-426c-a0ab-275b25147d67" containerName="main"
Aug 01 00:46:18.459063 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:18.458967    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ca4268b-50a4-426c-a0ab-275b25147d67" containerName="main"
Aug 01 00:46:18.463161 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:18.463137    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-52mqj"
Aug 01 00:46:18.465629 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:18.465606    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Aug 01 00:46:18.465780 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:18.465756    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Aug 01 00:46:18.467114 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:18.467091    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Aug 01 00:46:18.467375 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:18.467350    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-mfwrn\""
Aug 01 00:46:18.469351 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:18.469327    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-52mqj"]
Aug 01 00:46:18.495716 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:18.495682    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d711d94e-2255-4cba-a1b7-90b918630b86-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-52mqj\" (UID: \"d711d94e-2255-4cba-a1b7-90b918630b86\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-52mqj"
Aug 01 00:46:18.495916 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:18.495727    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d711d94e-2255-4cba-a1b7-90b918630b86-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-52mqj\" (UID: \"d711d94e-2255-4cba-a1b7-90b918630b86\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-52mqj"
Aug 01 00:46:18.495916 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:18.495780    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d711d94e-2255-4cba-a1b7-90b918630b86-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-52mqj\" (UID: \"d711d94e-2255-4cba-a1b7-90b918630b86\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-52mqj"
Aug 01 00:46:18.495916 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:18.495844    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d711d94e-2255-4cba-a1b7-90b918630b86-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-52mqj\" (UID: \"d711d94e-2255-4cba-a1b7-90b918630b86\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-52mqj"
Aug 01 00:46:18.495916 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:18.495869    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc45n\" (UniqueName: \"kubernetes.io/projected/d711d94e-2255-4cba-a1b7-90b918630b86-kube-api-access-cc45n\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-52mqj\" (UID: \"d711d94e-2255-4cba-a1b7-90b918630b86\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-52mqj"
Aug 01 00:46:18.495916 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:18.495898    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d711d94e-2255-4cba-a1b7-90b918630b86-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-52mqj\" (UID: \"d711d94e-2255-4cba-a1b7-90b918630b86\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-52mqj"
Aug 01 00:46:18.596971 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:18.596932    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d711d94e-2255-4cba-a1b7-90b918630b86-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-52mqj\" (UID: \"d711d94e-2255-4cba-a1b7-90b918630b86\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-52mqj"
Aug 01 00:46:18.596971 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:18.596976    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d711d94e-2255-4cba-a1b7-90b918630b86-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-52mqj\" (UID: \"d711d94e-2255-4cba-a1b7-90b918630b86\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-52mqj"
Aug 01 00:46:18.597234 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:18.597017    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d711d94e-2255-4cba-a1b7-90b918630b86-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-52mqj\" (UID: \"d711d94e-2255-4cba-a1b7-90b918630b86\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-52mqj"
Aug 01 00:46:18.597234 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:18.597050    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d711d94e-2255-4cba-a1b7-90b918630b86-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-52mqj\" (UID: \"d711d94e-2255-4cba-a1b7-90b918630b86\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-52mqj"
Aug 01 00:46:18.597234 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:18.597108    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d711d94e-2255-4cba-a1b7-90b918630b86-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-52mqj\" (UID: \"d711d94e-2255-4cba-a1b7-90b918630b86\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-52mqj"
Aug 01 00:46:18.597234 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:18.597135    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc45n\" (UniqueName: \"kubernetes.io/projected/d711d94e-2255-4cba-a1b7-90b918630b86-kube-api-access-cc45n\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-52mqj\" (UID: \"d711d94e-2255-4cba-a1b7-90b918630b86\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-52mqj"
Aug 01 00:46:18.597436 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:18.597373    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d711d94e-2255-4cba-a1b7-90b918630b86-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-52mqj\" (UID: \"d711d94e-2255-4cba-a1b7-90b918630b86\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-52mqj"
Aug 01 00:46:18.597489 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:18.597468    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d711d94e-2255-4cba-a1b7-90b918630b86-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-52mqj\" (UID: \"d711d94e-2255-4cba-a1b7-90b918630b86\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-52mqj"
Aug 01 00:46:18.597554 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:18.597469    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d711d94e-2255-4cba-a1b7-90b918630b86-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-52mqj\" (UID: \"d711d94e-2255-4cba-a1b7-90b918630b86\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-52mqj"
Aug 01 00:46:18.599265 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:18.599242    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d711d94e-2255-4cba-a1b7-90b918630b86-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-52mqj\" (UID: \"d711d94e-2255-4cba-a1b7-90b918630b86\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-52mqj"
Aug 01 00:46:18.599577 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:18.599561    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d711d94e-2255-4cba-a1b7-90b918630b86-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-52mqj\" (UID: \"d711d94e-2255-4cba-a1b7-90b918630b86\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-52mqj"
Aug 01 00:46:18.604884 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:18.604861    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc45n\" (UniqueName: \"kubernetes.io/projected/d711d94e-2255-4cba-a1b7-90b918630b86-kube-api-access-cc45n\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-52mqj\" (UID: \"d711d94e-2255-4cba-a1b7-90b918630b86\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-52mqj"
Aug 01 00:46:18.774808 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:18.774703    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-52mqj"
Aug 01 00:46:18.908977 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:18.908953    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-52mqj"]
Aug 01 00:46:18.910830 ip-10-0-130-148 kubenswrapper[2559]: W0801 00:46:18.910802    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd711d94e_2255_4cba_a1b7_90b918630b86.slice/crio-545ba428ac5035b8a86c93845714b0cbdc3d9d94d97dff0ced644012581d76c1 WatchSource:0}: Error finding container 545ba428ac5035b8a86c93845714b0cbdc3d9d94d97dff0ced644012581d76c1: Status 404 returned error can't find the container with id 545ba428ac5035b8a86c93845714b0cbdc3d9d94d97dff0ced644012581d76c1
Aug 01 00:46:19.440308 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:19.440262    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-52mqj" event={"ID":"d711d94e-2255-4cba-a1b7-90b918630b86","Type":"ContainerStarted","Data":"28adc5e31ebf1919560678b2b5ebed20270f61061d8e261765ad4c2e16588623"}
Aug 01 00:46:19.440511 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:19.440343    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-52mqj" event={"ID":"d711d94e-2255-4cba-a1b7-90b918630b86","Type":"ContainerStarted","Data":"545ba428ac5035b8a86c93845714b0cbdc3d9d94d97dff0ced644012581d76c1"}
Aug 01 00:46:19.458398 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:19.458343    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-52mqj" podStartSLOduration=1.458328114 podStartE2EDuration="1.458328114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:46:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:46:19.45679731 +0000 UTC m=+3642.176597158" watchObservedRunningTime="2026-08-01 00:46:19.458328114 +0000 UTC m=+3642.178127959"
Aug 01 00:46:19.775794 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:19.775733    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-52mqj"
Aug 01 00:46:23.105112 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:23.105065    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw" podUID="cfea3687-e01f-4cfa-bc88-7ee55bc7bed1" containerName="main" containerID="cri-o://b9f5f6f25570555768c52e385822d47b843ee62316dd3bb1f71bc3bf57538e48" gracePeriod=45
Aug 01 00:46:23.861982 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:23.861950    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw"
Aug 01 00:46:23.940489 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:23.940397    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfea3687-e01f-4cfa-bc88-7ee55bc7bed1-model-cache\") pod \"cfea3687-e01f-4cfa-bc88-7ee55bc7bed1\" (UID: \"cfea3687-e01f-4cfa-bc88-7ee55bc7bed1\") "
Aug 01 00:46:23.940489 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:23.940456    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzb78\" (UniqueName: \"kubernetes.io/projected/cfea3687-e01f-4cfa-bc88-7ee55bc7bed1-kube-api-access-jzb78\") pod \"cfea3687-e01f-4cfa-bc88-7ee55bc7bed1\" (UID: \"cfea3687-e01f-4cfa-bc88-7ee55bc7bed1\") "
Aug 01 00:46:23.940489 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:23.940486    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfea3687-e01f-4cfa-bc88-7ee55bc7bed1-tls-certs\") pod \"cfea3687-e01f-4cfa-bc88-7ee55bc7bed1\" (UID: \"cfea3687-e01f-4cfa-bc88-7ee55bc7bed1\") "
Aug 01 00:46:23.940807 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:23.940529    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfea3687-e01f-4cfa-bc88-7ee55bc7bed1-dshm\") pod \"cfea3687-e01f-4cfa-bc88-7ee55bc7bed1\" (UID: \"cfea3687-e01f-4cfa-bc88-7ee55bc7bed1\") "
Aug 01 00:46:23.940807 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:23.940553    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cfea3687-e01f-4cfa-bc88-7ee55bc7bed1-cabundle-cert\") pod \"cfea3687-e01f-4cfa-bc88-7ee55bc7bed1\" (UID: \"cfea3687-e01f-4cfa-bc88-7ee55bc7bed1\") "
Aug 01 00:46:23.940807 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:23.940581    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfea3687-e01f-4cfa-bc88-7ee55bc7bed1-home\") pod \"cfea3687-e01f-4cfa-bc88-7ee55bc7bed1\" (UID: \"cfea3687-e01f-4cfa-bc88-7ee55bc7bed1\") "
Aug 01 00:46:23.940807 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:23.940621    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cfea3687-e01f-4cfa-bc88-7ee55bc7bed1-kserve-provision-location\") pod \"cfea3687-e01f-4cfa-bc88-7ee55bc7bed1\" (UID: \"cfea3687-e01f-4cfa-bc88-7ee55bc7bed1\") "
Aug 01 00:46:23.940807 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:23.940693    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfea3687-e01f-4cfa-bc88-7ee55bc7bed1-model-cache" (OuterVolumeSpecName: "model-cache") pod "cfea3687-e01f-4cfa-bc88-7ee55bc7bed1" (UID: "cfea3687-e01f-4cfa-bc88-7ee55bc7bed1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:46:23.940807 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:23.940774    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfea3687-e01f-4cfa-bc88-7ee55bc7bed1-tmp-dir\") pod \"cfea3687-e01f-4cfa-bc88-7ee55bc7bed1\" (UID: \"cfea3687-e01f-4cfa-bc88-7ee55bc7bed1\") "
Aug 01 00:46:23.941123 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:23.941012    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfea3687-e01f-4cfa-bc88-7ee55bc7bed1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cfea3687-e01f-4cfa-bc88-7ee55bc7bed1" (UID: "cfea3687-e01f-4cfa-bc88-7ee55bc7bed1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:46:23.941123 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:23.941032    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfea3687-e01f-4cfa-bc88-7ee55bc7bed1-model-cache\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:46:23.941399 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:23.941372    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfea3687-e01f-4cfa-bc88-7ee55bc7bed1-home" (OuterVolumeSpecName: "home") pod "cfea3687-e01f-4cfa-bc88-7ee55bc7bed1" (UID: "cfea3687-e01f-4cfa-bc88-7ee55bc7bed1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:46:23.943418 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:23.943389    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfea3687-e01f-4cfa-bc88-7ee55bc7bed1-kube-api-access-jzb78" (OuterVolumeSpecName: "kube-api-access-jzb78") pod "cfea3687-e01f-4cfa-bc88-7ee55bc7bed1" (UID: "cfea3687-e01f-4cfa-bc88-7ee55bc7bed1"). InnerVolumeSpecName "kube-api-access-jzb78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:46:23.943418 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:23.943410    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfea3687-e01f-4cfa-bc88-7ee55bc7bed1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cfea3687-e01f-4cfa-bc88-7ee55bc7bed1" (UID: "cfea3687-e01f-4cfa-bc88-7ee55bc7bed1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:46:23.944921 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:23.944903    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfea3687-e01f-4cfa-bc88-7ee55bc7bed1-dshm" (OuterVolumeSpecName: "dshm") pod "cfea3687-e01f-4cfa-bc88-7ee55bc7bed1" (UID: "cfea3687-e01f-4cfa-bc88-7ee55bc7bed1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:46:23.955854 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:23.955809    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfea3687-e01f-4cfa-bc88-7ee55bc7bed1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cfea3687-e01f-4cfa-bc88-7ee55bc7bed1" (UID: "cfea3687-e01f-4cfa-bc88-7ee55bc7bed1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:46:23.964988 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:23.964959    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfea3687-e01f-4cfa-bc88-7ee55bc7bed1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cfea3687-e01f-4cfa-bc88-7ee55bc7bed1" (UID: "cfea3687-e01f-4cfa-bc88-7ee55bc7bed1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:46:24.041988 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:24.041953    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cfea3687-e01f-4cfa-bc88-7ee55bc7bed1-kserve-provision-location\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:46:24.041988 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:24.041982    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfea3687-e01f-4cfa-bc88-7ee55bc7bed1-tmp-dir\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:46:24.041988 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:24.041994    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzb78\" (UniqueName: \"kubernetes.io/projected/cfea3687-e01f-4cfa-bc88-7ee55bc7bed1-kube-api-access-jzb78\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:46:24.042215 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:24.042003    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfea3687-e01f-4cfa-bc88-7ee55bc7bed1-tls-certs\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:46:24.042215 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:24.042011    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfea3687-e01f-4cfa-bc88-7ee55bc7bed1-dshm\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:46:24.042215 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:24.042019    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cfea3687-e01f-4cfa-bc88-7ee55bc7bed1-cabundle-cert\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:46:24.042215 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:24.042027    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfea3687-e01f-4cfa-bc88-7ee55bc7bed1-home\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:46:24.462779 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:24.462724    2559 generic.go:356] "Generic (PLEG): container finished" podID="cfea3687-e01f-4cfa-bc88-7ee55bc7bed1" containerID="b9f5f6f25570555768c52e385822d47b843ee62316dd3bb1f71bc3bf57538e48" exitCode=0
Aug 01 00:46:24.462779 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:24.462779    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw" event={"ID":"cfea3687-e01f-4cfa-bc88-7ee55bc7bed1","Type":"ContainerDied","Data":"b9f5f6f25570555768c52e385822d47b843ee62316dd3bb1f71bc3bf57538e48"}
Aug 01 00:46:24.463303 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:24.462794    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw"
Aug 01 00:46:24.463303 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:24.462822    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw" event={"ID":"cfea3687-e01f-4cfa-bc88-7ee55bc7bed1","Type":"ContainerDied","Data":"0972b957a7bd46723358f2781265f67734dbfe73cd70a55a8c7ea43ed1e3a466"}
Aug 01 00:46:24.463303 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:24.462844    2559 scope.go:117] "RemoveContainer" containerID="b9f5f6f25570555768c52e385822d47b843ee62316dd3bb1f71bc3bf57538e48"
Aug 01 00:46:24.472342 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:24.472324    2559 scope.go:117] "RemoveContainer" containerID="190dededaabbf7fec8b7d0b722923280d410372ae6c3011c1f3c64249a09d577"
Aug 01 00:46:24.480081 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:24.480063    2559 scope.go:117] "RemoveContainer" containerID="b9f5f6f25570555768c52e385822d47b843ee62316dd3bb1f71bc3bf57538e48"
Aug 01 00:46:24.480339 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:46:24.480318    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9f5f6f25570555768c52e385822d47b843ee62316dd3bb1f71bc3bf57538e48\": container with ID starting with b9f5f6f25570555768c52e385822d47b843ee62316dd3bb1f71bc3bf57538e48 not found: ID does not exist" containerID="b9f5f6f25570555768c52e385822d47b843ee62316dd3bb1f71bc3bf57538e48"
Aug 01 00:46:24.480383 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:24.480350    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9f5f6f25570555768c52e385822d47b843ee62316dd3bb1f71bc3bf57538e48"} err="failed to get container status \"b9f5f6f25570555768c52e385822d47b843ee62316dd3bb1f71bc3bf57538e48\": rpc error: code = NotFound desc = could not find container \"b9f5f6f25570555768c52e385822d47b843ee62316dd3bb1f71bc3bf57538e48\": container with ID starting with b9f5f6f25570555768c52e385822d47b843ee62316dd3bb1f71bc3bf57538e48 not found: ID does not exist"
Aug 01 00:46:24.480383 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:24.480368    2559 scope.go:117] "RemoveContainer" containerID="190dededaabbf7fec8b7d0b722923280d410372ae6c3011c1f3c64249a09d577"
Aug 01 00:46:24.480624 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:46:24.480606    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"190dededaabbf7fec8b7d0b722923280d410372ae6c3011c1f3c64249a09d577\": container with ID starting with 190dededaabbf7fec8b7d0b722923280d410372ae6c3011c1f3c64249a09d577 not found: ID does not exist" containerID="190dededaabbf7fec8b7d0b722923280d410372ae6c3011c1f3c64249a09d577"
Aug 01 00:46:24.480673 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:24.480631    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"190dededaabbf7fec8b7d0b722923280d410372ae6c3011c1f3c64249a09d577"} err="failed to get container status \"190dededaabbf7fec8b7d0b722923280d410372ae6c3011c1f3c64249a09d577\": rpc error: code = NotFound desc = could not find container \"190dededaabbf7fec8b7d0b722923280d410372ae6c3011c1f3c64249a09d577\": container with ID starting with 190dededaabbf7fec8b7d0b722923280d410372ae6c3011c1f3c64249a09d577 not found: ID does not exist"
Aug 01 00:46:24.485178 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:24.485158    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw"]
Aug 01 00:46:24.489188 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:24.489168    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-fccfb56ff-m2jkw"]
Aug 01 00:46:25.786734 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:25.786694    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfea3687-e01f-4cfa-bc88-7ee55bc7bed1" path="/var/lib/kubelet/pods/cfea3687-e01f-4cfa-bc88-7ee55bc7bed1/volumes"
Aug 01 00:46:28.775852 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:28.775804    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-52mqj"
Aug 01 00:46:28.788348 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:28.788318    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-52mqj"
Aug 01 00:46:29.496985 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:29.496954    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-52mqj"
Aug 01 00:46:41.662738 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:41.662705    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-52mqj"]
Aug 01 00:46:41.671511 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:46:41.671475    2559 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 01 00:46:41.671511 ip-10-0-130-148 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:46:41.671511 ip-10-0-130-148 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Aug 01 00:46:41.671511 ip-10-0-130-148 kubenswrapper[2559]:  > containerID="28adc5e31ebf1919560678b2b5ebed20270f61061d8e261765ad4c2e16588623" cmd=["/bin/sleep","15"]
Aug 01 00:46:41.671873 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:41.671535    2559 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-52mqj" message="" err=<
Aug 01 00:46:41.671873 ip-10-0-130-148 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:46:41.671873 ip-10-0-130-148 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Aug 01 00:46:41.671873 ip-10-0-130-148 kubenswrapper[2559]:  >
Aug 01 00:46:41.671873 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:46:41.671549    2559 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 01 00:46:41.671873 ip-10-0-130-148 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:46:41.671873 ip-10-0-130-148 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Aug 01 00:46:41.671873 ip-10-0-130-148 kubenswrapper[2559]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-52mqj" podUID="d711d94e-2255-4cba-a1b7-90b918630b86" containerName="main" containerID="cri-o://28adc5e31ebf1919560678b2b5ebed20270f61061d8e261765ad4c2e16588623"
Aug 01 00:46:41.671873 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:41.671614    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-52mqj" podUID="d711d94e-2255-4cba-a1b7-90b918630b86" containerName="main" containerID="cri-o://28adc5e31ebf1919560678b2b5ebed20270f61061d8e261765ad4c2e16588623" gracePeriod=60
Aug 01 00:46:42.024442 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:42.024417    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-52mqj"
Aug 01 00:46:42.091053 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:42.091017    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d711d94e-2255-4cba-a1b7-90b918630b86-tls-certs\") pod \"d711d94e-2255-4cba-a1b7-90b918630b86\" (UID: \"d711d94e-2255-4cba-a1b7-90b918630b86\") "
Aug 01 00:46:42.091243 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:42.091058    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d711d94e-2255-4cba-a1b7-90b918630b86-tmp-dir\") pod \"d711d94e-2255-4cba-a1b7-90b918630b86\" (UID: \"d711d94e-2255-4cba-a1b7-90b918630b86\") "
Aug 01 00:46:42.091243 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:42.091201    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d711d94e-2255-4cba-a1b7-90b918630b86-home\") pod \"d711d94e-2255-4cba-a1b7-90b918630b86\" (UID: \"d711d94e-2255-4cba-a1b7-90b918630b86\") "
Aug 01 00:46:42.091243 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:42.091242    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d711d94e-2255-4cba-a1b7-90b918630b86-model-cache\") pod \"d711d94e-2255-4cba-a1b7-90b918630b86\" (UID: \"d711d94e-2255-4cba-a1b7-90b918630b86\") "
Aug 01 00:46:42.091420 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:42.091285    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cc45n\" (UniqueName: \"kubernetes.io/projected/d711d94e-2255-4cba-a1b7-90b918630b86-kube-api-access-cc45n\") pod \"d711d94e-2255-4cba-a1b7-90b918630b86\" (UID: \"d711d94e-2255-4cba-a1b7-90b918630b86\") "
Aug 01 00:46:42.091420 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:42.091339    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d711d94e-2255-4cba-a1b7-90b918630b86-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d711d94e-2255-4cba-a1b7-90b918630b86" (UID: "d711d94e-2255-4cba-a1b7-90b918630b86"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:46:42.091420 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:42.091356    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d711d94e-2255-4cba-a1b7-90b918630b86-dshm\") pod \"d711d94e-2255-4cba-a1b7-90b918630b86\" (UID: \"d711d94e-2255-4cba-a1b7-90b918630b86\") "
Aug 01 00:46:42.091584 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:42.091448    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d711d94e-2255-4cba-a1b7-90b918630b86-home" (OuterVolumeSpecName: "home") pod "d711d94e-2255-4cba-a1b7-90b918630b86" (UID: "d711d94e-2255-4cba-a1b7-90b918630b86"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:46:42.091584 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:42.091455    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d711d94e-2255-4cba-a1b7-90b918630b86-model-cache" (OuterVolumeSpecName: "model-cache") pod "d711d94e-2255-4cba-a1b7-90b918630b86" (UID: "d711d94e-2255-4cba-a1b7-90b918630b86"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:46:42.091688 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:42.091620    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d711d94e-2255-4cba-a1b7-90b918630b86-tmp-dir\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:46:42.091688 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:42.091636    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d711d94e-2255-4cba-a1b7-90b918630b86-home\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:46:42.091688 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:42.091645    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d711d94e-2255-4cba-a1b7-90b918630b86-model-cache\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:46:42.093324 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:42.093299    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d711d94e-2255-4cba-a1b7-90b918630b86-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d711d94e-2255-4cba-a1b7-90b918630b86" (UID: "d711d94e-2255-4cba-a1b7-90b918630b86"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:46:42.093458 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:42.093435    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d711d94e-2255-4cba-a1b7-90b918630b86-dshm" (OuterVolumeSpecName: "dshm") pod "d711d94e-2255-4cba-a1b7-90b918630b86" (UID: "d711d94e-2255-4cba-a1b7-90b918630b86"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:46:42.093521 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:42.093468    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d711d94e-2255-4cba-a1b7-90b918630b86-kube-api-access-cc45n" (OuterVolumeSpecName: "kube-api-access-cc45n") pod "d711d94e-2255-4cba-a1b7-90b918630b86" (UID: "d711d94e-2255-4cba-a1b7-90b918630b86"). InnerVolumeSpecName "kube-api-access-cc45n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:46:42.192186 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:42.192100    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cc45n\" (UniqueName: \"kubernetes.io/projected/d711d94e-2255-4cba-a1b7-90b918630b86-kube-api-access-cc45n\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:46:42.192186 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:42.192131    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d711d94e-2255-4cba-a1b7-90b918630b86-dshm\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:46:42.192186 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:42.192142    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d711d94e-2255-4cba-a1b7-90b918630b86-tls-certs\") on node \"ip-10-0-130-148.ec2.internal\" DevicePath \"\""
Aug 01 00:46:42.547225 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:42.547188    2559 generic.go:356] "Generic (PLEG): container finished" podID="d711d94e-2255-4cba-a1b7-90b918630b86" containerID="28adc5e31ebf1919560678b2b5ebed20270f61061d8e261765ad4c2e16588623" exitCode=0
Aug 01 00:46:42.547448 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:42.547240    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-52mqj"
Aug 01 00:46:42.547448 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:42.547280    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-52mqj" event={"ID":"d711d94e-2255-4cba-a1b7-90b918630b86","Type":"ContainerDied","Data":"28adc5e31ebf1919560678b2b5ebed20270f61061d8e261765ad4c2e16588623"}
Aug 01 00:46:42.547448 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:42.547351    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-52mqj" event={"ID":"d711d94e-2255-4cba-a1b7-90b918630b86","Type":"ContainerDied","Data":"545ba428ac5035b8a86c93845714b0cbdc3d9d94d97dff0ced644012581d76c1"}
Aug 01 00:46:42.547448 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:42.547381    2559 scope.go:117] "RemoveContainer" containerID="28adc5e31ebf1919560678b2b5ebed20270f61061d8e261765ad4c2e16588623"
Aug 01 00:46:42.556944 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:42.556916    2559 scope.go:117] "RemoveContainer" containerID="28adc5e31ebf1919560678b2b5ebed20270f61061d8e261765ad4c2e16588623"
Aug 01 00:46:42.557174 ip-10-0-130-148 kubenswrapper[2559]: E0801 00:46:42.557156    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28adc5e31ebf1919560678b2b5ebed20270f61061d8e261765ad4c2e16588623\": container with ID starting with 28adc5e31ebf1919560678b2b5ebed20270f61061d8e261765ad4c2e16588623 not found: ID does not exist" containerID="28adc5e31ebf1919560678b2b5ebed20270f61061d8e261765ad4c2e16588623"
Aug 01 00:46:42.557221 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:42.557184    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28adc5e31ebf1919560678b2b5ebed20270f61061d8e261765ad4c2e16588623"} err="failed to get container status \"28adc5e31ebf1919560678b2b5ebed20270f61061d8e261765ad4c2e16588623\": rpc error: code = NotFound desc = could not find container \"28adc5e31ebf1919560678b2b5ebed20270f61061d8e261765ad4c2e16588623\": container with ID starting with 28adc5e31ebf1919560678b2b5ebed20270f61061d8e261765ad4c2e16588623 not found: ID does not exist"
Aug 01 00:46:42.568613 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:42.568589    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-52mqj"]
Aug 01 00:46:42.570007 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:42.569988    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-52mqj"]
Aug 01 00:46:43.786040 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:46:43.786007    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d711d94e-2255-4cba-a1b7-90b918630b86" path="/var/lib/kubelet/pods/d711d94e-2255-4cba-a1b7-90b918630b86/volumes"
Aug 01 00:49:30.015429 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:49:30.015400    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5579c88784-ntwhl_c33c7c85-f678-49ff-bcc0-5b1de21c563b/router/0.log"
Aug 01 00:49:30.976282 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:49:30.976251    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5579c88784-ntwhl_c33c7c85-f678-49ff-bcc0-5b1de21c563b/router/0.log"
Aug 01 00:49:31.924035 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:49:31.924005    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5579c88784-ntwhl_c33c7c85-f678-49ff-bcc0-5b1de21c563b/router/0.log"
Aug 01 00:49:32.900436 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:49:32.900405    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5579c88784-ntwhl_c33c7c85-f678-49ff-bcc0-5b1de21c563b/router/0.log"
Aug 01 00:49:34.163894 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:49:34.163861    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-4pvmb_12c3ad76-0e8b-4c7a-9402-f56d9519d156/limitador/0.log"
Aug 01 00:49:35.792087 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:49:35.792056    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-4pvmb_12c3ad76-0e8b-4c7a-9402-f56d9519d156/limitador/0.log"
Aug 01 00:49:37.438339 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:49:37.438306    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-4pvmb_12c3ad76-0e8b-4c7a-9402-f56d9519d156/limitador/0.log"
Aug 01 00:49:38.832989 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:49:38.832957    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-4pvmb_12c3ad76-0e8b-4c7a-9402-f56d9519d156/limitador/0.log"
Aug 01 00:49:40.113837 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:49:40.113801    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-4pvmb_12c3ad76-0e8b-4c7a-9402-f56d9519d156/limitador/0.log"
Aug 01 00:49:41.433152 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:49:41.433121    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-4pvmb_12c3ad76-0e8b-4c7a-9402-f56d9519d156/limitador/0.log"
Aug 01 00:49:42.741732 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:49:42.741701    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-4pvmb_12c3ad76-0e8b-4c7a-9402-f56d9519d156/limitador/0.log"
Aug 01 00:49:44.034605 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:49:44.034570    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-4pvmb_12c3ad76-0e8b-4c7a-9402-f56d9519d156/limitador/0.log"
Aug 01 00:49:45.333109 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:49:45.333078    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-4pvmb_12c3ad76-0e8b-4c7a-9402-f56d9519d156/limitador/0.log"
Aug 01 00:49:46.657572 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:49:46.657538    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-4pvmb_12c3ad76-0e8b-4c7a-9402-f56d9519d156/limitador/0.log"
Aug 01 00:49:47.974853 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:49:47.974824    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-4pvmb_12c3ad76-0e8b-4c7a-9402-f56d9519d156/limitador/0.log"
Aug 01 00:49:49.270214 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:49:49.270181    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-4pvmb_12c3ad76-0e8b-4c7a-9402-f56d9519d156/limitador/0.log"
Aug 01 00:49:50.599845 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:49:50.599811    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-4pvmb_12c3ad76-0e8b-4c7a-9402-f56d9519d156/limitador/0.log"
Aug 01 00:49:51.893454 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:49:51.893423    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-4pvmb_12c3ad76-0e8b-4c7a-9402-f56d9519d156/limitador/0.log"
Aug 01 00:49:53.189925 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:49:53.189883    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-4pvmb_12c3ad76-0e8b-4c7a-9402-f56d9519d156/limitador/0.log"
Aug 01 00:49:54.495783 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:49:54.495751    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-4pvmb_12c3ad76-0e8b-4c7a-9402-f56d9519d156/limitador/0.log"
Aug 01 00:49:55.790500 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:49:55.790467    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-4pvmb_12c3ad76-0e8b-4c7a-9402-f56d9519d156/limitador/0.log"
Aug 01 00:49:57.066823 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:49:57.066788    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-4pvmb_12c3ad76-0e8b-4c7a-9402-f56d9519d156/limitador/0.log"
Aug 01 00:49:58.388244 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:49:58.388214    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-4pvmb_12c3ad76-0e8b-4c7a-9402-f56d9519d156/limitador/0.log"
Aug 01 00:49:59.691447 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:49:59.691416    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-4pvmb_12c3ad76-0e8b-4c7a-9402-f56d9519d156/limitador/0.log"
Aug 01 00:50:00.983256 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:00.983166    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-4pvmb_12c3ad76-0e8b-4c7a-9402-f56d9519d156/limitador/0.log"
Aug 01 00:50:02.303338 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:02.303313    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-4pvmb_12c3ad76-0e8b-4c7a-9402-f56d9519d156/limitador/0.log"
Aug 01 00:50:03.636690 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:03.636659    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-4pvmb_12c3ad76-0e8b-4c7a-9402-f56d9519d156/limitador/0.log"
Aug 01 00:50:04.979024 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:04.978992    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-4pvmb_12c3ad76-0e8b-4c7a-9402-f56d9519d156/limitador/0.log"
Aug 01 00:50:06.300390 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:06.300358    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-4pvmb_12c3ad76-0e8b-4c7a-9402-f56d9519d156/limitador/0.log"
Aug 01 00:50:08.436284 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:08.436249    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kbl7c/must-gather-t2vpq"]
Aug 01 00:50:08.436710 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:08.436536    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d711d94e-2255-4cba-a1b7-90b918630b86" containerName="main"
Aug 01 00:50:08.436710 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:08.436547    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d711d94e-2255-4cba-a1b7-90b918630b86" containerName="main"
Aug 01 00:50:08.436710 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:08.436563    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfea3687-e01f-4cfa-bc88-7ee55bc7bed1" containerName="main"
Aug 01 00:50:08.436710 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:08.436568    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfea3687-e01f-4cfa-bc88-7ee55bc7bed1" containerName="main"
Aug 01 00:50:08.436710 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:08.436584    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfea3687-e01f-4cfa-bc88-7ee55bc7bed1" containerName="storage-initializer"
Aug 01 00:50:08.436710 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:08.436590    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfea3687-e01f-4cfa-bc88-7ee55bc7bed1" containerName="storage-initializer"
Aug 01 00:50:08.436710 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:08.436637    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d711d94e-2255-4cba-a1b7-90b918630b86" containerName="main"
Aug 01 00:50:08.436710 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:08.436644    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfea3687-e01f-4cfa-bc88-7ee55bc7bed1" containerName="main"
Aug 01 00:50:08.439463 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:08.439445    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kbl7c/must-gather-t2vpq"
Aug 01 00:50:08.441944 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:08.441920    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kbl7c\"/\"openshift-service-ca.crt\""
Aug 01 00:50:08.442057 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:08.441940    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kbl7c\"/\"kube-root-ca.crt\""
Aug 01 00:50:08.442933 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:08.442917    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kbl7c\"/\"default-dockercfg-txszj\""
Aug 01 00:50:08.448709 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:08.448686    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kbl7c/must-gather-t2vpq"]
Aug 01 00:50:08.560336 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:08.560307    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9c89\" (UniqueName: \"kubernetes.io/projected/0c8df006-7029-4025-8fc1-aae4db17944f-kube-api-access-h9c89\") pod \"must-gather-t2vpq\" (UID: \"0c8df006-7029-4025-8fc1-aae4db17944f\") " pod="openshift-must-gather-kbl7c/must-gather-t2vpq"
Aug 01 00:50:08.560493 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:08.560357    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0c8df006-7029-4025-8fc1-aae4db17944f-must-gather-output\") pod \"must-gather-t2vpq\" (UID: \"0c8df006-7029-4025-8fc1-aae4db17944f\") " pod="openshift-must-gather-kbl7c/must-gather-t2vpq"
Aug 01 00:50:08.660767 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:08.660716    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9c89\" (UniqueName: \"kubernetes.io/projected/0c8df006-7029-4025-8fc1-aae4db17944f-kube-api-access-h9c89\") pod \"must-gather-t2vpq\" (UID: \"0c8df006-7029-4025-8fc1-aae4db17944f\") " pod="openshift-must-gather-kbl7c/must-gather-t2vpq"
Aug 01 00:50:08.660899 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:08.660787    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0c8df006-7029-4025-8fc1-aae4db17944f-must-gather-output\") pod \"must-gather-t2vpq\" (UID: \"0c8df006-7029-4025-8fc1-aae4db17944f\") " pod="openshift-must-gather-kbl7c/must-gather-t2vpq"
Aug 01 00:50:08.661115 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:08.661098    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0c8df006-7029-4025-8fc1-aae4db17944f-must-gather-output\") pod \"must-gather-t2vpq\" (UID: \"0c8df006-7029-4025-8fc1-aae4db17944f\") " pod="openshift-must-gather-kbl7c/must-gather-t2vpq"
Aug 01 00:50:08.670929 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:08.670912    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9c89\" (UniqueName: \"kubernetes.io/projected/0c8df006-7029-4025-8fc1-aae4db17944f-kube-api-access-h9c89\") pod \"must-gather-t2vpq\" (UID: \"0c8df006-7029-4025-8fc1-aae4db17944f\") " pod="openshift-must-gather-kbl7c/must-gather-t2vpq"
Aug 01 00:50:08.749681 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:08.749658    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kbl7c/must-gather-t2vpq"
Aug 01 00:50:08.877172 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:08.877075    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kbl7c/must-gather-t2vpq"]
Aug 01 00:50:08.879756 ip-10-0-130-148 kubenswrapper[2559]: W0801 00:50:08.879716    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c8df006_7029_4025_8fc1_aae4db17944f.slice/crio-bd8197f5af5de39ab6e1935e1f1fc88db08368e79863b005a87ac4861a0bd5ba WatchSource:0}: Error finding container bd8197f5af5de39ab6e1935e1f1fc88db08368e79863b005a87ac4861a0bd5ba: Status 404 returned error can't find the container with id bd8197f5af5de39ab6e1935e1f1fc88db08368e79863b005a87ac4861a0bd5ba
Aug 01 00:50:08.881349 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:08.881331    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 01 00:50:09.430728 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:09.430692    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kbl7c/must-gather-t2vpq" event={"ID":"0c8df006-7029-4025-8fc1-aae4db17944f","Type":"ContainerStarted","Data":"bd8197f5af5de39ab6e1935e1f1fc88db08368e79863b005a87ac4861a0bd5ba"}
Aug 01 00:50:10.438522 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:10.438410    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kbl7c/must-gather-t2vpq" event={"ID":"0c8df006-7029-4025-8fc1-aae4db17944f","Type":"ContainerStarted","Data":"7b3e3fa320fe27888e6b06387d287702120d7d4ce0cafce4104e5c72ea64fed7"}
Aug 01 00:50:10.438522 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:10.438476    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kbl7c/must-gather-t2vpq" event={"ID":"0c8df006-7029-4025-8fc1-aae4db17944f","Type":"ContainerStarted","Data":"fdb40f6bbf646e0701e40d3e1777c7b0505a0f8fbfb291dab3433353192f5bb1"}
Aug 01 00:50:10.454932 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:10.454887    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kbl7c/must-gather-t2vpq" podStartSLOduration=1.621035756 podStartE2EDuration="2.454871809s" totalImagesPullingTime="833.836053ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:50:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:50:10.453126806 +0000 UTC m=+3873.172926719" watchObservedRunningTime="2026-08-01 00:50:10.454871809 +0000 UTC m=+3873.174671662"
Aug 01 00:50:11.165523 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:11.165491    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jqggp_e31cdcb9-5ed4-4cc2-94d4-be29c89e8e06/global-pull-secret-syncer/0.log"
Aug 01 00:50:11.313523 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:11.313483    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-92v2f_539b6c48-1833-4461-8ab1-580974198fb3/konnectivity-agent/0.log"
Aug 01 00:50:11.336194 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:11.336162    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-148.ec2.internal_4016d03779e0d094ad64607388fac106/haproxy/0.log"
Aug 01 00:50:17.191761 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:17.191716    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-4pvmb_12c3ad76-0e8b-4c7a-9402-f56d9519d156/limitador/0.log"
Aug 01 00:50:18.415631 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:18.415599    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-vknnx_13eeae3e-dd1f-4829-8fd8-9a867df2630c/cluster-monitoring-operator/0.log"
Aug 01 00:50:18.578788 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:18.578729    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6thpn_d3f711a4-1b8e-4e4a-b23e-ff0282a73571/node-exporter/0.log"
Aug 01 00:50:18.598323 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:18.598291    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6thpn_d3f711a4-1b8e-4e4a-b23e-ff0282a73571/kube-rbac-proxy/0.log"
Aug 01 00:50:18.620186 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:18.620146    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6thpn_d3f711a4-1b8e-4e4a-b23e-ff0282a73571/init-textfile/0.log"
Aug 01 00:50:20.439899 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:20.439867    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-f77f48c6d-fp7hq_7eeb5980-be60-42c9-b61b-639811f83942/networking-console-plugin/0.log"
Aug 01 00:50:20.848889 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:20.848857    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-spblg_c2e50946-aea4-4886-82ec-0bd5a31196cf/console-operator/1.log"
Aug 01 00:50:20.857592 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:20.857563    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-spblg_c2e50946-aea4-4886-82ec-0bd5a31196cf/console-operator/2.log"
Aug 01 00:50:21.275591 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:21.275556    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kbl7c/perf-node-gather-daemonset-w2t4v"]
Aug 01 00:50:21.279291 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:21.279265    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-w2t4v"
Aug 01 00:50:21.288302 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:21.288271    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kbl7c/perf-node-gather-daemonset-w2t4v"]
Aug 01 00:50:21.341379 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:21.341343    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-77c974ffcf-8znrf_08e9f54f-4554-4f00-8c74-5d45fc73b17f/download-server/0.log"
Aug 01 00:50:21.381921 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:21.381857    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/eac0c388-e4a5-4498-95c6-ff1a47e36b21-proc\") pod \"perf-node-gather-daemonset-w2t4v\" (UID: \"eac0c388-e4a5-4498-95c6-ff1a47e36b21\") " pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-w2t4v"
Aug 01 00:50:21.382283 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:21.382250    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eac0c388-e4a5-4498-95c6-ff1a47e36b21-sys\") pod \"perf-node-gather-daemonset-w2t4v\" (UID: \"eac0c388-e4a5-4498-95c6-ff1a47e36b21\") " pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-w2t4v"
Aug 01 00:50:21.382402 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:21.382336    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eac0c388-e4a5-4498-95c6-ff1a47e36b21-lib-modules\") pod \"perf-node-gather-daemonset-w2t4v\" (UID: \"eac0c388-e4a5-4498-95c6-ff1a47e36b21\") " pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-w2t4v"
Aug 01 00:50:21.382402 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:21.382387    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmrv5\" (UniqueName: \"kubernetes.io/projected/eac0c388-e4a5-4498-95c6-ff1a47e36b21-kube-api-access-hmrv5\") pod \"perf-node-gather-daemonset-w2t4v\" (UID: \"eac0c388-e4a5-4498-95c6-ff1a47e36b21\") " pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-w2t4v"
Aug 01 00:50:21.382523 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:21.382420    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/eac0c388-e4a5-4498-95c6-ff1a47e36b21-podres\") pod \"perf-node-gather-daemonset-w2t4v\" (UID: \"eac0c388-e4a5-4498-95c6-ff1a47e36b21\") " pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-w2t4v"
Aug 01 00:50:21.483442 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:21.483398    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eac0c388-e4a5-4498-95c6-ff1a47e36b21-sys\") pod \"perf-node-gather-daemonset-w2t4v\" (UID: \"eac0c388-e4a5-4498-95c6-ff1a47e36b21\") " pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-w2t4v"
Aug 01 00:50:21.483919 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:21.483469    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eac0c388-e4a5-4498-95c6-ff1a47e36b21-lib-modules\") pod \"perf-node-gather-daemonset-w2t4v\" (UID: \"eac0c388-e4a5-4498-95c6-ff1a47e36b21\") " pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-w2t4v"
Aug 01 00:50:21.483919 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:21.483507    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmrv5\" (UniqueName: \"kubernetes.io/projected/eac0c388-e4a5-4498-95c6-ff1a47e36b21-kube-api-access-hmrv5\") pod \"perf-node-gather-daemonset-w2t4v\" (UID: \"eac0c388-e4a5-4498-95c6-ff1a47e36b21\") " pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-w2t4v"
Aug 01 00:50:21.483919 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:21.483522    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eac0c388-e4a5-4498-95c6-ff1a47e36b21-sys\") pod \"perf-node-gather-daemonset-w2t4v\" (UID: \"eac0c388-e4a5-4498-95c6-ff1a47e36b21\") " pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-w2t4v"
Aug 01 00:50:21.483919 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:21.483530    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/eac0c388-e4a5-4498-95c6-ff1a47e36b21-podres\") pod \"perf-node-gather-daemonset-w2t4v\" (UID: \"eac0c388-e4a5-4498-95c6-ff1a47e36b21\") " pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-w2t4v"
Aug 01 00:50:21.483919 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:21.483619    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/eac0c388-e4a5-4498-95c6-ff1a47e36b21-proc\") pod \"perf-node-gather-daemonset-w2t4v\" (UID: \"eac0c388-e4a5-4498-95c6-ff1a47e36b21\") " pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-w2t4v"
Aug 01 00:50:21.483919 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:21.483650    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eac0c388-e4a5-4498-95c6-ff1a47e36b21-lib-modules\") pod \"perf-node-gather-daemonset-w2t4v\" (UID: \"eac0c388-e4a5-4498-95c6-ff1a47e36b21\") " pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-w2t4v"
Aug 01 00:50:21.483919 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:21.483619    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/eac0c388-e4a5-4498-95c6-ff1a47e36b21-podres\") pod \"perf-node-gather-daemonset-w2t4v\" (UID: \"eac0c388-e4a5-4498-95c6-ff1a47e36b21\") " pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-w2t4v"
Aug 01 00:50:21.483919 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:21.483719    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/eac0c388-e4a5-4498-95c6-ff1a47e36b21-proc\") pod \"perf-node-gather-daemonset-w2t4v\" (UID: \"eac0c388-e4a5-4498-95c6-ff1a47e36b21\") " pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-w2t4v"
Aug 01 00:50:21.492017 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:21.491995    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmrv5\" (UniqueName: \"kubernetes.io/projected/eac0c388-e4a5-4498-95c6-ff1a47e36b21-kube-api-access-hmrv5\") pod \"perf-node-gather-daemonset-w2t4v\" (UID: \"eac0c388-e4a5-4498-95c6-ff1a47e36b21\") " pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-w2t4v"
Aug 01 00:50:21.591579 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:21.591482    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-w2t4v"
Aug 01 00:50:21.760596 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:21.760569    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kbl7c/perf-node-gather-daemonset-w2t4v"]
Aug 01 00:50:21.763445 ip-10-0-130-148 kubenswrapper[2559]: W0801 00:50:21.763408    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podeac0c388_e4a5_4498_95c6_ff1a47e36b21.slice/crio-460aabc00ed12bbedefaecf460dc0d2baa2222eec125d56710682217d432ecf9 WatchSource:0}: Error finding container 460aabc00ed12bbedefaecf460dc0d2baa2222eec125d56710682217d432ecf9: Status 404 returned error can't find the container with id 460aabc00ed12bbedefaecf460dc0d2baa2222eec125d56710682217d432ecf9
Aug 01 00:50:21.937231 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:21.937145    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5b8776669-4zdc5_7d4b598c-cec0-4607-95bb-63f05f1eab65/volume-data-source-validator/0.log"
Aug 01 00:50:22.515963 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:22.515916    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-w2t4v" event={"ID":"eac0c388-e4a5-4498-95c6-ff1a47e36b21","Type":"ContainerStarted","Data":"ea1560b37f3ac8465b6dc7b99edf5e634306a843c28859e93b503bfc6d2808b4"}
Aug 01 00:50:22.516503 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:22.515989    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-w2t4v" event={"ID":"eac0c388-e4a5-4498-95c6-ff1a47e36b21","Type":"ContainerStarted","Data":"460aabc00ed12bbedefaecf460dc0d2baa2222eec125d56710682217d432ecf9"}
Aug 01 00:50:22.516503 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:22.516040    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-w2t4v"
Aug 01 00:50:22.533126 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:22.533069    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-w2t4v" podStartSLOduration=1.533047654 podStartE2EDuration="1.533047654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-01 00:50:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:50:22.531614299 +0000 UTC m=+3885.251414140" watchObservedRunningTime="2026-08-01 00:50:22.533047654 +0000 UTC m=+3885.252847501"
Aug 01 00:50:22.930718 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:22.930645    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fxfvr_0ae80080-0f96-4a21-b13c-87738f73a578/dns/0.log"
Aug 01 00:50:22.950995 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:22.950958    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fxfvr_0ae80080-0f96-4a21-b13c-87738f73a578/kube-rbac-proxy/0.log"
Aug 01 00:50:23.023061 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:23.023033    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ld78r_96d86114-06a1-4c32-9d63-c7f4262fa871/dns-node-resolver/0.log"
Aug 01 00:50:23.495137 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:23.495106    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-pruner-29759040-9x7p9_8243c84a-74f8-424f-833d-8eee80046f7a/image-pruner/0.log"
Aug 01 00:50:23.558700 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:23.558671    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7mgjz_482f5dcb-a14d-426f-93e6-b1ce4c9b89e6/node-ca/0.log"
Aug 01 00:50:24.746787 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:24.746724    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5579c88784-ntwhl_c33c7c85-f678-49ff-bcc0-5b1de21c563b/router/0.log"
Aug 01 00:50:25.254132 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:25.254099    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9zcz2_1d81c21b-5141-40fb-bb77-b676e9461c82/serve-healthcheck-canary/0.log"
Aug 01 00:50:25.793212 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:25.793178    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-77696685b-xjqzb_967d9bc9-7bc0-496f-b18b-3b7b95cab4f2/insights-operator/0.log"
Aug 01 00:50:25.795254 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:25.795231    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-77696685b-xjqzb_967d9bc9-7bc0-496f-b18b-3b7b95cab4f2/insights-operator/1.log"
Aug 01 00:50:26.152182 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:26.152103    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vqpfb_41ab6526-f493-4ea8-881a-fda14abc3525/kube-rbac-proxy/0.log"
Aug 01 00:50:26.216891 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:26.216860    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vqpfb_41ab6526-f493-4ea8-881a-fda14abc3525/exporter/0.log"
Aug 01 00:50:26.280086 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:26.280054    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vqpfb_41ab6526-f493-4ea8-881a-fda14abc3525/extractor/0.log"
Aug 01 00:50:28.533218 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:28.533186    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-w2t4v"
Aug 01 00:50:29.366867 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:29.366836    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6cb8c69f9-2q8vn_da8cc73b-8926-4a78-8fde-6647144e1781/manager/0.log"
Aug 01 00:50:30.571354 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:30.571281    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5579c88784-ntwhl_c33c7c85-f678-49ff-bcc0-5b1de21c563b/router/0.log"
Aug 01 00:50:31.381224 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:31.381138    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-t7mpc_e3f7c45f-1bee-41f1-8880-4820a748f7a7/s3-init/0.log"
Aug 01 00:50:31.398072 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:31.398042    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-t7mpc_e3f7c45f-1bee-41f1-8880-4820a748f7a7/download-hf-model/0.log"
Aug 01 00:50:31.993685 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:31.993647    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5579c88784-ntwhl_c33c7c85-f678-49ff-bcc0-5b1de21c563b/router/0.log"
Aug 01 00:50:35.813926 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:35.813891    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-swnwf_234c2a7e-69c4-4dff-8d02-8d0568136b0d/migrator/0.log"
Aug 01 00:50:35.832095 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:35.832065    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-swnwf_234c2a7e-69c4-4dff-8d02-8d0568136b0d/graceful-termination/0.log"
Aug 01 00:50:36.161771 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:36.161669    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7878f4b88c-slttz_e709e147-60bb-46dc-be7d-3d5269dbcd8a/kube-storage-version-migrator-operator/1.log"
Aug 01 00:50:36.163360 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:36.163329    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7878f4b88c-slttz_e709e147-60bb-46dc-be7d-3d5269dbcd8a/kube-storage-version-migrator-operator/0.log"
Aug 01 00:50:37.360904 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:37.360872    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r9jcj_ee8bfdcd-3661-46f6-93a0-93bb3facacd4/kube-multus-additional-cni-plugins/0.log"
Aug 01 00:50:37.380205 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:37.380174    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r9jcj_ee8bfdcd-3661-46f6-93a0-93bb3facacd4/egress-router-binary-copy/0.log"
Aug 01 00:50:37.399499 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:37.399472    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r9jcj_ee8bfdcd-3661-46f6-93a0-93bb3facacd4/cni-plugins/0.log"
Aug 01 00:50:37.419953 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:37.419932    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r9jcj_ee8bfdcd-3661-46f6-93a0-93bb3facacd4/bond-cni-plugin/0.log"
Aug 01 00:50:37.440460 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:37.440438    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r9jcj_ee8bfdcd-3661-46f6-93a0-93bb3facacd4/routeoverride-cni/0.log"
Aug 01 00:50:37.459730 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:37.459709    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r9jcj_ee8bfdcd-3661-46f6-93a0-93bb3facacd4/whereabouts-cni-bincopy/0.log"
Aug 01 00:50:37.478083 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:37.478061    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r9jcj_ee8bfdcd-3661-46f6-93a0-93bb3facacd4/whereabouts-cni/0.log"
Aug 01 00:50:37.525967 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:37.525943    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-c8d8d_b490be26-0249-4036-8841-93b022ae3b1f/kube-multus/0.log"
Aug 01 00:50:37.581973 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:37.581947    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-bpv44_450206d3-1b10-45ae-b6e9-6575c0d0be94/network-metrics-daemon/0.log"
Aug 01 00:50:37.596642 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:37.596619    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-bpv44_450206d3-1b10-45ae-b6e9-6575c0d0be94/kube-rbac-proxy/0.log"
Aug 01 00:50:38.025712 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:38.025678    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-spblg_c2e50946-aea4-4886-82ec-0bd5a31196cf/console-operator/1.log"
Aug 01 00:50:38.049638 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:38.049612    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-spblg_c2e50946-aea4-4886-82ec-0bd5a31196cf/console-operator/1.log"
Aug 01 00:50:38.638510 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:38.638476    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nmsz6_71cbd9c3-1692-4ded-a7fc-bf48635b0c29/ovn-controller/0.log"
Aug 01 00:50:38.654448 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:38.654423    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nmsz6_71cbd9c3-1692-4ded-a7fc-bf48635b0c29/ovn-acl-logging/0.log"
Aug 01 00:50:38.678595 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:38.678571    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nmsz6_71cbd9c3-1692-4ded-a7fc-bf48635b0c29/kube-rbac-proxy-node/0.log"
Aug 01 00:50:38.697522 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:38.697498    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nmsz6_71cbd9c3-1692-4ded-a7fc-bf48635b0c29/kube-rbac-proxy-ovn-metrics/0.log"
Aug 01 00:50:38.713903 ip-10-0-130-148 kubenswrapper[2559]: I0801 00:50:38.713880    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nmsz6_71cbd9c3-1692-4ded-a7fc-bf48635b0c29/northd/0.log"